Scaling

Performance

The performance of the Encap server depends on many factors. Different Encap configurations have different performance characteristics, therefore we recommend performance testing each implementation.

This page provides guidelines for tuning common Encap use cases.

Infrastructure

Higher availability

Note

This is a generic example of infrastructure. It does not show setup specifics such as a load balancer heartbeat and shared IP.

Depending on the specific solution, the choice details may vary.

Diagram showing example of basic infrastructure for higher availability

Higher peak loads

Diagram showing example of horizontally scaling the infrastructure for higher peak loads

Encap server

Hardware

Usually, the limiting resource on the Encap server is CPU. When selecting Encap server hardware, you should prioritise CPU, then memory.

You should ensure that you always have enough physical memory to get the entire Encap process in memory without swapping. As the Encap server runs in a JVM, the max size of the Encap process can be roughly estimated as JAVA_MAX_HEAP (see below) + 500 MB.

The network bandwidth between the server and the database should be high, as Encap reads and writes a significant amount of data during high load.

OS

On most Linux versions, the default open files limit is too low for larger Encap installation. Since the cost of increasing this limit is very low, we recommend setting it relatively high; at least double the expected number of concurrent connections.

In RedHat, this is done in limits.conf.

sudo vim /etc/security/limits.conf
...
<username> soft nofile 65535
<username> hard nofile 65535
...

Encap software tuning

You can tune for stable performance at high load by:

• Limiting the database connections to what the database can handle.
• Limiting jetty threads to ensure predictable degradation.
• Limiting the queue to ensure fail fast when beyond load acceptance.

1. Database cleanup should be enabled and configured properly. Keeping a large set of outdated data could hurt performance significantly.
   • The default settings should be tuned to only keep the least possible amount of required data. By default, it is set to 10 years to ensure legal obligations are met.
   • The cleanup should also be scheduled and ordered to run when the server load is expected to be low. This is typically at night.
   • If your legal obligations permit, the value of admin.dbCleanup.daysToKeep in encap-cluster.properties should ideally be set to between 10-31 days.
2. Java heap size is set by using the java.vm.options configuration parameter in encap-instance.properties.
   1. Set maximum Java heap size by changing -Xmx2G to correspond to 80% of available physical memory on the server (assuming it only runs Encap), or at least 10 MB x jetty.threads.max + 1GB
   2. Set minimum Java heap size by changing the -Xms512M to half of what is set for the maximum above.
3. Set jetty.threads.max in encap-instance.properties to the minimum number needed to fully utilise all available CPU. This is normally in the 50-300 range, depending on the number of server CPU cores.
4. The main database connection pool is configured using encap.db.maxPoolSize in encap-db.properties. The default is set to 0.5 x jetty.threads.max.
   • There are also two more properties encap.db.mTrans.mainPoolFactor=0.20 and encap.db.log.mainPoolFactor=0.25, which are multiplied by the encap.db.maxPoolSize to control the poolSize for non-core transaction groups.

Database

Hardware

The two most common limiting factors for the database in an Encap use case is memory cache and disk write IOPS.

Database cleanup

In order to clean up old sessions, the Encap server runs a cleanup job at a configurable interval default once at night. From a performance perspective, it is important that the retention time is not set too long for systems with high load, as this might hurt the database performance significantly.

Database upgrade

In the case of upgrading the Encap server to a new version, you should be aware that a huge database takes time to upgrade. Therefore, it is important to configure the database cleanup job as mentioned above, so that old outdated data is removed from database before the upgrade job is started.

Database tuning

• You should scale connectors to match the aggregate connector count of all Encap servers.
• You should set the buffer to a size that allows all active session date in buffer. With a properly tuned buffer almost the entire IO use of Encap will be disk writes.

Database graph examples

Good

We are within the load capabilities of the database. With constant load on the Encap servers we want to see stable performance from the database. The minor amounts of jitter in the graphs is just telling us we are close to the load limit of the database. Also note how disk writes are at 50 MB while disk read is at less than 1 MB. This is typical for the database use of Encap when the database buffer is properly configured.

Graph showing example of being within the load capabilities of the database

Bad

Lots of jitter in the database graphs. With constant load on the Encap servers this is an indication that the database is being pushed beyond it's load limits. If possible, increase the database resource that is bottlenecking performance. Reducing the number of database connections might also help improve performance, but this is a balancing act between having enough but not to many.

Graph showing example of being pushed beyond the load limits of the database

Scaling

Encap supports both vertical and horizontal scaling. Below is some examples taken from our Amazon performance test environments. In all cases the Encap servers are backed by the same Amazon RDS database and load balanced by an EC2 load balancer instance.

Thanks to the support for stateless load-balancing across multiple Encap servers, it scales near perfectly with the number of virtual cores in both cases until it hits the database limit in this environment. Because of the simplicity in scaling the Encap server itself, very large Encap installations quickly become a question of scaling the database more than a question of scaling Encap.

Vertical scaling

Scaling by increasing single Encap server resources. Since the Encap server usually is bottlenecked by CPU increasing the computational power of the Encap server is one common way to scale out. When growing the Encap server instance we recommend reconfiguring the following parameters, see above for guidance on what they should be set to.

• encap-instance.properties
  • Max and min heap settings for java.vm.options
  • jetty.threads.max
• encap-db.properties
  • encap.db.maxPoolSize

Example data

Diagram showing example data for vertical scaling

Horizontal scaling

Scaling by adding additional Encap servers. Since Encap supports stateless load balancing across servers adding a new Encap server is very simple. The only thing to be aware of is that adding additional server increases the number of database connections has to handle and increased parallelism against the database usually lead to reduced database performance.

Example data

Note

In the example data below, each server is a c3.large server.

Diagram showing example data for horizontal scaling

Storage

Storage use is dependent on multiple factors

• Type of operation done (activation, authentication).
• Configuration of server.
• Context information used.
• Device information gathered.
• Document to sign when signing is used.

The information below is just an indication of what it looks like in a normal case

Encap server

With default log levels the only file of a significant size is the audit log.

<ENCAP_HOME>/core/logs/audit.log

Note

This file is rotated daily.

Size increase

The table below illustrates the size increase per 1000 sessions.

Database

Most of the database growth in normal use cases consist of session and audit related information. This is information that has no functional value except for reporting and statistics when the session has expired.

An effective way of limiting database growth is to configure the database cleanup job to periodically clean out old session and audit data.

The use case will determine how long the data should be kept before it is cleaned out, but we recommend keeping data for at least a couple of months to facilitate customer and solution support.

Size increase

The table below illustrates the size increase per 1000 sessions.