Server maintenance

Logs

All logs are found in <ENCAP_HOME>/core/logs. Logs are rotated daily and logs older than the history limit for the specific log type are deleted permanently from the file system.

If you wish to keep all logs for archival purposes, you must move old logs out of the Encap server log folder. This can be done when the Encap server has rolled the logs of the current day and the server has started writing the logs of the next day, or at any point after this.

You can find further information about log configuration, log types, and available metrics on the Encap server logs page.

Database cleanup

In order to reduce the database size, and to speed up backup procedures and Encap upgrades, it can be advantageous to regularly remove data that is no longer required. To facilitate this, the Encap server provides an optional database cleanup service as part of the Encap server. This is enabled by default. You can find out how to configure this in the next section.

Removal of information

The cleanup job removes information related to registrations, sessions and audits.

During normal operation, the Encap server stores session information for each operation and an audit trail in the database. This information is used by the reporting service and for auditing purposes, but is not necessary for authentication purposes once the session has expired and the session, and all the session related data, will be removed by the cleanup job.

During deletion of a registration, the Encap server tombstones the registration so it can no longer be used and is therefore no longer required. The cleanup job will remove all deleted registrations, and all the data related to the registrations, and the audit trail.

Below is a high level diagram of Encap's data model, the boxes marked in green are cleaned during the cleanup job.

Diagram showing database cleanup model

Database cleanup configuration

The cleanup service is configured in encap-cluster.properties. See the Server configuration page for information on how to configure.

The result of each cleanup run can be seen in the Encap REST APIs.

Regular maintenance

Recommended regular maintenance.

Daily

Log rotation

• Proxy (dependent on proxy setup)
• Encap server:
  • <ENCAP_HOME>/core/logs
  • Handled by the server itself.
  Want to learn more?

  For more information, see the Encap server logs page.

• Database

Backup

• Database
• Encap server and proxy server configuration.
• Encap server and proxy server logs (if deemed necessary from a security standpoint, the logs are not necessary to restore an Encap server installation).

Archive old Encap server and proxy logs. The age at which the logs should be archived is up to operations.

Weekly

None.

Monthly

None.

At certificate expiry

An Encap server environment contains several certificates all with a set expiry date. It is necessary to renew and install certificates well in advance of the expiry dates.

• SSL certificate on proxy (dependant on proxy installation).
  • Validity dependent on provider.
• SSL certificate on Encap server (<ENCAP_HOME>/core/config/ssl.jks).
  • Validity dependent on provider.
  • Not used if https is terminated on proxy.
• WS and SAML signing key on Encap server (configured in manageKey.sh shell script).
  • Validity dependent on provider.
• Apple APNs certificate on Encap server (configured using the REST API).
  • Valid one year.
  • Not used if push is not used.

Monitoring

The Encap server can be deployed in a huge amount of different environments and configurations.

The following offers basic recommendations for monitoring an Encap server installation and its dependencies.

Network (ping, telnet... )

Monitor all network connections used by Encap server. If any of these go down Encap server will not work as expected.

• Proxy to Encap server.
• Encap server to database server.
• Encap server to Android Push (if push is used).
  • fcm.googleapis.com:443
  • oauth2.googleapis.com:443 for access token
• Encap server to Apple (if push is used).
  • api.push.apple.com:443 (PRODUCTION)
  • api.sandbox.push.apple.com:443 (SANDBOX)
• Internet to proxy.
• Service provider to Encap server (REST API).
• Encap server to service provider (if REST-callback is used).

Disk status

Monitor disk status to ensure necessary mounts are available and have free space on these servers

• Proxy server
• Encap server
• Database server

JVM on Encap server

Monitor basic JVM health parameters. These need to be monitored by a local agent as remote JMX access has been disabled for security reasons. Primary parameters to monitor is:

• Heap use not exceeding 90% across GC boundaries. This indicates the application is about about to go out of memory, and will usually be accompanied by frequent GC and high GC CPU usage.
• JVM CPU use not using 90% or more of total for extended time, peaks are OK.

CPU (OS)

Monitor CPU usage all servers involved to detect high load problems that can lead to slow response times.

Database

Exactly what to monitor on the database is dependent on which database and version is used on the respective installation. Our general recommendations involves monitoring the following parameters:

• Free space if not auto extend. If the database goes full Encap server will fail.
• Connections in use
• Memory in use

URL (GET)

Regularly polling the below URL can give a good basic indication if Encap server is up and running. This does a basic health check of the Encap server.

<Encap server url>/api/admin/v1/status

Logs

No messages with log level WARN, ERROR or FATAL in <ENCAP_HOME>/core/logs/stderrout.log. For information on file and statsD logging, see the Encap server logs page.

Device statistics

The Encap server supports extracting information about which versions the end customer devices are using.

This is explained more in depth in the API documentation: <Encap server url>/api/docs/index.html#_statistics_api.

Troubleshooting

This chapter lists some known error conditions, and suggested resolutions.

Too many open files (Soap fault: can't trust X509Certificate chain in WS-Security header)

The following error messages in the log may indicate that the server has reached the limit of open files:

2012-09-27 04:03:41,952 ERROR [JIoEndpoint] Socket accept failed: java.net.SocketException: Too many open files
:
2012-09-27 04:03:41,988 ERROR |-[net.link.util.config.DefaultConfigFactory] While loading config from: vfs:/home/qa2/installs/latest-snapshot/core/config/encap-cluster.properties
java.io.FileNotFoundException: /home/qa2/installs/latest-snapshot/core/config/encap-cluster.properties (Too many open files)
:
2012-09-27 04:03:42,060 WARN  |-[net.link.util.ws.security.SOAPUtils] Soap fault: can't trust X509Certificate chain in WS-Security header

This can be solved by increasing the number of open files, to a value that does not cause too many open files under the desired load. Typically, this involves using ulimit -n to monitor the number of open files during load and editing /etc/security/limits.conf to change the open files limit.

Example: Set the limit of open files to 8192 for user named encapuser on Ubuntu server

1. In file /etc/security/limits.conf:

   encapuser    soft nofile 8192
   encapuser    hard nofile 8192

2. Uncommenting /etc/pam.d/su:

   session    required    pam_limits.so 

3. Verifying the number of open files, logout/login to effectuate the configuration changes above:

   encapuser@encapserver:~$ ulimit -n
   8192

No MAC address available on server

During startup encap server will try to fetch the MAC address of the provided server in encap-instance.properties, if there is no MAC address fallback method will fetch first network interface with a MAC address.

If there is no network interface with a MAC address the server will print the following error and exit.

ERROR n.e.s.m.audit.AuditContextFilter - No MAC found for this server! System will exit.

This can be solved by assuring that a network interface for the server has an MAC address.