Monitoring the High Availability System
The ThingWorx High Availability (HA) configuration has many components. Monitoring all of them can be complex; therefore, it is a best practice to use a monitoring tool to make administrative tasks easier.
General Monitoring Information
PTC recommends monitoring the following metrics on each server in the ThingWorx HA configuration:
• CPU utilization of each machine or process
• Memory usage
• Hard disk space
• Open ports
Each component within the landscape has additional metrics to monitor. For more information on component metrics, see the following sections.
ZooKeeper
The methods to monitor ZooKeeper are provided here:
ZooKeeper supports monitoring through the following methods:
• Calls to its command port and the use of specified four-letter words.
• JMX
In a default configuration, ZooKeeper communicates on the following ports:
• Port 2181 for client connections
• Port 2888 for follower heartbeat messages
• Port 3888 for communication with other ZooKeeper nodes during leader election
Pgpool-II
By default, Pgpool-II communicates watchdog messages on port 9000 and heartbeats on port 9694.
Pgpool-II with watchdog runs as a process called pgpool at the file location /usr/sbin/pgpool. You can check the status of the processes using the following command:
ps -elf | grep pgpool
A list of processes should be returned, which indicates that Pgpool-II is running.
Any events or errors that occur are logged in the syslog.
The following messages indicate that Pgpool-II with watchdog has started successfully:
...
wd_init: start watchdog
pgpool-II successfully started. version x.x.x (...)
find_primary_node: primary node id is 0
...
PostgreSQL
PostgreSQL includes a subsystem for monitoring called the Statistics Collector, which is configured in the postgresql.conf file. For more information, see
http://www.postgresql.org/docs/current/static/monitoring-stats.html. For details about PostgreSQL monitoring options, see
https://wiki.postgresql.org/wiki/Monitoring.By default, PostgreSQL accepts database connections on port 5432.
In an Ubuntu environment, PostgreSQL runs on a service called postgresql.
PostgreSQL stores its logs in the /var/log/postgresql/postgresql-10.x-main.log file. To confirm that the system is running, the following message should be logged:
...
(...) LOG: database system is ready to accept connections
...
HAProxy
HAProxy has a built-in status web page that you can access by adding configuration settings to the haproxy.cfg file and navigating to a configured port in a browser. The default configuration enables navigation to the HAProxy status page on port 1936.
HAProxy port usage depends on the configuration in your environment, but the example configuration of HAProxy and ThingWorx uses port 8080. In an environment with SSL, it should be configured to pass port 8443. When using the configuration file in the default setup, the HAProxy status page communicates on port 1936.
HAProxy runs on a service called haproxy.
HAProxy logs service details to /var/log/syslog. It also has its own logs at /var/log/haproxy.log to which errors are written if issues occur for HAProxy at startup or during runtime.
Ignite
Starting with ThingWorx 9.6, Ignite 2.16 is supported.
Ignite metrics reporting is available in JMX and Prometheus for the following categories:
• Cache
• Partition map exchange
• Compute jobs
• Thread pools
• Communication IO
• Discovery IO
To enable metrics reporting, set the following configuration:
|
Configuration
|
Type
|
Default Value
|
Accepted Value
|
|
default-enable-metrics
|
BOOLEAN
|
false
|
• true – metrics reporting is enabled
• false – metrics reporting is not enabled
|
Selected cache reporting is supported with the configuration under ignite:
Example
"cache" : {
"ignite" : {
"caches" : {
"thing_propertyValueCache" : {
"enable-metrics" : true
}
}
}
}
• Ignite metrics are exposed as MBean objects and attributes using JMX and can be visualized in visualization tools like JConsole.
• These metrics are also available for visualization on Prometheus and Grafana.
Use the following syntax to access metrics available in Prometheus.
• Cache - ignite_<name_of_the_cache>_<metric_name>
• Partition map exchange (PME) – ignite_pme_<metric_name>
• Compute Jobs – ignite_jobs_<metrics_name>
• Communication - ignite_communication_<metrics_name>
• Discovery - ignite_discovery_<metrics_name>
• Cluster – ignite_cluster_<metric_name>
• Threadpool - ignite_threadpools_<metric_name>