Monitoring the High Availability System
The ThingWorx HA configuration has many components. Monitoring all of them is complex and a best practice would be to use a monitoring tool that can make administrative tasks easier.
PTC System Monitor
The PTC System Monitor (PSM) is a separate, independent application performance monitoring system that is powered by the Production Edition of Dynatrace. PTC has tailored the Dynatrace Production Edition to meet the needs for monitoring PTC products such as ThingWorx. The PTC System Monitor provides useful dashboards and instrumentation that allows monitoring while maintaining critical performance requirements.
Documentation on the PTC System Monitor:
A default deployment of PSM should result in a monitoring solution for each ThingWorx server and the database servers. PSM can also be used to monitor other components of the ThingWorx HA configuration through some additional plug ins.
PSM Monitoring Database
PSM has built in monitoring for Microsoft SQL Server.
There is an optional monitoring plug in for PostgreSQL as well. To obtain the latest plug in, go to
Dynatrace’s GitHub repository and search for ‘postgresql’. The results should offer links to download the plug in, installation, and configuration instructions and discussions on the metrics that are monitored.
PSM Monitoring ZooKeeper
The
Dynatrace JMX monitor can be used to gather metrics from ZooKeeper.
PSM Monitoring HAProxy
The
Dynatrace JMX monitor can be used to gather metrics from HAProxy.
General Monitoring Information
PTC recommends monitoring the following metric on each server in the ThingWorx HA configuration.
• CPU utilization of each machine or process
• Memory usage
• Hard disk space
• Open ports
Each component within the landscape has additional metrics to monitor. For more information on component metrics, see the following sections.
ZooKeeper
The methods to monitor ZooKeeper are provided here:
ZooKeeper supports monitoring through two methods
1. Calls to its command port and providing specific four letter words.
2. JMX
In a default configuration, ZooKeeper communicates on the following ports:
• Port 2181 for client connections
• Port 2888 for follower heartbeat messages
• Port 3888 for communication with other ZooKeeper nodes during leader election
ZooKeeper uses Log4j as its logging solution. More information is provided here:
https://zookeeper.apache.org/doc/r3.4.12/zookeeperAdmin.html#sc_loggingPgpool-II
By default, Pgpool-II communicates watchdog messages on port 9000 and heartbeats on port 9694.
Pgpool-II with watchdog runs as a process called pgpool at the file location /usr/sbin/pgpool. You can check the status of the processes using the following command:
ps -elf | grep pgpool
A list of processes should be returned, which indicates that Pgpool-II is running.
Any events or errors that occur are logged in the syslog.
The following messages indicate that Pgpool-II with watchdog has started successfully:
...
wd_init: start watchdog
pgpool-II successfully started. version x.x.x (...)
find_primary_node: primary node id is 0
...
PostgreSQL
PostgreSQL includes a subsystem for monitoring called the Statistics Collector, which is configured in the postgresql.conf file. For more information, see
http://www.postgresql.org/docs/current/static/monitoring-stats.html. For details about PostgreSQL monitoring options, see
https://wiki.postgresql.org/wiki/Monitoring.By default, PostgreSQL accepts database connections on port 5432.
In an Ubuntu environment, PostgreSQL runs on a service called postgresql.
PostgreSQL stores its logs in the /var/log/postgresql/postgresql-10.x-main.log file. To confirm that the system is running, the following message should be logged:
...
(...) LOG: database system is ready to accept connections
...
HAProxy
HAProxy has a built-in status web page that you can access by adding configuration settings to the haproxy.cfg file and navigating to a configured port in a browser. The default configuration enables navigation to the HAProxy status page on port 1936.
HAProxy port usage depends on the configuration in your environment, but the example configuration of HAProxy and ThingWorx uses port 8080. In an environment with SSL, it should be configured to pass port 8443. When using the configuration file in the default setup, the HAProxy status page communicates on port 1936.
HAProxy runs on a service called haproxy.
HAProxy logs service details to /var/log/syslog. It also has its own logs at /var/log/haproxy.log to which errors are written if issues occur for HAProxy at startup or during runtime.