ThingWorx Cluster Sizing Considerations
Sizing ThingWorx Foundation Nodes
In clustered operations, the sizing for individual ThingWorx Foundation nodes is largely unchanged. Each cluster node must have enough memory to load the entire Thing model. Therefore, the size of each cluster node must match the size of a single-server node. For example, a medium-sized single server is 16 vCPUs and 32 GiB RAM. Similarly, for a two-node cluster, each cluster node would be 16 vCPUs and 32 GiB RAM.
If Apache Ignite is being deployed separately, Foundation Server memory consumption may be reduced slightly as property state information has been transferred to the Ignite nodes.
CPU utilization will increase to execute background cluster synchronization tasks. Individual operations can also be slightly slower in clustered configurations due to the added latency of a shared cache. This is offset by the ability to run a scale a larger number of operations in parallel (business logic and/or user requests) across multiple nodes.
For high availability, once the correct size of an individual node has been determined, it is recommended that you deploy at least one more ThingWorx Foundation node than needed to handle the peak load on your application. This will allow the cluster to continue to meet performance expectations in the event of a single node failure.
Sizing Connection Server Nodes
In clustered operations, Connection Servers are required in order to distribute device load across the cluster, or to re-distribute if a node failure occurs.
As with Foundation nodes, it is recommended to deploy at least one more Connection Server than needed to handle your expected device count. This allows the Connection Servers to maintain connectivity with all devices if a single Connection Server node fails.
Review the ThingWorx Connection Services Help Center for system requirements for individual Connection Server options.
Sizing Apache Zookeeper Nodes
Apache ZooKeeper is an open source solution for synchronization of distributed applications. It provides monitoring and leader election for ThingWorx nodes.
A three-node set, each with 2 vCPUs and 2 GiB RAM would be recommended. An odd number of instances is required to maintain a quorum.
Review the Apache Zookeeper System Requirements for more details: https://zookeeper.apache.org/doc/r3.1.2/zookeeperAdmin.html#sc_systemReq
Sizing Database Nodes
Beyond high availability, database load increases in clustered ThingWorx configurations. To illustrate this, identical load tests were performed with five different Medium deployments using Microsoft Azure Virtual Machines:
|
Single-Node and Cluster Performance Comparison (PostgreSQL only)
|
|||||
|---|---|---|---|---|---|
|
Foundation
|
1 × F16s_v2
|
2 × F16s_v2
|
3 × F16s_v2
|
4 × F16s_v2
|
5 × F16s_v2
|
|
Ignite
|
None
|
3 × F8s_v2
|
3 × F8s_v2
|
3 × F8s_v2
|
3 × F8s_v2
|
|
PostgreSQL
|
1 × F16s_v2
|
1 × F16s_v2
|
1 × F16s_v2
|
1 × F16s_v2
|
1 × F16s_v2
|
|
InfluxDB
|
None
|
None
|
None
|
None
|
None
|
|
Results Per Node (Avg.)
|
26,000 WPS
19 HTTP Ops
|
12,000 WPS
10 HTTP Ops
|
10,000 WPS
6 HTTP Ops
|
7,000 WPS
6 HTTP Ops
|
6,000 WPS
2 HTTP Ops
|
|
Results Total
|
24,000 WPS
19 HTTP Ops
|
30,000 WPS
24 HTTP Ops
|
28,000 WPS
22 HTTP Ops
|
30,000 WPS
12 HTTP Ops
|
|
|
Reminder: Sizing Guide recommendations are intended for use initial baselines to size ThingWorx implementations. Individual results will vary based on edge configuration, application load, etc.
|
|||||
While the tests above show a small data ingestion improvement, HTTP Request performance does not improve due to inadequate database resources.
To address this, scalability can improve with larger RDBMS instances, and/or by using InfluxDB as indicated below.
|
Single-Node and Cluster Performance Comparison (PostgreSQL + InfluxDB)
|
|||||
|---|---|---|---|---|---|
|
Foundation
|
1 × F16s_v2
|
2 × F16s_v2
|
3 × F16s_v2
|
4 × F16s_v2
|
5 × F16s_v2
|
|
Ignite
|
None
|
3 × F8s_v2
|
3 × F8s_v2
|
3 × F8s_v2
|
3 × F8s_v2
|
|
PostgreSQL
|
1 × F8s_v2
|
1 × F8s_v2
|
1 × F8s_v2
|
1 × F8s_v2
|
1 × F8s_v2
|
|
InfluxDB
|
1 × F8s_v2
|
1 × F8s_v2
|
1 × F8s_v2
|
1 × F8s_v2
|
1 × F8s_v2
|
|
Results Per Node (Avg.)
|
90,000 WPS
120 HTTP Ops
|
53,000 WPS
187 HTTP Ops
|
39,000 WPS
148 HTTP Ops
|
31,500 WPS
127 HTTP Ops
|
27,000 WPS
108 HTTP Ops
|
|
Results Total
|
106,000 WPS
375 HTTP Ops
|
118,000 WPS
445 HTTP Ops
|
126,000 WPS
508 HTTP Ops
|
135,000 WPS
540 HTTP Ops
|
|
|
Reminder: Sizing Guide recommendations are intended for use initial baselines to size ThingWorx implementations. Individual results will vary based on edge configuration, application load, etc.
|
|||||
 |
InfluxDB open source was used for these tests, which does not provide high availability. InfluxDB Enterprise would be recommended for production ThingWorx cluster implementations. For InfluxDB Enterprise sizing, plan for two InfluxDB “Data” nodes as indicated, plus three “Meta” nodes, typically 1-2 vCPUs and 0.5-1GiB RAM each. For more guidance on InfluxDB sizing, please review https://docs.influxdata.com/influxdb/v1.8/guides/hardware_sizing/.
|
Sizing Apache Ignite Nodes
In a ThingWorx cluster, Apache Ignite provides a shared cache for property values and additional data such as file transfer jobs.
Ignite can be embedded within the ThingWorx Foundation process, which does not require a separate installation. Alternatively, for greater load distribution Ignite can also be deployed as a separate cluster.
Ignite can also be run in one of two modes:
• PARTITIONED Mode - data is divided equally into partitions and split equally between participating nodes. Best cache-write performance.
• REPLICATED Mode - each Ignite instance has a copy of every data point. Best cache-read performance.
Review the Apache Ignite documentation of cache modes for more details: https://apacheignite.readme.io/docs/cache-modes
The primary load from Ignite is network throughput between each ThingWorx Foundation and Ignite instance. While CPU, Disk or Memory requirements could indicate smaller VM sizes from cloud providers, it is important to monitor for limited or throttled network throughput.
If throughput issues are encountered, they can be addressed either by moving to larger systems with greater network capacity, or by adding additional Ignite nodes (using PARTITONED mode) to distribute load.
As a rough starting point, a memory size equal to your ThingWorx Foundation nodes would be a safe conservative estimate.
To calculate the memory footprint of your shared cache more precisely:
1. Calculate the VTQ (value, timestamp and quality) memory footprint for your ThingWorx property cache.
a. For each type of thing, determine the data type size in memory. For example, if your property is a 50-character string, and each character requires 2 bytes, that string requires 100 bytes of memory.
b. Add 8 bytes. (4 for the date/time of that value, and 4 for the quality of the value).
c. Multiply by 2. ThingWorx caches the current and previous value of each property.
d. Multiply by the number of things of that type.
e. Add together the result for each thing type.
2. Add 30% for Ignite’s in-memory indexes of values.
3. Multiply by the number of data copies you want across the Ignite cluster. For example, if you want one backup of the data so that no data is lost if a single Ignite node fails, then multiply by 2.
4. Divide by the number of Ignite cluster nodes you plan to deploy.
5. Each Ignite node will require an additional ~300 MB for its own operation.
Review the Apache Ignite documentation for additional details: https://apacheignite.readme.io/docs/capacity-planning