Using InfluxDB as the Persistence Provider
Overview
If your system intensively deals with time series data and your implementation heavily depends on value streams or streams for persistence/retrieval of data, we recommend using InfluxDB as the persistence provider in ThingWorx. InfluxDB is a high-performance data store written specifically for time series data. It allows for high throughput ingest, compression, and real-time querying of that same data. InfluxDB is used as a data store for any use case involving large amounts of time-stamped data, including DevOps monitoring, log data, application metrics, IoT sensor data, and real-time analytics. It also provides other capabilities, including Data Retention Policies (RP) and so on. The InfluxDB enterprise offers high availability and highly scalable clustering solution for time series data needs.
InfluxDB data persistence storage providers are available as part of default installation for PostgreSQL, MSSQL, or Azure SQL used for property providers:
• InfluxPersistenceProviderPackage
• Influx2PersistenceProviderPackage
ThingWorx supports OSS, Enterprise, and Cloud versions.
• InfluxDB OSS 1.x and InfluxDB Enterprise 1.x. are supported via InfluxPersistenceProviderPackage. InfluxDB Enterprise is supported only with INfluxDB 1.x
• InfluxDB OSS 2.x is supported via Influx2PersistenceProviderPackage.
• InfluxDB Cloud is supported only via Influx2PersistenceProviderPackage
Refer to Release Advisor for the specific supported versions.
 |
Support for Influx 2.0 is added from 9.3.1 to ThingWorx
|
|
|
The InfluxDB data provider currently supports value streams and streams only. The support for data tables, wikis, and blogs is not available.
|
|
|
Data Export is support with InfluxPersistenceProviderPackage and from ThingWorx 9.3.8 onwards Data Export is now supported with Influx2PersistenceProviderPackage.
|
|
|
InfluxDB is not supported as a property provider.
|
|
|
Customers who want to upgrade to ThingWorx 9.3.9 and later, and ThingWorx 9.4.0 and later, and using InfluxDB OSS 2.0, to leverage Influx2PersistenceProviderPackage, first need to upgrade to ThingWorx 9.3.8 in order to move to InfluxDB OSS v2.6 as export is required prior upgrade to InfluxDB v2.6. For InfluxDB Cloud, no need to export prior upgrade, hence, no need to move to ThingWorx 9.3.8 prior upgrade to ThingWorx 9.3.9 and later and ThingWorx 9.4.0 and later.
|
InfluxDB Enterprise and ThingWorx Starting Landscape
| | The diagram above includes InfluxDB Enterprise. For InfluxDB open source, the architecture diagram would be the same, except that it only works with one node. And for InfluxDB Cloud, the database is hosted outside the ThingWorx environment and managed by InfluxData company. |
The following terms are used in this documentation when referring to the configuration for InfluxDB Enterprise:
• Load Balancer— InfluxDB Enterprise does not function as a load balancer. An administrator needs to configure it.
• Cluster— An InfluxDB Enterprise cluster is comprised of two types of nodes: meta nodes and data nodes.
• Data Nodes— All the raw time series data lives there. For high availability, you need a replication factor of at least two.
• Meta nodes— These nodes have a simple job, to keep state consistent. They contain only basic information about state, such as retention policies, users, and databases. In a high availability environment, at least three meta nodes are needed.
Find more information about HA at https://www.influxdata.com/blog/understanding-influxenterprise-what-is-a-cluster/.
Choosing the Right Influx Database Option
A high level overview of the two InfluxDB options currently available for PTC on-premise customers follows. The database software options and underlying support options should be carefully considered when choosing an Influx product. For any questions when choosing the correct database options, contact Influx Support: https://www.influxdata.com/contact-sales. PTC customers should identify themselves as PTC ThingWorx users.
• InfluxDB open source OSS (single node)
◦ Single node only, not scalable.
◦ Free
◦ Community-based Influx support available using the Influx Community site at InfluxData Community Forums.
◦ Not recommended for production instances, due to the limit of Influx technical support (no live Influx support for urgent situations).
◦ PTC customer assumes responsibility for database maintenance and monitoring.
◦ PTC will provide best effort Influx recommendations, however, some situations may require escalation to Influx technical resources.
• InfluxDB Enterprise
If you are looking for a data store for higher volumes and velocity data than what is currently available with other databases, then the following benefits are possible with InfluxDB Enterprise:
◦ Higher rate of ingestion of data.
◦ You can have more than one data repository for run time data. For example, you can keep relational data in PostgreSQL, while using InfluxDB for high volume stream and value stream data. When you define a stream or value stream, ThingWorx uses the default run time data store provider, but you can configure it to use any defined persistence provider. You can still export data from other data providers and import to InfluxDB. ThingWorx handles the data abstraction.
◦ Cloud-friendly architecture (horizontal scale, only with InfluxDB Enterprise).
◦ High availability available.
◦ Influx technical support available with Influx contract using the Influx Support site at InfluxData.
◦ PTC customer can open tickets with Influx as needed for PTC/Influx collaboration.
◦ PTC customer assumes responsibility for database maintenance and monitoring.
• InfluxDB Cloud (Hosted outside the ThingWorx environment and managed by InfluxData company)
The following benefits are possible with InfluxDB Cloud:
◦ InfluxDB Cloud is built as a cloud-native, multi-tenant, elastic scale, serverless platform.
◦ Offers availability, durability, and scalability across multiple regions on AWS, Google, and Azure.
◦ It is secure and hardened with all the latest security features and patches installed and configured.
◦ InfluxDB Cloud automatically rebalances clusters, handles sharding, and leverages built-in replication factors.
Best Practices
Series Limit
Series is the total number of properties for all the Things logged into InfluxDB. InfluxDB performs well with high data volume directed to a small amount of Things and Thing properties, such as 10Ks or 100Ks. The total number of series is limited to 1 million by default in InfluxDB. You can increase the limit, but the performance of InfluxDB decreases as the number of series goes beyond this limit.
If you have a large number of Things and properties, you can choose the ones with the highest data volume and point only them to InfluxDB to take the pressure off from PostgreSQL or MSSQL.
Alternatively, if you want to divide the series to multiple servers, you can have multiple instances of InfluxDB Data Provider pointing to different InfluxDB server instances.
Write Limit
There are 100K writes per second with a 60gb memory VM with 32 cores. Going beyond this limit can cause problems on ThingWorx and it can run out of resources to handle any request or job, such as writing to the database. At that point, ThingWorx becomes stalled while InfluxDB is still writing to the database. This is not an issue with PostgreSQL because PostgreSQL becomes the bottleneck and ThingWorx is never able to get to the point of being out of resources to handle the tasks inside.
SSL/Secure Connection
InfluxDB supports SSL and HTTPS connections. You can enable SSL and HTTPS connections to increase security if the network between ThingWorx and InfluxDB is not supported. A self-signed certificate is adequate, if the signing private key is kept secure.
Known InfluxDB Limitations on Property Base Types
You can not change the base type of a property after it has been logged to a value stream. See Type conflict on insert int value in float field#3460 for more information.
Purging Properties
The PurgeAllPropertyHistory, PurgeSelectedPropertyHistory, and PurgePropertyHistory services can be used to purge properties from InfluxDB. Use the startDate and endDate parameters to specify a range.
| | Purging is not supported when using Influx2PersistenceProviderPackage provider. The PurgePropertyHistory, PurgeAllPropertyHistory, PurgeSelectedPropertyHistory services have no effect. |