Durable Queues
Durable queues are queues in the ThingWorx Platform that use Apache Kafka (for on-premises deployments) or Azure Event Hubs (for PTC cloud deployments) as a distributed event streaming platform. These platforms serve as message brokers and data pipelines. They externalize and transfer the in-memory queues in ThingWorx nodes to disk-based storage in the form of durable queues. This process improves internal messaging in ThingWorx Platform in the following ways:
• Enhanced ThingWorx stability with guaranteed at-most-once execution (each message is delivered either once or not at all) for steady state processing. Durable queues ensure there is no data loss due to overflow.
• Improved reliability of event and data processing if ThingWorx nodes fail. For more information, see Batch Processing with Durable Queues.
• Reduced event processing latency by half as compared to in-memory queues.
ThingWorx supports durable queues for the following:
• Unordered events (events subscribed by stateless subscriptions)
• Persistent properties
• Logged properties
The following diagram shows the functional flow of in-memory queues versus durable queues. For more information about in-memory queues, see In-Memory Queues.
After enabling durable queues for events, persistent properties, or logged properties, the property writes are processed to the durable queue and then the in-memory queue.
Durable Queues in High Availability Configurations
Durable queues also support High Availability (HA) configurations. There should be one Apache Kafka or Azure Event Hubs instance configured with an HA setup. For more information about HA configurations, see Sizing for Durable Queues.
The following diagram shows the flow of IoT Streams through Apache Kafka in an HA environment:
Important Considerations
• Only unordered events (events subscribed by stateless subscriptions) are pushed to Kafka or Event Hubs. Stateful events are routed to existing Akka Flows.
• Events and persistent or logged properties are consumed at-most-once.
• For Kafka and Event Hubs, data retention is 24 hours by default.
• The platform can connect to Kafka, create topics, and send and receive messages. If durable queues are enabled, Kafka is considered critical infrastructure. If issues occur when trying to connect to Kafka (such as mismatched credentials), the ThingWorx Platform will not start.