Data Ordering
 |
From ThingWorx 9.7 onwards, data ordering is available.
|
With the streaming data from devices in the field, events and subscriptions are in place for business logic, data processing, and so on. Data processed through subscription is ingested into the supported database. Data processing in ThingWorx is executed in parallel to support many Things and associated property values.
While data streams to ThingWorx and is processed through events and subscriptions, ordered data is expected for the sequential processing of events on properties. Processed data should also be ingested in sequential order in the database.
Problem Statement
If data order is missed, it impacts events generation, subscriptions, and ultimately, data ingestion. This scenario is observed during Kepware disconnection and reconnection, in High Availability setup (in presence of Connection Server for balancing WebSocket connections). Data Ordering is supported on Single server and High Availability setup deployments.
Solution
Data ordering helps with the following:
• For Connection Server, maintain stickiness for a particular Thing to Pair of ThingWorx ID and websocket (WS) channel ID. This ensures that property value updates for a particular thing are always sent to specific ThingWorx and specific WebSocket channels.
• The eMessage Connector(eMC) internally uses the Connection server to communicate with the ThingWorx Platform. The changes for the connection server help eMessage Connector(eMC) to send ordered data to ThingWorx.
• On the ThingWorx Platform, implement Akka actor per thing. With each incoming property value for a Thing, an actor is created, and the property value is pushed to the Actor’s mailbox. Akka actor then pulls the messages from the queue and processes them sequentially, guaranteeing Ordered data processing on the platform side.
• To enable Data Ordering in BasicSettings of platform-settings.json the following setting can be used.
Setting | Type | Value | Default Value |
|---|---|---|---|
EnableDataOrdering | BOOLEAN | true OR false | false |
• The thingworx_WSExecutionProcessingSubsystem_isDataOrderingEnabled metric at /Metrics endpoint helps you monitor data ordering setting:
Metric Name | Type | Example |
|---|---|---|
thingworx_WSExecutionProcessingSubsystem_isDataOrderingEnabled | gauge | # HELP thingworx_WSExecutionProcessingSubsystem_isDataOrderingEnabled Data ordering enabled # TYPE thingworx_WSExecutionProcessingSubsystem_isDataOrderingEnabled gauge thingworx_WSExecutionProcessingSubsystem_isDataOrderingEnabled{prefix="Platform.Core.Subsystems",category="WSExecutionProcessingSubsystem",platformid=""} 0.0 |
| | Event ordering is a functionality that is delivered and available in 9.5. To turn on this configuration on the Subscription definition, click Execute Events Sequentially. |
Other Component Impact
• Java and C SDK can send property values to ThingWorx with the following APIs:
VirtualThing.setProperty()
VirtualThing.updateSubscribedProperties();
Using the previous APIs, properties are sent to the ThingWorx Platform in sequence.
• To achieve data ordering, if a Connection Server is behind the load balancer, tweak the load balancer configuration to have stickiness. Without this data, ordering can’t be achieved.
• Data Ordering fixes are available in the following components:
◦ Connection Server 9.3.0 and later
◦ eMessage Connector 2.5.0 and later
◦ Kepware 6.16 and later
Benefits
Data ordering has the following benefits:
• Data is processed in order allowing for reliable data storage and processing.
• Enables customers to choose between high throughput parallel processing versus serialized and ordered data processing.
• Enables ThingWorx Solutions, ingesting data from Kepware, to receive and process data in order.