Data Ordering
 |
Data Ordering is beta feature for ThingWorx 9.6 and will be available generally in upcoming release.
|
With the streaming data from devices in the field, events and subscriptions are in place for business logic, data processing, and so on. Data processed through subscription is ingested into the supported database. Data processing in ThingWorx is executed in parallel to support many Things and associated property values.
While data streams to ThingWorx and is processed through events and subscriptions, ordered data is expected for the sequential processing of events on properties. Processed data should also be ingested in sequential order in the database.
Problem Statement
If data order is missed, it impacts events generation, subscriptions, and ultimately, data ingestion. This scenario is observed during Kepware disconnection and reconnection, in High Availability setup (in presence of Connection Server for balancing WebSocket connections).
Solution
In ThingWorx 9.6, the following changes are introduced:
1. For Connection Server, maintain stickiness for a particular Thing to Pair of ThingWorx ID and websocket (WS) channel ID. This ensures that property value updates for a particular thing are always sent to specific ThingWorx and specific WebSocket channels.
2. The eMessage Connector(eMC) internally uses the Connection server to communicate with the ThingWorx Platform. The changes for the connection server help eMessage Connector(eMC) to send ordered data to ThingWorx.
3. On the ThingWorx Platform, implement Akka actor per thing. With each incoming property value for a Thing, an actor is created, and the property value is pushed to the Actor’s mailbox. Akka actor then pulls the messages from the queue and processes them sequentially, guaranteeing Ordered data processing on the platform side.
4. A new flag is exposed to enable Data Ordering in BasicSettings of platform-settings.json.
Setting | Type | Value | Default Value |
|---|---|---|---|
EnableDataOrdering | BOOLEAN | true OR false | false |
| | Event ordering is a functionality that is delivered and available in 9.5. To turn on this configuration on the Subscription definition, click Execute Events Sequentially. |
Other Component Impact
• Java and C SDK can send property values to ThingWorx with the following APIs:
VirtualThing.setProperty()
VirtualThing.updateSubscribedProperties();
Using the previous APIs, properties are sent to the ThingWorx Platform in sequence.
• To achieve data ordering, if a Connection Server is behind the load balancer, tweak the load balancer configuration to have stickiness. Without this data, ordering can’t be achieved.