Property Transform Architecture

The PropertyTransform microserver provides a set of analytic transformation services that can be used to derive value from streaming data entering ThingWorx. The transformed output from these services can be used to visualize data, create alerts, or run subsequent machine learning processes. Interaction with these property transform services requires that ThingWorx Foundation is already installed and running.

1. PropertyTransform Microserver – This microserver includes services to provide the following statistical transformations: Maximum, Mean, Median, Minimum, Mode, Standard Deviation, Range, Threshold Count, Range Count, and Trend Count. The microserver monitors the RabbitMQ message exchange for changes to properties on user-created Things, Thing Templates, or Thing Shapes, and is responsible for starting and stopping transform jobs in the external Apache Flink processing engine.

During installation of the PropertyTransform microserver, a Java service starts and spins up a Virtual Thing. The Virtual Thing contacts ThingWorx and creates a corresponding Remote Thing, called PropertyTransformThing.

2. PropertyTransform Thing – The Remote Thing is an extension of the corresponding microserver. You can see it in your list of Things in ThingWorx but you don’t need to interact with it, except to verify installation. When a call is made to create a transform property, the Remote Thing is contacted behind the scenes to validate that the PropertyTransform microserver is connected and the Flink cluster is running.

3. User-created Thing/Thing Template/Thing Shape – When you create a transform property on a Thing, Thing Template, or Thing Shape, you can define the parameters of the property transformation, including what computation to perform, how frequently, and over what amount of data. Changes to the property definition are sent as ongoing metadata, via the RabbitMQ message exchange, to the PropertyTransform microserver.

Subsequent updates to the source property data are sent continuously to the Flink transform job, via the RabbitMQ exchange.

4. Flink – Apache Flink is a stream processing engine, external to ThingWorx, that the PropertyTransform microserver uses to handle computations. Flink manages the data stream and performs the transformations based on the user-provided property definition. To support the streaming process, Flink communicates through the RabbitMQ message exchange, both to receive data and to return computation results.

5. RabbitMQ Exchange – This external message broker handles event messaging between the property transform components. The exchange passes property transform definitions from ThingWorx to the PropertyTransform microserver. It also continuously passes updated data from the source property in ThingWorx to the Flink transform job. When data is transformed, the exchange returns the calculation results from Flink, via a result queue, to the transformed property in ThingWorx. Each Thing that includes a transformed property has its own result queue and calculation results are written to the appropriate queue.

When you add a Property Transform to a Thing, Thing Template, or Thing Shape in ThingWorx, you define various aspects of the property. This property definition includes information about which source property to base the transformation on, how often to launch the transformation, and how much data to include. The property definition is sent through the RabbitMQ exchange to the PropertyTransform microserver. When the microserver receives the property definition, it starts a Flink transform job.

Meanwhile, the RabbitMQ exchange monitors the source property in ThingWorx for any updates. It continuously sends the updated source property data to Flink for use in the transform job. Flink monitors the incoming data and performs the transform calculations according to the property definition. The computation results are returned, via RabbitMQ, from Flink to the property transform in ThingWorx.