Tuning and Scaling the ThingWorx Flow Engine
The ThingWorx Flow Engine service contains a number of configuration options that can have an outsized effect on the performance of ThingWorx Flow and its execution of workflows. These options can be found in the deploymentConfig.json file in the engine module. If any of these configuration options are changed, then the ThingWorxFlow service needs to be restarted using the native service control tools for your Operating System (Services or sc for Windows, sysctl for Red Hat). The following sections describe each of the relevant settings and provides recommendations on how to configure them based on your environment and system resources.
Execution Throughput (CHANNEL_MAX_FETCH)
• Type—Integer
• Default—10
• Units—Total messages
Controls the maximum number of workflow execution messages the Engine service will handle. These messages are transmitted by the ThingWorx Foundation whenever a Workflow service is executed, including standalone workflows that are manually executed or that use a trigger. Lowering this number will limit the number of workflows that the Engine can execute simultaneously.
Development environments should configure this to match the number of cores on your system. Production systems should configure this between 50% and 200% the number of cores, depending on the number of other services that run on the same machine.
Worker Memory Allocation (ENGINE_SIZE)
• Type—Integer
• Default—1802
• Units—Megabytes (10242 bytes)
This represents the maximum amount of memory in megabytes an Engine worker process can allocate for objects. Factors that should be taken into account for configuring this value includes:
• Number of activities in a workflow
• Complexity of inline expressions or Node.js actions
• Amount of data retrieved or processed per action
Deployment Recommendations—It is recommended to leave it at its default in development environments. For production environments, you should configure ENGINE_SIZE to be approximately the maximum amount of system RAM divided by the setting used for CHANNEL_MAX_FETCH.
Maximum Execution Time (MAX_FLOW_RUN_TIME)
• Type—Integer
• Default—300000
• Units—Milliseconds
Controls the total amount of time in milliseconds a workflow is allowed to execute per run. This setting will prevent an Engine worker from executing a workflow for an overly long period of time. This value will vary based on deployment and the network environment present; however, it is recommended to keep this value low to keep workers from remaining unavailable for long periods of time, as this can cause a degradation in overall performance of Flow. An appropriate balance will need to be made between providing enough time to allow the workflow to complete execution, and keeping the value low enough such that engine workers are made available as soon as possible.
Deployment Recommendations—It is recommended to leave it at its default for development environments, or longer values if needed for debugging purposes. For production environments, it is recommended to set the value to the length of your longest running workflow plus 15%.
Worker Recovery Period (SLEEP_BETWEEN_FLOW_EXECUTION)
• Type—Integer
• Default—3000
• Units—Milliseconds
Inserts an additional delay in milliseconds before an Engine worker is able to execute a workflow. This is primarily meant to allow for the operating system to reclaim memory from killed Engine workers, allowing for increased security in environments that require not sharing memory space between workflow executions.
Deployment Recommendations—For development and most production environments, it is recommended to set this value to 0. For multi-tenant environments that have sensitive information that cannot be intermingled, it is recommended to leave this at the default or increase as needed. There is no value that guarantees the operating system has properly recycled the memory.
Kill Engine Workers After Flow Execution (KILL_WORKER_AFTER_RUN)
• Type—Boolean
• Default—false
Whenever a workflow is executed, the Engine service will create a worker process that handles the actual execution of the workflow (up to the value set by CHANNEL_MAX_FETCH). By default, the worker processes will remain active to service additional execution requests. This setting modifies that behavior such that once the execution of the workflow completes, the worker process terminates.
Deployment Recommendations—For development and most production environments, it is recommended to leave this value at its default. For multi-tenant environments that have sensitive information that cannot be intermingled, it is recommended to set this to true.
Worker Availability Retry Count (AVAILABLE_WORKER_CHECK_TRIES)
• Type—Integer
• Default—10
• Units—Retries
If all worker processes are currently in use but a workflow request is pending, the Engine service will continue to attempt to reserve a worker process as many times as configured. If the maximum number of retries has been attempted, then Engine will fail the execution request.
Deployment Recommendations—For development purpose, it is sufficient to keep this value at its default. For production environments, it is recommended to set this value at a reasonable value; in conjunction with the AVAILABLE_WORKER_CHECK_INTERVAL setting, you will not want the total delay to exceed approximately half the total request service time.
Worker Availability Retry Delay (AVAILABLE_WORKER_CHECK_INTERVAL)
• Type—Integer
• Default—3000
• Units—Milliseconds
Determines how long the Engine service waits before attempting to reserve a worker process for workflow execution, under the assumption that the first attempt failed.
Deployment Recommendations—For development purposes, it is sufficient to keep this value at its default. For production environments, it is recommended to set this value at a reasonable value; in conjunction with the AVAILABLE_WORKER_CHECK_TRIES setting, you will not want the total delay to exceed approximately half the total request service time.
Worker Process Spindown Delay (WORKER_DISMISS_INTERVAL)
• Type—Integer
• Default—1800
• Units—Seconds
Specifies how long the Engine service waits for additional work before it begins terminating the worker processes. This value does not have any meaning if KILL_WORKER_AFTER_RUN is set to true.
Deployment Recommendations—In On-premise environments, it is generally sufficient to leave this value at its default. For cloud deployments or memory-constrained environments, it may be desirable to reduce this value, particularly if workflows are only occasionally used.
Sample Engine Configuration
The sample engine deploymentConfig.json file is as follows:
{
"MAX_FLOW_RUN_TIME": 180000, // Only allows a workflow to run for up to 3 minutes.
"EXCHANGE_STATUS_CHECK_INTERVAL": 120000, // Checks if it can communicate with the Exchange service every 2 minutes.
"REFRESH_ON_INTERVAL_MINUTES": 60,
"ENGINE_PORT": 2006, // The port Engine will reserve for health check purposes
"ENGINE_HOST": "localhost", // The hostname Engine will reserve the port against
"SLEEP_BETWEEN_FLOW_EXECUTION": 3000, // Waits 3 seconds before a worker can execute the next workflow
"ENGINE_DATA_PATH": "C:/ThingWorxOrchestration/modules/cache", // Location where files are stored during workflow execution
"EXCHANGE": { // The host and port for the Exchange service that this Engine should communicate with
"HOST": "localhost",
"PORT": "7822"
},
"logLevel":"error", // valid levels are 'trace', 'debug', 'info', 'warn', and 'error'; default is 'error'
"QUEUE": {
"MAX_SOCKET": 100,
"QUEUE_CONSUMPTION_UNIT": {
"256": 1
},
"DEFAULT_FLOW_QUEUE": "256",
"ACTIVE_ADAPTER_DEFAULT": "amqp",
"ACTIVE_ADAPTER_FLOW": "amqp",
"ACTIVE_ADAPTER_STOP": "amqp",
"ACTIVE_ADAPTER_WEBHOOK": "amqp",
"ADAPTERS": {
"AMQP": {
"CONFIG": { // The host and port for the RabbitMQ server
"port": "5672",
"protocol": "amqp",
"host": "localhost",
"vhost": "orchestration",
"CHANNEL_MAX_FETCH": "5" // The maximum number of messages this Engine will fetch from the queue
},
"FLOW_QUEUE": {
"QUEUE_NAMES": {
"256m": "256mb"
},
"QUEUE_NAME": "256mb"
}
}
}
}
}