Edge-Controlled File Transfers
Axeda Agent / Platform Initiated File Upload
As shown in the illustration,
• At step no. 2, the following input validations are performed.
◦ If the job is Edge-controlled, validate that the maximum concurrent transfers and concurrent transfer per Thing are exceeding.
◦ If not exceeding, the job is queued for later transfer. If not, Copy fails, and the job is not queued.
• During Egress, in step no. 5:
◦ Determine if there are any downloads, uploads, SCM Operations, or remote property writes operations to perform.
◦ File transfers can only start if the transfer limit is not exceeded.
• During Create or Write to binary file, in step no. 8, the file is locked for writing. If this is the last chunk and checksum is provided, then it is validated.
• Mark the job as failed or complete. Additionally, validate checksum, if provided.
ThingWorx Platform Initiated File Download
As shown in the illustration,
• During file Copy, in step no. 1, the following input validations are performed.
◦ If job is edge controlled, validate max concurrent transfers and concurrent transfer per thing are exceeded.
◦ If not exceeded, job is queued for later transfer else Copy fails and job is not queued.
• During Egress i.e.step no. 4,
◦ Determine if there are any downloads/uploads/SCM Operations/remote property writes/operations to perform.
◦ File transfers can only start if the limits on transfers is not exceeded.
• During Create or Write to Binary file i.e. step no. 7, File is locked for reading.
• Platform accordingly marks the job as failed or complete.
Important tunable parameters
• Configuration settings on the File Transfer Subsystem must be carefully tuned to achieve a good concurrency for file transfers. Some of the settings are listed below. For more details, see File Transfer Subsystem
◦ Total Max Edge-Controlled File Transfers Allowed → Max parallel file transfers available in the Platform for dequeuing a job and moving to active queue. If you need more parallelism, this value must be higher.
◦ Total Max Edge-Controlled File Transfers Allowed Per Thing → Max parallel file transfers per thing available in the Platform for dequeuing a job and moving to active queue. If you have fewer thing with large number of files to be transferred per thing, increasing this might be useful.
 | Increasing this value may contribute to unequal distribution of available positions from the maximum parallel file transfers among devices. |
◦ Max File Transfers Allowed in Offline Queue → If the file transfer upload rate is very high and processing is comparatively slower, then this value must be higher so that the jobs stay in the offline job, instead of being rejected.
◦ Max File Transfers Allowed Per Thing in Offline Queue → If the file transfer upload rate per thing is very high and processing is comparatively slower, then this value must be higher. Then the jobs will stay in the offline job instead of being rejected.
• Edge-controlled file transfers are executed using the WSExecutionProcessor thread pool. Therefore, sufficient threads must be available for file transfers to happen. Use the Execution Processor Settings in the WSExecutionProcessingSubsystem to tune this thread pool. For more information, see WebSocket Execution Processing Subsystem.
• File transfer statistics can be monitored by looking at the different metrics exposed at the /Metrics endpoint. To know more about mfile tramsfer metrics, see Monitoring the ThingWorx Subsystems.
• For more information on tuning the eMessage Connector, see Fine-Tuning the eMessage Connector.
Best Practices
• ThingWorx Platform supports concurrent reading and writing of files in File Repositories. To prevent data corruption and ensure consistency the Platform has to acquire locks on files and directories for duration of read/write operations. Avoiding writing to the same directory or files concurrently allows more concurrency and increases the throughput of file operations.
• For high volumes of file transfers, the file input/ouput must be capable of handling concurrent read/write operations, and must be placed on a separate storage than /ThingworxPlatform.
• Monitor file transfer performance continuously. Refer to file metrics documentation.
• Application developers should be conscious of the relationship between ThingWorx Subscriptions and file transfer operations, as service execution with heavy input/ouput tied to file transfer event completion will utilize the limited TWEventProcessor threads meant for fast running operations. Adapting TWEventProcessor thread pool settings accordingly, as well as ensuring infrastructure performance so that other subscription execution performance is not impacted.