File Transfers
A file transfer moves a file from one file repository to another. File repositories may exist on the ThingWorx Platform or on a remote device. On a remote device, the file repository is the file system on the device. On ThingWorx Platform, file repositories are things that implement the required services for reading from them and writing to them.
In a ThingWorx environment, files are transferred by calling the Copy() service on the File Transfer Subsystem (FTSS) of the ThingWorx Platform. The FTSS orchestrates the movement of files. When a transfer completes or if it results in an error, the FTSS notifies the requester.
File Transfers with ThingWorx Edge SDKs
The ThingWorx Platform and the Edge SDKs transfer files using the AlwaysOn protocol, which allows chunking of messages. You can use chunking to avoid generating continuation frames. Note that file transfer is a protocol on top of websockets and AlwaysOn. To help you understand what this means, here are some definitions:
• Messages — Deliver AlwaysOn requests and responses. For best performance, you can and should set messageSize and maxMessageSize (AlwaysOn) to 1,000,000 bytes.
• Frames — WebSocket messages. The size is fixed at 8192 bytes. Do NOT change this value.
• Chunks — Portions of a message that fit in the frames. Chunking allows messages to be larger than the frame size. In general the chunk size is the same is the frame size.
• Blocks — Packages that contain portions of files that have been broken up so that they can be sent as AlwaysOn messages. Blocks must fit inside messages. The default size of a file transfer block is 124KB. For better performance, change the block size to 1MB. The BlockSize and MaxFileSize properties are specific to file transfers. MaxFileSize protects you from someone filling your repository and perhaps your file system with a huge file, potentially creating a denial of service (DoS) attack. BlockSize defines how the size of your file is broken up into chunks for transmission. A 1000 byte file sent with 100 byte BlockSize is sent in 10 service calls.
• MaxMessageSize and BlockSize are two properties that can be changed to improve performance. They are related, with MaxMessageSize set larger than BlockSize to allow your service call to be sent because a service call must fit into the maximum message size. More specifically MaxMessageSize should be at least 50*(BlockSize/ChunkSize) bytes greater than BlockSize. For example if chunk size is 8192 bytes and BlockSize is 1MB, then MaxMessageSize should be at least 50*(1000000/8192) or approximately 6104 bytes greater than BlockSize. In this example, the MaxMessageSize would be at least 1,006,104 bytes.
 |
Testing has shown that a Block size of 1MB for file transfers (upload and download) is the best setting for most network speeds. In addition, if you lower the block size, performance improves until the size is 1MB, and then performance does not change. In addition, downloads were always faster than uploads.
|
Queueable File Transfers
Unless specified, file transfers between an edge client and the ThingWorx Platform require the client to be connected from the start of the transfer to the completion. To initiate a file transfer from a ThingWorx Platform while an edge client is disconnected, you can use the queueable functionality. Queueable file transfers are placed in an offline queue and execute when the client connects.
After the transfer starts, the edge client cannot disconnect, or the transfer will fail. File transfers that are queued expire if they do not begin before the number of seconds set for the Time To Live (sec) of a Queueable File Transfer configuration setting for the FTSS.
If the device goes offline before a queued transfer is performed, the transfer job is re-queued. If a file transfer job is re-queued frequently, the File Transfer Subsystem generates an event notification that says the job is stalled. The event is generated after the transfer has been queued for 12 hours, then again after 18 hours, after 22 hours, and after 23 hours. After 24 hours, the job is discarded, and the file transfer must be requested again.
Before submitting a file transfer job to the queue, the File Transfer Subsystem checks whether the related device is connected. If not, the job is placed in the offline queue. If that queue is full or another exception occurs, the state of the transfer job is updated and the exception is thrown.
A queued file transfer is re-queued only if the edge client has not yet been notified of the transfer request. Once the agent is notified, the job completes or fails, depending on the status returned from the client. Both the destination and the source for a file transfer must be online for a transfer job to start. More specifically, the edge client must be connected and the FileRepository on the ThingWorx Platform specified for the transfer must be online.
Edge-Controlled File Transfers
For an edge client to control file transfers, the Thing that represents the client must implement the EdgeControlled Thing Shape You can add this Thing Shape from ThingWorx Composer. Edge-controlled file transfers must have either the sourceRepo or the targetRepo set as a FileRepository Thing on the ThingWorx Platform. In an edge-controlled file transfer, the edge invokes the Touch service to update the timestamp of the file transfer as it progresses. If the edge client does not do this, the ThingWorx Platform times out the file transfer. The edge client indicates whether the transfer was successful or completed with an error in its log.
Additional Resources
For information about initiating file transfers from a ThingWorx Platform using the Copy service, see Transfer Files Using the Copy Service.