File Transfer Subsystem
The file transfer subsystem provides the required methods to manage file transfers between Remote Things, file repositories, and federated servers.
Unless specified, file transfers between an edge client and the ThingWorx Platform require the client to be connected from the beginning of the transfer through completion. ThingWorx Platform file transfers can be initiated while the edge client is disconnected by instructing the ThingWorx Platform to perform the copy using the queueable functionality. Queueable file transfers are placed in an offline queue and execute once the agent is connected.
 |
If you are using PostgreSQL as your persistence provider, the offline queue is persisted to support failover for ThingWorx high availability.
|
After the transfer is initiated, the edge client cannot disconnect or the transfer will fail. File transfers that are queued will expire if they do not begin before the Time To Live (sec) of a Queueable File Transfer configuration setting value.
Generally, file transfers are controlled by the ThingWorx Platform. However, you can have edge-controlled file transfers, which are activated by adding the EdgeControlled Thing Shape to the Remote Thing that is participating in the file transfer. Edge-controlled file transfers must have either the source or the destination set as a FileRepository Thing on the ThingWorx Platform. In an edge-controlled file transfer, the transfer is controlled by the edge client. The edge invokes the Touch service to update the timestamp of the file transfer as it progresses. If the edge does not do this, the ThingWorx Platform times out the file transfer. The edge indicates whether the transfer was successful or completed with an error.
You can embed context by providing the metadata parameter when initiating the file transfer. This field is a JSON object, which can contain any arbitrary JSON. Certain base functionality will result in the ThingWorx Platform using this field to supply additional instructions to the edge client. For these transfers, the information in the metadata field should not be modified by any process other than the ThingWorx Platform.
File Transfer between Federated Servers
To transfer files between federated servers, do the following:
|
|
A forward slash (/) is the recommended path delimiter for file repositories.
|
1. Configure a federation between ThingWorx ServerA and ThingWorx ServerB. For more information, see Configuring a Federation.
2. Add ThingA to ServerA using the FileRepository Thing Template.
a. Select the Published checkbox for ThingA so it is accessible by other ThingWorx servers.
3. Add RemoteThingA to ServerB using the RemoteThingWithFileTransfer Thing Template.
a. In the Identifier field, enter ThingA@ServerA.
4. Copy your file (for example, test.txt) to the SystemRepository root folder of ServerB.
5. Open the FileTransferSubsystem on ServerB and execute the copy service with the following parameter values:
◦ sourceRepo: SystemRepository
◦ sourcePath: /
◦ sourceFile: test.txt
◦ targetRepo: RemoteThingA
◦ targetPath: /
◦ targetFile: test.txt
◦ Do not change the other parameters from their default values.
6. Go to the /ThingworxStorage/repository/ThingA folder on ServerA and verify that the test.txt file was successfully copied there.
Configuration
|
File Transfer Settings
|
Data Type
|
Default
|
Notes
|
||
|---|---|---|---|---|---|
|
Min Threads Allocated to File Transfer Pool
|
NUMBER
|
10
|
Defines the core pool size for ThreadPoolExecutor. This thread pool is used to coordinate the platform controlled file transfer logic.
|
||
|
Max Threads Allocated to File Transfer Pool
|
NUMBER
|
10
|
Defines the maximum thread pool size for ThreadPoolExecutor.
Asynchronous file transfers can be lost if ThingWorx goes down. For example, suppose this setting has the default value of 10 and 50 long-running file transfers are submitted. If ThingWorx goes down, 40 files will be lost.
|
||
|
Max Queue Entries Before Adding New Working Thread
|
NUMBER
|
100
|
Defines the upper limit for the number of entries in the queue used in ThreadPoolExecutor.
This restricts the number of active transfers allowed at a time.
|
||
|
Idle Thread Timeout (sec)
|
NUMBER
|
600000
|
Defines how long to keep idle threads alive in the ThreadPoolExecutor. The pool will terminate threads and move back to the core pools size after the specified time.
|
||
|
FileTransfer Idle Timeout (sec)
|
NUMBER
|
30
|
Between every step during the file transfer process (checksum, ReadFromBinaryFile, WriteToBinaryFile, validation) idle timeout is checked. If the step is taking longer than the defined timeout time, the transfer is cancelled.
|
||
|
Max FileTransfer block size (bytes)
|
NUMBER
|
128000
|
Defines the number of bytes requested for ReadFromBinaryFile and WriteToBinaryFile operations. This represents the chunk size for each write.
This variable imposes a maximum block size during a file transfer at the system level.
The EMS configuration still takes precedence. However, if the EMS is configured with buffer_size larger than what is specified in this variable, then this variable will cap this block size. If larger block sizes (128KB) are configured through the EMS, this value must be increased. The maximum compile level is 1MB.
|
||
|
Max FileTransfer size (bytes)
|
NUMBER
|
100000000
|
Defines the maximum number of bytes that a Copy operation supports.
If the source file is larger than this value, the transfer fails and an error message is displayed.
|
||
|
Max File Transfers Allowed in Offline Queue
|
NUMBER
|
50000
|
Defines the maximum number of offline queued file transfers allowed in the system.
|
||
|
Max File Transfers Allowed Per Thing in Offline Queue
|
NUMBER
|
10
|
Defines the maximum number of offline queued file transfers allowed per thing.
|
||
|
Time to Live (sec) of a Queueable File Transfer
|
NUMBER
|
86400
|
Defines the maximum time that a queued file transfer can stay in the offline queue.
A file transfer is be removed from the offline queue after this specified amount of time.
|
||
|
Total Max Edge-Controlled File Transfers Allowed
|
NUMBER
|
500
|
Defines the maximum active edge-controlled file transfers that is allowed in the system.
The total concurrent edge-controlled transfers across the platform. This is separate from the maximum allowed always-on transfers, which are controlled by the Max Queue Entries Before Adding New Working Thread setting.
The upper limit of this property is controlled by MaxConcurrentFileTransfersEdgeCtrl in platform-settings.json file. The default value is 1000.
|
||
|
Total Max Edge-Controlled File Transfers Allowed Per Thing
|
NUMBER
|
2
|
Defines the maximum number of concurrent transfers allowed to or from an edge-controlled Thing. For example, a value of 2 means that one edge-controlled Thing can have only two active transfers (upload or download) at a given time. Subsequent requests to dequeue a file transfer egress are rejected until there is sufficient capacity.
|
||
|
Idle Timeout for Edge-Controlled File Transfers (sec)
|
NUMBER
|
600
|
Defines the maximum time an active job can remain active without being acted on (for example, by a data transfer or a job state update). The range is 1–3600 seconds. This is similar to the idle timeout for always-on transfer jobs, but usually it takes longer to account for the ping cycle of polling devices.
|
||
|
File Transfer Cleanup Frequency (sec)
|
NUMBER
|
10
|
Defines the frequency of the cleanup task for evaluating the file transfer operations.
The cleanup task removes stale and expired jobs from the active jobs table. This action releases the file transfer slots, which will help other queued operations. The recommended minimum and maximum values must be between 1 second and 60 seconds, respectively.
The value for this setting must be set judicially. The default value, 10 sec, is expected to work for most of the use cases.
• A very small value will start the background task too frequently. It will get all currently active file transfer jobs from the cache and evaluate for expiration. This will also add unnecessarily computation and load on caches.
• A very high value will start this task less frequently and there could be stuck file transfers holding the valid reservation and possibly not releasing the reservation in timely manner. This could possibly impact overall file transfer operations if there are many stuck operations like unstable network, device not connecting, and so on.
|