Purging Runtime Data
Overview
It is easy to underestimate the amount of data that is generated and stored in a production system. While PTC does not have an official recommendation on how much data to store, since it is heavily dependent on the use case, it is recommended to have a plan in place for any production system to purge the old data for performance reasons. When planning for data purging, the following should be considered:
• How many properties are logged?
• How many Things have logged properties?
• How frequently are those properties logged?
• Should the data be archived for records? Refer to the Runtime Data section.
Very generally, if data is logged every couple of minutes for a few properties on a couple hundred edge devices, data could be kept for several months to a year. If data is logged several times a minute for 100 properties on a thousand different Things, even several weeks of old data may need to be purged. It is recommended to query and purge data in small chunks on a continual basis. Creating a scheduled purge on a more frequent basis (daily instead of weekly) will help with performance.
If data storage is required for longer than one year, PTC recommends using an external database for storage.
How to Purge Data
While the JavaScript to purge data will be different depending on the scenario, the following general steps will be the same:
1. Create a new Thing and set its Base Thing Template to Scheduler.
2. Open the Configuration section and set the runAsUser and schedule properties.
◦ Refer to https://docs.oracle.com/cd/E12058_01/doc/doc.1014/e12030/cron_expressions.htm for information on proper formatting for the schedule property.
▪ Depending on the purge, expect JVM usage to increase during the job. Therefore, it is best to break up large jobs into smaller ones during off-peak hours.
3. Select Subscriptions and click the Add button.
4. In the Subscription Info section, select Me and check Enabled.
5. Click Inputs.
6. In the Event field, select ScheduledEvent.
7. Add JavaScript specifics for the purge job.
 |
For value stream and stream services, if a start date is not provided, all values will be purged, beginning with epoch. If an end date is not provided, all properties up until the time the service is executed will be purged.
|
Scheduler Best Practices
• Do not subscribe to a timer or scheduler event at the Thing Template level. Instead, subscribe on the Thing.
• Do not query vast amounts of data in timers that fire frequently. Reference this article on the PTC Community for additional information.
• Whenever possible, do not query and then update the same data source in a subscription. Instead, query a stream of a data table once in each subscription, and again once at the end of the scheduler job if necessary (using batch update statements).
Purging Value Streams, Streams, File Repositories
Reference Best Practice for Purging Value Streams, Streams, and Old File Repository Files in ThingWorx for more information on purging data.