Online and Offline Audit Data (Data Table)
Online Audit Data
When the Data Table implementation is the active implementation, the online audit entries are maintained online based on the values specified for the maxEntriesOnlineAuditData and DaysOnline parameters of the audit subsystem. The following services enable administrators to manage the amount of data in the online audit data table:
Archive and purge operations are triggered by the related scheduler Things in ThingWorx Platform. These scheduler Things are responsible for invoking these services periodically. You can configure the schedulers to suit the needs of your organization. By default, these services run daily. The online audit messages are persisted to files that are stored in the AuditArchive directory of the AuditArchiveFileRepository of the audit subsystem. You cannot configure this repository.
 |
Offline in the context of the audit subsystem means that the audit messages can no longer be filtered or searched, and that they are no longer stored in the data table.
|
As of v.8.5.0 of the ThingWorx Platform, the ArchiveAuditHistory service no longer deletes data that meet either of the retention criteria. Instead a separate PurgeAuditData service deletes this data if either criteria is met.
Refer to Searching Audit Messages for optional parameters of the QueryAuditHistory service.
Offline Audit Data
Offline audit data provides a storage location for audit data that you do not need to access often but that you want to keep for administrative purposes. The ArchiveAuditHistory service copies online audit data to offline storage. The following services enable administrators to manage the offline data:
The export operation calls the ArchiveAuditHistory service, which copies the requested online audit data from the AuditArchiveFileRepository to the specified export file.The export operation processes and translates that archived data and stores it in the specified export file.
The CleanUpOfflineAudit service typically runs based on a scheduler. It can also be run manually. You pass the service an olderThanDate to specify the archived files to be deleted from the AuditArchiveFileRepository.
Offline files are archived without localization. An offline reader service, called ExportAuditData, is available to retrieve them, based on a specified locale. Refer to Using the ExportAuditData Service for more information.
The following diagram shows the hierarchy for offline archive files. The paragraph following the diagram explains the node just below the FileRepository, "OfflineAudit folder for active implementation".
Hierarchy of Offline Archive
In the diagram above, the node, "OfflineAudit folder for active implementation", represents the main directory in the repository for the active implementation, Direct Persistence or Data Table. If you start with the Data Table implementation and later switch to the Direct Persistence implementation, there will be two directories here, AuditArchive for the Data Table implementation and AuditArchiveDirectPersistence for the Direct Persistence implementation. Both directories have the same subdirectory structure shown above. Archived audit data is preserved when you switch from one implementation to the other.
 | If you decide to switch from one implementation to the other, be sure that you have archived any audit data that you require to be preserved before making the switch. |
The dated folder is for the date of the Audit messages it contains, not the date that the archive was run. As shown in the diagram above, the format of the dated folder name is YYYY-MM-DD (ISO 8601) format, with a new folder for every day of audit messages. The "Offline Audit Chunks" are in a zipped JSON file that contains audit messages for that date. The default chunk size is 50,000 rows (the minimum size). A system administrator can change the default size of chunks in the configuration of the audit subsystem. For example, the system administrator could specify a chunk size of 70,000 rows. A zipped chunk file would exist for that many rows of messages in the dated folder. If every audit entry were ~1KB, the size of a chunk file would be around ~70MB unzipped or in memory.
Data Retention and Performance
To maximize performance, a balance of online audit data and offline data storage is required. The audit subsystem provides archiving and purging operations to help you achieve the optimum balance for your system.
You can use the following properties of the subsystem to control the retention of online audit data:
• DaysOnline — The purge operation checks the DaysOnline parameter of the audit subsystem to determine which audit messages should be archived and which should remain available online. The DaysOnline parameter specifies how may days’ worth of audit data should be retained online. The default number of days to retain online audit data is 60.
• maxEntriesOnlineAuditData — For the Data Table implementation only, this threshold setting for the audit subsystem specifies the number of rows that can exist in the online audit data table before the purge operation is invoked.
When its scheduler calls the ArchiveAuditHistory service, the current date is passed as the olderThanDate. All online audit messages older than this date are archived. This service does not check the DaysOnline parameter of the audit subsystem.
The PurgeAuditData service checks the value of the DaysOnline parameter of the audit subsystem to determine which online audit messages should be purged and which should remain available online. Before deleting audit messages, this service checks if there are any online audit messages marked for deletion that have not been archived. If so, this service calls the ArchiveAuditHistory service to store the messages offline. These actions ensure that any data that is deleted is archived before being deleted.
| | If the value of DaysOnline is 0 and the purge service is called, all the data is moved to the offline repository and then deleted from the online repository. |
Refer to Exporting Audit Data for information on using the ExportAuditData service to export online and offline audit data for the Data Table implementation.