Online and Offline Audit Data (Direct Persistence)
The services and their parameters in the Direct Persistence implementation differ from those in the Data Table implementation. This topic explains online and offline audit data, the services you can use to extract it, and the parameters that are used in the Direct Persistence implementation.
Online Audit Data
When the Direct Persistence implementation is active, the online audit entries are maintained online based on the values specified for the DaysOnline parameter of the audit subsystem. Services are provided to manage the amount of data in the online audit data store:
Archive and purge operations are triggered by the related scheduler Things in ThingWorx Platform. These scheduler Things are responsible for invoking these services periodically. You can configure the schedulers to suit the needs of your organization. By default, these services run daily. You can also invoke them manually when necessary. The online audit messages are persisted to files that are stored in the AuditArchiveDirectPersistence directory of the AuditArchiveFileRepository of the audit subsystem. You cannot configure this repository.
 |
Offline in the context of the audit subsystem means that the audit messages can no longer be filtered or searched, and that they are no longer stored in the online database.
|
The retention criterion for online audit data is the number of days to retain audit data online configuration parameter for the audit subsystem. The ArchiveAuditHistoryDirectPersistence service does not delete data that meet this retention criterion. Instead a separate PurgeAuditData service can be run on a schedule or manually to delete online audit data if the criterion is met.
To search online audit data using the QueryAuditHistoryWithQueryCriteria service, refer to
Offline Audit Data
Offline audit data provides a storage location for audit data that you do not need to access often but that you want to keep for administrative purposes. The ArchiveAuditHistoryDirectPersistence service copies online audit data to offline storage. The following services enable administrators to manage the offline data:
The ExportAuditData service calls the ArchiveAuditHistoryDirectPersistence service, which copies the requested online audit data from the AuditArchiveFileRepository to the specified export file.The export operation processes and translates that archived data and stores it in the specified export file.
The CleanUpOfflineAudit service typically runs based on a scheduler. It can also be run manually. You pass the service an olderThanDate to specify the archived files to be deleted from the AuditArchiveFileRepository.
Offline files are archived without localization. An offline reader service, ExportAuditData, is available to retrieve them, based on a specified locale. Refer to Exporting Audit Data for more information.
The following diagram shows the hierarchy for offline archive files. The paragraph following the diagram explains the node just below the FileRepository, "OfflineAudit folder for active implementation".
Hierarchy of Offline Archive
In the diagram above, the node, "OfflineAudit folder for active implementation", represents the main directory in the repository for the active implementation, Direct Persistence and/or Data Table. If you began with the Data Table implementation and later switched to the Direct Persistence implementation, there will be two directories here, AuditArchive for the Data Table implementation and AuditArchiveDirectPersistence for the Direct Persistence implementation. Both directories have the same subdirectory structure shown above. Archived audit data is preserved when you switch from one implementation to the other.
 | If you decide to switch from one implementation to the other, be sure that you have archived any audit data that you require to be preserved before making the switch. |
The dated folders are for the date of the Audit messages they contain, not the date that the archive was run. As shown in the diagram above, the format of the dated folder name is YYYY-MM-DD (ISO 8601) format, with a new folder for every day of audit messages. The "Offline Audit Chunks" are in a zipped JSON file that contains audit messages for that date. The default chunk size is 50,000 rows. Note that the minimum size is 10,000 rows. A system administrator can change the default size of chunks in the configuration of the audit subsystem. For example, the system administrator could specify a chunk size of 70,000 rows. A zipped chunk file would exist for that many rows of messages in the dated folder. If every audit entry were ~1KB, the size of a chunk file would be around ~70MB unzipped or in memory.
Data Retention and Performance
To maximize performance, a balance of online audit data and offline data storage is required. The audit subsystem provides both archiving and purging operations to help you achieve the optimum balance for your system. While the archive operation does not check the DaysOnline parameter of the subsystem, the PurgeAuditData service does check it.
When its scheduler calls the ArchiveAuditHistoryDirectPersistence service, the current date is passed as the olderThanDate. All online audit messages older than this date are archived. This service does not check the DaysOnline parameter of the audit subsystem.
The PurgeAuditData service checks the value of the DaysOnline parameter of the audit subsystem to determine which online audit messages should be purged and which should remain available online. Before deleting audit messages, this service checks if there are any online audit messages marked for deletion that have not been archived. If so, this service calls the ArchiveAuditHistoryDirectPersistence service. These actions ensure that any data that is deleted is archived before being deleted.
| | If the value of DaysOnline is 0 and the purge service is called, all the data is moved to the offline repository and then deleted from the online repository. |
Exporting Online and Offline Audit Data
As of ThingWorx Platform v.9.0.0, the Audit Subsystem has two export services, one for online audit data only and one for both online and offline audit data. Refer to Exporting Online Audit Data and Exporting Audit Data for information on using these services to export audit data.