Custom Log Retrieval Strategy for Querying Logs from External Sources

ThingWorx Model Definition in Composer > System > Logs > OpenTelemetry for ThingWorx Platform Logging > Custom Log Retrieval Strategy for Querying Logs from External Sources

Logs observability can be achieved through different third-party enterprise logging provider applications available in the market. These applications have multiple ways to push and retrieve logs.

With OpenTelemetry, logs are pushed to logging provider applications.

The primary purpose of this Log Retrieval Strategy is to retrieve logs from 3rd party source and display these logs on the Composer > Monitoring > Logs page.

As part of the overall implementation, ThingWorx provides the following:

• Default strategy Thing DefaultLogRetrievalStrategyThing which retrieves logs from ThingWorx File logs and Sumo Logic depending on the configuration.

• Extension SDK to develop a custom Log Retrieval strategy.

DefaultLogRetrievalStrategyThing is a system object. Only its configuration can be updated. Users are not supposed to use this DefaultLogRetrievalStrategyThing template for implementing custom log retrieval strategies.

Log retrieval is handled by RetrieveLogs service available with DefaultLogRetrievalStrategyThing. Log retrieval is based on,

• If file logging is disabled (DisableFileLogging is set to true), and OpenTelemetry is enabled (EnableOpenTelemetryForLogging is set to true), only then logs are retrieved from sumo logic.

• For all other cases, logs will be retrieved from files.

In case of custom implementation for Log Retrieval, RetrieveLogs Service can be implemented in following 2 ways.

• Through Composer using JavaScript Service

a. Create a thing in Composer with base template as LogRetrievalStrategy.

b. Go to Services and override the RetriveLogs service.

c. Provide your JavaScript code to retrieve logs from third-party logging software.

Usually Content loader functions or similar functions are used to communicate with third-party.

• By creating an Extension in ThingWorx using Extension SDK

a. Create an extension in Eclipse using latest SDK (>=9.6.0)

b. Create a thing template (java class) with base template as LogRetrievalStrategy.

Your class should extend Log Retrieval Strategy class.

c. Override the Retrieve Logs java method, and implement your custom code to retrieve logs from third-party logging software.

d. (Optional) Update metadata.xml of the extension and create a thing. You can also create thing later in Composer after importing extension.

e. Build extension and import it into ThingWorx.

In both the cases, once you have a thing, you need to set this thing as LogRetrievalStrategy thing in Logging subsystem.

This tutorial does not cover the basics of creating ThingWorx extensions. It assumes you:

• Know how to build a ThingWorx extension.

• Have the Extension SDK, v.9.6.0 or later, which includes LogRetrievalStrategy thing template.

• Know how to build and import an extension into the ThingWorx Platform.

Implementing a Custom Log Retrieval Strategy

1. To create a custom Log Retrieval strategy in Java, create a strategy Thing Template. This Thing Template serves as the code base for a Java-based algorithm. The class definition for the Thing Template is as follows:

@ThingworxBaseTemplateDefinition(name = "LogRetrievalStrategy")
public class CustomLogRetrievalStrategy extends LogRetrievalStrategy{

2. Implement and override the RetrieveLogs Method.

@ThingworxServiceDefinition(
name = LoggingConstants.RETRIEVE_LOGS_STRATEGY_SERVICE_NAME,
isAllowOverride = true,
description = "Retrieve Logs from the given source",
category = "Queries")
@ThingworxServiceResult(name = "result", description = "Table entries", baseType = "INFOTABLE", aspects = { "dataShape:LogEntry" })
@Override
public InfoTable RetrieveLogs(
@ThingworxServiceParameter(
name = LoggingConstants.MAX_ITEMS,
description = "Maximum number of items to return",
baseType = "NUMBER",
aspects = { "defaultValue:" + RESTAPIConstants.DEFAULT_ITEM_COUNT }) Double maxItems,
@ThingworxServiceParameter(
name = LoggingConstants.START_DATE,
description = "Start time",
baseType = "DATETIME") DateTime startDate,
@ThingworxServiceParameter(
name = LoggingConstants.END_DATE,
description = "End time",
baseType = "DATETIME") DateTime endDate,
@ThingworxServiceParameter(
name = LoggingConstants.FROM_LOG_LEVEL,
description = "From log level",
baseType = "STRING") String fromLogLevel,
@ThingworxServiceParameter(
name = LoggingConstants.TO_LOG_LEVEL,
description = "To log level",
baseType = "STRING") String toLogLevel,
@ThingworxServiceParameter(
name = LoggingConstants.USER,
description = "Log messages for a specific user",
baseType = "USERNAME") String user,
@ThingworxServiceParameter(
name = LoggingConstants.THREAD,
description = "Log messages for a specific thread",
baseType = "STRING") String thread,
@ThingworxServiceParameter(
name = LoggingConstants.PLATFORM_ID,
description = "Log message from a specific instance",
baseType = "STRING") String platformId,
@ThingworxServiceParameter(
name = LoggingConstants.ORIGIN,
description = "Specific class or log source",
baseType = "STRING") String origin,
@ThingworxServiceParameter(
name = LoggingConstants.INSTANCE,
description = "Specific class or log source",
baseType = "STRING") String instance,
@ThingworxServiceParameter(
name = LoggingConstants.SEARCH_EXPRESSION,
description = "Keywords to search content",
baseType = "STRING") String searchExpression,
@ThingworxServiceParameter(
name = LoggingConstants.SORT_FIELD_NAME,
description = "Sort field name",
baseType = "STRING") String sortFieldName,
@ThingworxServiceParameter(
name = LoggingConstants.ASCENDING_SEARCH,
description = "Ascending search indicator (true = ascending, false = descending)",
baseType = "BOOLEAN") Boolean ascendingSearch,
@ThingworxServiceParameter(
name = LoggingConstants.OLDEST_FIRST,
description = "Search/sort from oldest to newest",
baseType = "BOOLEAN") Boolean oldestFirst,
@ThingworxServiceParameter(
name = LoggingConstants.IS_REGEX,
description = "The Search expression should be treated as REGEX",
baseType = "BOOLEAN",
aspects = { "defaultValue:" + false }) Boolean isRegex,
@ThingworxServiceParameter(
name = "logName",
description = "Keywords to search content",
baseType = "STRING",
aspects = { Aspects.ASPECT_ISREQUIRED + ":true", Aspects.ASPECT_SELECTOPTIONS +
":ApplicationLog:ApplicationLog|CommunicationLog:CommunicationLog|" +
"ConfigurationLog:ConfigurationLog|ScriptLog:ScriptLog|SecurityLog:SecurityLog" }) String logName)
throws Exception {

// Create InfoTable to hold results
InfoTable logEntriesTable = new InfoTable();

// Add Field Definitions to InfoTable.
logEntriesTable.addField(new FieldDefinition(LoggingConstants.CONTENT, BaseTypes.STRING));
logEntriesTable.addField(new FieldDefinition(LoggingConstants.USER, BaseTypes.STRING));
logEntriesTable.addField(new FieldDefinition(LoggingConstants.THREAD, BaseTypes.STRING));
logEntriesTable.addField(new FieldDefinition(LoggingConstants.PLATFORM_ID, BaseTypes.STRING));
logEntriesTable.addField(new FieldDefinition(LoggingConstants.ORIGIN, BaseTypes.STRING));
logEntriesTable.addField(new FieldDefinition(LoggingConstants.INSTANCE, BaseTypes.STRING));

// TODO: Custom Log Retrieval Strategy implementation here. Add each log entry as row to logEntriesTable

return logEntriesTable;

3. Create a Strategy Thing in the metadata.xml file of your extension.

4. Package and build an extension.

5. Import the extension through ThingWorx Composer.

6. Create a Thing from the ThingTemplate CustomLogRetrievalStrategy.

7. Go to the LoggingSubsystem > Configuration page.

8. Select a Thing created in step 6 as Log Retrieval Strategy.

9. Go to Composer > Monitoring > ApplicationLog and verify if the logs are retrieved from the external source.

Was this helpful?