ThingWorx Model Definition and Composer > Things > Content Crawler
  
Content Crawler
A Content Crawler Thing is used to call a service on another entity. It is used to retrieve data and store the data in the data table of the Content Crawler Thing.
On a separate entity from the Content Crawler Thing, a service must be defined that fetches data and returns an INFOTABLE of that data back to the Content Crawler. The Content Crawler then maps the incoming fields as well as the tags to the fields used in the Data Shape for the Content Crawler. Each row is added as a new entry to the Data Table on the Content Crawler Thing. The index of the Content Crawler’s Data Table functions in the same way as a normal Data Table entity.
* 
For information on creating a Content Crawler using the legacy Composer, see Content Crawler (Legacy Composer).
Creating a Content Crawler
To retrieve data from the Data Table of an entity into the Data Table of the Content Crawler Thing, do the following:
1. Create a data shape and define fields to use in a data table. To create a Data Shape from Composer, browse MODELING > Data Shapes, and then click the New button.
a. Enter a name and description.
b. In the Field Definitions area, click the Add button.
c. In the new field definition pane, enter appropriate information, and then click .
2. Create a data table with the data shape created in the previous step. To create a Data Table from Composer, browse DATA STORAGE > Data Tables, and then click the New button.
a. Select a Data Table template, and then click OK.
b. Enter the name, description, and select the Data Shape you created in the previous step.
c. In the Services area, create a custom service by clicking Add.
d. In the Output area, select INFOTABLE from the drop-down list.
e. Select the Data Shape created in the previous step.
f. Set the Infotable Type as Is Content Crawler Entry, and then click the Done button.
3. Create a new data shape for the Content Crawler thing.
* 
You can create a new Content Crawler-specific data shape, or you can use the same data shape that was used in the data table created in step 1. Although this step is optional, we will use a new data shape for the Content Crawler Thing in this example.
a. Create a new Content Crawler thing:
a. From Composer, browse MODELING > Things, and then click the New button.
b. Enter a name, and in the Base Thing Template field, select Content Crawler.
c. In the Data Shape field, select the data shape you created in the previous step, and then click Save.
Content Crawler Configuration
The Configuration area for the Content Crawler Thing contains configuration tables that allow you to map fields from the retrieved data.
The Field to Tag Mappings configuration table maps the values of a field to tags in a data tag vocabulary.
When the data tag vocabulary is dynamic, any value that is mapped from the data will have a term automatically entered into the vocabulary.
When the data tag vocabulary is not dynamic, any value that is mapped from the data will need a pre-defined term representing the value to be mapped properly.
For example: TestingVocab:false;TestingVocab:iAmAString. The first part is the boolProp value and the second part is the stringProp value.
The Index Settings configuration for a data table allows you to define additional table indexes. This is similar to a relational database table, where in addition to the primary key (the primary key is defined in the data shape), you need to query the table based on other fields. You should create an index for each set of filter criteria commonly used. It has a significant impact on query performance.
The Field to Field Mappings configuration table maps the fields from the retrieved data to the fields defined on the data shape of the Content Crawler Thing.
* 
If the same Data Shape is used on the Content Crawler Thing and for the INFOTABLE returned from the Content Crawler service, field mappings are not necessary as they are handled automatically.
Content Crawler Services
The following services are unique to the Content Crawler Thing:
CrawlEntries: Purges all of the Data Table entries for the Content Crawler, and then executes GetExternalContent.
GetExternalContent: Executes the service defined on the General Information area of the Content Crawler Thing. An INFOTABLE of retrieved values is returned from the service. No modifications to the Data Table for the Content Crawler are performed.
GetExternalContentDetail: Retrieves a specific content item by key.