Tutorial B: ThingWorx Flow Connectors SDK Tutorial
Prerequisites
The following are the prerequisites required for this tutorial:
• LTS version of Node.js supported with the installed ThingWorx Flow version
To know the node.js version currently in use, open Command Prompt, and run the following command:
node -v
• An Integrated Development Environment (IDE) that can handle the Node js projects.
The tutorial uses WebStorm as an IDE.
• An intermediate level understanding of the Node js code
• Samples folder
|
It is recommended to use the same machine which has ThingWorx Flow installed to make use of the installed components.
|
Introduction
The tutorial explains the steps required to build a typical connector. We will build a connector for Gmail as most developers are likely to be familiar with how Gmail works. Building the connector will familiarize you with the tools, artifacts, and processes involved in building a connector, and help you to build a connector for your application. Building a new connector does not require a deployment of ThingWorx Flow. After the connector is built and tested locally, it can be deployed to a test on-premise installation for testing.
The tutorial covers the following:
1. Creating a connector project.
2. An OAuth2 configuration
3. Creating a lookup that searches for filters related in your Gmail account.
4. Creating an action to get the id and subject of email messages that match the selected filter.
5. Creating a trigger that allows a new flow to be initiated when a new mail is received.
6. Adding icons
7. Deploying connectors to an on-premise instance of ThingWorx Flow.
Step 1: Creating a Connector Project
A connector is a specialized npm package that consist of various artifacts such as oauth configuration, action, lookups, triggers. The project should follow the patterns for building a npm package. It should have a package.json , a lib folder, a test folder and folders for each of the artifacts and their versions.
Open a terminal/cmd windows and navigate to a folder where you would like to create a new connector project. In this tutorial we will create the connector in the directory c:\test
1. Change directory to the test directory: cd c:\test
2. Run the following command to create a project: flow init <projectname>
You should see the following:
cd c:\test
[2018-10-21T15:35:11.519] [INFO] default - Project gmail created successfully in .
3. Open the project directory in an IDE to review the artifacts created by default.
Every project contains an index.js file and a package.json file. Do not change the index.js file as the file is needed to load the connectors in the ThingWorx Flow server.
| The package.json file refers to ptc-flow-sdk. Without this dependency it is not be possible to test, deploy, or run the connector. It also provides a few APIs that make connector development easier and more consistent. |
2. Install the async package using the following command:
npm install async
3. Install the request module using the following commands. The request module contains an API for making HTTP requests.
npm install request
npm install request-promise
4. Install the lodash module that provide a host of useful functions:
npm install lodash
npm install ptc-flow-sdk@<Tool version> -g
Check the
<Tool version> for your version of ThingWorx Flow
here.
| All commands in the tutorial are run from the connector project directory c:\test\gmail. This helps avoid providing the complete path to the connector directory using the -d/--projectDir options. |
Step 2: Creating an OAuth2 configuration
| Before you begin, make sure that you have followed the steps provided in the section Configuration for Testing OAuths and Triggers. |
Gmail uses OAuth 2.0 for authentication. Therefore the sample connector uses OAuth 2.0 as an authentication mechanism. All OAuth2 providers require an application to create a client id and client secret. The application can then request a user to log in to the OAuth provider using these credentials. The user can grant the application permissions to access data on his behalf. The permission are defined by scopes. The user can selectively provide the application access to his data. For details on how this is implemented for Gmail and other google apps, visit the links that follow.
Provide a redirect URL. The redirect URL:
https://<hostname>:<port>//Thingworx/Oauths/oauth/return looks like the following:
https://flow.local.rnd.ptc.com/Thingworx/Oauths/oauth/return
Some systems allow multiple redirects.
| Make sure that the host you use is added to the .flow\flow.json file in the user home directory. If this is not the actual host name but an alias to the local host, add it to the host file of your Operating System. An example of flow.json is as follows. { "hostname": "localhost", "port":443 } • The hostname value must be localhost. • The port value is the ThingWorx Flow hosted port number. By default it is 443. |
For example,
1. On Windows, right-click the editor, and then select Run as Administrator.
2. Edit the host file located at c:\windows\system32\drivers\etc\hosts and add or update an entry as follows:
127.0.0.1 flow.local.rnd.ptc.com
where, flow.local.rnd.ptc.com should be replaced with the alias.
After completing the previous steps, you created an application and a client id and client secret for the application.
1. Create an OAuth configuration using the following command.
c:\test\gmail>flow add oauth gmail
[2018-10-21T16:26:54.398] [INFO] add-oauth - Oauth configuration gmail added successfully.
This command creates the folder \auth\oauth\gmail-gmail in the project folder. The folder will have a single config.json file.
In this case, the config.json file is a template that must be customized to work with a specific OAuth provider such as Google.
2. Open it in a text editor or IDE, and then make the changes as described below. Make sure that the file is edited correctly and stays a valid JSON file.
3. Set category, name, title, and icons as required. See the section
Adding Icons to your Connector below for details about creating and setting icons.
4. Search for the key oauth2_params_other and replace it with the value
[
"{\"access_type\":\"offline\"}"
]
5. Search for key oauth2_params_scope and replace its value with the following JSON array:
[
"{\"https://www.googleapis.com/auth/gmail.labels\":\"Manage mailbox labels\"}",
"{\"https://www.googleapis.com/auth/gmail.insert\":\"Insert mail into your mailbox\"}",
"{\"https://www.googleapis.com/auth/gmail.modify\":\"All read/write operations except immediate, permanent deletion of threads and messages, bypassing Trash.\"}",
"{\"https://www.googleapis.com/auth/gmail.readonly\":\"Read all resources and their metadata—no write operations.\"}",
"{\"https://www.googleapis.com/auth/gmail.compose\":\"Create, read, update, and delete drafts. Send messages and drafts.\"}",
"{\"https://www.googleapis.com/auth/gmail.send\":\"Send email on your behalf\"}"
]
Note the value of
env_local_params. Here the local is a name of the environment. There can be five possible environments and are represented by the keys
env_production_params, env_pre_prod_params, env_staging_params, env_local_params. The value are of the form
{{{local.CLIENT_SECRET}}}. The
local.CLIENT_SECRET is a variable that stands in for the name of the environment. It should be enclosed in the three open and close braces. These values are replaced with customer-provided values while running the
LoadOAuthConfiguration service in
ThingWorx Composer to load OAuths to the
ThingWorx Flow server. The values are provided through a JSON file. The value is substituted when generating an OAuth configuration only for the environment that ThingWorx Flow server is running in. If the server has
NODE_ENV environment variable set to production, the production value
CLIENT_SECRET is populated with the values provided in the OAuth data file. This allows the OAuth configuration to be customized for each customer deployment.
7. Look for oauth2_auth_url and replace its value with /auth.
Step 3: Testing an OAuth Configuration
An OAuth configuration can be tested before it is deployed. To do this, ThingWorx Flow CLI launches a browser allowing you to sign into the OAuth provider using your credentials. If authentication succeeds, the access token is generated and can be saved to the disk for later use. It is also possible to test each deployment configuration by selecting the appropriate configuration in the embedded chrome browser window.
Before we can run the test command we need to create a test data file that contains the actual client id and secret that will be substituted for the env_<environment-name>_params as mentioned above. To do this, create a new JSON file, testOauthData.json file in c:\test, and add the following content to it.
{
"gmail-gmail": {
"Test Gmail Oauth": {
"CLIENT_ID": "<your-client-id-here>"
"CLIENT_SECRET": ""<your-client-secret-here>"
}
}
}
Make sure you replace the <your-client-id-here> and <your-client-secret-here> with the previously retrieved values from Google console.
Next, ensure that the .flow\flow.json file contained in the user’s home directory contains the correct host name, port and certificate passphrase. This is required only if one was provided while creating the self-signed certificate.
Run the following command in a command prompt, set FLOW_ALLOW_UNTRUSTED_CERTIFICATES=true
This variable is required since we are using a self-signed certificate to run the tool.
1. Run the following command to test the OAuth.
flow test oauth gmail-gmail -f ..\testOauthData.json -t "Test Gmail Oauth"
2. Select a configuration to test and click Validate OAuth Configuration.
3. Select the scopes that you want to allow the application to access and click Allow.
4. Enter the user name for your google account. This can be any google account, not necessarily the one used to create the client-id and secret.
5. Enter your password. If authentication succeeds you should see the access token.
6. Click Save and Exit to save the access token for future use.
The following message appears on the console.
[2018-10-22T12:51:07.643] [INFO] oauth - Access token saved with ID :664ec37a-9720-460c-8536-fbf372198cb1
The id from this message will be used to test other artifacts that depend on the access token such as lookups, actions, and triggers. The id produced above should be provided to all test command using the -a/--access_token option.
| If you receive an error like Redirect URI mismatch you should ensure that the flow.json configuration is correct and the URL that the flow uses matches the one registered with google. The redirect URL used by the flow is printed on the console when the test starts. For example [2018-10-22T11:07:53.868] [INFO] oauthServer- Server started, redirect_uri used for oauth = https://flow.local.rnd.ptc.com:443/Thingworx/Oauths/oauth/return |
| OAuth access tokens are short lived, before using the access token to test an action, lookup, trigger you should run the flow test oauth command to regenerate a new token. |
Step 4: Creating a lookup
Lookups are a search mechanism. They are typically used to dynamically get a list of options for certain input fields of actions or triggers. For example. email messages in Gmail can have labels associated with them. This helps to quickly search for related email that are all tagged using the label. However, an end user is unlikely to remember the names of all labels created in his account. The lookup we will build here produces an id and value pairs for labels found in the Gmail account. One or more of these can then be used to get the ids and subject of emails with those labels.
Lookups do not have metadata instead they are represented by a single index.js file. They are also not versioned. There can be any number of lookup methods in the index.js file.
To create a lookup run the following command:
c:\test\gmail>flow add lookup
[2018-10-22T14:41:52.298] [INFO] add-lookup - Lookup added successfully
Running the command will generate a index.js in the lookup\gmail folder.
Open the index.js file in your IDE and add the code listed below to it.
At the top of the file import the following modules.
const async = require('async')
const request = require('request')
const logger = require('ptc-flow-sdk').getLogger('gmail-lookup')
Copy the code below after all existing code in the file.
gmailAPIs.getLabels = function (input, options, output) {
logger.debug('Trying to fetch labels')
// Gmail API to fetch th labels in the gmail account
options.url = 'https://www.googleapis.com/gmail/v1/users/me/labels'
// Validating that the authorization is done and we have a access_token to //make a request else throw the error
options.validateDependencies(input.access_token, function (err, status) {
if (err) {
return output({
error: err
})
}
// If the authorization is proper then execute the request
execute(input, options, output)
})
}
// common function for api to execute the request module
function execute (input, options, output) {
async.waterfall([
function (cb) {
// In input we get auth code which we can use to get access token
getAccessToken(input, options, cb)
}
], function (err, accessToken) {
if (err) {
return output({
error: err
})
} else {
// Execute the request to get the labels from the gmail api
requestExecute(input, accessToken, options, output)
}
})
}
// Extract the acces_token from the input and return it
function getAccessToken (input, options, cb) {
options.getAccessToken(input.access_token, function (err, oauth) {
if (err) {
cb(err)
} else {
oauth = JSON.parse(oauth)
cb(null, oauth.access_token)
}
})
}
// Make a request to the gmail url to get the labels
function requestExecute (input, accessToken, options, output) {
request({
method: 'GET',
headers: {
'Authorization': 'Bearer ' + accessToken,
'Content-Type': 'application/json'
},
url: options.url
}, function (err, res, body) {
let result = []
if (err) {
return output({
error: err
})
}
if (res.statusCode === 401) {
return output({
error: 'Your access token is Invalid or Expired'
})
}
if (res.statusCode >= 200 && res.statusCode < 400) {
if (typeof (body) === 'string') {
body = JSON.parse(body)
}
result = filterResult(body.messages || body.labels || body, input)
var pageDetails = options.getPage(result, input, options)
var data = {
'results': result.slice(pageDetails.page, (pageDetails.page + options.maxResults))
}
data.next_page = options.getNextPage(data.results.length >= options.maxResults)
return output(null, data)
}
return output({
error: body
})
})
}
// filter result on the basis of output demand
function filterResult (body, input) {
var result = []
if (body && Array.isArray(body)) {
body.forEach(function (item, index) {
result.push({
'id': String(item.name),
'value': item.name
})
})
}
return result
}
To test the lookup, execute the following commands:
1. Get a new access token using the command that follows:
flow test oauth gmail-gmail -f ..\testOauthData.json -t "Test Gmail Oauth"
2. Save the token and exit.
3. Test the lookup using the command that follows:
flow test lookup gmail getLabels -a cecd33a3-8a33-4c0c-b298-178fd80a9261 -l trace. You should see the output generated by the lookup. The output is a JSON array of IDs and values.
Step 5: Creating and testing an action
An action interacts with the external connected system to perform some operation. Usually it is a Create Read Update Delete operation. Actions take inputs and produce some output. The inputs and outputs produced by an action need to be specified in the action.json file The input and output properties are special JSON schema, that are used for rendering the input form, and output schema is used to render the mapping user interface. Other useful properties include the tag and the icon which are used to group actions and display an icon for the action in the user interface.
• To create a new action, execute the command below:
c:\test\gmail>flow add action get-messages
[2018-10-22T16:26:53.925] [INFO] add-action - Action get-messages, version v1 added successfully
• Next open the action.json file C:\test\gmail\action\gmail-get-messages\v1\action.json in your IDE.
• Set the input and icon property to gmail. The input property JSON schema is as shown in the code that follows:
Set the input property to {
"title": "Get email messages",
"type": "object",
"properties": {
"auth": {
"title": "Authorize gmail",
"type": "string",
"oauth": "gmail-gmail",
"minLength": 1
},
"label": {
"title": "Label",
"type": "string",
"minLength": 1,
"lookup": {
"id": "getLabels",
"service": "gmail",
"enabled": true,
"searchable": false,
"auth": "oauth",
"dependencies": [
"auth"
]
}
}
}
}
The input schema defines a single input field. This is a list since it has a lookup specified for it. When used in the flow composer, the lookup is executed, and the returned values appears in the list. Note how dependencies of the lookup are described. This tells the system that the auth field must be filled before the invoking the lookup.
The auth property describes the authentication method used by this action. In this case, we use an OAuth configuration Gmail.
Set the “output” property to {
"messages": {
"title": "Messages",
"type": "array",
"items": {
"type": "object",
"properties": {
"messageId": {
"type": "string",
"title": "Message Detail ID"
},
"subject": {
"type": "string",
"title": "Message Detail Sub"
}
}
}
}
}
| The output produced by the action is a JSON array of JSON objects. Each object will have messageIdand subject fields. |
Open the index.js file in the IDE. At the top of the file add the following code:
const rp = require('request-promise')
const logger = require('ptc-flow-sdk').getLogger('gmail-get-messages')
Replace the execute method with the code below:
this.execute = function (input, output) {
// Create a request header to get messages under a label selected from
const options = {
method: 'GET',
url: 'https://www.googleapis.com/gmail/v1/users/me/messages',
useQuerystring: true,
qs: {labelIds: ['STARRED']},
headers:
{
'Authorization': 'Bearer ' + input.auth,
'Content-Type': 'application/json'
}
}
rp(options).then((rawData) => {
const data = JSON.parse(rawData)
let mailRequests = []
if (data && data.messages) {
data.messages.forEach((msg) => {
mailRequests.push(this.fetchMessage(msg.id, input.auth))
})
return Promise.all(mailRequests)
} else {
logger.warn('No messages found')
Promise.resolve()
}
}).then((results) => {
if (results) {
let arr = []
results.forEach((result) => {
let resData = JSON.parse(result)
let msgHeader = resData.payload.headers.find(header => header.name === 'Subject')
arr.push({messageId: resData.id, subject: msgHeader ? msgHeader.value : 'No subject'})
})
return output(null, {messages: arr})
} else {
return output(null)
}
}).catch(err => {
return output(err)
})
}
Add the new method given below:
this.fetchMessage = function (msgId, authToken) {
const options = {
method: 'GET',
url: 'https://www.googleapis.com/gmail/v1/users/me/messages/' + msgId,
qs: { format: 'metadata', metadataHeaders: 'Subject' },
headers:
{
'Authorization': 'Bearer ' + authToken,
'Content-Type': 'application/json'
}
}
return rp(options)
}
Make sure that the file is a valid JavaScript file.
1. Acquire the access token again using the following steps:
◦ Set FLOW_ALLOW_UNTRUSTED_CERTIFICATES=true
◦ Execute the command that follows, and then note the id that is returned.
flow test oauth gmail-gmail -f ..\testOauthData.json -t "Test Gmail Oauth"
2. Create a new JavaScript file C:\test\gmail\test\actionTestData.js and add the following code to it and save it.
module.exports = {
testInput: {
label: 'STARRED'//use a label that applies to a few emails.
},
testOutputFn: function (input, actualOutput) {
// validate the actualOutput against expected output
// return a rejected promise to fail the validation or a resolved promise for success
// return Promise.reject(new Error('Validation successful'))
return Promise.reject(new Error('Validation failed'))
}
}
| The -i/--input and -o/--output parameters accept data or function that return a promise. |
When this is run on a server, the label is available as a part of the input parameter and can be accessed as input.label. The test utility does not execute a lookup, therefore, the output of the lookup needs to be passed to it.
3. Run the test command as follows:
C:\test\gmail>flow test action gmail-get-messages -a 685b7377-7000-
4a92-8679-8630c68a3265 -l debug -i testInput -f
C:\test\gmail\test\actionTestData.js -o testOutput
The test run should succeed.
Step 6: Creating and Testing a Polling Trigger
1. To create a new polling trigger, execute the following command:
flow add trigger -p
[2019-02-28T17:03:23.823] [INFO] add-trigger - Trigger gmail, version v1 added successfully
2. From your IDE, open the trigger.json file located at C:\test\gmail\trigger\poll\gmail \v1\trigger.json.
3. Set the icon property for gmail as follows:
"icon": "gmail",
4. Set the input and output properties.
Set the input property to
{
"properties": {
"auth": {
"type": "string",
"title": "Authorize gmail",
"minLength": 1,
"oauth": "gmail-gmail",
"propertyOrder": 1
},
"search": {
"type": "string",
"title": "Search Text",
"propertyOrder": 2,
"description": "Enter the search text to search for a specific email. E.g., from:john or subject:Christmas"
},
"customFilters": {
"type": "array",
"propertyOrder": 5,
"title": "Custom Filters",
"items": {
"type": "object",
"title": "Filter",
"properties": {
"input": {
"type": "string",
"title": "Input",
"minLength": 1
},
"operator": {
"type": "string",
"title": "Condition",
"enum": [
"Equals",
"GreaterThan"
],
"enumNames": [
"Equals",
"Greater Than"
]
},
"expected": {
"type": "string",
"title": "Expected",
"minLength": 1
}
}
}
}
},
"oneOf": [
{
"type": "object",
"title": "New Email",
"description": "Triggers when a new email is received",
"properties": {
"event": {
"type": "string",
"readonly": true,
"enum": [
"new_mail"
],
"options": {
"hidden": true
},
"propertyOrder": 3
},
"label": {
"type": "string",
"title": "Label",
"propertyOrder": 4,
"description": "Select a label for which you wish to set a trigger. E.g., If you select label as ‘Trash’, the trigger will fire off every time a new mail is moved to label named ‘Trash’ in your Gmail account."
}
}
},
{
"type": "object",
"title": "New Attachment",
"description": "Triggers when a new attachment is received",
"properties": {
"event": {
"type": "string",
"readonly": true,
"enum": [
"new_attachment"
],
"options": {
"hidden": true
},
"propertyOrder": 3
},
"label": {
"type": "string",
"title": "Label",
"propertyOrder": 4,
"description": "Select a label for which you wish to set a trigger. E.g., If you select label as ‘Trash’, the trigger will fire off every time a new mail is moved to label named ‘Trash’ in your Gmail account."
}
}
}
]
}
The input schema defines a single input field. The oneOf object defines the events supported by the trigger. The above schema defines two trigger events:
◦ New Email
◦ New Attachment
The auth property describes the authentication method used by this trigger. In this case, we use an OAuth configuration Gmail.
Set the “output” property to
{
"new_mail": {
"type": "object",
"properties": {
"messageId": {
"type": "string",
"title": "ID",
"displayTitle": "ID"
},
"subject": {
"title": "Subject",
"displayTitle": "Subject",
"type": "string"
}
}
},
"new_attachment": {
"type": "object",
"properties": {
"attachments": {
"title": "Attachments",
"displayTitle": "Attachments",
"type": "array",
"items": {
"type": "object",
"properties": {
"mimeType": {
"title": "Mime Type",
"displayTitle": "Mime Type",
"type": "string"
},
"filename": {
"title": "File Name",
"displayTitle": "File Name",
"type": "string"
},
"attachmentId": {
"title": "Attachment ID",
"displayTitle": "Attachment ID",
"type": "string"
}
}
}
}
}
}
}
The above output schema defines output properties for two trigger events defined in the trigger input schema.
1. Open the index.js file in the IDE, and then add the following code at the top of the file:
const rp = require('request-promise')
const logger = require('ptc-flow-sdk').getLogger('gmail-trigger')
2. Replace the execute method with the code that follows:
Trigger.execute = function (input, options, output) {
// Create a request header to get messages under a label selected from
var date = new Date(options.unixTime * 1000)
const httpOpts = {
method: 'GET',
url: 'https://www.googleapis.com/gmail/v1/users/me/messages',
useQuerystring: true,
qs: { q: 'newer:' + date.getFullYear() + '/' + date.getMonth() + '/' + date.getDay() },
headers:
{
'Authorization': 'Bearer ' + input.auth,
'Content-Type': 'application/json'
}
}
rp(httpOpts).then((rawData) => {
const data = JSON.parse(rawData)
let mailRequests = []
if (data && data.messages) {
data.messages.forEach((msg) => {
mailRequests.push(this.fetchMessage(msg.id, input.auth))
})
return Promise.all(mailRequests)
} else {
logger.warn('No messages found')
Promise.resolve()
}
}).then((results) => {
if (results) {
let arr = []
results.forEach((result) => {
let resData = JSON.parse(result)
let msgHeader = resData.payload.headers.find(header => header.name === 'Subject')
arr.push({ messageId: resData.id, subject: msgHeader ? msgHeader.value : 'No subject' })
})
return output(null, { messages: arr })
} else {
return output(null)
}
}).catch(err => {
return output(err)
})
}
3. Add the new method given below:
Trigger.fetchMessage = function (msgId, authToken) {
const options = {
method: 'GET',
url: 'https://www.googleapis.com/gmail/v1/users/me/messages/' + msgId,
qs: { format: 'metadata', metadataHeaders: 'Subject' },
headers:
{
'Authorization': 'Bearer ' + authToken,
'Content-Type': 'application/json'
}
}
return rp(options)
}
| Make sure that the file is a valid JavaScript file. |
Step 7: Acquiring the Access Token
To acquire the access token again, follow these steps:
1. Set the following property: FLOW_ALLOW_UNTRUSTED_CERTIFICATES=true
2. Execute the command that follows, and then note the id that is returned.
flow test oauth gmail-gmail -f ..\testOauthData.json -t "Test Gmail Oauth"
3. Run the test command as follows:
flow test trigger gmail execute -p -e new_mail -a e0d56340-2fc4-4618-931c-ad6c983ae0e5 --stopAfter 1 --unixTime 1551312000
Step 8: Adding Icons to the Connector
1. Create a common folder in the connector project.
2. Under the common folder, create the folder named css.
3. Add a JSON file with the following format:
{ "name": "connector-name",
"background": "#FFFFFF",
"png128pxBase64": "base64encodedbinarydata"
}
• The name property should be the name of the connector and should match the icon property of the actions in their metadata json files.
• The background should be a HEX color code as shown in the example.
• Icon images should be in the .png format and the binary data should be base64 encoded. For more information, search the internet on how to convert a PNG to its base64 encoded binary representation.