Troubleshooting eMessage Connectors in an HA Cluster
This topic provides troubleshooting information for the eMessage Connector running in a ThingWorx High Availability (HA) Clustering environment.
Rapidly Starting Multiple eMessage Connectors in HA Can Cause Sync Issues
If two or more eMessage Connectors initially start up at the same time, they can cause contention with the ThingWorx Platform instance.
SYMPTOM: eMessage Connector 1 starts up and determines it needs to apply the AxedaProtocolAdapterServices Thing Shape to the Connection Services Hub. As a result, the Connection Services Hub tis created and begins initializing.
Before the Connection Services Hub fully starts, eMessage Connector 2 starts up and attempts to determine if the Connection Services Hub exists by calling the GetMetadata service. On the ThingWorx Platform, the Connection Services Hub is not running, so it throws an exception. The exception is written to the ApplicationLog of the ThingWorx Platform instance. For example:
/ApplicationLog.2020-02-07.0.log:2020-02-07 12:55:58.789+0000 [L: ERROR] [O: c.t.w.p.APIProcessor] [I: ]
[U: EMessageConnectorUser] [S: ] [P: platform1] [T: WSExecutionProcessor-3]
Unable to dispatch [ uri = /Things/ConnectionServicesHub/Services/GetMetadata/]:
Unable to Invoke Service GetMetadata on ConnectionServicesHub :
Thing [ConnectionServicesHub] is not running
As a result, eMessage Connector 2 throws the following exception (excerpted from the log)
2020-02-07 12:55:58.799 [pool-1-thread-1] WARN c.t.c.p.s.hub.impl.HubChannelImpl -
Unable to successfully initialize hub channel
com.thingworx.connectionserver.protocol.services.hub.HubChannelException:
Connection Server Hub thing named ConnectionServicesHub doesn't exist
at com.thingworx.connectionserver.protocol.services.hub.impl.HubChannelImpl.initializeOneTime(HubChannelImpl.java:92)
...
com.thingworx.connectionserver.protocol.services.hub.HubChannelException: Connection Server Hub thing named
ConnectionServicesHub doesn't exist
at com.thingworx.connectionserver.protocol.services.hub.impl.HubChannelImpl.initializeOneTime(HubChannelImpl.java:92)
...
SOLUTION: Restart any eMessage Connector that is showing the exception. If more than one eMessage Connector shows the exception, start the first eMessage Connector and wait until it is fully up before starting the rest of the eMessage Connectors. To prevent this situation in the future, always wait for the first Connector to be fully up before starting the rest of them.