Installing and Configuring Ignite for ThingWorx HA
Apache Ignite is a horizontally-scalable and fault-tolerant distributed in-memory computing platform for building real-time applications that can process terabytes of data with in-memory speed. In a ThingWorx HA clustered environment, Ignite is used as distributed memory for storing shared state, such as current property values and binding information, between ThingWorx Foundation nodes.
Prerequisites
You must have the following:
• ZooKeeper installed and running
• Network configurations allowing access to ZooKeeper nodes
Installation (Ignite 2.14 and Newer)
2. Download the ThingWorx libraries cache-provider-ignite-[version].zip and cache-provider-api-[version].zip, extract the zip, and put the jar files into the Ignite folder $IGNITE_HOME/libs.
3. Use the ZooKeeper service discovery mechanism for the cluster to connect with the platform.
b. Move the ignite-zookeeper, ignite-log4j2, and ignite-slf4j directories from the $IGNITE_HOME/libs/optional folder to the $IGNITE_HOME/libs folder.
c. If on Linux, do the following to update files to latest version:
# remove the existing zookeeper jar file in the ignite-zookeeper folder
rm -rf "${IGNITE_HOME}"/libs/ignite-zookeeper/zookeeper-*
# remove unneeded library
rm -rf "${IGNITE_HOME}"/libs/ignite-indexing/
# download the latest zookeeper plus other required libraries and place in ignite-zookeeper folder
wget -q --no-check-certificate https://repo1.maven.org/maven2/org/apache/zookeeper/zookeeper/3.8.1/zookeeper-3.8.1.jar -P "${IGNITE_HOME}"/libs/ignite-zookeeper/
wget -q --no-check-certificate https://repo1.maven.org/maven2/org/apache/zookeeper/zookeeper-jute/3.8.1/zookeeper-jute-3.8.1.jar -P "${IGNITE_HOME}"/libs/ignite-zookeeper/
wget -q --no-check-certificate https://repo1.maven.org/maven2/org/apache/curator/curator-client/5.2.1/curator-client-5.2.1.jar -P "${IGNITE_HOME}"/libs/ignite-zookeeper/
wget -q --no-check-certificate https://repo1.maven.org/maven2/org/apache/curator/curator-framework/5.2.1/curator-framework-5.2.1.jar -P "${IGNITE_HOME}"/libs/ignite-zookeeper/
wget -q --no-check-certificate https://repo1.maven.org/maven2/org/apache/curator/curator-recipes/5.2.1/curator-recipes-5.2.1.jar -P "${IGNITE_HOME}"/libs/ignite-zookeeper/
wget -q --no-check-certificate https://repo1.maven.org/maven2/org/apache/curator/curator-x-discovery/5.2.1/curator-x-discovery-5.2.1.jar -P "${IGNITE_HOME}"/libs/ignite-zookeeper/
wget -q --no-check-certificate https://repo1.maven.org/maven2/io/netty/netty-handler/4.1.86.Final/netty-handler-4.1.86.Final.jar -P "${IGNITE_HOME}"/libs/ignite-zookeeper/
wget -q --no-check-certificate https://repo1.maven.org/maven2/io/netty/netty-buffer/4.1.86.Final/netty-buffer-4.1.86.Final.jar -P "${IGNITE_HOME}"/libs/ignite-zookeeper/
wget -q --no-check-certificate https://repo1.maven.org/maven2/io/netty/netty-codec/4.1.86.Final/netty-codec-4.1.86.Final.jar -P "${IGNITE_HOME}"/libs/ignite-zookeeper/
wget -q --no-check-certificate https://repo1.maven.org/maven2/io/netty/netty-common/4.1.86.Final/netty-common-4.1.86.Final.jar -P "${IGNITE_HOME}"/libs/ignite-zookeeper/
wget -q --no-check-certificate https://repo1.maven.org/maven2/io/netty/netty-resolver/4.1.86.Final/netty-resolver-4.1.86.Final.jar -P "${IGNITE_HOME}"/libs/ignite-zookeeper/
wget -q --no-check-certificate https://repo1.maven.org/maven2/io/netty/netty-transport-native-epoll/4.1.86.Final/netty-transport-native-epoll-4.1.86.Final.jar -P "${IGNITE_HOME}"/libs/ignite-zookeeper/
wget -q --no-check-certificate https://repo1.maven.org/maven2/io/netty/netty-transport-classes-epoll/4.1.86.Final/netty-transport-classes-epoll-4.1.86.Final.jar -P "${IGNITE_HOME}"/libs/ignite-zookeeper/
wget -q --no-check-certificate https://repo1.maven.org/maven2/io/netty/netty-transport-native-unix-common/4.1.86.Final/netty-transport-native-unix-common-4.1.86.Final.jar -P "${IGNITE_HOME}"/libs/ignite-zookeeper/
wget -q --no-check-certificate https://repo1.maven.org/maven2/io/netty/netty-transport/4.1.86.Final/netty-transport-4.1.86.Final.jar -P "${IGNITE_HOME}"/libs/ignite-zookeeper/
wget -q --no-check-certificate https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-databind/2.12.7/jackson-databind-2.12.7.jar -P "${IGNITE_HOME}"/libs/ignite-zookeeper/
wget -q --no-check-certificate https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-core/2.12.7/jackson-core-2.12.7.jar -P "${IGNITE_HOME}"/libs/ignite-zookeeper/
wget -q --no-check-certificate https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-annotations/2.12.7/jackson-annotations-2.12.7.jar -P "${IGNITE_HOME}"/libs/ignite-zookeeper/
wget -q --no-check-certificate https://repo1.maven.org/maven2/com/google/guava/guava/30.1-jre/guava-30.1-jre.jar -P "${IGNITE_HOME}"/libs/ignite-zookeeper/
wget -q --no-check-certificate https://repo1.maven.org/maven2/org/apache/ignite/ignite-zookeeper-ip-finder-ext/1.0.0/ignite-zookeeper-ip-finder-ext-1.0.0.jar -P "${IGNITE_HOME}"/libs/ignite-zookeeper/
d. If on Windows, do the following to update the files to the latest version: removelibs/ignite-zookeeper/zookeepr *jar files and libs/ignite-indexing/ directory.
i. Download the following jar files and place them in the libs/ignite-zookeeper/ directory.
▪ https://repo1.maven.org/maven2/org/apache/zookeeper/zookeeper/3.8.1/zookeeper-3.8.1.jar
▪ https://repo1.maven.org/maven2/org/apache/zookeeper/zookeeper-jute/3.8.1/zookeeper-jute-3.8.1.jar
▪ https://repo1.maven.org/maven2/org/apache/curator/curator-client/5.2.1/curator-client-5.2.1.jar
▪ https://repo1.maven.org/maven2/org/apache/curator/curator-framework/5.2.1/curator-framework-5.2.1.jar
▪ https://repo1.maven.org/maven2/org/apache/curator/curator-recipes/5.2.1/curator-recipes-5.2.1.jar
▪ https://repo1.maven.org/maven2/org/apache/curator/curator-x-discovery/5.2.1/curator-x-discovery-5.2.1.jar
▪ https://repo1.maven.org/maven2/io/netty/netty-handler/4.1.86.Final/netty-handler-4.1.86.Final.jar
▪ https://repo1.maven.org/maven2/io/netty/netty-buffer/4.1.86.Final/netty-buffer-4.1.86.Final.jar
▪ https://repo1.maven.org/maven2/io/netty/netty-codec/4.1.86.Final/netty-codec-4.1.86.Final.jar
▪ https://repo1.maven.org/maven2/io/netty/netty-common/4.1.86.Final/netty-common-4.1.86.Final.jar
▪ https://repo1.maven.org/maven2/io/netty/netty-resolver/4.1.86.Final/netty-resolver-4.1.86.Final.jar
▪ https://repo1.maven.org/maven2/io/netty/netty-transport-native-epoll/4.1.86.Final/netty-transport-native-epoll-4.1.86.Final.jar
▪ https://repo1.maven.org/maven2/io/netty/netty-transport-native-unix-common/4.1.86.Final/netty-transport-native-unix-common-4.1.86.Final.jar
▪ https://repo1.maven.org/maven2/io/netty/netty-transport/4.1.86.Final/netty-transport-4.1.86.Final.jar
▪ https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-databind/2.12.7/jackson-databind-2.12.7.jar
▪ https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-core/2.12.7/jackson-core-2.12.7.jar
▪ https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-annotations/2.12.7/jackson-annotations-2.12.7.jar
▪ https://repo1.maven.org/maven2/com/google/guava/guava/30.1-jre/guava-30.1-jre.jar
▪ https://repo1.maven.org/maven2/org/apache/ignite/ignite-zookeeper-ip-finder-ext/1.0.0/ignite-zookeeper-ip-finder-ext-1.0.0.jar
4. Update the default-config.xml file to enable ZooKeeper service discovery. You must place the ZooKeeper server address in the zkConnectionString connection property.
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd">
<bean class="org.apache.ignite.configuration.IgniteConfiguration">
<property name="discoverySpi">
<bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
<property name="ipFinder">
<bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.zk.TcpDiscoveryZookeeperIpFinder">
<property name="zkConnectionString" value="ZOOKEEPER_IPADDRESS:2181"/>
</bean>
</property>
</bean>
</property>
<property name="peerClassLoadingEnabled" value="true"/>
</bean>
</beans>
Installing ThingWorx 9.2x through 9.3x (Ignite 2.11 and Older)
2. Download the ThingWorx libraries cache-provider-ignite-[version].zip and cache-provider-api-[version].zip, extract the zip, and put the jar files into the Ignite folder $IGNITE_HOME/libs.
3. Use the ZooKeeper service discovery mechanism for the cluster to connect with the platform.
b. Move the ignite-zookeeper, ignite-log4j2, and ignite-slf4j directories from the $IGNITE_HOME/libs/optional folder to the $IGNITE_HOME/libs folder.
c. If on Linux, do the following to update files to latest version:
# remove the existing zookeeper jar file in the ignite-zookeeper folder
rm -rf "${IGNITE_HOME}"/libs/ignite-zookeeper/zookeeper-*
# remove libs/ignite-zookeeper/curator-*
rm -rf "${IGNITE_HOME}"/libs/ignite-zookeeper/curator*
# remove libs/ignite-zookeeper/guava-*
rm -rf "${IGNITE_HOME}"/libs/ignite-zookeeper/guava*
# download the latest zookeeper plus other required libraries and place in ignite-zookeeper folder
wget -q --no-check-certificate https://repo1.maven.org/maven2/org/apache/zookeeper/zookeeper/3.8.1/zookeeper-3.8.1.jar -P "${IGNITE_HOME}"/libs/ignite-zookeeper/
wget -q --no-check-certificate https://repo1.maven.org/maven2/org/apache/zookeeper/zookeeper-jute/3.8.1/zookeeper-jute-3.8.1.jar -P "${IGNITE_HOME}"/libs/ignite-zookeeper/
wget -q --no-check-certificate https://repo1.maven.org/maven2/org/apache/curator/curator-client/5.2.1/curator-client-5.2.1.jar -P "${IGNITE_HOME}"/libs/ignite-zookeeper/
wget -q --no-check-certificate https://repo1.maven.org/maven2/org/apache/curator/curator-framework/5.2.1/curator-framework-5.2.1.jar -P "${IGNITE_HOME}"/libs/ignite-zookeeper/
wget -q --no-check-certificate https://repo1.maven.org/maven2/org/apache/curator/curator-recipes/5.2.1/curator-recipes-5.2.1.jar -P "${IGNITE_HOME}"/libs/ignite-zookeeper/
wget -q --no-check-certificate https://repo1.maven.org/maven2/org/apache/curator/curator-x-discovery/5.2.1/curator-x-discovery-5.2.1.jar -P "${IGNITE_HOME}"/libs/ignite-zookeeper/
wget -q --no-check-certificate https://repo1.maven.org/maven2/io/netty/netty-handler/4.1.86.Final/netty-handler-4.1.86.Final.jar -P "${IGNITE_HOME}"/libs/ignite-zookeeper/
wget -q --no-check-certificate https://repo1.maven.org/maven2/io/netty/netty-buffer/4.1.86.Final/netty-buffer-4.1.86.Final.jar -P "${IGNITE_HOME}"/libs/ignite-zookeeper/
wget -q --no-check-certificate https://repo1.maven.org/maven2/io/netty/netty-codec/4.1.86.Final/netty-codec-4.1.86.Final.jar -P "${IGNITE_HOME}"/libs/ignite-zookeeper/
wget -q --no-check-certificate https://repo1.maven.org/maven2/io/netty/netty-common/4.1.86.Final/netty-common-4.1.86.Final.jar -P "${IGNITE_HOME}"/libs/ignite-zookeeper/
wget -q --no-check-certificate https://repo1.maven.org/maven2/io/netty/netty-resolver/4.1.86.Final/netty-resolver-4.1.86.Final.jar -P "${IGNITE_HOME}"/libs/ignite-zookeeper/
wget -q --no-check-certificate https://repo1.maven.org/maven2/io/netty/netty-transport-native-epoll/4.1.86.Final/netty-transport-native-epoll-4.1.86.Final.jar -P "${IGNITE_HOME}"/libs/ignite-zookeeper/
wget -q --no-check-certificate https://repo1.maven.org/maven2/io/netty/netty-transport-classes-epoll/4.1.86.Final/netty-transport-classes-epoll-4.1.86.Final.jar -P "${IGNITE_HOME}"/libs/ignite-zookeeper/
wget -q --no-check-certificate https://repo1.maven.org/maven2/io/netty/netty-transport-native-unix-common/4.1.86.Final/netty-transport-native-unix-common-4.1.86.Final.jar -P "${IGNITE_HOME}"/libs/ignite-zookeeper/
wget -q --no-check-certificate https://repo1.maven.org/maven2/io/netty/netty-transport/4.1.86.Final/netty-transport-4.1.86.Final.jar -P "${IGNITE_HOME}"/libs/ignite-zookeeper/
wget -q --no-check-certificate https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-databind/2.12.7/jackson-databind-2.12.7.jar -P "${IGNITE_HOME}"/libs/ignite-zookeeper/
wget -q --no-check-certificate https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-core/2.12.7/jackson-core-2.12.7.jar -P "${IGNITE_HOME}"/libs/ignite-zookeeper/
wget -q --no-check-certificate https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-annotations/2.12.7/jackson-annotations-2.12.7.jar -P "${IGNITE_HOME}"/libs/ignite-zookeeper/
wget -q --no-check-certificate https://repo1.maven.org/maven2/com/google/guava/guava/30.1-jre/guava-30.1-jre.jar -P "${IGNITE_HOME}"/libs/ignite-zookeeper/
wget -q --no-check-certificate https://repo1.maven.org/maven2/org/apache/ignite/ignite-zookeeper-ip-finder-ext/1.0.0/ignite-zookeeper-ip-finder-ext-1.0.0.jar -P "${IGNITE_HOME}"/libs/ignite-zookeeper/
d. If on Windows, perform the following to update the files to the latest version: removelibs/ignite-zookeeper/zookeepr *jar files and libs/ignite-indexing/ directory.
i. Download the following jar files and place them in the libs/ignite-zookeeper/ directory.
▪ https://repo1.maven.org/maven2/org/apache/zookeeper/zookeeper/3.8.1/zookeeper-3.8.1.jar
▪ https://repo1.maven.org/maven2/org/apache/zookeeper/zookeeper-jute/3.8.1/zookeeper-jute-3.8.1.jar
▪ https://repo1.maven.org/maven2/org/apache/curator/curator-client/5.1.0/curator-client-5.1.0.jar
▪ https://repo1.maven.org/maven2/org/apache/curator/curator-framework/5.1.0/curator-framework-5.1.0.jar
▪ https://repo1.maven.org/maven2/org/apache/curator/curator-recipes/5.1.0/curator-recipes-5.1.0.jar
▪ https://repo1.maven.org/maven2/org/apache/curator/curator-x-discovery/5.1.0/curator-x-discovery-5.1.0.jar
▪ https://repo1.maven.org/maven2/io/netty/netty-handler/4.1.86.Final/netty-handler-4.1.86.Final.jar
▪ https://repo1.maven.org/maven2/io/netty/netty-buffer/4.1.86.Final/netty-buffer-4.1.86.Final.jar
▪ https://repo1.maven.org/maven2/io/netty/netty-codec/4.1.86.Final/netty-codec-4.1.86.Final.jar
▪ https://repo1.maven.org/maven2/io/netty/netty-common/4.1.86.Final/netty-common-4.1.86.Final.jar
▪ https://repo1.maven.org/maven2/io/netty/netty-resolver/4.1.86.Final/netty-resolver-4.1.86.Final.jar
▪ https://repo1.maven.org/maven2/io/netty/netty-transport-native-epoll/4.1.86.Final/netty-transport-native-epoll-4.1.86.Final.jar
▪ https://repo1.maven.org/maven2/io/netty/netty-transport-native-unix-common/4.1.86.Final/netty-transport-native-unix-common-4.1.86.Final.jar
▪ https://repo1.maven.org/maven2/io/netty/netty-transport/4.1.86.Final/netty-transport-4.1.86.Final.jar
▪ https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-databind/2.12.7/jackson-databind-2.12.7.jar
▪ https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-core/2.12.7/jackson-core-2.12.7.jar
▪ https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-annotations/2.12.7/jackson-annotations-2.12.7.jar
▪ https://repo1.maven.org/maven2/com/google/guava/guava/30.1-jre/guava-30.1-jre.jar
4. Update the default-config.xml file to enable ZooKeeper service discovery. You must place the ZooKeeper server address in the zkConnectionString connection property.
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd">
<bean class="org.apache.ignite.configuration.IgniteConfiguration">
<property name="discoverySpi">
<bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
<property name="ipFinder">
<bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.zk.TcpDiscoveryZookeeperIpFinder">
<property name="zkConnectionString" value="ZOOKEEPER_IPADDRESS:2181"/>
</bean>
</property>
</bean>
</property>
<property name="peerClassLoadingEnabled" value="true"/>
</bean>
</beans>
Networking
The following ports must be open between the Ignite servers and the ThingWorx servers:
Port
|
Usage
|
47500-47600
|
Discovery port
|
47100-47200
|
Communication port
|
8080
|
Default port for REST API
|
49128
|
Default port for JMX connection
|