ThingWorx High Availability > Installing and Configuring Ignite for ThingWorx HA
Installing and Configuring Ignite for ThingWorx HA
Apache Ignite is a horizontally-scalable and fault-tolerant distributed in-memory computing platform for building real-time applications that can process terabytes of data with in-memory speed. In a ThingWorx HA clustered environment, Ignite is used as distributed memory for storing shared state, such as current property values and binding information, between ThingWorx Foundation nodes.
Prerequisites
You must have the following:
ZooKeeper installed and running
Network configurations allowing access to ZooKeeper nodes
Installation (Ignite 2.14 and Newer)
* 
If you’re using ThingWorx 9.2x through 9.3x, see Installing ThingWorx 9.2x through 9.3x (Ignite 2.11 and Older).
1. Download Ignite from their website: https://ignite.apache.org/download.cgi#binaries. For version compatibility information, see the PTC Release Advisor.
2. Download the ThingWorx libraries cache-provider-ignite-[version].zip and cache-provider-api-[version].zip, extract the zip, and put the jar files into the Ignite folder $IGNITE_HOME/libs.
3. Use the ZooKeeper service discovery mechanism for the cluster to connect with the platform.
b. Move the ignite-zookeeper, ignite-log4j2, and ignite-slf4j directories from the $IGNITE_HOME/libs/optional folder to the $IGNITE_HOME/libs folder.
c. If on Linux, do the following to update files to latest version:
# remove the existing zookeeper jar file in the ignite-zookeeper folder
rm -rf "${IGNITE_HOME}"/libs/ignite-zookeeper/zookeeper-*

# remove unneeded library
rm -rf "${IGNITE_HOME}"/libs/ignite-indexing/

# download the latest zookeeper plus other required libraries and place in ignite-zookeeper folder
wget -q --no-check-certificate https://repo1.maven.org/maven2/org/apache/zookeeper/zookeeper/3.8.1/zookeeper-3.8.1.jar -P "${IGNITE_HOME}"/libs/ignite-zookeeper/
wget -q --no-check-certificate https://repo1.maven.org/maven2/org/apache/zookeeper/zookeeper-jute/3.8.1/zookeeper-jute-3.8.1.jar -P "${IGNITE_HOME}"/libs/ignite-zookeeper/
wget -q --no-check-certificate https://repo1.maven.org/maven2/org/apache/curator/curator-client/5.2.1/curator-client-5.2.1.jar -P "${IGNITE_HOME}"/libs/ignite-zookeeper/
wget -q --no-check-certificate https://repo1.maven.org/maven2/org/apache/curator/curator-framework/5.2.1/curator-framework-5.2.1.jar -P "${IGNITE_HOME}"/libs/ignite-zookeeper/
wget -q --no-check-certificate https://repo1.maven.org/maven2/org/apache/curator/curator-recipes/5.2.1/curator-recipes-5.2.1.jar -P "${IGNITE_HOME}"/libs/ignite-zookeeper/
wget -q --no-check-certificate https://repo1.maven.org/maven2/org/apache/curator/curator-x-discovery/5.2.1/curator-x-discovery-5.2.1.jar -P "${IGNITE_HOME}"/libs/ignite-zookeeper/
wget -q --no-check-certificate https://repo1.maven.org/maven2/io/netty/netty-handler/4.1.86.Final/netty-handler-4.1.86.Final.jar -P "${IGNITE_HOME}"/libs/ignite-zookeeper/
wget -q --no-check-certificate https://repo1.maven.org/maven2/io/netty/netty-buffer/4.1.86.Final/netty-buffer-4.1.86.Final.jar -P "${IGNITE_HOME}"/libs/ignite-zookeeper/
wget -q --no-check-certificate https://repo1.maven.org/maven2/io/netty/netty-codec/4.1.86.Final/netty-codec-4.1.86.Final.jar -P "${IGNITE_HOME}"/libs/ignite-zookeeper/
wget -q --no-check-certificate https://repo1.maven.org/maven2/io/netty/netty-common/4.1.86.Final/netty-common-4.1.86.Final.jar -P "${IGNITE_HOME}"/libs/ignite-zookeeper/
wget -q --no-check-certificate https://repo1.maven.org/maven2/io/netty/netty-resolver/4.1.86.Final/netty-resolver-4.1.86.Final.jar -P "${IGNITE_HOME}"/libs/ignite-zookeeper/
wget -q --no-check-certificate https://repo1.maven.org/maven2/io/netty/netty-transport-native-epoll/4.1.86.Final/netty-transport-native-epoll-4.1.86.Final.jar -P "${IGNITE_HOME}"/libs/ignite-zookeeper/
wget -q --no-check-certificate https://repo1.maven.org/maven2/io/netty/netty-transport-classes-epoll/4.1.86.Final/netty-transport-classes-epoll-4.1.86.Final.jar -P "${IGNITE_HOME}"/libs/ignite-zookeeper/
wget -q --no-check-certificate https://repo1.maven.org/maven2/io/netty/netty-transport-native-unix-common/4.1.86.Final/netty-transport-native-unix-common-4.1.86.Final.jar -P "${IGNITE_HOME}"/libs/ignite-zookeeper/
wget -q --no-check-certificate https://repo1.maven.org/maven2/io/netty/netty-transport/4.1.86.Final/netty-transport-4.1.86.Final.jar -P "${IGNITE_HOME}"/libs/ignite-zookeeper/
wget -q --no-check-certificate https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-databind/2.12.7/jackson-databind-2.12.7.jar -P "${IGNITE_HOME}"/libs/ignite-zookeeper/
wget -q --no-check-certificate https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-core/2.12.7/jackson-core-2.12.7.jar -P "${IGNITE_HOME}"/libs/ignite-zookeeper/
wget -q --no-check-certificate https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-annotations/2.12.7/jackson-annotations-2.12.7.jar -P "${IGNITE_HOME}"/libs/ignite-zookeeper/
wget -q --no-check-certificate https://repo1.maven.org/maven2/com/google/guava/guava/30.1-jre/guava-30.1-jre.jar -P "${IGNITE_HOME}"/libs/ignite-zookeeper/
d. If on Windows, do the following to update the files to the latest version: removelibs/ignite-zookeeper/zookeepr *jar files and libs/ignite-indexing/ directory.
i. Download the following jar files and place them in the libs/ignite-zookeeper/ directory.
https://repo1.maven.org/maven2/org/apache/zookeeper/zookeeper/3.8.1/zookeeper-3.8.1.jar
https://repo1.maven.org/maven2/org/apache/zookeeper/zookeeper-jute/3.8.1/zookeeper-jute-3.8.1.jar
https://repo1.maven.org/maven2/org/apache/curator/curator-client/5.2.1/curator-client-5.2.1.jar
https://repo1.maven.org/maven2/org/apache/curator/curator-framework/5.2.1/curator-framework-5.2.1.jar
https://repo1.maven.org/maven2/org/apache/curator/curator-recipes/5.2.1/curator-recipes-5.2.1.jar
https://repo1.maven.org/maven2/org/apache/curator/curator-x-discovery/5.2.1/curator-x-discovery-5.2.1.jar
https://repo1.maven.org/maven2/io/netty/netty-handler/4.1.86.Final/netty-handler-4.1.86.Final.jar
https://repo1.maven.org/maven2/io/netty/netty-buffer/4.1.86.Final/netty-buffer-4.1.86.Final.jar
https://repo1.maven.org/maven2/io/netty/netty-codec/4.1.86.Final/netty-codec-4.1.86.Final.jar
https://repo1.maven.org/maven2/io/netty/netty-common/4.1.86.Final/netty-common-4.1.86.Final.jar
https://repo1.maven.org/maven2/io/netty/netty-resolver/4.1.86.Final/netty-resolver-4.1.86.Final.jar
https://repo1.maven.org/maven2/io/netty/netty-transport-native-epoll/4.1.86.Final/netty-transport-native-epoll-4.1.86.Final.jar
https://repo1.maven.org/maven2/io/netty/netty-transport-native-unix-common/4.1.86.Final/netty-transport-native-unix-common-4.1.86.Final.jar
https://repo1.maven.org/maven2/io/netty/netty-transport/4.1.86.Final/netty-transport-4.1.86.Final.jar
https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-databind/2.12.7/jackson-databind-2.12.7.jar
https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-core/2.12.7/jackson-core-2.12.7.jar
https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-annotations/2.12.7/jackson-annotations-2.12.7.jar
https://repo1.maven.org/maven2/com/google/guava/guava/30.1-jre/guava-30.1-jre.jar
4. Update the default-config.xml file to enable ZooKeeper service discovery. You must place the ZooKeeper server address in the zkConnectionString connection property.
<?xml version="1.0" encoding="UTF-8"?>

<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd">

<bean class="org.apache.ignite.configuration.IgniteConfiguration">
<property name="discoverySpi">
<bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
<property name="ipFinder">
<bean class="com.thingworx.cache.ignite.discovery.TcpDiscoveryZookeeperIpFinder">
<property name="zkConnectionString" value="ZOOKEEPER_IPADDRESS:2181"/>
</bean>
</property>
</bean>
</property>
<property name="peerClassLoadingEnabled" value="true"/>
</bean>
</beans>
5. If you are running IPv6 on the system, you must modify the Ignite startup script to pass the Java option Djava.net.preferIPv4Stack=true. For more information, see Network Configuration | Ignite Documentation.
Installing ThingWorx 9.2x through 9.3x (Ignite 2.11 and Older)
1. Download Ignite from their website: https://ignite.apache.org/download.cgi#binaries. For version compatibility information, see the PTC Release Advisor.
2. Download the ThingWorx libraries cache-provider-ignite-[version].zip and cache-provider-api-[version].zip, extract the zip, and put the jar files into the Ignite folder $IGNITE_HOME/libs.
3. Use the ZooKeeper service discovery mechanism for the cluster to connect with the platform.
b. Move the ignite-zookeeper, ignite-log4j2, and ignite-slf4j directories from the $IGNITE_HOME/libs/optional folder to the $IGNITE_HOME/libs folder.
c. If on Linux, do the following to update files to latest version:
# remove the existing zookeeper jar file in the ignite-zookeeper folder
rm -rf "${IGNITE_HOME}"/libs/ignite-zookeeper/zookeeper-*

# remove libs/ignite-zookeeper/curator-*
rm -rf "${IGNITE_HOME}"/libs/ignite-zookeeper/curator*

# remove libs/ignite-zookeeper/guava-*
rm -rf "${IGNITE_HOME}"/libs/ignite-zookeeper/guava*

# download the latest zookeeper plus other required libraries and place in ignite-zookeeper folder
wget -q --no-check-certificate https://repo1.maven.org/maven2/org/apache/zookeeper/zookeeper/3.8.1/zookeeper-3.8.1.jar -P "${IGNITE_HOME}"/libs/ignite-zookeeper/
wget -q --no-check-certificate https://repo1.maven.org/maven2/org/apache/zookeeper/zookeeper-jute/3.8.1/zookeeper-jute-3.8.1.jar -P "${IGNITE_HOME}"/libs/ignite-zookeeper/
wget -q --no-check-certificate https://repo1.maven.org/maven2/org/apache/curator/curator-client/5.2.1/curator-client-5.2.1.jar -P "${IGNITE_HOME}"/libs/ignite-zookeeper/
wget -q --no-check-certificate https://repo1.maven.org/maven2/org/apache/curator/curator-framework/5.2.1/curator-framework-5.2.1.jar -P "${IGNITE_HOME}"/libs/ignite-zookeeper/
wget -q --no-check-certificate https://repo1.maven.org/maven2/org/apache/curator/curator-recipes/5.2.1/curator-recipes-5.2.1.jar -P "${IGNITE_HOME}"/libs/ignite-zookeeper/
wget -q --no-check-certificate https://repo1.maven.org/maven2/org/apache/curator/curator-x-discovery/5.2.1/curator-x-discovery-5.2.1.jar -P "${IGNITE_HOME}"/libs/ignite-zookeeper/
wget -q --no-check-certificate https://repo1.maven.org/maven2/io/netty/netty-handler/4.1.86.Final/netty-handler-4.1.86.Final.jar -P "${IGNITE_HOME}"/libs/ignite-zookeeper/
wget -q --no-check-certificate https://repo1.maven.org/maven2/io/netty/netty-buffer/4.1.86.Final/netty-buffer-4.1.86.Final.jar -P "${IGNITE_HOME}"/libs/ignite-zookeeper/
wget -q --no-check-certificate https://repo1.maven.org/maven2/io/netty/netty-codec/4.1.86.Final/netty-codec-4.1.86.Final.jar -P "${IGNITE_HOME}"/libs/ignite-zookeeper/
wget -q --no-check-certificate https://repo1.maven.org/maven2/io/netty/netty-common/4.1.86.Final/netty-common-4.1.86.Final.jar -P "${IGNITE_HOME}"/libs/ignite-zookeeper/
wget -q --no-check-certificate https://repo1.maven.org/maven2/io/netty/netty-resolver/4.1.86.Final/netty-resolver-4.1.86.Final.jar -P "${IGNITE_HOME}"/libs/ignite-zookeeper/
wget -q --no-check-certificate https://repo1.maven.org/maven2/io/netty/netty-transport-native-epoll/4.1.86.Final/netty-transport-native-epoll-4.1.86.Final.jar -P "${IGNITE_HOME}"/libs/ignite-zookeeper/
wget -q --no-check-certificate https://repo1.maven.org/maven2/io/netty/netty-transport-classes-epoll/4.1.86.Final/netty-transport-classes-epoll-4.1.86.Final.jar -P "${IGNITE_HOME}"/libs/ignite-zookeeper/
wget -q --no-check-certificate https://repo1.maven.org/maven2/io/netty/netty-transport-native-unix-common/4.1.86.Final/netty-transport-native-unix-common-4.1.86.Final.jar -P "${IGNITE_HOME}"/libs/ignite-zookeeper/
wget -q --no-check-certificate https://repo1.maven.org/maven2/io/netty/netty-transport/4.1.86.Final/netty-transport-4.1.86.Final.jar -P "${IGNITE_HOME}"/libs/ignite-zookeeper/
wget -q --no-check-certificate https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-databind/2.12.7/jackson-databind-2.12.7.jar -P "${IGNITE_HOME}"/libs/ignite-zookeeper/
wget -q --no-check-certificate https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-core/2.12.7/jackson-core-2.12.7.jar -P "${IGNITE_HOME}"/libs/ignite-zookeeper/
wget -q --no-check-certificate https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-annotations/2.12.7/jackson-annotations-2.12.7.jar -P "${IGNITE_HOME}"/libs/ignite-zookeeper/
wget -q --no-check-certificate https://repo1.maven.org/maven2/com/google/guava/guava/30.1-jre/guava-30.1-jre.jar -P "${IGNITE_HOME}"/libs/ignite-zookeeper/
wget -q --no-check-certificate https://repo1.maven.org/maven2/org/apache/ignite/ignite-zookeeper-ip-finder-ext/1.0.0/ignite-zookeeper-ip-finder-ext-1.0.0.jar -P "${IGNITE_HOME}"/libs/ignite-zookeeper/
d. If on Windows, perform the following to update the files to the latest version: removelibs/ignite-zookeeper/zookeepr *jar files and libs/ignite-indexing/ directory.
i. Download the following jar files and place them in the libs/ignite-zookeeper/ directory.
https://repo1.maven.org/maven2/org/apache/zookeeper/zookeeper/3.8.1/zookeeper-3.8.1.jar
https://repo1.maven.org/maven2/org/apache/zookeeper/zookeeper-jute/3.8.1/zookeeper-jute-3.8.1.jar
https://repo1.maven.org/maven2/org/apache/curator/curator-client/5.1.0/curator-client-5.1.0.jar
https://repo1.maven.org/maven2/org/apache/curator/curator-framework/5.1.0/curator-framework-5.1.0.jar
https://repo1.maven.org/maven2/org/apache/curator/curator-recipes/5.1.0/curator-recipes-5.1.0.jar
https://repo1.maven.org/maven2/org/apache/curator/curator-x-discovery/5.1.0/curator-x-discovery-5.1.0.jar
https://repo1.maven.org/maven2/io/netty/netty-handler/4.1.86.Final/netty-handler-4.1.86.Final.jar
https://repo1.maven.org/maven2/io/netty/netty-buffer/4.1.86.Final/netty-buffer-4.1.86.Final.jar
https://repo1.maven.org/maven2/io/netty/netty-codec/4.1.86.Final/netty-codec-4.1.86.Final.jar
https://repo1.maven.org/maven2/io/netty/netty-common/4.1.86.Final/netty-common-4.1.86.Final.jar
https://repo1.maven.org/maven2/io/netty/netty-resolver/4.1.86.Final/netty-resolver-4.1.86.Final.jar
https://repo1.maven.org/maven2/io/netty/netty-transport-native-epoll/4.1.86.Final/netty-transport-native-epoll-4.1.86.Final.jar
https://repo1.maven.org/maven2/io/netty/netty-transport-native-unix-common/4.1.86.Final/netty-transport-native-unix-common-4.1.86.Final.jar
https://repo1.maven.org/maven2/io/netty/netty-transport/4.1.86.Final/netty-transport-4.1.86.Final.jar
https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-databind/2.12.7/jackson-databind-2.12.7.jar
https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-core/2.12.7/jackson-core-2.12.7.jar
https://repo1.maven.org/maven2/com/fasterxml/jackson/core/jackson-annotations/2.12.7/jackson-annotations-2.12.7.jar
https://repo1.maven.org/maven2/com/google/guava/guava/30.1-jre/guava-30.1-jre.jar
4. Update the default-config.xml file to enable ZooKeeper service discovery. You must place the ZooKeeper server address in the zkConnectionString connection property.
<?xml version="1.0" encoding="UTF-8"?>

<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd">

<bean class="org.apache.ignite.configuration.IgniteConfiguration">
<property name="discoverySpi">
<bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
<property name="ipFinder">
<bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.zk.TcpDiscoveryZookeeperIpFinder">
<property name="zkConnectionString" value="ZOOKEEPER_IPADDRESS:2181"/>
</bean>
</property>
</bean>
</property>
<property name="peerClassLoadingEnabled" value="true"/>
</bean>
</beans>
5. If you are running IPv6 on the system, you must modify the Ignite startup script to pass the Java option Djava.net.preferIPv4Stack=true. For more information, see Network Configuration | Ignite Documentation.
Networking
The following ports must be open between the Ignite servers and the ThingWorx servers:
Port
Usage
47500-47600
Discovery port
47100-47200
Communication port
8080
Default port for REST API
49128
Default port for JMX connection
Configuring SSL/TLS for Ignite
To configure SSL/TLS for Ignite, see Configuring SSL/TLS for Ignite.
Was this helpful?