Configuring SSL/TLS for ZooKeeper
You can configure SSL or TLS for ZooKeeper when using ThingWorx HA Clustering:
Configuring ZooKeeper
1. Make sure you are running a version of ZooKeeper that supports SSL or TLS.
2. Get your SSL certificate and the truststore. Perform the following steps:
1. Create the ZooKeeper keystore.
2. Use the public key of the ZooKeeper keystore and create the ZooKeeper truststore.
 |
The only acceptable extensions for certificates are JKS, PEM, and PKCS12(p12).
|
3. Go to apache-zookeeper-[version]-bin/conf and update or create zoo.cfg.
4. Add the following entries:
dataDir=/<path-to-zookeeper-data>/data
dataLogDir=/<path-to-zookeeper-datalog>/datalog
secureClientPort=2281
tickTime=2000
initLimit=5
syncLimit=2
autopurge.snapRetainCount=3
autopurge.purgeInterval=0
maxClientCnxns=60
admin.enableServer=true
standaloneEnabled=false
quorumListenOnAllIPs=true
sslQuorum=true
ssl.clientAuth=none
To activate the quorum between ZooKeeper nodes, set the variable sslQuorum=true in thezoo.cfg file. The nodes will use the auto-generated SSL to secure the quorum.
ZooKeeper uses one-way TLS, not mutual TLS (mTLS). The ssl.clientAuth property accepts:
◦ none–No client certificate is required (one-way SSL).
◦ want–ZooKeeper requests a client certificate but does not enforce it.
◦ need–ZooKeeper requires a valid client certificate (full mTLS).
5. In the directory specified by the dataDir path in zoo.cfg, create a file named myid for each node. The file must contain a single numeric ID that corresponds to the node:
◦ For node 1, enter 1 on the first line, then save and close the file.
◦ Repeat for each node (for example, node 2 should have 2, node 3 should have 3, and so on).
Ensure the file contains only digits with no extra whitespace and is readable by the ZooKeeper service user (typically zookeeper). If the ID does not match the configuration, the node will fail to join the cluster.
6. At the end of zoo.cfg, add entries in the following format: server.X=<hostname or IP>:<peerPort>:<electionPort>
Example for a 3-node ensemble:
# 3-Node ensemble configuration
# Format: server.X=<hostname or IP>:<peerPort>:<electionPort>
server.1=10.0.0.1:2888:3888
server.2=10.0.0.2:2888:3888
server.3=10.0.0.3:2888:3888
Ensure the server.X numbers match the myid file on each node. The dataDir and dataLogDir directories must exist and be writable by the ZooKeeper user. Auto-purge is strongly recommended in production to prevent disk growth.
7. Modify the ZooKeeper startup script.
◦ For Linux deployments:
Modify <path to zookeeper>/bin/zkServer.sh and add the following: export SERVER_JVMFLAGS="...".
◦ For Windows deployments:
If deploying on Windows, follow these adjusted instructions:
a. Modify <path to zookeeper>/bin/zkServer.cmd.
b. Use the set command instead of export: set SERVER_JVMFLAGS=....
c. The script is not preconfigured to handle this variable. You must declare the variable and reference it in the Java launch command. For example:
export SERVER_JVMFLAGS="
-Dzookeeper.serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory
-Dzookeeper.ssl.keyStore.location=<path-to-zookeeper-certificates>/zookeeper.p12
-Dzookeeper.ssl.keyStore.password=<certificate-password>
-Dzookeeper.ssl.trustStore.location=<path-to-zookeeper-certificates>/truststore.p12
-Dzookeeper.ssl.trustStore.password=<truststore-password>
-Dzookeeper.ssl.quorum.keyStore.location=<path-to-zookeeper-certificates>/zookeeper.p12
-Dzookeeper.ssl.quorum.keyStore.password=<certificate-password>
-Dzookeeper.ssl.quorum.trustStore.location=<path-to-zookeeper-certificates>/truststore.p12
-Dzookeeper.ssl.quorum.trustStore.password=<truststore-password>
-Dzookeeper.ssl.quorum.hostnameVerification=false"
8. Start ZooKeeper:
./zkServer.sh start
9. In the logs, verify that the configuration is correct:
tail -f apache-zookeeper-3.5.6-bin/logs/<zookeeper-log-file>
Configuring ThingWorx
1. Copy the ZooKeeper certificates to your instance or make sure they are available on the machine where ThingWorx is running.
|
|
Tomcat users must have access to the certificates or copy them into the ThingworxPlatform directory.
|
2. Modify your platform-settings.json to include the following as a root element at the end of the file, on the same level as PlatformSettingsConfig.
"ZookeeperSettings": {
"SSLEnabled": "true",
# If SSL is enabled, you must include the following; trust store is optional:
"KeyStorePath": "<path-to-zookeeper-certificates>/zookeeper.p12",
"KeyStorePass": "<certificate-password>",
"TrustStorePath": "<path-to-zookeeper-certificates>/truststore.p12",
"TrustStorePass": "<truststore-password>"
"SASLEnabled": "false",
# If SASL is enabled, you must include the following:
"JaasConfPath": "/tmp1/jaas.conf",
"Krb5ConfPath": "/tmp1/krb5.conf"
}
3. Search for the default ZooKeeper port 2181 and replace it with the secure port 2281.
4. Make sure all of the CoordinatorHosts entries and address-resolver > connection ports are updated to match the secureClientPort value from zoo.cfg.
Configuring Ignite
1. Copy the ZooKeeper keystore and truststore to the Ignite machine or make sure they are available on the machine where the Ignite server is running.
2. Set the ZOOKEEPER_CONNECTION environment variable, and find the JVM_XOPTS environment variable used to start Ignite and update it as follows:
# zookeeper1 represents the host name where zookeeper is available and 2281 the secure port from zoo.cfg
export ZOOKEEPER_CONNECTION=zookeeper1:2281,zookeeper2:2281,zookeeper3:2281
# update the JVM_XOPTS
JVM_XOPTS="-Dzookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty -Dzookeeper.client.secure=true -Dzookeeper.ssl.keyStore.location=<path-to-zookeeper-certificates>/zookeeper.p12 -Dzookeeper.ssl.keyStore.password=<keystore-password> -Dzookeeper.ssl.trustStore.location=<path-to-zookeeper-certificates>/truststore.p12 -Dzookeeper.ssl.trustStore.password=<truststore-password>"
Configuring the Connection Server
1. Copy the ZooKeeper certificates to your instance or make sure they are available on the machine where the Connection Server is running.
2. Update the port for the cx-server.discovery.connectionString in the Connection Server configuration file to use the secure port.
For example, cx-server.discovery.connectionString = "{zookeeper-host}:2281".
3. Add the following system properties to the CONNECTION_SERVER_OPTS environment variable.
4. If ThingWorx is using TLS, set the environment variable for it. For more information, see Setting an Environment Variable When Using SSL/TLS.
For example:
export CONNECTION_SERVER_OPTS="
-Dzookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty
-Dzookeeper.client.secure=true
-Dzookeeper.ssl.keyStore.location=<path-to-zookeeper-certificates>/zookeeper.p12
-Dzookeeper.ssl.keyStore.password=<keystore-password>
-Dzookeeper.ssl.trustStore.location=<path-to-zookeeper-certificates>/truststore.p12
-Dzookeeper.ssl.trustStore.password=<truststore-password>"
Encrypting Passwords Using the ThingWorx Security Management Tool
|
|
After completing the server setup with all components, we recommend encrypting passwords to enhance security.
|
If you want to avoid inserting plain passwords into the platform-settings.json file, you can encrypt passwords inside twx-keystore using the security tool. You must encrypt the password using the encrypt.zk.keystore.password and encrypt.zk.truststore.password for key store and trust store passwords respectively.
./security-common-cli keystore.conf set encrypt.zk.keystore.password "ptcptc"
Then change the platform-settings.json file to have ThingWorx pick the password from the key store:
"KeyStorePass": "encrypt.zk.keystore.password",
"TrustStorePass": "encrypt.zk.truststore.password"
Next Steps
To complete your Ignite setup, refer to the following topics: