Handling Large Datasets
If you are working in ThingWorx Analytics Server with a large dataset (for example, larger than 9 million rows by 30 columns), you may encounter a com.esotericsoftware.kryo.KryoException: Buffer overflow error when you run some analytics operations. If this type of error occurs, you can resolve it by increasing the maximum buffer size of a specific flag in the system-environment-variables.properties file.
To increase the maximum buffer size:
1. In the installation directory where your ThingWorx Analytics server is installed, navigate to the config folder and open the system-environment-variables.properties file.
2. Find the property called # Maximum allowable size of Kryo serialization buffer and update the size of the spark.kryoserializer.buffer.max flag to 128m.
3. Save the file and restart your ThingWorx Analytics server in order for the change to take effect.