Use Cases
Following use cases are supported by Codebeamer's cluster architecture:
• Graceful Addition: This process involves adding a new node to a running cluster, ensuring its seamless integration, recognition, request processing, and data replication.
• Graceful Removal: This process involves removing a node from a running cluster, ensuring its seamless removal, workload redistribution to other nodes, and the maintenance of cluster operation and data consistency.
• Sudden Removal: This process involves abruptly removing a node, simulating a hardware failure, and verifying the cluster's ability to automatically detect the failure, redistribute the workload, and maintain connections and data consistency.
• Multiple Node Failures: This process involves simulating simultaneous failures of multiple nodes and verifying the cluster's ability to handle them, rebalance the workload, maintain data consistency, and remain operational.
What happens during the graceful addition
1. The cluster is up and running.
2. A new node has been initiated.
3. The new node integrates itself into the cache synchronization process.
1. In earlier versions of Codebeamer, a new file was created in the config/cache/peerDiscovery directory.
2. In Codebeamer, cache peer information is added to the cache_peer_discovery table.
4. The new node chooses a random node in the cluster to pre-populate its caches.
5. Existing nodes in the cluster load cache peer information and register the new node into the cluster.
6. Cache synchronization events are sent to the new node.
7. The new node is added to the load balancer and is ready to serve requests.
What happens during the graceful removal
1. The cluster is up and running.
2. An existing node has been shut down gracefully.
3. The node removes itself from the cache synchronization process.
a. In the earlier versions of Codebeamer, a peer discovery file was removed from theconfig/cache/peerDiscovery directory.
b. In the current versions of Codebeamer, cache peer is removed from thecache_peer_discovery table.
4. Existing nodes in the cluster load the cache peer information and deregister the stopped node from the cluster.
5. Cache synchronization events are not sent to the stopped node.
6. The stopped node is removed from the load balancer and is stopped to serve requests.
What happens during the sudden removal
1. The cluster is up and running.
2. An existing node has been shut down gracefully.
3. The node cannot remove itself from the cache synchronization process.
◦ In the earlier versions of Codebeamer, the peer discovery file was not updated in the config/cache/peerDiscovery directory.
◦ In the current versions of Codebeamer, cache peer is not updated in the cache_peer_discovery table.
4. Existing nodes in the cluster load cache peer information and deregister the stopped node from the cluster.
5. Cache synchronization events are not sent to the stopped node.
6. The stopped node is removed from the load balancer and is stopped to serve requests.
Alive state of a cache peer
The parameters CB_RMI_CACHE_TIME_TO_LIVE and CB_RMI_CACHE_SOCKET_TIMEOUT_MILLIS influence how quickly a cache peer is removed from the cluster. The application tries to send cache synchronization requests to a cache peer if it is considered alive. The application retrieves cache peers information from the database, which includes the last_alivetimestamp. If this timestamp falls within the CB_RMI_CACHE_TIME_TO_LIVEinterval, the application tries to send a cache replication request to the node. If the node is down, the request waits until the CB_RMI_CACHE_SOCKET_TIMEOUT_MILLIS expires.
Alive state of an application server
The application's health is assessed based on the status of the health check URL. A 200-status code indicates the application is healthy, while a 5xx status code signifies it is unhealthy. The application returns a 500-status code if cache replication between nodes fails or if the database is inaccessible. If the application crashes due to an out-of-memory (OOM) error or hardware issue, it does not send any response.
It is advisable to remove any unhealthy application from the cluster, as these nodes cannot handle requests, including cache replications.