This article provides step-by-step instructions to replace and re-add a failed node (Analytics or Search) in a Versa cluster.

Prerequisites

Before starting, ensure the following for the replacement node:

  • Same Versa software version as other nodes in the cluster.

  • Same number of network interfaces.

  • Similar CPU, memory, and hyper-threading profile.

  • IP addresses and routes are identical to the failed node.

  • Hostname and DNS entries are correctly set in /etc/hosts.


Step 1: Prepare the Node

  1. Make sure the following files are the same from an existing node of the same personality (Analytics or Search):

  2. Make sure the below on vansetup.conf on the new node:

    • Update the rpc address and listener address fields.

    • zookeeper parameter should be set correctly.

    • Replication factor set as per your cluster configuration.

    • For Analytics nodes only, set the seeds parameter to the listener IP of an existing Analytics node:

      seeds="a.b.c.d"

Step 2: Perform Pre-Setup Based on Node Type

  • Analytics Node:

    • Run the following on any existing Analytics node to remove the old node's host ID:

      # nodetool status  # Note the host-id of the failed node 

# nodetool removenode <host-id>

  • Search Node:

    • Ensure the Zookeeper cluster is up and has a leader before running the setup. Run below command on node running zookeeper.

      # vsh dbstatus

Step 3: Run vansetup.py

Execute the setup script on the new node:

# cd /opt/versa/scripts/van-scripts 
# sudo ./vansetup.py

Step 4: Sync Certificates from the Director node

# sudo su versa 
# cd /opt/versa/vnms/scripts/
# ./vnms-cert-sync.sh --sync 
# ./vd-van-cert-upgrade.sh --pull

When prompted for "postpone restart", select y.

Step 5: Restart Directors (HA Setup Only)

  1. On Secondary Director: vsh stop

  2. On Primary Director: vsh restart

  3. On Secondary Director: vsh start

Verify HA sync between directors

Post-Checks

For Analytics Node

  • Run nodetool status on both the new and existing Analytics nodes.

  • New node will initially show UJ (Joining). It should change to UN (Up/Normal) once data sync completes.

For Search Node

  1. Check search DB status:

    # vsh dbstatus

    • Ensure live-nodes count is correct.

    • collections should be healthy

  2. Check Solr cluster health:

    # sudo /opt/versa/scripts/van-install/cluster-install.sh solr cluster_status

            All replicas should show as active.

Confirm the new node is reachable via the Director UI.

Troubleshooting

If the node fails to join properly or services remain inactive:

  • Review nodetool status, vsh dbstatus, and Solr status.

If issues persist, open a ticket with Versa TAC and include:

  • Node type and personality

  • Output of post-check commands

  • Capture the tech-support and shell session logs