Check Cassandra Service Status
- On the server where Cassandra is hosted, run the following command to check if the Cassandra service is active:# nodetool statusnodetool: Failed to connect to '127.0.0.1:7199' - ConnectException: 'Connection refused (Connection refused)'.
- If the service is not running, you can start it with:# sudo service cassandra restart
- If it fails to start, check the logs located at /var/log/cassandra/system.log for any errors.
- Binding issues (e.g., Cassandra unable to reach to the desired IP address or port).
- JVM-related errors (e.g., out-of-memory).
- Authentication or file-related issues.
Check NTP Service Status
Make sure the NTP is running, and time is in sync between the analytics nodes using the below command.
# timedatectlLocal time: Sun 2025-02-09 20:08:13 PST
Universal time: Mon 2025-02-10 04:08:13 UTC
RTC time: Mon 2025-02-10 04:08:14
Time zone: America/Los_Angeles (PST, -0800)
System clock synchronized: yes
systemd-timesyncd.service active: no
RTC in local TZ: no
If NTP is not running, follow the below KB to add NTP configuration through analytics admin GUI.
How to configure NTP on Versa Analytics nodes : Versa Support
Common observation in logs
1. System debug logs not updating
If the /var/log/cassandra/system.log are not updating with latest logs post restarting of the Cassandra service.
# dateSun Feb 9 19:23:25 PST 2025
# sudo service cassandra restart
# tail -f /var/log/cassandra/system.log
INFO [StorageServiceShutdownHook] 2025-02-09 20:20:31,064 HintsService.java:210 - Paused hints dispatch
INFO [StorageServiceShutdownHook] 2025-02-09 20:20:31,093 Server.java:179 - Stop listening for CQL clients
INFO [StorageServiceShutdownHook] 2025-02-09 20:20:31,093 Gossiper.java:1720 - Announcing shutdown
INFO [StorageServiceShutdownHook] 2025-02-09 20:20:31,095 StorageService.java:2484 - Node /172.168.10.3 state jump to shutdown
To recover this, replace the cassandra-env.sh file with the latest configuration.
If the service is not running, you can start it with:
# sudo su# cd /etc/cassandra/
# ls -l
# cp cassandra-env.sh /home/admin
# cp cassandra-env.sh.dpkg-dist cassandra-env.sh
2. Commit log corruption.
If the /var/log/cassandra/system.log is failing with commitlog file errors.
ERROR [CommitLogAllocator:1] 2025-02-10 15:22:45,056 CommitLogAllocator.java:417 - Corrupt commit log file /var/lib/cassandra/commitlog/CommitLog-1234567890.log detected. Ignoring this file and continuing. ERROR [CommitLogAllocator:1] 2025-02-10 15:22:46,045 CommitLog.java:523 - Failed to recover commit log segment /var/lib/cassandra/commitlog/CommitLog-1234567890.log. Exception: java.io.IOException: Unable to read file: /var/lib/cassandra/commitlog/CommitLog-1234567890.log ERROR [CommitLogAllocator:1] 2025-02-10 15:22:47,128 CommitLog.java:284 - Error during commit log recovery. Will retry after backoff. Exception: java.nio.file.InvalidPathException: Malformed input or invalid file path for /var/lib/cassandra/commitlog/CommitLog-1234567890.log WARN [CommitLogAllocator:1] 2025-02-10 15:22:50,903 CommitLog.java:134 - Commit log file /var/lib/cassandra/commitlog/CommitLog-1234567890.log corrupted. Data may be lost. ERROR [CommitLogAllocator:1] 2025-02-10 15:22:51,235 CommitLogAllocator.java:292 - CommitLog segment corruption detected, Cassandra may not recover from this failure cleanly. ERROR [CommitLogAllocator:1] 2025-02-10 15:22:52,004 CassandraDaemon.java:490 - Exiting Cassandra due to commit log corruption.
To recover this, remove the corrupted commit log files and restart Cassandra service.
# sudo rm -rf /var/lib/cassandra/commitlog/CommitLog-1234567890.log