If an Analytic cluster has been around since 20.2.x release, and has sustained upgrades into 21.x without it's Solr (Search) DB ever being deleted/re-created, it can contain Solr DB objects of N-2 Version (Lucene 6.0) which are not compatible with Lucene 8.0 version used by Solr in 22.1.x
Hence post upgrade to 22.1.x you can observe the below error in the output of "vsh dbstatus" on the Solr nodes where "searchlogs" collection would be stuck in "down" state
While checking the cluster_status output you will see the below
cd /opt/versa/scripts/van-install
sudo ./cluster_install.sh solr cluster_status | python -m json.tool
<snip>
You will see the below errors in solr.log (under /var/log/versa/solr/ directory) - this is definitive indication that you are facing the issue discussed here
This issue is due to the incompatibility of Lucene 6.0 objects with Lucene 8.0 version used in 22.1.x Solr
Prior to upgrade you can check if Lucene 6.0 elements are present by executing the below - if Lucene5* files are present it would indicate that the Lucene 6.0 objects will "possibly" be present in the Solr DB and you may hit this issue.
root@versa-analytics:/etc/solr/bin# ls -tlr /var/lib/solr/data/data/searchlogs_shard1_replica1/data/index | grep Lu
-rw-rw---- 1 solr solr 4000 Jun 13 11:05 _3_Lucene54_0.dvm
-rw-rw---- 1 solr solr 12776 Jun 13 11:05 _3_Lucene54_0.dvd
-rw-rw---- 1 solr solr 12261 Jun 13 11:05 _3_Lucene50_0.tim
-rw-rw---- 1 solr solr 776 Jun 13 11:05 _3_Lucene50_0.tip
You can also check solr.log to verify if lucene 6.0 reference is present (however it's possible that initialization logs have rolled over in which case this may not be deterministic)
root@versa-analytics:# grep lucene.version=6 /var/log/versa/solr/solr.log*
2024-06-13 18:40:57.322 INFO (searcherExecutor-14-thread-1) [ ] o.a.s.c.SolrCore [searchlogs_shard1_replica1] Registered new searcher Searcher@15ce5a24[searchlogs_shard1_replica1] main{ExitableDirectoryReader(UninvertingDirectoryReader(_b(6.6.1):C761:[diagnostics={os=Linux, java.vendor=Oracle Corporation, java.version=1.8.0_281, java.vm.version=25.281-b09, lucene.version=6.6.1, mergeMaxNumSegments=-1, os.arch=amd64, java.runtime.version=1.8.0_281-b09, source=merge, mergeFactor=10, os.version=4.4.0-151-generic, timestamp=1718301916088}]:[attributes={Lucene50StoredFieldsFormat.mode=BEST_SPEED}] _l(6.6.1):C1319:[diagnostics={os=Linux, java.vendor=Oracle Corporation, java.version=1.8.0_281, java.vm.version=25.281-b09, lucene.version=6.6.1, mergeMaxNumSegments=-1, os.arch=amd64, java.runtime.version=1.8.0_281-b09, source=merge, mergeFactor=10, os.version=4.4.0-151-generic,
Solution: Unfortunately the only option is to delete the "searchlogs" collection and re-create it so that the old objects are dropped, which essentially means that all the search logs will be deleted (alarms, access logs, traffic monitoring logs etc), this will not affect the "Analytic" logs (Cassandra DB is not affected by this, and hence all the Analytic stats will be available)
Procedure to delete and re-create the searchlogs collection is as below. This should be executed on any "one" of the Search nodes in the cluster - in other words it should be executed on just 1 search node (any one of them)
cd /opt/versa/scripts/van-install
sudo su
./cluster_install.sh solr delete_alias globallogs
./cluster_install.sh solr delete_alias alarmlogsRead
./cluster_install.sh solr upd_alias alarmlogsRead alarmlogs
./cluster_install.sh solr delete_collection searchlogs
Check if the "searchlogs" collection has been removed
./cluster_install.sh solr cluster_status
Now, create the "searchlogs" collection as bellow
./cluster_install.sh solr create_collection searchlogs
./cluster_install.sh solr upd_alias globallogs alarmlogs/searchlogs
Check if the "searchlogs" collection has been re-added
./cluster_install.sh solr cluster_status
The search log retention is usually for 3 days (or 7 days for alarms), these are the only logs that would be deleted.
Please reach out to Versa TAC for any further clarifications
