Issue Description


After performing Analytics hardening and regenerating database passwords, the Search (Solr) service may fail authentication. This typically occurs when the Solr service continues using old credentials even though the Analytics and Search database passwords have been updated.


Root Cause

The password regeneration script successfully updates the Analytics and Search database credentials, but the Solr service on the Search node does not automatically pick up the new password. As a result, Solr continues attempting authentication with the old Cassandra credentials.


Script used to change passwords:

sudo ./van_cluster_installer.py --regen-analytics-db-password --regen-search-db-password --force

When running vsh dbstatus on the Search node, the command may return an authentication error similar to the following:

  • HTTP 401 – require authentication

  • Failure to retrieve system information from Solr

  • JSON parsing errors caused by Solr returning an HTML error page instead of JSON

ERROR: Failed to get system information from http://localhost:8983/solr

HTTP ERROR 401 require authentication

URI: /solr/admin/info/system


Example:

$ vsh dbstatus

=====================

Datacenter: Search

=====================

ERROR: Failed to get system information from http://localhost:8983/solr due to: org.apache.solr.common.SolrException: Parse error : <html>

<head>

<meta data-fr-http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>

<title>Error 401 require authentication</title>

</head>

<body><h2>HTTP ERROR 401 require authentication</h2>

<table>

<tr><th>URI:</th><td>/solr/admin/info/system</td></tr>

<tr><th>STATUS:</th><td>401</td></tr>

<tr><th>MESSAGE:</th><td>require authentication</td></tr>

<tr><th>SERVLET:</th><td>default</td></tr>

</table>

</body>

</html>

Found 1 Solr nodes:

Solr process 6016 running on port 8983

--- Logging error ---

Traceback (most recent call last):

File "/opt/versa/scripts/van-install/solr_op.py", line 21, in get_json_val_generic

data = json.load(json_file)

File "/usr/lib/python3.8/json/__init__.py", line 293, in load

return loads(fp.read(),

File "/usr/lib/python3.8/json/__init__.py", line 357, in loads

return _default_decoder.decode(s)

File "/usr/lib/python3.8/json/decoder.py", line 337, in decode

obj, end = self.raw_decode(s, idx=_w(s, 0).end())

File "/usr/lib/python3.8/json/decoder.py", line 355, in raw_decode

raise JSONDecodeError("Expecting value", s, err.value) from None

json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

File "/usr/lib/python3.8/logging/handlers.py", line 69, in emit

if self.shouldRollover(record):

File "/usr/lib/python3.8/logging/handlers.py", line 185, in shouldRollover

msg = "%s\n" % self.format(record)

File "/usr/lib/python3.8/logging/__init__.py", line 925, in format

return fmt.format(record)

File "/usr/lib/python3.8/logging/__init__.py", line 664, in format

record.message = record.getMessage()

File "/usr/lib/python3.8/logging/__init__.py", line 369, in getMessage

msg = msg % self.args

TypeError: not all arguments converted during string formatting

Call stack:

File "/opt/versa/scripts/van-install/solr_op.py", line 429, in <module>

sys.exit(main())

File "/opt/versa/scripts/van-install/solr_op.py", line 409, in main

find_unhealthy_coll_shard_replica(cluster_status_file)

File "/opt/versa/scripts/van-install/solr_op.py", line 95, in find_unhealthy_coll_shard_replica

collections = get_json_val_generic(cluster_status_file, key_vals)

File "/opt/versa/scripts/van-install/solr_op.py", line 36, in get_json_val_generic

log.error("Opening file:", input_file)

Message: 'Opening file:'

Arguments: ('/tmp/cluster_status_file',)

All collections/shards/replicas are healthy!


Verification Steps:

Note: execute below commands with sudo 'sudo su'

  1. confirm password update - Ensure the output reflects the new Cassandra password.

    vsh print-auth

  2. Test Solr authentication with the new password - If this command succeeds, Solr is using the updated credentials. If it fails, proceed to the next step

    curl -s --user cassandra:NEW_PASSWORD 'http://localhost:8983/solr/admin/collections?action=clusterstatus&wt=json'

  3. Test using the default Cassandra password - If this works, it confirms Solr has not yet applied the new password

    curl -s --user cassandra:cassandra 'http://localhost:8983/solr/admin/collections?action=clusterstatus&wt=json'



Resolution:

Manually update the Cassandra password used by Solr via the Solr security API.
Note: execute below commands with sudo 'sudo su'


Step 1: Update Solr authentication password

Run the following command using the current working credentials (typically the default password):

curl --user cassandra:cassandra http://localhost:8983/api/cluster/security/authentication -H "Content-type:application/json" -d "{'set-user': {'cassandra': 'NEW_PASSWORD'}}"




Step 2: Verify the fix

Run the database status check again:

vsh dbstatus


The output should now complete without authentication or parsing errors.