Problem description: When a flexvnf is brought up (either from the factory), or post an upgrade or usb based image installation, the services are found to be "stopped" state as shown below OR if you find inconsisent packages while executing sudo dpkg -l | grep -v "ii" (as part of pre-check before upgrade)
Execute the below command check if the image installation is corrupted - if the output shows "half-configuration" it implies that the previous installation was not successful and it has left the system in an inconsistent state.
From the "shell" prompt execute the below
sudo dpkg -s versa-flexvnf
OR if you find packages is an inconsistent state (any state other than "ii" is deemed inconsistent)
As a means of repairing the inconsistent packages you are trying executing the below
sudo dpkg --configure -a
Post executing this, you can re-check sudo dpkg -l | grep -v "ii" to confirm if packages are still in an inconsistent state
If inconsistent packages continue to exist, one of the means to recover is to re-install the image via shell ("sudo /home/versa/packages/<image>") however before re-installing you would have to execute the below steps to prepare the system for a re-installation
Note that re-installation would purge the current configuration and the site would b
Re-installation procedure
Step 1: Execute the below and kill any tasks that show up
0 S admin 18006 17853 0 80 0 - 3000 pipe_w 11:02 pts/0 00:00:00 flexvnf stop --color=auto flexvnf
0 S admin 18007 17856 0 80 0 - 3000 pipe_w 11:02 pts/0 00:00:00 flexvnf run --color=auto flexvnf
18006 is the pid of this taks, you can kill is as below (kill all the tasks that show up in the output)
[admin@Flex1: ~] $ sudo kill -9 18006
[admin@Flex1: ~] $ sudo kill -9 18007
Step 2: Stop the monit and fail2ban services
sudo service monit stop
sudo service fail2ban stop
Step 3: Remove the monit.pid and fail2ban.pid (if it exists) from /var/run
[admin@Flex1: ~] $ cd /var/run
[admin@Flex1: run] $ ls -lrt | grep monit
-rw-r--r-- 1 root root 5 Jul 13 12:40 monit.pid
[admin@Flex1: run] $ ls -lrt | grep fail2ban
drwxr-xr-x 2 root root 80 Jul 13 12:40 fail2ban << there is no fail2ban.pid so just removing monit.pid alone
[admin@Flex1: run] $ sudo rm -rf monit.pid
Step 4: Make sure there are not "versa" tasks running
[admin@Flex1: run] $ ps -aefl | grep versa | grep -v "grep"
If there are any versa tasks running you can kill them using "sudo kill -9 <pid>" similar to how we killed the tasks in step 1
Step 5: Modify the file /etc/init.d/versa-flex vnf
[admin@Flex1: run] $ sudo vi /etc/init.d/versa-flexvnf
press "shift g" to go to bottom the page and look for the below line "stop)" and add "exit 0" just below it
Modified as below (make sure you don't disturb the indentation of the other lines)
Save the file using "esc" followed by ":wq!"
Step 6: Remove the files under /opt/versa
[admin@Flex1: run] $ sudo rm -rf /opt/versa
Step 7: Proceed with image installation using the step below (we use nohup to ensure that the installation task runs in the background - so that it continues to run even if we lose the ssh connection)
Go to the /home/versa/packages directory and look for the image that you want to re-install, and execute the installation using "nohup sudo ./<image> &" as shown below
cd /home/versa/packages/
ls -lrt
nohup sudo ./versa-flexvnf-20200606-211402-63668a2-20.2.2-wsm.bin &
Execute the below to check the progress of the installation
tail -f nohup.out
Step 8: If you lose the "ssh connection" in the middle of the installation, you can re-login and check the progress of the installation by executing the below
tail -f /home/versa/packages/nohup.out
Step 9: When it reaches the below step of the installation (the logs will stop moving further) - perform a "sudo reboot"
Step 10: Wait for 7-10 mins to allow the reboot to complete, post reboot check the services and check output of "show interfaces brief" - if the services are running and "show interfaces brief" shows all the vni interfaces, you can go ahead with the staging