GZTP Troubleshooting

Step 1: Check the connectivity to EJBCA Server https://EJBCA-IP:8443/ejbca/adminweb/

Step 2: If connectivity to EJBCA is fine then ask the customer to verify the internet connectivity from the appliance.

Step 3: If the connectivity to EJBCA is down usually NOC will receive an automated e-mail from EJBCA server as below.

> Follow the steps provided on the mail

NOTE: Do not stop these process IDs unless there is a proper reason.

             All the logs related EJBCA are stored in server.log

             /opt/wildfly-9.0.2.Final/standalone/log/

Phases of GZTP:

Currently the device deployment involves phase 1 and phase 4

Phase 1(Provider pre-staging): During this phase the device will connect to EJBCA server to register and download the certificate to the device. The device will connect to GZTP controller, authenticate the IKE by PSK/Versa CA certificate then it notified to GZTP director to push the provider pre-staging configuration then the device goes into reboot.

Phase 2 

Phase 3: Device connects provider’s CA server to register the serial and download the certificate

Phase 4 (provider post staging) : Once the device is up, it will connect the provider controller and authenticate the IKE using Versa CA / PSK based authentication then notified to provider director to push provider staging configuration , this time again the device will go into reboot and come up with post staging configuration (IKE and MPBGP will up with the provider controller).

Pre-checks 

• NOTE: DHCP must be enabled for the Internet link
• Device should reachable to internet after loading the default configuration.

Required Details

• Customer name or Controller name
• Region
• Device serial number
• “show interface brief “output from device CLI

E.g.

VERSANOC184-IND

Customer name: VERSANOC184

Region: IND

• Connectivity issue to EJBCA server
• Connectivity issue to Versa staging server / Versa Controller
• Task on GZTP VD is getting failed

Phase 1 tshoot steps :

Step 1: With details provided verify the entries on the Botesh server (10.192.78.102)

 /home/versa/payloads

/home/versa/payloads/VERSANOC184-IND

Step 2: Check the connect alarms on both GZTP controllers for the serial number.

If there are no entries,

• Check the internet connectivity from the device after loading the default configuration
• Check if the certificate generated on EJBCA server, if there is no entry add the device serial again from chatbot/botesh server.

Registered itself and download certificate from CA.

You can find certificate is present like below:

Step 3: Check the logs under /home/versa/logs and task created for that device name on GZTP VD,

Check the workflow template status on the provider director and make sure it should be in deployed state.

Go to Administrator > Inventory > Hardware > check the device status > make sure it should not be in UNKNOWN state.

Step 4: Check the “show interface brief “from device CLI, CPE should receive mgmt. as shown below.

Step 5: Check the IKE/IPSEC state on the device

Step 6: Once the IKE is done, controller will connect and notifies the director, device will go into reboot and will push versa-staging configuration to the device, this can be verified under tasks created on GZTP VD.

Once the device receives versa-prestaging from GZTP VD then it should connect to provider’s controller Below are the reported issues in phase 4

• Connectivity issue between device and provider’s controller
• Connectivity issue between provider’s controller and director
• Connectivity issue between device and provider’s director
• PSK parameters mismatch between provider controller and device
• Global Organization ID mismatch

Step 1: Check the connectivity between device and provider’s controller WAN interface

Step 2: Check if the device receiving Mgmt IP from provider’s controller.

Step 3: Check IKE / IPSEC state on the device and alarms on provider controller for the device serial

If the IKE/IPSEC fails, check for the error and follow step 4

If the IKE/IPSEC is done, go to step 5

Step 4: Verify the templates, DG and the PSK parameters from device bind data and json file from Botesh.

umesh@Botesh:/home/versa$ cat JFSF12345-VERSANOC184-IND-0001-2.json

Step 5: Once the IKE/IPSEC state is done, check for the task on provider director

If there is no task created on VD

• Check for the IP conflict on VD LAN and device LAN
• Verify Global organization ID value on provider org and <device_step.2.json>

 End