Steps to Troubleshoot VAST CSI Driver

Run Basic Troubleshooting Commands

Run the following commands to troubleshoot VAST CSI Driver:

Display all pods:

kubectl get pods --all-namespaces -o wide

Check logs from the controller.
Typically, there are four containers: csi-provisioner, csi-attacher, csi-resizer, and csi-vast-plugin. Check all of them.
Run the following command to display the most important information:
```
kubectl logs csi-vast-controller-0 --namespace vast-csi -c csi-vast-plugin | less
```
Check logs from the node.
There are probably two containers: csi-node-driver-registrar and csi-vast-plugin . Check both.
Run the following command to display the most important information:
```
kubectl logs csi-vast-node-<NODE_ID> --namespace vast-csi -c csi-vast-plugin
```
Check to see if there are any system-level events, such as errors.
Run the following command to list events sorted by timestamp:
```
kubectl get events --sort-by=.metadata.creationTimestamp
```

Display persistent volume claims:

kubectl get pvc

The output is similar to the following:

NAME         STATUS VOLUME                                   CAPACITY ACCESS MODES STORAGECLASS        AGE
shared-claim Bound  pvc-41e3cd02-1a80-4dfb-bb50-6d272a9e649e 1Mi      RWO          managed-nfs-storage 12d

Display persistent volumes:

kubectl get pv

The output is similar to the following:

NAME                                     CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM                STORAGECLASS        REASON AGE
pvc-41e3cd02-1a80-4dfb-bb50-6d272a9e649e 1Mi      RWO          Delete         Bound  default/shared-claim managed-nfs-storage        12d

Display the defined storage classes:

kubectl get sc

The output is similar to the following:

NAME                PROVISIONER AGE
managed-nfs-storage nfs-client  13d

Note
To get more detailed information, use the describe option on each command.

Check Kubernetes Logs

If none of the above helps, check the underlying Kubernetes logs.

The location of Kubernetes logs depends on your operating system. On CentOS, the typical location is /var/log/messages.

These logs often contain more detailed information about the container startup process and provide an indication of errors.

Check VMS Logs

VAST CSI Driver contacts VAST Management Service (VMS) to create directories and quotas on behalf of the Kubernetes cluster. So checking VMS logs can help troubleshoot your environment.

VMS logs can be found in /vast/vman/vms/log on the VMS node.

Useful information is often found in vms.log and vapi.log. For example, the vms.log can provide the following information (the mount for VAST CSI Driver is /keystest/k8s):

grep k8s vms.log
[2022-01-05 17:18:47,313] INFO [services.py:382/430] AUDIT - VMS - Manager: admin (10.61.203.63) POST /api/quotas/ with body {'create_dir': True, 'name': 'csi:nfs-vol-claim:nfs-vol-claim:pvc-0f5d878e-4fa4-4c91-a915-7a5c', 'path': '/keystest/k8s/pvc-0f5d878e-4fa4-4c91-a915-7a5c9fcdfc2c', 'hard_limit': 1048576}
[2022-01-05 17:18:47,562] INFO [services.py:837/430] Creating Quota with {'create_dir': True, 'name': 'csi:nfs-vol-claim:nfs-vol-claim:pvc-0f5d878e-4fa4-4c91-a915-7a5c', 'path': '/keystest/k8s/pvc-0f5d878e-4fa4-4c91-a915-7a5c9fcdfc2c', 'hard_limit': 1048576}
[2022-01-05 17:18:47,573] INFO [services.py:843/430] Creating directory path: /keystest/k8s/pvc-0f5d878e-4fa4-4c91-a915-7a5c9fcdfc2c
[2022-01-05 17:18:47,578] INFO [services.py:852/430] Created directory path: /keystest/k8s/pvc-0f5d878e-4fa4-4c91-a915-7a5c9fcdfc2c