Monitor and Troubleshoot VAST CSI Driver

Export VAST CSI Driver Metrics to Prometheus

VAST CSI Driver can be configured to expose CSI node and controller metrics in Prometheus format. The node metrics include total counts and average durations for CSI RPCs and mounts/umounts, and also NFS transport (xprt) statistics. The controller metrics include total counts and average durations for CSI RPCs.

Enable Export of CSI Metrics

By default, the driver does not expose any metrics.

To enable export of metrics:

Add the following to the driver's Helm chart configuration file:

node:
  metrics:
    enabled: true
    port: 9090
controller:
  metrics:
    enabled: true
    port: 9091

Install or upgrade the driver's Helm chart.

Exposed CSI Metrics Endpoints and Ports

When metrics export is enabled:

A headless service is created that serves metrics requests at two endpoints:
- GET /metrics for getting the metrics in Prometheus format (counters, histograms, gauges),
- GET /health for health checks.
The node's DaemonSet pods expose the node metrics port 9090.
The controller's Deployment/StatefulSet pods expose the controller metrics port 9091.

Note: You can override default ports by specifying a different value in the port entry under node or controller metrics in the driver's Helm chart configuration file.

Exported CSI Metrics

Tip: For a complete reference on CSI metrics, see here.

CSI Node Metrics

Mounts/umounts


`csi_node_mount_operations_total`	Total number of mounts (of a PVC to a pod)
`csi_node_mount_duration_seconds`	Duration of mounts (in seconds)
`csi_node_umount_operations_total`	Total number of umounts
`csi_node_umount_duration_seconds`	Duration of umounts (in seconds)

NFS transport (xprt) status per cluster


`csi_node_nfs_xprt_total`	Total number of active NFS transports
`csi_node_nfs_xprt_connected`	Number of connected NFS transports
`csi_node_nfs_xprt_pending_requests_total`	Total number of pending requests across all mounts
`csi_node_nfs_xprt_backlog_total`	Total number of backlog requests across all mounts
`csi_node_nfs_xprt_unhealthy`	Number of unhealthy NFS transports

NFS transport (xprt) status per virtual IP

These metrics can only be exported while the virtual IP is connected.


`csi_node_nfs_xprt_connected_state`	`1.0` indicates a healthy connection. `0.0` means disconnected.
`csi_node_nfs_xprt_congested_state`	`1.0` means flow control is active.
`csi_node_nfs_xprt_locked_state`	`1.0` indicates that the connection is locked.
`csi_node_nfs_xprt_pending_requests`	Number of RPC calls waiting for a response
`csi_node_nfs_xprt_backlog_depth`	Backlog queue depth for this virtual IP
`csi_node_nfs_xprt_mounts`	Number of active NFS mounts for this virtual IP

CSI Controller Metrics


`csi_plugin_operations_total`	Total number of all CSI gRPC method calls (CreateVolume, DeleteVolume, ControllerPublishVolume, and so on)
`csi_plugin_operations_seconds`	Average duration of a CSI gRPC method call

Metrics Details

In addition to the measured value per metric type, a metric may include labels that provide additional information about the measured operation. For example, the following metric:

csi_node_mount_operations_total{operation_type="nfs",status="success",node_name="worker-node-1",pvc_namespace="prod"} 15

specifies that the measured value was taken for the mounts that:

were made through the NFS access protocol,
completed successfully,
occurred on a worker node named worker-node-1,
targeted the prod namespace.

Access Exported CSI Metrics

Tip: For more detailed guidance, see here.

Run the following commands to verify that the metrics endpoints work as expected:

For node metrics:

kubectl get pods -n vast-csi -l app.kubernetes.io/component=csi-node
kubectl port-forward -n vast-csi pod/<CSI node pod name> 9090:9090
curl -s http://localhost:9090/metrics
curl -s http://localhost:9090/health

For controller metrics:

kubectl get pods -n vast-csi -l app.kubernetes.io/component=csi-controller
kubectl port-forward -n vast-csi pod/<CSI controller pod name> 9091:9091
curl -s http://localhost:9091/metrics
curl -s http://localhost:9091/health

Sample Metrics Values for Common Scenarios

The following illustrates typical metrics values in common scenarios.

Tip: For more examples, see here.

Upon creating a pod:
- Aggregate metrics are set to 0.
- Counters are not reported.
- csi_node_nfs_xprt_connected_state, csi_node_nfs_xprt_congested_state and csi_node_nfs_xprt_pending_requests are not reported.

After mounting one PVC to virtual IP 192.168.1.10:

csi_node_mount_operations_total{node_name="worker-1",operation_type="nfs",pvc_namespace="default",status="success"} 1.0
csi_node_mount_duration_seconds_sum{...} 0.823
csi_node_mount_duration_seconds_count{...} 1.0
csi_node_nfs_xprt_total 1.0
csi_node_nfs_xprt_connected 1.0
csi_node_nfs_xprt_connected_state{destination="192.168.1.10"} 1.0

If the VAST cluster's virtual IP becomes unreachable:
- csi_node_mount_duration_seconds is at approx. 30 seconds (or your configured timeout)
- csi_node_nfs_xprt_total is 0.0.
In case of network congestion/high latency:
- csi_node_nfs_xprt_congested_state is 1.0.
- csi_node_nfs_xprt_pending_requests exceeds 100.
- csi_node_nfs_xprt_unhealthy is set to a non-zero value.
On pod deletion:
- csi_node_mount_operations_total stays at its last value.
- csi_node_umount_operations_total is reported with a non-zero value.
- After approx. 30 seconds, csi_node_nfs_xprt_total drops to 0.0.

Performance Tuning for VAST CSI Driver

To fine-tune VAST CSI Driver performance on large-scale Kubernetes clusters:

Adjust volume attachment
Disable sending of usage statistics
Adjust QPS and burst limits
Use API tokens for VMS authentication
Increase the number of workers
Use ephemeral volumes

Adjust Volume Attachment

Use the attachRequired parameter in the driver's Helm chart configuration file to fine-tune how the driver attaches volumes to workload nodes prior to mounting the volume into a pod.

attachRequired: true (default): The driver performs a separate ControllerPublishVolume RPC to attach the volume to the node.
attachRequired: false: The driver attaches the volume to the node as part of the NodePublishVolume operation, which can significantly speed up volume attachment. NOTE: This setup cannot be used with nodes where HTTP/HTTPS ports are not open.

Disable Sending of Usage Stats

Use the disableUsageStats parameter in the driver's Helm chart configuration file to prevent the driver from sending usage metrics to the VMS.

disableUsageStats: false (default): The driver sends usage metrics to the VMS.
disableUsageStats: true: The driver does not send usage metrics to the VMS.

Adjust QPS and Burst Limits

Adjust the kube-api-qps and kube-api-burst parameters of the CSI provisioner in the driver's Helm chart configuration file:

kube-api-qps sets the maximum allowed number of requests per second that the CSI provisioner can send to the Kubernetes API server. The default value is 5.
kube-api-burst sets the maximum allowed number of requests that can be sent at any given time after the kube-api-qps limit is exceeded. The default value is 10.

For example:

controller:
  extraArgs:
    csiProvisioner:
      - kube-api-qps=500
      - kube-api-burst=1000

Use API Tokens for VMS Authentication

Using Kubernetes secrets with API tokens for authentication on the VAST cluster may improve performance compared to using secrets with VMS credentials.

Increase Number of Workers

Adjust the numWorkers parameter in the driver's Helm chart configuration file to Increase the number of CSI workers.

Use Ephemeral Volumes

With ephemeral volumes, the driver's interaction with the VMS is not bound to the CSI controller but is distributed among the CSI nodes. Using ephemeral volumes allows for similar performance improvements as running with attachRequired: false.

Steps to Troubleshoot VAST CSI Driver

Run basic troubleshooting commmands
Check Kubernetes logs
Check VMS logs

Run Basic Troubleshooting Commands

Run the following commands to troubleshoot VAST CSI Driver:

Display all pods:

kubectl get pods --all-namespaces -o wide

Check logs from the controller.

Typically, there are four containers: csi-provisioner, csi-attacher, csi-resizer, and csi-vast-plugin. Check all of them.

Run the following command to display the most important information:
```
kubectl logs csi-vast-controller-0 --namespace vast-csi -c csi-vast-plugin | less
```
Check logs from the node.

There are probably two containers: csi-node-driver-registrar and csi-vast-plugin . Check both.

Run the following command to display the most important information:
```
kubectl logs csi-vast-node-<NODE_ID> --namespace vast-csi -c csi-vast-plugin
```
Check to see if there are any system-level events, such as errors.

Run the following command to list events sorted by timestamp:
```
kubectl get events --sort-by=.metadata.creationTimestamp
```

Display persistent volume claims:

kubectl get pvc

The output is similar to the following:

NAME         STATUS VOLUME                                   CAPACITY ACCESS MODES STORAGECLASS        AGE
shared-claim Bound  pvc-41e3cd02-1a80-4dfb-bb50-6d272a9e649e 1Mi      RWO          managed-nfs-storage 12d

Display persistent volumes:

kubectl get pv

The output is similar to the following:

NAME                                     CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM                STORAGECLASS        REASON AGE
pvc-41e3cd02-1a80-4dfb-bb50-6d272a9e649e 1Mi      RWO          Delete         Bound  default/shared-claim managed-nfs-storage        12d

Display the defined storage classes:

kubectl get sc

The output is similar to the following:

NAME                PROVISIONER AGE
managed-nfs-storage nfs-client  13d

Note: To get more detailed information, use the describe option on each command.

Check Kubernetes Logs

If none of the above helps, check the underlying Kubernetes logs.

The location of Kubernetes logs depends on your operating system. On CentOS, the typical location is /var/log/messages.

These logs often contain more detailed information about the container startup process and provide an indication of errors.

Check VMS Logs

VAST CSI Driver contacts VAST Management Service (VMS) to create directories and quotas on behalf of the Kubernetes cluster. So checking VMS logs can help troubleshoot your environment.

VMS logs can be found in /vast/vman/vms/log on the VMS node.

Useful information is often found in vms.log and vapi.log. For example, the vms.log can provide the following information (the mount for VAST CSI Driver is /keystest/k8s):

grep k8s vms.log
[2022-01-05 17:18:47,313] INFO [services.py:382/430] AUDIT - VMS - Manager: admin (10.61.203.63) POST /api/quotas/ with body {'create_dir': True, 'name': 'csi:nfs-vol-claim:nfs-vol-claim:pvc-0f5d878e-4fa4-4c91-a915-7a5c', 'path': '/keystest/k8s/pvc-0f5d878e-4fa4-4c91-a915-7a5c9fcdfc2c', 'hard_limit': 1048576}
[2022-01-05 17:18:47,562] INFO [services.py:837/430] Creating Quota with {'create_dir': True, 'name': 'csi:nfs-vol-claim:nfs-vol-claim:pvc-0f5d878e-4fa4-4c91-a915-7a5c', 'path': '/keystest/k8s/pvc-0f5d878e-4fa4-4c91-a915-7a5c9fcdfc2c', 'hard_limit': 1048576}
[2022-01-05 17:18:47,573] INFO [services.py:843/430] Creating directory path: /keystest/k8s/pvc-0f5d878e-4fa4-4c91-a915-7a5c9fcdfc2c
[2022-01-05 17:18:47,578] INFO [services.py:852/430] Created directory path: /keystest/k8s/pvc-0f5d878e-4fa4-4c91-a915-7a5c9fcdfc2c

Resolve PVC Timeouts

PVC Timeout due to Missing NFS Client on Kubernetes Worker Node

If a PVC does not mount with a timeout, a likely outcome is the Kubernetes worker node is missing the NFS client.

Run the kubectl logs csi-vast-node-<NODE_ID> --namespace vast-csi -c csi-vast-plugin command to check the node logs.

The output can be similar to the following:

Warning  FailedMount             50s        kubelet, k8s-a-node03    Unable to attach or mount volumes: unmounted volumes=[data], unattached volumes=[mariadb-credentials default-token-zpkp5 data config]: timed out waiting for the condition 
Warning  FailedMount             37s        kubelet, k8s-a-node03    MountVolume.SetUp failed for volume "pvc-4ebe1936-3613-4c37-957c-4d21e623aa98" : kubernetes.io/csi: mounter.SetupAt failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded

To resolve the problem, install the NFS client and test mounts locally on the client, then retry via Kubernetes.

To install the NFS client:

sudo yum -y install nfs-utils

PVC Timeout due to No Communication with VMS

Kubernetes nodes can be on the VAST data network and not on the VAST management network. In this case, the return path from VMS to the Kubernetes node will fail.

Ensure that you have valid routes on the correct network.

On the Kubernetes node, run the following command and verify that the correct interface is used:

ip route get <vms_ip>

Run this command on the VMS node:

ip route get <k8s_node_ip>

Documentation Index

Monitor and Troubleshoot VAST CSI Driver

Export VAST CSI Driver Metrics to Prometheus

Enable Export of CSI Metrics

Exposed CSI Metrics Endpoints and Ports

Exported CSI Metrics

CSI Node Metrics

CSI Controller Metrics

Metrics Details

Access Exported CSI Metrics

Sample Metrics Values for Common Scenarios

Performance Tuning for VAST CSI Driver

Adjust Volume Attachment

Disable Sending of Usage Stats

Adjust QPS and Burst Limits

Use API Tokens for VMS Authentication

Increase Number of Workers

Use Ephemeral Volumes

Steps to Troubleshoot VAST CSI Driver

Run Basic Troubleshooting Commands

Check Kubernetes Logs

Check VMS Logs

Resolve PVC Timeouts

PVC Timeout due to Missing NFS Client on Kubernetes Worker Node

PVC Timeout due to No Communication with VMS