VAST Block CSI Driver Workflow
To let your applications use block storage on a VAST cluster, define a Kubernetes storage class for block storage and reference it in the PVC. Based on the storage class definition, VAST Block CSI Driver will provision a block volume for the PVC using a preconfigured view on the VAST cluster.
Ensure that the following prerequisites are met:
Add a storage class that will be used to provision block volumes. In the storage class definition:
-
Specify the
block.csi.vastdata.comprovisioner. -
Add the
subsystemparameter to specify the name of the NVMe subsystem where block volumes will be created. This NVMe subsystem must be associated with a VAST cluster view preconfigured for block storage. If the VAST cluster exposes multiple subsystems that share the same name across VAST tenants, specify thetenantNameoption to for the driver to pick the subsystem associated with the tenant you need. -
Optionally, define volume naming using the
volumeGroupparameter. -
Ensure that the virtual IP pool belongs to the same tenant as the VAST cluster's view configured for block storage provisioning.
Tip: For more detailed guidelines, see Create Block Storage Classes. For more examples, see here.
For example:
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: vastdata-filesystem
parameters:
csi.storage.k8s.io/controller-expand-secret-name: vast-mgmt
csi.storage.k8s.io/controller-expand-secret-namespace: default
csi.storage.k8s.io/controller-publish-secret-name: vast-mgmt
csi.storage.k8s.io/controller-publish-secret-namespace: default
csi.storage.k8s.io/node-publish-secret-name: vast-mgmt
csi.storage.k8s.io/node-publish-secret-namespace: default
csi.storage.k8s.io/node-stage-secret-name: vast-mgmt
csi.storage.k8s.io/node-stage-secret-namespace: default
csi.storage.k8s.io/provisioner-secret-name: vast-mgmt
csi.storage.k8s.io/provisioner-secret-namespace: default
fsType: xfs
subsystem: BlockView
vip_pool_fqdn: 'MyDomain'
volume_group: folder1/folder2/block-{namespace}-{id}
provisioner: block.csi.vastdata.com
reclaimPolicy: Delete
volumeBindingMode: Immediate
allowVolumeExpansion: true
Create a Persistent Volume Claim (PVC) as follows:
-
Specify the
volumeModeasBlockfor raw block storage orFileSystem(default) for formatted volumes. For more information on volume modes, see Volume Modes. -
Reference the newly created block storage class in
storageClassName.
For example:
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: pvc-raw-block
spec:
volumeMode: Block
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
storageClassName: vast-block-sc
Reference the newly created PVC on the claimName parameter in the pod definition.
For example:
apiVersion: v1
kind: Pod
metadata:
name: test-pod
spec:
containers:
- name: app-container
image: nginx
volumeMounts:
- name: data
mountPath: /mnt/data
volumes:
- name: data
persistentVolumeClaim:
claimName: pvc-raw-block
VAST Cluster Requirements and Preconfigurations
VAST Cluster 5.3 or later is required to provision block volumes.
The VAST cluster must be configured for block storage provisioning as follows:
-
Block storage support is enabled for the tenant where block volumes will be created.
-
Under the tenant, there exists a view that exposes the NVMe subsystem where block volumes will be created. The view must have the block storage protocol enabled and the NVMe subsystem defined.
Note: Note that the block storage view is not automatically created by VAST Block CSI Driver. It must be created manually beforehand.
Kubernetes Requirements
-
A host connecting to block storage must have an NVMe driver installed and configured as needed. This is done outside of VAST Block CSI Driver.
-
The
nvme-clipackage must be installed on Kubernetes nodes where volumes are to be created.
Volume Modes
VAST Block CSI Driver supports the following volume modes, which can be specified using the volumeMode option in the PVC definition:
-
Block: A volume is presented to the pod as a raw block device without any filesystem on it.Ephemeral volumes cannot be provisioned in this mode.
-
Filesystem(default): A volume is formatted using the filesystem type specified in the storage class definition. The following filesystem types are supported:ext4(default),ext3andxfs.In this mode, the volume can be attached to multiple pods simultaneously, but all pods must be scheduled on the same node.
Create Block Storage Classes
VAST Block CSI Driver supports multiple Kubernetes storage classes. During initial deployment of VAST Block CSI Driver, you define one or more storage classes in the driver's Helm chart configuration file. Later you can add more storage classes by creating and applying a Kubernetes YAML configuration file.
Add Storage Class
To add a storage class using a Kubernetes YAML configuration file:
-
Create a YAML configuration file that defines a new storage class with the following required parameters:
Tip: For a complete list of storage class options, and also for detailed information about required and optional parameters, see Storage Class Option Reference.
apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: <storage class name> provisioner: block.csi.vastdata.com parameters: subsystem: <NVMe subsystem name> vip_pool_fqdn: <virtual IP pool FQDN> | vip_pool_name: <virtual IP pool name> <optional: pairs of secrets and secret namespaces for each volume processing stage>For example:
apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: block-on-vast provisioner: block.csi.vastdata.com parameters: subsystem: BlockView vip_pool_fqdn: 'MyDomain' csi.storage.k8s.io/controller-expand-secret-name: vast-mgmt csi.storage.k8s.io/controller-expand-secret-namespace: default csi.storage.k8s.io/controller-publish-secret-name: vast-mgmt csi.storage.k8s.io/controller-publish-secret-namespace: default csi.storage.k8s.io/node-publish-secret-name: vast-mgmt csi.storage.k8s.io/node-publish-secret-namespace: default csi.storage.k8s.io/node-stage-secret-name: vast-mgmt csi.storage.k8s.io/node-stage-secret-namespace: default csi.storage.k8s.io/provisioner-secret-name: vast-mgmt csi.storage.k8s.io/provisioner-secret-namespace: default -
Deploy the YAML configuration file:
kubectl apply -f <filename>.yaml -
Verify that the storage class has been added:
kubectl get storageclassesThe output is similar to the following:
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE standard (default) k8s.io/minikube-hostpath Delete Immediate false 4d1h block-on-vast block.csi.vastdata.com Delete Immediate true 58m
Storage Class Option Reference
You can specify storage class options as follows:
-
In the Helm chart configuration file created for VAST Block CSI Driver during initial deployment,
-
In a Kubernetes YAML configuration file deployed at a later stage.
-
In VAST CSI Operator's
VastStoragecustom resource definition, when deploying in an OpenShift environment using VAST CSI Operator.
allowVolumeExpansion
(Optional) Determines whether volume expansion is allowed (default, true) or not (false).
Syntax for the Helm chart configuration file and VAST CSI Operator's VastStorage:
allowVolumeExpansion: true|false
Syntax for the Kubernetes YAML configuration file:
allow_volume_expansion: true|false
blockingClones
(Optional) Determines whether VAST Block CSI Driver waits for VAST snapshot or clone operations to complete before allowing Kubernetes to proceed with volume provisioning:
-
If true, the driver waits for all snapshot and clone operations to complete before the PVC can be attached to a pod. This ensures that the volume is fully usable and consistent at the time of provisioning, but may add a significant latency.
-
If false (default), the driver proceeds with provisioning right after the snapshot or clone operation is initiated.
Syntax for the Helm chart configuration file and VAST CSI Operator's VastStorage:
blockingClones: true|false
Syntax for the Kubernetes YAML configuration file:
blocking_clones: true|false
ephemeralVolumeNameFormat
(Optional) A format string that controls naming of Kubernetes ephemeral volumes created through VAST Block CSI Driver.
If not specified, the default format csi:{namespace}:{name}:{id} is used, where:
-
namespaceis the Kubernetes namespace where volume parent workload is located. -
nameis the volume name. -
idis the volume ID set by Kubernetes.
Syntax for the Helm chart configuration file and VAST CSI Operator's VastStorage:
ephemeralVolumeNameFormat: "<format>"
Syntax for the Kubernetes YAML configuration file:
eph_volume_name_fmt: "<format>"
fsType
(Optional) The filesystem type to format the volume. The default is ext4.
This option is applicable only when the PVC has volumeMode set to FileSystem. For more information about volume modes, see Volume Modes.
Syntax for the Helm chart configuration file and VAST CSI Operator's VastStorage:
fsType: "ext4" | "ext3" |"xfs"
Syntax for the Kubernetes YAML configuration file:
fs_type: "ext4" | "ext3" |"xfs"
hostEncryption
(Optional) Sets parameters for LUKS-based host encryption:
-
luksType: "<type>"sets the LUKS version. Valid values:luks2(default) orluks1, -
cipher: "<cipher>"specifies the cipher to be used for host encryption. Valid values:-
aes-xts-plain64(default) -
aes-cbc-essiv:sha256 -
serpent-xts-plain64 -
twofish-cbc-essiv:sha256
-
-
keySize: "<bits>"sets the length of the key for the selected cipher, in bits. Default is 512. -
hash: "<algorithm>"specifies the hashing algorithm. Valid values:-
sha1 -
sha256(default) -
sha512 -
ripemd160 -
whirlpool
-
-
pdkdfMemory: "<kilobytes>"is the memory cost for PBKDF. Default is 65536. -
Performance parameters are all
trueby default:-
perf-same_cpu_crypt: {true|false}- Use same CPU for encryption work. -
perf-submit_from_crypt_cpus: {true|false}- Submit IO from crypt CPUs. -
perf-no_read_workqueue: {true|false}- Bypass read workqueue. -
perf-no_write_workqueue: {true|false}- Bypass write workqueue.
Note: Performance parameters are available starting with VAST Block CSI Driver 2.6.4.
-
Syntax for the Helm chart configuration file and VAST CSI Operator's VastStorage:
hostEncryption:
luksType: "<type>"
cipher: "<cipher>"
keySize: "<bits>"
hash: "<algorithm>"
pbkdfMemory: "<kilobytes>"
perf-same_cpu_crypt: {true|false}
perf-submit_from_crypt_cpus: {true|false}
perf-no_read_workqueue: {true|false}
perf-no_write_workqueue: {true|false}
Syntax for the Kubernetes YAML configuration file:
host_encryption: '{"luks_type":"<type>","cipher":"<cipher>","key_size": "<bits>","hash":"<algorithm>","pdkdf_memory":"<kilobytes>", perf_same_cpu_crypt: {true|false}, perf_submit_from_crypt_cpus: {true|false}, perf_no_read_workqueue: {true|false}, perf_no_write_workqueue: {true|false}}'
qosPolicy
The name of a Quality of Service (QoS) policy to be associated with automatically created views. The QoS policy can be passed either by its name or by its ID (in qos_policy_id).
A QoS policy sets performance limits per view. For more information, see Configure QoS Policy.
Syntax for the Helm chart configuration file and VAST CSI Operator's VastStorage:
qosPolicy: "<QoS policy name>"
Syntax for the Kubernetes YAML configuration file:
qos_policy: "<QoS policy name>"
qosPolicyId
The ID of a Quality of Service (QoS) policy to be associated with automatically created views. The QoS policy can be passed either by its ID or by its name (in qos_policy_name).
A QoS policy sets performance limits per view. For more information, see Configure QoS Policy.
Syntax for the Helm chart configuration file and VAST CSI Operator's VastStorage:
qosPolicyId: "<QoS policy ID>"
Syntax for the Kubernetes YAML configuration file:
qos_policy_id: "<QoS policy ID>"
reclaimPolicy
(Optional) Determines whether to delete ( Delete , default) or not (Retain) the dynamically provisioned volumes in case their PVCs get deleted.
Syntax for the Helm chart configuration file and VAST CSI Operator's VastStorage:
reclaimPolicy: "Delete"|"Retain"
Syntax for the Kubernetes YAML configuration file:
reclaim_policy: "Delete"|"Retain"
secretName and secretNamespace
These options lets you supply information for communicating with the VAST cluster:
-
<secret name>is the name of the Kubernetes secret that contains information about the VAST cluster on which to provision volumes for this particular storage class, the corresponding VMS user credentials or authentication token and, optionally, the SSL certificate. For more information, see Provisioning Block Volumes on Multiple VAST Clusters. -
<secret namespace>: if the storage class Kubernetes secret was created in a namespace that is different from that used to install the driver's Helm chart, add this parameter to specify the namespace of the Kubernetes secret.
Syntax for the Helm chart configuration file and VAST CSI Operator's VastStorage:
secretName: "<secret name>"
secretNamespace: "<secret's namespace>"
Syntax for the Kubernetes YAML configuration file:
<pairs of secrets and secret namespaces for each provisioning stage>
If the secret and its namespace are defined as global options in the Helm chart configuration file, they are automatically propagated to each storage class and each provisioning stage therein. In this case, you do not need to explicitly include the per-stage secrets with their corresponding namespaces in the storage class definition.
If no global settings exist for the secret and its namespace, you need to specify them directly in the storage class definition in the following format. Note that you can specify a different value for each stage:
csi.storage.k8s.io/controller-expand-secret-name: <secret name>
csi.storage.k8s.io/controller-expand-secret-namespace: <secret namespace>
csi.storage.k8s.io/controller-publish-secret-name: <secret name>
csi.storage.k8s.io/controller-publish-secret-namespace: <secret namespace>
csi.storage.k8s.io/node-publish-secret-name: <secret name>
csi.storage.k8s.io/node-publish-secret-namespace: <secret namespace>
csi.storage.k8s.io/node-stage-secret-name: <secret name>
csi.storage.k8s.io/node-stage-secret-namespace: <secret namespace>
csi.storage.k8s.io/provisioner-secret-name: <secret name>
csi.storage.k8s.io/provisioner-secret-namespace: <secret namespace>
setDefaultStorageClass
(Optional) If set to true, VAST Block CSI Driver uses the storage class as the default storage class for all volumes provisioned on the cluster. The default value is false.
Syntax for the Helm chart configuration file and VAST CSI Operator's VastStorage:
setDefaultStorageClass: true|false
Syntax for the Kubernetes YAML configuration file:
set_default_storage_class: true|false
subsystem
(Required) The name of the NVMe subsystem where block volumes will be created. This is the subsystem exposed through the VAST cluster view preconfigured for block storage.
Syntax for the Helm chart configuration file and VAST CSI Operator's VastStorage:
subsystem: "<subsystem name>"
Syntax for the Kubernetes YAML configuration file:
subsystem: "<subsystem name>"
tenantName
(Optional) Specify the name of the VAST tenant which is associated with the subsystem. This option is used to determine the correct tenant in case multiple subsystems on the VAST cluster share the same name across tenants.
Syntax for the Helm chart configuration file and VAST CSI Operator's VastStorage:
tenantName: "<tenant name>"
Syntax for the Kubernetes YAML configuration file:
tenant_name: "<tenant name>"
vipPool
The name of the virtual IP pool to be used by VAST Block CSI Driver.
The virtual IP pool that you specify for a storage class must belong to the same VAST Cluster tenant as the VAST Cluster view specified on the subsystem parameter of the storage class.
Either vipPool or vipPoolFQDN option is required. These options are mutually exclusive. When vipPool is used, VAST Block CSI Driver makes an additional call to the VMS to obtain the IP, which may impact performance when mounting volumes. For more information, see Configure VAST Cluster for VAST Block CSI Driver and Set Up DNS-Based Virtual IP Resolution for Block Volumes.
Syntax for the Helm chart configuration file and VAST CSI Operator's VastStorage:
vipPool: "<virtual IP pool name>"
Syntax for the Kubernetes YAML configuration file:
vip_pool_name: "<virtual IP pool name>"
vipPoolFQDN
The domain name of the virtual IP pool to be used by VAST Block CSI Driver.
The virtual IP pool that you specify for a storage class must belong to the same VAST Cluster tenant as the VAST Cluster view specified on the subsystem parameter of the storage class.
Either vipPoolFQDN or vipPool option is required. These options are mutually exclusive. With vipPoolFQDN, the IP for volume mounting is obtained through DNS, which improves mounting times. For more information, see Configure VAST Cluster for VAST Block CSI Driver and Set Up DNS-Based Virtual IP Resolution for Block Volumes.
Syntax for the Helm chart configuration file and VAST CSI Operator's VastStorage:
vipPoolFQDN: "virtual IP pool's domain name"
Syntax for the Kubernetes YAML configuration file:
vip_pool_fqdn: "virtual IP pool's domain name"
volumeNameFormat
(Optional) A format string that controls naming of volumes created through VAST Block CSI Driver. If not specified, the default format csi:{namespace}:{name}:{id} is used, where:
-
namespaceis the Kubernetes namespace where volume parent workload is located. -
nameis the volume name. -
idis the volume ID set by Kubernetes.
This parameter can be used to provide a nested directory structure for the volumes, for example: /folder1/folder2/block-{namespace}-{id}
Syntax for the Helm chart configuration file and VAST CSI Operator's VastStorage:
volumeNameFormat: "<format>"
Syntax for the Kubernetes YAML configuration file:
volume_name_fmt: "<format>"
Provision Ephemeral Block Volumes
VAST Block CSI Driver supports Ephemeral Volumes (EVs). You can provision PVCs directly from pod definitions, which are discarded once the pod is terminated.
To provision ephemeral volumes:
-
In the PVC, ensure that
volumeModeis set toFilesystem. -
In the pod definition:
-
Specify the
volumeAttributesas follows:-
subsystem: as in the storage class definition. -
vip_pool_fqdnorvip_pool_name: as in the storage class definition. -
size: as in the PVC.
-
-
If you have not specified a global secret in the VAST Block CSI Driver's Helm chart configuration file, specify the secret to use for the ephemeral volumes on the
nodePublishSecretRefoption. -
Specify the filesystem type on the
fsTypeparameter.
-
For example:
apiVersion: v1
kind: Pod
metadata:
name: some-pod
spec:
containers:
- command:
- sh
- -c
- while true; do date -Iseconds >> /shared/$HOSTNAME; sleep 1; done
image: busybox
name: my-frontend
volumeMounts:
- mountPath: /shared
name: my-eph-vol
volumes:
- csi:
driver: block.csi.vastdata.com
volumeAttributes:
size: 5G
subsystem: myblock
vip_pool_fqdn: 'MyDomain'
nodePublishSecretRef:
name: vast-mgmt
fsType: xfs
name: my-eph-vol
Provision Static Block Volumes
VAST Block CSI Driver can expose existing data as statically provisioned persistent volumes.
Below is an example of a YAML definition for a statically provisioned persistent volume:
apiVersion: v1
kind: PersistentVolume
metadata:
name: csi-pv-static
spec:
storageClassName: vastdata-filesystem
capacity:
storage: 1Gi
accessModes:
- ReadWriteOnce
csi:
driver: block.csi.vastdata.com
volumeAttributes:
vip_pool_name: vippool-1
subsystem: myblock
controllerPublishSecretRef:
name: vast-mgmt
namespace: default
volumeHandle: /static/volume
The PVC for a static volume is similar to the following:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: csi-pvc-static
spec:
storageClassName: vastdata-filesystem
volumeName: csi-pv-static
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
Provision Block Volumes on Multiple VAST Clusters
VAST Block CSI Driver enables you to provision volumes on multiple VAST clusters.
The VAST cluster to use can be specified per storage class, together with the user credentials needed to connect to the VAST cluster. You can also configure multiple storage classes to connect to the same VAST cluster using different user credentials.
Note: Class-specific parameters take precedence over global parameters that apply to all classes.
To configure a storage class to use a particular combination of a VAST cluster and user credentials:
-
Create a Kubernetes secret that will be used to provision storage on multiple VAST clusters. The secret specifies the following:
-
The hostname of the VAST cluster on which you want to provision volumes
-
The VMS user's username and password
-
If using SSL encryption with a self-signed SSL certificate:
- The SSL certificate to be used to protect the connection
-
If you are creating the secret in a namespace that is different from the namespace used to install the driver's Helm chart:
- The namespace where the secret is created
Note: When provisioning storage on multiple VAST clusters for Kubernetes ephemeral volumes, the namespace where the Kubernetes secret is created must be the same as the namespace where the driver's Helm chart is installed.
-
-
Create a new storage class and supply the newly created secret using the
secretNameparameter of the class.If you have creating the secret in a namespace that is different from the namespace used to install the driver's Helm chart, specify the secret's namespace on the
secretNamespaceparameter.
If a storage class does not contain the secretName parameter, VAST Block CSI Driver uses the information from the secretName and endpoint parameters specified in the driver's Helm chart configuration file (values.yaml) for all storage classes.
Set Up DNS-Based Virtual IP Resolution for Block Volumes
When mounting block volumes on the VAST cluster, VAST Block CSI Driver assigns a virtual IP to each volume being mounted. The virtual IP is taken from one or more virtual IP pools defined on the VAST cluster.
You can control how VAST Block CSI Driver obtains the IP to use, per storage class:
-
By using DNS to resolve a virtual IP pool's FQDN into an IP. This method improves performance when mounting volumes.
Specify the FQDN of the virtual IP pool on the
vipPoolFQDNoption in the driver's configuration file.Ensure that the VAST cluster has DNS configured, and the virtual IP pool has Virtual IP Pool Domain Name defined in its settings.
-
By making an additional call to VMS to retrieve an IP. This is a legacy method.
Specify the name of the virtual IP pool on the
vipPoolparameter in the driver's configuration file.
Either vipPool or vipPoolFQDN option is required when defining a storage class. These options are mutually exclusive.
Support of VAST Cluster Multi-Tenancy for Block Storage
VAST Block CSI Driver can operate on multiple VAST Cluster tenants.
To determine from which tenant to provision block storage for a claim, VAST Cluster looks at the NVMe subsystem exposed by the view that is preconfigured for block storage.
Manage Block Volume Access Modes
A block volume can be accessible in read/write or read-only mode depending on the following controls:
-
The
accessModesparameter specified in the PVC definition. The parameter lists allowed PVC access modes for any pods mounting the volume. VAST Block CSI Driver supports the following modes:-
For raw block volumes (
volumeMode: block): ReadOnlyMany (ROX), ReadWriteOnce (RWO), ReadWriteOncePod (RWOP), ReadWriteMany (RWX)Note that the ReadWriteMany (RWX) mode for raw block devices can only be implemented with cluster-aware file systems.
-
For filesystem volumes (
volumeMode: filesystem): ReadOnlyMany (ROX), ReadWriteOnce (RWO), ReadWriteOncePod (RWOP)
Note: See here for a detailed explanation of these access modes.
-
-
The
readOnlyflag of a pod (set to true or false). This flag lets you restrict access by a particular pod to read-only, which may override the access mode specified by the PVC'saccessModesparameter.The
readOnlyflag can only be applied for filesystem volumes (volumeMode: filesystem). If set for a raw block volume (volumeMode: block), the driver ignores the flag.
To allow concurrent read access by multiple pods to a raw block volume:
-
Specify in the PVC:
spec: accessModes: - ReadOnlyMany volumeMode: Block -
Ensure that none of the pods has the
readOnlyflag set.
To allow read/write access to a raw or filesystem block volume by any one pod on the same node at a time:
-
Specify in the PVC:
spec: accessModes: - ReadWriteOnce volumeMode: {Block|Filesystem} -
Ensure that none of the pods has the
readOnlyflag set.
To allow concurrent read/write access by multiple pods to the same raw block volume:
-
Specify in the PVC:
spec: accessModes: - ReadWriteMany volumeMode: Block -
Ensure that none of the pods has the
readOnlyflag set.
To allow concurrent read/write access to a filesystem volume by multiple pods except one:
-
Specify in the PVC:
spec: accessModes: - ReadWriteOnce volumeMode: Filesystem -
Specify in the pod that is not allowed to write:
spec: containers: volumeMounts: - readOnly: true
To allow read/write access to a raw or filesystem block volume by exactly one pod at a time (cluster-wide):
-
Specify in the PVC:
spec: accessModes: - ReadWriteOncePod volumeMode: {Block|Filesystem}