Replicating Block Volumes with VAST CSI Replication Operator

Starting with version 2.6.6, VAST Block CSI Driver supports asynchronous replication of block volumes between VAST clusters, with an ability to perform a failover to the secondary replication peer. Data from one source cluster can be replicated to multiple destination clusters.

VAST CSI Replication Operator is a Kubernetes operator that simplifies configuring asynchronous replication of volumes provisioned on the VAST cluster by providing automated replication management and PVC labeling based on the replication CRDs.

Using the information provided in the replication CRDs, VAST CSI Replication Operator can perform the following operations:

Create a protection policy of the NATIVE_REPLICATION type on the source VAST cluster, for each pair of VAST clusters listed in the CRD.
Create a protected path on the source peer and associate it with a replication stream for each destination peer.
Create VolumeReplicationClass and VolumeGroupReplicationClass definitions required by the driver.
Start replication according to the replication schedule specified in the CRD.
On a failover:
- Create/delete PVC-PV pairs on each former destination cluster (so that the workloads can mount the replica).
- Create replica objects (volumes)
- Restart the pods that are using source PVCs and redirect them to the appropriate PVCs at the former destination cluster.

The following replication types are supported:

One-volume replication is configured on a single volume by means of the VastVolumeReplication CRD.
Multi-volume replication lets you replicate all volumes created with a particular storage class. This replication type is implemented using the VastStorageClassReplication CRD, with the ability to restrict replication to a certain group of volumes only.
Subsystem replication lets you replicate volumes of a particular subsystem.

To set up replication with VAST CSI Replication Operator:

Ensure that VAST cluster and Kubernetes requirements are met.
Complete preconfiguration steps on the source and destination VAST clusters.
Install VAST CSI Replication Operator.
(Optional) Refine the name format for destination PVCs and PVs.
Create the replication CRD.
(Optional) Specify a subpath to replicate.
(Optional) Create setup for subsystem replication.
Create PVCs to be replicated.
After the source PVC are created, the replication starts according to the schedule defined in the replication CRD.

VAST CSI Replication Operator comes with a CLI tool that lets you check the replication status, initiate a failover, run on-demand synchronization, or remove a replication configuration.

NOTE: A failover can also be initiated manually, by updating the replication CRDs.

VAST Cluster Requirements

VAST clusters running release 5.5.0 or later, with network connectivity through their replication ports.

Kubernetes Requirements

Kubernetes 1.26 or later
A Kubernetes secret containing VMS credentials for source and destination clusters.
NOTE: Secrets with VMS API tokens are not supported.

Preconfigurations on VAST Clusters

Ensure that the following items are preconfigured on the source VAST cluster:

An Element Store view to expose the subsystem with the source volumes
A replication virtual IP pool to be used for asynchronous replication of block volumes
An asynchronous replication peer for the destination cluster(s)

Ensure that the following items are preconfigured on the destination VAST cluster(s):

A replication virtual IP pool to be used for asynchronous replication of block volumes.
To implement single-volume or multi-volume replication: a view with the same subsystem name as the one created on the source cluster.
For subsystem-level replication, there is no need to manually create a view on the destination peer(s). The destination view will be created automatically based on the subsystem specification in the storage class.

NOTE: The preconfiguration steps are performed using VAST Web UI, VAST CLI or VAST REST API as described in the VAST Cluster Administrator's Guide.

Install VAST CSI Replication Operator

To install VAST CSI Replication Operator:

NOTE: The PVC and PV name formats may vary. For details, see Name Format for Destination PVCs and PVs.

Add the following to the driver's Helm chart configuration file to enable the extensions controller:

extensions:
  enabled: true
  replication:
    pvcNameFormat: "{pvc_name}-repl-{endpoint}"
    pvNameFormat: "{pv_name}-repl-{endpoint}"

image:
  vastExtensionController:
    repository: vastdataorg/csi
    tag: dev

If you want to use destination PVC/PV naming rules that are different from the default, adjust the pvcNameFormat and pvNameFormat as needed.
Deploy the driver with the updated Helm chart.

Name Format for Destination PVCs and PVs

You can specify the name format for the PVCs and PVs created at the destination cluster by adding the pvcNameFormat and pvNameFormat entries to the driver's Helm chart configuration file (see Install VAST CSI Replication Operator above).

By default, the following format is used:

{pvc_name}-repl-{endpoint}
{pv_name}-repl-{endpoint}

For example: myapp-data-repl-16-0-0-2

You can alter the pattern as needed using any of the following:

`{pvc_name}`	Full name of the source PVC.
`{pvc_name_suf:<N>}`	Last N characters of the source PVC name.
`{pvc_name_pref:<N>}`	First N characters of the source PVC name.
`{pv_name}`	Full name of the source PV.
`{pv_name_suf:<N>}`	Last N characters of the source PV name.
`{pv_name_pref:<N>}`	First N characters of the source PV name.
`{endpoint}`	VAST cluster endpoint IP where the dots are replaced with hyphens.
`{sc_name}`	The name of the destination storage class.
`{sc_name_suf:<N>}`	Last N characters of the destination storage class name.
`{sc_name_pref:<N>}`	First N characters of the destination storage class name.

Create Replication CRDs

Create either of the CRDs depending on the number of volumes to be replicated:
- VastStorageClassReplication (VSCR) to set up replication for all of the volumes created with a particular storage class.
- With this CRD, the operator waits at least one replication cycle before creating the PVC on each destination cluster specified in the CRD.
- VastVolumeReplication (VVR) to set up replication for a single volume.
- With this CRD, the operator creates one VolumeGroupReplication and one VolumeGroupReplicationClass per storage class specified in the CRD. The PVC on the destination cluster is created without any additional wait time.
After the YAML content is created, apply the CRDs with the kubectl apply command.

VastStorageClassReplication CRD

The VastStorageClassReplication CRD sets up replication for all volumes created with a particular storage class.

Below is an example of the CRD used to replicate all volumes created with storage class vastdata-filesystem1 to destinations controlled with storage classes vastdata-filesystem2 and vastdata-filesystem3. The replicas are created every 15 minutes, with the local snapshots kept for 2 days and the remote snapshots kept for 7 days.

NOTE: For detailed information on each of the parameters, see Replication CRD Spec Field Reference.

apiVersion: vastdata.com/v1alpha1
kind: VastStorageClassReplication
metadata:
  name: app-replication
  namespace: default
spec:
  primaryStorageClass: vastdata-filesystem1

  protectionTopology:
    - source: vastdata-filesystem1
      destination: vastdata-filesystem2
    - source: vastdata-filesystem1
      destination: vastdata-filesystem3
    - source: vastdata-filesystem2
      destination: vastdata-filesystem3

  protectionPolicyTemplate:
    params:
      - every: 15M
        keepLocal: 2D
        keepRemote: 7D
        
  failoverType: ungracefulFailover
  syncIntervalSeconds: 900
  syncPVCPV: true
  syncVastObjects: true
  pvcRemap: false

VastVolumeReplication CRD

The VastVolumeReplication CRD sets up replication for a single volume. It provides the same set of parameters as the VastStorageClassReplication CRD, and also the volumeName parameter used to identify the replicated volume.

NOTE: For detailed information on each of the parameters, see Replication CRD Spec Field Reference.

Below is an example of the CRD used to replicate all volumes created with storage class vastdata-filesystem1 to destinations controlled with storage classes vastdata-filesystem2. The replicas are created every hour, with the local snapshots kept for 3 weeks and the remote snapshots kept for 4 weeks.

apiVersion: vastdata.com/v1alpha1
kind: VastVolumeReplication
metadata:
  name: app-replication
  namespace: default
spec:
  volumeName: myapp-data
  primaryStorageClass: vastdata-filesystem1

  protectionTopology:
    - source: vastdata-filesystem1
      destination: vastdata-filesystem2

  protectionPolicyTemplate:
    params:
      - every: 1H
        keepLocal: 3W
        keepRemote: 4w
        
  failoverType: gracefulFailover
  syncIntervalSeconds: 900
  syncPVCPV: true
  pvcRemap: false
  destVolReclaimPolicy: Retain

Replication CRD Spec Field Reference

The spec section of the VastVolumeReplication and/or VastStorageClassReplication CRD can contain the following fields:

Field	Description
`primaryStorageClass: <storage class name>`	The name of the storage class that creates volumes to be replicated (initially on the source VAST cluster).
`protectionTopology: - source: <storage class name 1> destination: <storage class name 2> peerName: <peer name 1> - source: <storage class name 1> destination: <storage class name 3> peerName: <peer name 2> - ...`	Pairs of source and destination VAST clusters represented by the names of the respective storage classes. NOTE: The clusters involved must be configured as replication peers in the VMS. `source` refers to the VAST cluster where volumes to be replicated are hosted. Unless a failover occurs, the `source` is typically the same as the `primaryStorageClass`. `destination` refers to the VAST cluster to which the volumes are to be replicated. `peerName` specifies the name of the replication configuration (the one involving the `source` and `destination` in the same protection topology entry). This value corresponds to the Peer Name property of a replication peer in the VAST cluster's VMS. This parameter is optional. If not specified, the value is retrieved during auto-discovery. Note that to provide a complete topology definition to replicate from cluster A to clusters B and C, you need to specify three source-destination pairs to allow for proper synchronization of all involved clusters in case of a failover. For example: `protectionTopology: - source: A destination: B - source: A destination: C - source: B destination: C`
`protectionPolicyTemplate: params: - every: <time interval> keepLocal: <time interval> keepRemote: <time interval> startAt: <time> - every: <time interval> keepLocal: <time interval> keepRemote: <time interval> startAt: "<time>" - ...`	Sets one or more replication schedules. Required entries include: `every` defines the replication interval. `keepLocal` sets the period for keeping local snapshots. `keepRemote` sets the period for keeping remote snapshots. Specify time interval as a number followed by one-letter unit of measure: `S` or `s` for seconds, `m` for minutes (lowercase only), `H` or `h` for hours, `D` or `d` for days, `W` or `w` for weeks, `M` for months (uppercase only; one month is 30 days), `Y` or `y` for years (one year is 365 days). The optional `startAt` parameter determines when to start the replication according to the defined schedule. Specify time in YYYY-MM-DD HH:MM:SS or HH:MM format, for example: `startAt: "02:00"`.
`failoverType: <type>`	Determines the failover type. Valid values: `ungracefulFailover` `gracefulFailover` For an explanation of each of the failover types, see VAST Cluster Administrator's Guide.
`resync: true`	Initiates resynchronization between the source and the destination. Set to true to trigger immediate resynchronization. The operator resets it back to false automatically.
`syncIntervalSeconds: <number>`	The maximum allowed gap (in seconds) between sync points (snapshots that are shared between the replication peers). The default value is 900 seconds.
`syncPVCPV: {true\|false}`	Determines whether to create/delete static PV-PVC pairs on each former destination cluster after a failover (so that the workloads can mount the replica). The default value is `true`.
`pvcRemap: {true\|false}`	Controls whether to remap PVCs to the new primary on a failover. If set to `true`, after a failover, the pods that are using source PVCs are automatically restarted and redirected to the appropriate PVCs at the former destination cluster. The default value is `false`.
`destVolReclaimPolicy: {Retain\|Delete}`	Determines whether to retain (`Retain`, default) or delete (`Delete`) destination volumes when the `VastVolumeReplication` and/or `VastStorageClassReplication` is deleted.

(Optional) Specify a Group of Volumes to Replicate

If you are going to use VastStorageClassReplication for bulk replication of volumes, you can restrict replication to a certain group of volumes by specifying the volume group name in the storage class definition used for source volumes.

To specify a volume group, add the volume_group parameter to the storage class definition. For example, to replicate only volumes in mygroup:

...
parameters:
   <secret-related parameters>
   tenant_name: <name>
   subsystem: <name>
   view_policy: <name>
   vip_pool_name: <name>
   volume_group: mygroup
...

(Optional) Create Setup for Subsystem Replication

To implement subsystem-level replication:

Ensure that the subsystem to be replicated does not exist on the destination clusters(s).
Ensure that the storage classes to be replicated contain the same subsystem name.

This setup allows for automatic creation of the same subsystem on all destination clusters.

Create PVCs to Be Replicated

To create a single PVC:

kubectl apply -f - <<EOF
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-block-pvc
  namespace: production
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: vastdata-block
EOF

To create a group of PVCs:

kubectl apply -f - <<EOF
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: postgres-data
  namespace: production
  labels:
    app: postgres
    tier: database
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 500Gi
  storageClassName: vastdata-block
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: postgres-wal
  namespace: production
  labels:
    app: postgres
    tier: database
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 100Gi
  storageClassName: vastdata-block
EOF

Verify Replication Status with CSI CLI

To display replication status for all replication configurations:

vcsi list [-n <namespace>]

The output is similar to the following:

KIND  NAMESPACE  NAME             STORAGE CLASSES                                  PRIMARY               FAILOVER TYPE       SYNC STATUS  AGE
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
VSCR  default    app-replication  ["vastdata-filesystem", "vastdata-filesystem2"]  vastdata-filesystem   ungracefulFailover  Completed    26m
VVR   default    myapp-data-repl  ["vastdata-filesystem", "vastdata-filesystem2"]  vastdata-filesystem   ungracefulFailover  InProgress   3m

To display details for a particular replication configuration:

vcsi status {--vscr|vvr} <CRD name> [-n <namespace>]