Configuring Secure Replication Certificates

VAST's Secure replication setting uses "Mutual TLS", often referred to as "mTLS".  mTLS allows both sides of the replication connection to authenticate each other whilst encrypting all traffic between the two clusters.

Unlike normal TLS, where there is generally a need for certificates to be signed by an external Certificate Authority (CA) to confer trust, with point-to-point mTLS this is not required.  The CA Certificate used to sign the replication certificate should be a unique certificate that is ONLY used to sign the replication certificates.

Using an external (or corporate) CA Certificate breaks the security model used for mTLS authentication - any certificate signed by the same CA will be authorized for replication, which is not the desired outcome.

The steps below walk through creating a CA Certificate, creating and signing the Cluster-level TLS Certificates, and then installing these certificates on the VAST Clusters.

Creating a Root Certificate Authority Key/Certificate

The following commands will create a new password-protected CA Key and CA Certificate (also known as a Root Certificate).  You will be prompted for a password, which will be required whenever using this CA Key, which would normally only be when creating the cluster certificates in the following step.

openssl genrsa -aes256 -out ca.key 4096
openssl req -new -x509 -days 3650 -key ca.key -out ca.crt -subj '/CN=Replication-Root-CA'

These commands will result in 2 files being created - ca.key, which contains the (password-protected) CA Key, and ca.crt, which contains the CA Certificate.  You can optionally view the details of the CA Certificate (such as expiry details/etc) using the command :

openssl x509 -in ca.crt -noout -text

Creating the VAST Cluster Keys/Certificates

The following commands will create a new Key and Certificate pair for a VAST cluster, and sign the certificate using the CA Key created above. You will be prompted for the CA Key password you set above during the final (signing) step of this process.

Whilst it is possible to use the same Certificate/Key across multiple VAST clusters, it's normally recommended to create one per cluster. Set the CLUSTER variable to the relevant values and re-run the commands to generate each Key/Certificate pair.  Note that the cluster name does NOT need to match the actual name of the cluster - with mTLS verification is done based on the certificates, not the hostname/cluster name as with normal TLS.

CLUSTER=myclustername
openssl genrsa -out ${CLUSTER}.key 2048
openssl req -new -key ${CLUSTER}.key -out ${CLUSTER}.csr -nodes -subj "/CN=Replication-$CLUSTER"
openssl x509 -req -CA ca.crt -CAkey ca.key -CAcreateserial -in ${CLUSTER}.csr -out ${CLUSTER}.crt -days 1095

These commands will result in 3 files being created for each cluster - myclustername.key which contains the cluster Key, clustername.crt which contains the clusters Certificate, and a third file myclustername.csr which is only used during the creation process and can be deleted.  You can optionally view the details of the clusters Certificate (such as expiry details/etc) using the command :

openssl x509 -in myclustername.crt -noout -text

In addition to the certificate details similar to what we saw in the CA certificate, the cluster certificates should also show that they were 'Issued' by the CA certificate.

   Issuer: CN=Replication-Root-CA

Installing the Certificates

It is generally easiest to install the certificates using the VAST GUI, although the CLI or REST API can also be used.

In the GUI, browse to Settings -> Certificates, and then in the "Certificates for" dropdown box select "Replication".  Three input boxes will be shown, one for each of the cluster certificate, the cluster key, and the CA/Root Certificate.  Paste or Upload the contents of each of the certificates into the relevant boxes, making sure to use the correct cluster certificate/key for the cluster you're installing onto (the CA/Root Certificate is the same for all clusters).  If using cut-n-paste, make sure to include everything in the file, including the --- BEGIN/END CERTIFICATE ---  lines.

When all three boxes have been entered, click on Upload.  If you receive an error, then make sure that you have entered the correct details in each box, and that you are using matching certificate/key pairs, and try again.

Repeat the above process for each cluster, using the relevant cluster certificate/key for that cluster.

Testing the Certificates

At this point, you can simply connect the two clusters for Replication (Data Protection -> Replication Peers -> Create Peer and set the Secure mode to “Secure”), if you want to make sure that the correct certificates are being used, you can use the commands below.

These can be run from any host with access to the Replication VIP pool on the cluster, including from the CNodes themselves.

The following command will show the certificate being used by the cluster. Replace the IP address in the command with an IP address from a Replication VIP Pool on the cluster, but leave the port as 49002, as this is the TCP port used for secure replication traffic.

$ echo | openssl s_client -connect 172.31.1.1:49002
CONNECTED(00000003)
depth=1 CN = Replication-Root-CA
verify error:num=19:self signed certificate in certificate chain
139997221902224:error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:s3_pkt.c:1493:SSL alert number 40
139997221902224:error:140790E5:SSL routines:ssl23_write:ssl handshake failure:s23_lib.c:177:

Certificate chain
 0 s:/CN=Replication-vast1
   i:/CN=Replication-Root-CA
 1 s:/CN=Replication-Root-CA
   i:/CN=Replication-Root-CA
[...]

The command itself will return a “handshake” error, as shown, because we didn't provide the required certificate to authenticate the mTLS connection.  However, we can see from the output that the VIP we connected to is using the correct certificate ('Replication-vast1') and that this certificate was signed by our CA Certificate ('Replication-Root-CA').

If we want to take testing a step further, we can use OpenSSL to verify the mTLS authentication process as well by providing it with the cluster certificate and keys, and the CA/Root Certificate.  This time, there should be no errors displayed as the handshake and authentication should be fully successful.

$ echo | openssl s_client -connect 172.31.2.1:49002 -key vast1.key -cert vast1.crt -CAfile ca.crt

Using External CA Signed Certificates

Using certificates signed by an external Certificate Authority (CA) is not supported for VAST Secure Replication.

The mTLS security model used by Secure Replication uses the fact that a presented client certificate is signed by the CA certificate configured on the cluster as being an indication that the client is trusted.  If an external CA certificate is used, then ANY TLS certificate also signed by that CA will be considered trusted and accepted for replication, which is not the designed outcome.

It is possible to use corporate CA infrastructure to generate the required certificates, as long as it can be guaranteed that the CA Key/Certificate used will never be used to sign any certificates other than those that will be used for VAST Secure Replication.

If using a tool other than OpenSSL to generate the certificates, additional configuration will likely be needed when creating the certificates to allow mTLS to function.

Corporate CA-signed certificates often include "X509v3 extensions" which limit the purposes a certificate can be used for.  As an example, certificates created for a website will normally have a "Cert Type" of "SSL Server" and/or a "X509v3 Extended Key Usage" of "TLS Web Server Authentication", meaning that these certificates can be used as a server certificate, but not as a client certificate.  (See RFC 3280 for more details on X509v3 extensions)

In the case of VAST Secure Replication/mTLS, each Replication Peer has a "client" end (the cluster on which the Replication Peer was configured) and a "server" end (the cluster whose details were entered to connect to), and thus the certificates created for each system will need to include (at least) those roles.

However, if 3 or more VAST Clusters are configured in a mesh configuration, then at least some of the clusters will be acting as both Clients AND Servers, and thus for these systems the certificates will need to include both roles. To simplify deployment, it's recommended that ALL certificates used have both the client and server roles enabled, even when only using 2 clusters.

The "X509v3 extensions" section of the certificate is added during certificate signing, so these will need to be set by whoever controls the process of signing the certificates.  The minimal values that are required are :

• "Certificate Type" (Sometimes called "Netscape Cert Type") should include both "SSL Client" and "SSL Server."

• "X509v3 Key Usage" should include "Digital Signature."

• "X509v3 Extended Key" Usage should include "TLS Web Server Authentication” and “TLS Web Client Authentication."

Alternatively, these fields can be excluded from the certificate entirely, indicating that all uses are allowed.

The x509 extensions contained within a signed certificate can be viewed using OpenSSL as follows

openssl x509 -in myclustername.pem -noout -text

In the middle of the output, there should be a section "X509v3 extensions" which includes some or all of the headings mentioned above. For example:

    X509v3 extensions:
        X509v3 Basic Constraints:
            CA:FALSE
        Netscape Cert Type:
            SSL Client, SSL Server
        X509v3 Subject Key Identifier:
            8A:E0:EF:4B:7A:8E:33:65:6E:78:AE:90:6F:A8:77:18:E5:B6:F8:1A

        X509v3 Key Usage: critical
            Digital Signature
        X509v3 Extended Key Usage:
            TLS Web Server Authentication, TLS Web Client Authentication

In this case, both the “Netscape Cert Type” and “X509v3 Extended Key Usage” fields are set to allow use as both a server and a client.

Intermediate Certificates

In general, there should never be a need to use Intermediate certificates with VAST Secure Replication.  The use of intermediate certificates generally implies that the CA Certificate is being used for multiple purposes, and as stated above, for Secure Replication, we require the CA Certificate to never be used for any purposes other than the signing of Secure Replication certificates.