Configuring a Load Balancer/Proxy Health Check URL

When configuring a load balancer or reverse web proxy to front-end a VAST cluster for S3 traffic, it may be desirable to configure a health check URL that the load balancer/proxy can use to determine if each of the backend servers is functioning correctly.

In general, health checks are normally not required for VAST as the cluster will reassign the IP Addresses allocated to a VIP pool between CNodes in the event of a failure/outage, so all IP addresses should be available and responding at all times; however, there can still be situations where having a health check can be beneficial.

The best mechanism for creating a test URL on VAST is to create an S3 object that can be accessed anonymously. Anonymous S3 URLs function as HTTP URLs with no special headers/options/etc required, so this object can be directly used for the health check URL, with the URL  http://<hostname>/<bucket-name>/<object-key>

In the steps below, we will create a bucket with the name “healthcheck” and upload an object with the object key “ping” into it, resulting in a health check URL for each IP address of http://<ip-address>/healthcheck/ping

Step 1: Create a Bucket

First, we need to create a bucket for this object. This could be done using an existing bucket, but it is recommended to create a separate bucket for this purpose, especially since this bucket will need to have anonymous access enabled, which is normally not required for most buckets.

In the VAST GUI, create a new ‘S3 Bucket’ view, with the S3 bucket name as desired (eg, 'healthcheck'), and with ‘ACLs Enabled’ so that we can configure our object to have an ACL allowing anonymous access.  (This can also be done with a Bucket Policy; however, ACLs are easier)

The Bucket Owner for the bucket must be set to a user with an S3 access key assigned (to upload the test object), but otherwise can be any user.

Be sure to enable ‘Allow anonymous access’ for the bucket, and then click on ‘Create’ to create the view.

Step 2: Upload a Test Object

Next, using the S3 access/secret key for the user defined as the bucket owner, upload an object to this bucket.  The contents of the object are not important unless the load balancer/proxy looks for specific content when it accesses the health check URL.

The required object must have an ACL that allows anonymous read access. The mechanism for this will vary depending on the S3 client being used.  For AWSCLI, the “--acl public-read” option can be used at the time of object upload :

$ echo "Healthy" > ping

$ aws s3 cp --acl public-read ping s3://healthcheck/

With s3cmd, the relevant option is “--acl-public”, which will also show the URL for the resulting object :

$ s3cmd -c s3cfg put --acl-public ping s3://healthcheck/
upload: 'ping' -> 's3://healthcheck/ping'  [1 of 1]
 8 of 8   100% in    0s   514.57 B/s  done
Public URL of the object is: http://s3.vast1.example.com/healthcheck/ping

Alternatively, you can upload the object without setting the ACL, and then apply it afterwards. For example, using AWSCLI :

$ aws s3 cp ping s3://healthcheck/
$ aws s3api put-object-acl --bucket healthcheck --key ping --grant-read uri=http://acs.amazonaws.com/groups/global/AllUsers

Step 3: Test Health Check URL

After uploading the object, we can test that it works as desired.

As stated above, the health check URL will take the form http://<hostname>/<bucket-name>/<object-key>, and we can use any HTTP client to access this URL.  For example, using curl :

$ curl http://s3.vast1.example.com/healthcheck/ping
Healthy

If you receive an “AccessDenied” error when testing this URL, confirm that the bucket (in the View Configuration) has 'Anonymous access' enabled, AND that the object has an ACL allowing public-read (aka AllUsers read) access.

Most load balancers will not look at the contents of the returned document, but instead look at the HTTP response code - normally expecting a “200” response code.  The HTTP response code can be viewed using curl as follows :

$ curl -w "HTTP Response Code : %{http_code}" http://s3.vast1.example.com/healthcheck/ping
Healthy
HTTP Response Code : 200

Load balancers typically monitor each IP address separately, so we can confirm that this also works when accessing via an IP address rather than a hostname.

$ curl http://172.31.1.1/healthcheck/ping
Healthy

Note that if you’re using HTTPS for the health check URL you’ll likely need to tell your HTTP client not to verify the TLS certificate when using the IP address. This can be done by using the --no-check-certificate option to curl.

Step 4: Configuring Load Balancer/Proxy

The process for configuring the Load Balancer/Proxy to use this health check URL will vary depending on the specific product being used - check the documentation for your load balancer to determine how to configure it.

As an example, for recent versions of HAProxy, the following backend configuration could be used :

backend vast-s3
  option httpchk
  http-check send meth GET uri /healthcheck/ping
  server server1 172.31.1.1:80 check
  server server2 172.31.1.2:80 check
  server server3 172.31.1.3:80 check
  [...]