To achieve the best performance for S3 traffic on VAST Data clusters, it is important that S3 connections are spread across as many CNodes as possible, and thus across as many Vast Virtual IPs as possible.
Different S3 clients and SDKs use different mechanisms to determine which IP address to connect to, so the preferred mechanism for spreading traffic can vary by client.
VAST generally recommends using the built-in VAST DNS Server to handle hostname-to-Virtual IP address translation. The VAST DNS Server simplifies DNS management by dynamically returning results for IP addresses in a specific VAST VIP Pool, and these IPs are automatically updated when changes are made to the VAST configuration.
Number of DNS Responses
By default, the VAST DNS Server responds to each query with a single IP address from the relevant VIP Pool. For S3 use cases, it is recommended to increase the number of entries returned with each query to at least 8 (up to 16 are supported).
The easiest way to determine the current configuration is to use either the dig (Unix) or nslookup (Unix or Windows) commands to query the relevant VAST VIP Pool hostname.
CMD DNS response
Looking at the ANSWER SECTION, we can see that 6 IP different IP addresses are being returned for this hostname – meaning that the VAST DNS server is configured to return (at least) 6 IP addresses for each query. (In this case, the server is configured to return 8 IP addresses, as there are only 6 in the VIP Pool; that is all that will be returned.
To modify the number of IP addresses returned per query, please contact VAST Support.
DNS Response TTL
The VAST DNS Server also allows configuring the “Time To Live” or “TTL” of a response. This value indicates to other DNS servers/caches how long the response to a query may be cached.
The VAST DNS Server TTL defaults to 0 seconds, indicating that the result should not be cached at all, and generally, this will provide the best results. Note that even when this value is set to 0, some DNS servers (in particular, Windows DNS Server) may still cache the results for around 1 second, as long as the VAST DNS Server is returning multiple responses, this will generally have minimal impact.
The TTL can be configured in the VAST GUI under Network Access → DNS → Edit.
TTL = 0
The TTL can also be confirmed using the dig command as per the previous example – the number listed after each of the hostnames in the ANSWER section is the TTL for that entry (which will be the same for all entries).
CMD response shows TTL
If the TTL returned by the dig command is different from that configured on the VAST cluster, then it’s likely that a DNS server in the path between the client and the VAST DNS Server is modifying the value and caching the results. If the TTL value returned is small (no more than a few seconds), this likely will not be a problem, but otherwise the system enacting this caching should be found and caching disabled - at least for the VAST DNS domain.
Client Behaviour
Most S3 clients do not have specific configuration around how they use multiple IP addresses; many will do so automatically when they are configured to open multiple parallel connections to the S3 server, which also generally improves performance significantly.
For example, the AWS Boto3 Python SDK has a configuration parameter max_concurrency which, as the name implies, controls the maximum number of concurrent requests – but it also sets the maximum number of IP addresses that will be used. If this is set to 1, then only a single IP address will be used for all connections. However, setting it to a higher number, such as 4 or 8, will result in multiple IP addresses from the DNS response being used, thus helping to balance traffic across multiple VAST CNodes.
The higher the number of physical clients being run in parallel, the more it will help balance traffic across all CNodes, regardless of the per-client parallelization.
Other Options
In a very small number of cases, in particular when very high performance is required from a single client that is not capable of using multiple IP addresses (which includes some Java-based clients) it may be optimal to use a host-side load balancer to route the traffic across multiple Virtual IPs, such as HAProxy or Envoy.
If considering this option, please reach out to your VAST Account team to discuss what may be the best available options.