How To Benchmark VAST Data with FIO

Configure the load-generating clients

To achieve sufficient concurrency, we need to run fio on many clients, all using the same VAST mount.

Mount the VAST cluster

To evenly spread the load across all CNodes in a VAST Data cluster and ensure that all NICs on all CNodes are used, use the VAST NFS driver on each client and mount the cluster with multipath.

Mount Option - spread_reads, spread_writes

These mount option parameters are not mandatory; the client can reach line rate with/without them. By default, all IO going to a specific file will go through a single connection. With TCP mounts, this may be the bottleneck for applications reading/writing to a single large file. By specifying spread_reads/spread_writes, the respective operation will be able to use all connections and result in increased throughput.

The vast majority of workloads benefit from spread_reads and spread_writes. The only known exceptions are single-thread applications reading files with O_DIRECT (not a default option). These applications have a single in-flight IO at a time, and having those go to the same connection/CNode results in lower latency (because of server-side prefetch).

For example:

sudo mkdir /mnt/vast
sudo mount -t nfs -o vers=3,nconnect=32,remoteports=172.200.201.1-172.200.201.16 172.200.201.1:/ /mnt/vast
mkdir -p /mnt/vast/fio

If you are unable to use the VAST NFS driver, mount the VAST Data cluster on each client to a unique CNode by a VIP on that CNode. You will need at least as many clients as you have CNodes. If your CNodes are dual-NIC, you will need 2x the clients.

sudo mkdir /mnt/vast
sudo mount -t nfs -o vers=3,nconnect=16 172.200.201.1:/ /mnt/vast
mkdir -p /mnt/vast/fio

Start fio in server mode

Now start fio in server mode on each client and daemonize it:

fio --server --daemonize=/tmp/fio.pid

Configure the job runner

On a host that could be a load-generating client, create the hosts.txt file and the job files.

Generate hosts.txt file

The job runner needs to know which hosts will be driving the load. Create a hosts.txt file with each host’s hostname or IP address on a separate line. For example:

10.143.11.111
10.143.11.113
10.143.11.115
10.143.11.117

Generate job files

In client/server mode, fio can only use job files, not command line arguments. To facilitate running an array of tests, we can dynamically generate job files with a shell script:

#!/bin/bash

for bs in 1m 512k 128k 64k; do
    for pattern in write read randread randwrite; do
        echo "
[global]
group_reporting
create_on_open=1
refill_buffers=1
randrepeat=0
ioengine=libaio
iodepth=16
direct=1
numjobs=16
nrfiles=1
directory=/mnt/vast/fio

[10g_files]
readwrite=$pattern
bs=$bs
filesize=10g
" > ${bs}_${pattern}.fio
    done
done

The [globals] section includes the best practices for benchmarking VAST, such as doing direct writes, an async engine, and a high iodepth. Because each thread on each host will generate a file of the specified size, it’s easiest to have each job just generate one file (nrfiles=1) and vary the filesize and number of threads (numjobs). You may want to customize numjobs and filesize to change how much data is written/read for the job.

Running the test

Now, we can run each job file with:

fio --client=hosts.txt 1m_write.fio
fio --client=hosts.txt 1m_read.fio
fio --client=hosts.txt 1m_randread.fio
fio --client=hosts.txt 1m_randwrite.fio

Test sequencing

The job files generated from the script will all use the same data files to save time between tests. An artifact of this is that smaller block writes will fragment the file and impact larger block reads. To address this testing artifact, either:

Delete and recreate the /mnt/vast/fio directory between each test run

Run the tests in this sequence: blocksize high → low; sequential write, sequential read, random read, random write, or in psudocode:

for bs in 1m 512k 128k 64k; do
    for pattern in write read randread randwrite; do
        fio --client=hosts.txt ${bs}_${pattern}.fio
    done
done