1. CNode VIP Balancing
Please refer to our knowledge base for up-to-date instructions on configuring your CNode’s VIPs for optimal performance. Client to Protocol Server (CNode) Balancing
2. VAST-NFSv3 Driver
Please refer to our knowledge base for up-to-date instructions on installing the VAST NFS driver, balancing your VIP pool configurations with the CNodes you have available, and setting up the mount on your host node.
3. Getting to know your machine’s network limitations
To use your host node properly, we need to inspect the hardware.
Example:
2x 400GbE Mellanox CX-7 Single-Port NICs
$ ibdev2netdev -v
0000:69:00.0 mlx5_0 (MT4129 - 30-100363-01) MCX715105AS-WEAT CX-7 1x400GbE QSFP112 PCIe Gen5 x16 VPI NIC fw 28.43.2026 port 1 (ACTIVE) ==> ens202np0 (Up)
0000:53:00.0 mlx5_4 (MT4129 - 30-100363-01) MCX715105AS-WEAT CX-7 1x400GbE QSFP112 PCIe Gen5 x16 VPI NIC fw 28.43.2026 port 1 (ACTIVE) ==> ens201np0 (Up)RDMA enabled
$ rdma link show
link mlx5_0/1 state ACTIVE physical_state LINK_UP netdev ens202np0
link mlx5_4/1 state ACTIVE physical_state LINK_UP netdev ens201np04. vperfcheck to validate performance baseline / Max line rate
Even though 800GbE is our line-rate maximum, and we believe our mount is fully optimized, we should always define a clear baseline for what is possible through the network/storage stack before diving into the actual LLM. That’s where our handy vperfcheck (based on elbencho) comes in handy.
Please refer to our knowledge base for up-to-date instructions on how to use vperfcheck and gather those metrics.
vperfcheck: Automated Performance Testing
5. Example
Assuming we had previously:
Set up a balanced vip pool with ranges
192.168.9.1 - 192.168.9.32(Example range).Installed the latest VAST-NFS driver on our host.
Our host supports RDMA traffic to our VAST cluster/cnodes.
Created the path
/mnt/kvcache-2ports.Know we have two NICs, each capable of 400GbE, and their names (
ens201np0andens202np0).
The following command will mount our view to /mnt/kvcache-2ports in an optimized manner:
mount -o rdma,vers=3,spread_writes,spread_reads,noidlexprt,localports_failover,localports=ens201np0~ens202np0,nconnect=32,remoteports=192.168.9.1-192.168.9.32 192.168.9.1:/kvcache /mnt/kvcache-2ports(You can read about the various settings in our VAST NFS mount params page here. VAST NFS Mount Options )