Press CTRL+K to open search

Deploying vLLM & LMCache In-Process on AMD

Updated
  • Updated on Sep 10, 2026
  • Published on Sep 1, 2026
  • 5 minute(s) read
Prev Next

vLLM & LMCache In-Process

As long-context LLM inference scales, GPU High Bandwidth Memory (HBM) rapidly becomes the primary bottleneck due to the linear expansion of the Key-Value (KV) cache. Integrating vLLM, LMCache, and VAST Data creates an ideal disaggregated infrastructure that offloads KVCache to a scale-out storage fabric, bypassing GPU memory bounds and slashing Time-To-First-Token (TTFT) by up to 10x.

This KB guide details how to configure LMCache and vLLM to mount high-performance VAST storage volumes over RDMA-enabled fabrics for production inference.

Prerequisites

Please make sure you have completed the File KVCache Prerequisites before starting this guide, as it can drastically impact performance.

Motivation

GPUs: The testing environment utilizes an AMD MI350X node.

  • Network:

    • The host is equipped with 2x Mellanox CX-7 400 GbE single-port NICs.

    • The network configuration for VAST storage is NFSv3 over RDMA.

  • Storage: The team tested two primary configurations for KV cache offloading:

    • Local host: offloading to local host RAM.

    • Remote VAST Data storage: Offloading to a VAST storage partition via NFSv3 over RDMA, with NFS multi-path.

  • Software: 

    • vLLM: Version 0.24

    • LMCache: Version 0.53

    • AMD ROCm: open foundation with programming models, compilers, libraries, runtimes, and deployment tools.

  • Model: GPT-oss 120B, TP-8

We evaluated a tp=8 configuration across concurrency levels ranging from 100 to 600, tuning the system according to the VAST Quick NFS Read Ahead Tuning guidelines to maximize performance:

  • nconnect: Set to 32

  • Read-ahead increased to 8192

  • vLLM & LMCache optimized configurations, detailed below

Results

TTFT was measured with and without the VAST storage backend (Layer 2). Both setups utilize L0/L1 caching (GPU & CPU).

~10x Warm tok/s gain

~10x Warm tok/s gain

~10x Warm mean TTFT gain

~10x Warm mean TTFT gain

Software Stack

We’ll be using the following AMD Stack:

  1. vLLM(*) - https://hub.docker.com/r/vllm/vllm-openai-rocm  (0.24.0+rocm723, vllm/vllm-openai-rocm@sha256:3832d79d9e514ce2e072580689da078726454596d833c8ab803f29f3cea5ea28)

  2. LMCache(*) - https://github.com/LMCache/LMCache/releases#release-v0.5.3-rocm (Release v0.5.3 · ROCm (gfx942, gfx950))

  3. Model - amd/gpt-oss-120b-w-mxfp4-a-fp8

 

Prerequisites

1. CNode VIP Balancing

Please refer to our knowledge base for up-to-date instructions on how to configure your CNodes VIPs in an optimized manner.  Client to Protocol Server (CNode) Balancing

2. VAST-NFSv3 Driver

Please refer to our knowledge base for up-to-date instructions on how to install the VAST NFS driver, balance your VIP pool configurations with the CNodes you have available, and finally set up the mount on your host node.

3. Getting to know your machine’s network limitations

To properly utilize your host node, we need to inspect the hardware.

Example:

2x 400GbE Mellanox CX-7 Single-Port NICs

$ ibdev2netdev -v
0000:69:00.0 mlx5_0 (MT4129 - 30-100363-01) MCX715105AS-WEAT CX-7 1x400GbE QSFP112 PCIe Gen5 x16 VPI NIC                                                                                                                           fw 28.43.2026 port 1 (ACTIVE) ==> ens202np0 (Up)
0000:53:00.0 mlx5_4 (MT4129 - 30-100363-01) MCX715105AS-WEAT CX-7 1x400GbE QSFP112 PCIe Gen5 x16 VPI NIC                                                                                                                           fw 28.43.2026 port 1 (ACTIVE) ==> ens201np0 (Up)

RDMA enabled

$ rdma link show
link mlx5_0/1 state ACTIVE physical_state LINK_UP netdev ens202np0
link mlx5_4/1 state ACTIVE physical_state LINK_UP netdev ens201np0

 

4. vperfcheck to validate performance baseline / Max line rate

Even though 800GbE is our maximum line rate, and we believe our mount is fully optimized, we should always establish a clear baseline for what is possible with the network/storage stack before diving into the actual LLM. That’s where our handy vperfcheck (based on elbencho) comes in handy.

Please refer to our knowledge base for up-to-date instructions on how to use vperfcheck and gather those metrics. vperfcheck: Automated Performance Testing

 

5. Example

Assuming we had previously:

  1. Set up a balanced vip pool with ranges 192.168.9.1 - 192.168.9.32 (Example range).

  2. Installed the latest VAST-NFS driver on our host.

  3. Our host supports RDMA traffic to our VAST cluster/cnodes.

  4. Created the path /mnt/kvcache-2ports.

  5. Now we have two NICs, each capable of 400GbE, and their names (ens201np0 and ens202np0).

 

The following command will mount our view to /mnt/kvcache-2ports in an optimized manner:

mount -o rdma,vers=3,spread_writes,spread_reads,noidlexprt,localports_failover,localports=ens201np0~ens202np0,nconnect=32,remoteports=192.168.9.1-192.168.9.32 192.168.9.1:/kvcache /mnt/kvcache-2ports

(You can read about the various settings in our VAST NFS mount params page here. VAST NFS Mount parameters )

 

Installation - AMD

 We’ll be working in the working directory /root/vastdata, and have cfg and modelsunderneath it. We’ll also be using the /mnt/kvcache-2ports from earlier, so be sure you have this tree ready

BASE_DIR="/root/vastdata"
MODEL_DIR="${BASE_DIR}/models/gpt-oss-120b-w-mxfp4-a-fp8"
CACHE_DIR="/mnt/kvcache-2ports"
CFG="${BASE_DIR}/cfg"   # contains lmcache.yaml + start.sh
mkdir -p ${BASE_DIR} ${MODEL_DIR} ${CFG}

 

Downloading the model locally

python3 -m venv .venv
source .venv/bin/activate
pip install "huggingface_hub[cli]"
hf download amd/gpt-oss-120b-w-mxfp4-a-fp8 --local-dir "${MODEL_DIR}"

 

./cfg/start.sh

#!/usr/bin/env bash
set -euo pipefail
exec vllm serve /models/model \
  --host 0.0.0.0 --port 8000 \
  --served-model-name gpt-oss-120b \
  --tensor-parallel-size 8 \
  --max-model-len 131072 \
  --enable-prefix-caching \
  --block-size=64 \
  --max-num-seqs 256 \
  --kv-transfer-config \
  '{"kv_connector":"LMCacheConnectorV1","kv_role":"kv_both"}'

./cfg/lmcache.yaml

chunk_size: 8192
save_decode_cache: false
enable_async_loading: true
blocking_timeout_secs: 120
lookup_timeout_ms: 1800000
local_cpu: true
max_local_cpu_size: 200.0
save_unfull_chunk: false
remote_storage_plugins:
  - fs
extra_config:
  remote_storage_plugin.fs.base_path: /kvcache
  remote_storage_plugin.fs.use_odirect: true
  remote_storage_plugin.fs.read_ahead_size: 8192
  fs_connector_read_ahead_size: 8192
  fs_connector_use_odirect: true
  save_chunk_meta: false

./run.sh

BASE_DIR="/root/vastdata"
MODEL_DIR="${BASE_DIR}/models/gpt-oss-120b-w-mxfp4-a-fp8"
CACHE_DIR="/mnt/kvcache-2ports"
CFG="${BASE_DIR}/cfg"   # contains lmcache.yaml + start.sh
mkdir -p "$CFG"
chmod +x "$CFG/start.sh"
docker run --name vllm-lmcache-ip-fs -d \
  --network host --ipc host \
  --ulimit memlock=-1 --ulimit stack=67108864 \
  --device /dev/kfd --device /dev/dri \
  --group-add video --group-add render \
  -v "${MODEL_DIR}:/models/model:ro" \
  -v "${CACHE_DIR}:/kvcache:rw" \
  -v "${CFG}:/cfg:ro" \
  -e LMCACHE_CONFIG_FILE=/cfg/lmcache.yaml \
  -e PYTHONHASHSEED=0 \
  -e HIP_FORCE_DEV_KERNARG=1 -e HSA_NO_SCRATCH_RECLAIM=1 \
  -e TORCH_BLAS_PREFER_HIPBLASLT=1 -e SAFETENSORS_FAST_GPU=1 \
  -e VLLM_ROCM_USE_AITER=1 -e VLLM_ROCM_USE_AITER_MHA=0 \
  -e VLLM_ROCM_USE_AITER_UNIFIED_ATTENTION=1 \
  -e VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4 \
  -e NCCL_MIN_NCHANNELS=112 \
  --entrypoint bash vllm-lmcache:0.24.0-rocm-lmcache0.5.3 /cfg/start.sh

./Dockerfile

# vllm-lmcache:0.24.0-rocm-lmcache0.5.3
FROM vllm/vllm-openai-rocm@sha256:3832d79d9e514ce2e072580689da078726454596d833c8ab803f29f3cea5ea28
ARG LMCACHE_VERSION=0.5.3
# PyPI deps, then overwrite with the official ROCm wheel (--no-deps keeps
# the base image's ROCm torch). Swap CUDA CuPy for ROCm CuPy.
RUN python3 -m pip install --no-cache-dir "lmcache==${LMCACHE_VERSION}" \
 && python3 -m pip uninstall -y cupy-cuda13x cupy-cuda12x cufile-python || true \
 && python3 -m pip install --no-cache-dir "cupy-rocm-7-0==14.1.1" \
 && python3 -m pip install --no-cache-dir --force-reinstall --no-deps --no-index \
      --find-links "https://github.com/LMCache/LMCache/releases/expanded_assets/v${LMCACHE_VERSION}-rocm" \
      "lmcache==${LMCACHE_VERSION}" \
 && python3 -c "import lmcache; print('lmcache', lmcache.__version__)"

 

Build the image & Run it

docker build -t vllm-lmcache:0.24.0-rocm-lmcache0.5.3 -f Dockerfile .
chmod +x run.sh
./run.sh

 

Additional Resources

https://hub.docker.com/r/rocm/vllm/tags (AMD published images, not used here but shared for knowledge)

https://github.com/ROCm/rocm-aic  (AMD AIC, not production-ready at time of posting)

(*) While optimized, they are frequently not the latest vLLM builds

Final Note

As workloads and software stacks frequently change, performance opportunities and compatibility shifts may occur. VAST actively expands its capabilities in the KV cache space—contact our team directly to leverage our latest updates and maximize your performance.

Related articles