Press CTRL+K to open search

Deploying vLLM AMD

New
  • Updated on Sep 10, 2026
  • Published on Sep 9, 2026
  • 3 minute(s) read
Prev Next

vLLM OffloadingConnector AMD

As long-context LLM inference scales, GPU High Bandwidth Memory (HBM) rapidly becomes the primary bottleneck due to linear Key-Value (KV) cache expansion. Integrating vLLM’s native OffloadingConnector, host DRAM, and VAST Data creates a disaggregated path that spills KV from GPU → CPU → NFS, bypassing GPU memory bounds and cutting warm Time-To-First-Token (TTFT) when  refixes hit the CPU/FS tiers.

This KB guide details how to configure vLLM OffloadingConnector against high-performance VAST storage volumes over RDMA-enabled fabrics for production inference — without LMCache.

Prerequisites

Please make sure you have completed the File KVCache Prerequisites before starting this guide, as it can drastically impact performance.

Motivation

GPUs: The testing environment utilizes an AMD MI350X node.

  • Network:

    • The host is equipped with 2x Mellanox CX-7 400GbE Single-Port NICs

    • The network configuration for VAST storage is NFSv3 over RDMA

  • Storage: The team tested two primary configurations for KV cache offloading:

    • Local host: OffloadingConnector CPU DRAM tier (/dev/shm mmap)

    • Remote VAST Data storage: FS secondary tier on a VAST partition via NFSv3 over RDMA, with NFS multipath

  • Software:

    • vLLM: Version 0.28.0 (ROCm)

    • AMD ROCm: open foundation with programming models, compilers, libraries, runtimes, and deployment tools

    • No LMCache — OffloadingConnector + TieringOffloadingSpec only

  • Model: GPT-oss 120B, TP-8
    (Maverick FP8 / TP-2 smokes use the same image and connector wiring)

We evaluated a tp=8 configuration across concurrency levels, tuning the system according to the VAST Quick NFS Read Ahead Tuning guidelines to maximize performance:

  • nconnect: Set to 32

  • Read-ahead increased to 8192

  • vLLM OffloadingConnector optimized configurations, detailed below

  • Rank-local host pin baked into the image (required for large CPU tiers on ROCm)

Results

Software Stack

We’ll be using the following AMD Stack:

  1. vLLM(*) - vllm/vllm-openai-rocm:v0.28.0

    (vllm/vllm-openai-rocm@sha256:e0a3b2bd3fe7ec563916c3a5d949898d133458c18d6b2f460c906885cfb32032)

  2. Model - amd/gpt-oss-120b-w-mxfp4-a-fp8

  3. Image overlay - vllm-fs:0.28.0-rocm-rankpin (this package’s Dockerfile; two patched files only)

Installation - AMD

We’ll be working in the working directory /root/vastdata, and have cfg and models underneath it. We’ll also be using the /mnt/kvcache-2ports from earlier, so be sure you have this tree ready

BASE_DIR="/root/vastdata"
MODEL_DIR="${BASE_DIR}/models/gpt-oss-120b-w-mxfp4-a-fp8"
CACHE_DIR="/mnt/kvcache-2ports"
CFG="${BASE_DIR}/cfg"   # contains start.sh
mkdir -p ${BASE_DIR} ${MODEL_DIR} ${CFG}

Copy this package into ${BASE_DIR} (Dockerfile, patches/, cfg/start.sh, run.sh).

Downloading the model locally

python3 -m venv .venv
source .venv/bin/activate
pip install "huggingface_hub[cli]"
hf download amd/gpt-oss-120b-w-mxfp4-a-fp8 --local-dir "${MODEL_DIR}"

Why the Dockerfile patches two files

Stock(Unpatched) vLLM pin_mmap_region registers the entire shared /dev/shm offload mmap in every tensor-parallel rank. At TP=8 with a 1600 GiB CPU tier that asks the driver for ~12.8 TiB of pinned mappings. On ROCm a failed cudaHostRegister leaves a sticky hipErrorInvalidValue; the next GPU op then crashes (often far from the pin site).

The files under patches/ change that to rank-local registration (page-aligned strided slots) and matching unregister in cleanup. Aggregate pinned bytes stay ~tier size. This is a source replacement in the image, not a runtime one (a layer that is applied in our Dockerfile).

Look for this line once per rank after engine start:

Rank-local host registration complete: rank=N rows=... ... GiB (shared region ... GiB)

./cfg/start.sh

#!/usr/bin/env bash
set -euo pipefail

CPU_GB="${VLLM_OFFLOAD_CPU_GB:-1600}"
CPU_BYTES=$((CPU_GB * 1024 * 1024 * 1024))
FS_ROOT="${VLLM_OFFLOAD_FS_ROOT:-/kvcache}"

KV_TRANSFER_CONFIG=$(cat <<EOF
{"kv_connector":"OffloadingConnector","kv_role":"kv_both","kv_connector_extra_config":{"spec_name":"TieringOffloadingSpec","cpu_bytes_to_use":${CPU_BYTES},"block_size":8192,"eviction_policy":"lru","offload_prompt_only":true,"secondary_tiers":[{"type":"fs","root_dir":"${FS_ROOT}","n_read_threads":32,"n_write_threads":16}]}}
EOF
)

exec vllm serve /models/model \
  --host 0.0.0.0 --port 8000 \
  --served-model-name gpt-oss-120b \
  --tensor-parallel-size 8 \
  --max-model-len 131072 \
  --enable-prefix-caching \
  --block-size=64 \
  --max-num-seqs 256 \
  --kv-transfer-config "${KV_TRANSFER_CONFIG}"

./run.sh

BASE_DIR="/root/vastdata"
MODEL_DIR="${BASE_DIR}/models/gpt-oss-120b-w-mxfp4-a-fp8"
CACHE_DIR="/mnt/kvcache-2ports"
CFG="${BASE_DIR}/cfg"   # contains start.sh
mkdir -p "$CFG"
chmod +x "$CFG/start.sh"

# Ensure /dev/shm can hold cpu_bytes_to_use (+ headroom). --ipc host uses the
# host tmpfs; remount is not reboot-persistent.
OFFLOAD_GB=1600
NEED_SHM_GB=$((OFFLOAD_GB + 32))
SHM_AVAIL_GB=$(df -BG /dev/shm | awk 'NR==2 {gsub(/G/,"",$4); print $4}')
if [[ "${SHM_AVAIL_GB}" -lt "${NEED_SHM_GB}" ]]; then
  mount -o remount,size="${NEED_SHM_GB}G" /dev/shm
fi

docker run --name vllm-fs-offload -d \
  --network host --ipc host \
  --ulimit memlock=-1 --ulimit stack=67108864 \
  --device /dev/kfd --device /dev/dri \
  --group-add video --group-add render \
  -v "${MODEL_DIR}:/models/model:ro" \
  -v "${CACHE_DIR}:/kvcache:rw" \
  -v "${CFG}:/cfg:ro" \
  -e PYTHONHASHSEED=0 \
  -e VLLM_OFFLOAD_CPU_GB=1600 \
  -e VLLM_OFFLOAD_FS_ROOT=/kvcache \
  -e HIP_FORCE_DEV_KERNARG=1 -e HSA_NO_SCRATCH_RECLAIM=1 \
  -e TORCH_BLAS_PREFER_HIPBLASLT=1 -e SAFETENSORS_FAST_GPU=1 \
  -e VLLM_ROCM_USE_AITER=1 -e VLLM_ROCM_USE_AITER_MHA=0 \
  -e VLLM_ROCM_USE_AITER_UNIFIED_ATTENTION=1 \
  -e VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4 \
  -e NCCL_MIN_NCHANNELS=112 \
  --entrypoint bash vllm-fs:0.28.0-rocm-rankpin /cfg/start.sh

./Dockerfile

# vllm-fs:0.28.0-rocm-rankpin
ARG BASE_IMAGE=vllm/vllm-openai-rocm@sha256:e0a3b2bd3fe7ec563916c3a5d949898d133458c18d6b2f460c906885cfb32032
FROM ${BASE_IMAGE}

COPY patches/gpu_worker.py \
     /usr/local/lib/python3.12/dist-packages/vllm/v1/kv_offload/cpu/gpu_worker.py
COPY patches/shared_offload_region.py \
     /usr/local/lib/python3.12/dist-packages/vllm/v1/kv_offload/cpu/shared_offload_region.py

RUN python3 - <<'PY'
from pathlib import Path
import vllm.v1.kv_offload.cpu.gpu_worker as gw
import vllm.v1.kv_offload.cpu.shared_offload_region as sor
assert "Rank-local host registration complete" in Path(gw.__file__).read_text()
assert "_rank_local_registered_ptrs" in Path(sor.__file__).read_text()
print("rank-local pin patch OK")
PY

Build the image & Run it

docker build -t vllm-fs:0.28.0-rocm-rankpin -f Dockerfile .
chmod +x run.sh cfg/start.sh
./run.sh

Additional Resources

(*) While optimized, they are frequently not the latest vLLM builds

Final Note

As workloads and software stacks frequently change, performance opportunities and compatibility shifts may occur. VAST actively expands its capabilities in the KV cache space—contact our team directly to leverage our latest updates and maximize your performance.

Related articles