Press CTRL+K to open search

Deploying vLLM NVIDIA

  • Updated on Sep 14, 2026
  • Published on Sep 14, 2026
  • 3 minute(s) read
Prev Next

vLLM OffloadingConnector

As long-context LLM inference scales, GPU High Bandwidth Memory (HBM) rapidly becomes the primary bottleneck due to linear Key-Value (KV) cache expansion. Integrating vLLM’s native OffloadingConnector, host DRAM, and VAST Data creates a disaggregated path that spills KV from GPU → CPU → NFS, bypassing GPU memory bounds and cutting warm Time-To-First-Token (TTFT) when prefixes hit the CPU/FS tiers.

This KB guide details how to configure vLLM OffloadingConnector against high-performance VAST storage volumes over RDMA-enabled fabrics for production inference — without LMCache.

Prerequisites

Please make sure you have completed the File KVCache Prerequisites before starting this guide, as it can drastically impact performance.

Motivation

GPUs: The testing environment utilizes 2x NVIDIA RTX PRO 6000 Blackwell

  • Network:

    • The host is equipped with 2x Mellanox CX-7 200GbE Single-Port NICs

    • The network configuration for VAST storage is NFSv3 over RDMA

  • Storage: The team tested two primary configurations for KV cache offloading:

    • Local host: OffloadingConnector CPU DRAM tier (/dev/shm mmap)

    • Remote VAST Data storage: FS secondary tier on a VAST partition via NFSv3 over RDMA, with NFS multipath

  • Software:

    • vLLM: Version 0.28.0

    • NVIDIA CUDA: proprietary accelerated computing platform with programming models, compilers, libraries, runtimes, and deployment tools

  • Model: Llama-4-Scout-17B-16E-Instruct-FP8, TP-2

We evaluated a tp=2 configuration across concurrency levels, tuning the system according to the VAST Quick NFS Read Ahead Tuning guidelines to maximize performance:

  • nconnect: Set to 32

  • Read-ahead increased to 8192

  • vLLM OffloadingConnector optimized configurations, detailed below

  • Rank-local host pin baked into the image (required for large CPU tiers on ROCm)

Results

TTFT was measured with and without the VAST storage backend (Layer 2). Both setups utilize L0/L1 caching (GPU & CPU).

~35x Warm tok/s gain

~35x Warm tok/s gain

~41x Warm mean TTFT gain

~41x Warm mean TTFT gain

Software Stack

We’ll be using the following NVIDIA Stack:

  1. vLLM(*) - vllm/vllm-openai:v0.28.0-cu129

    (vllm/vllm-openai@sha256:ac259a0111c6cf462a72e449962b84f7a624b5cbec24bd7d9ec3b67d40ffd1bf)

  2. Model - nvidia/Llama-4-Scout-17B-16E-Instruct-FP8

Installation - NVIDIA

We’ll be working in the working directory /root/vastdata, and have cfg and models underneath it. We’ll also be using the /mnt/kvcache-2ports from earlier, so be sure you have this tree ready

BASE_DIR="/root/vastdata"
MODEL_DIR="${BASE_DIR}/models/llama-4-scout"
CACHE_DIR="/mnt/kvcache-2ports"
CFG="${BASE_DIR}/cfg"   # contains start.sh
mkdir -p ${BASE_DIR} ${MODEL_DIR} ${CFG}

Copy this package into ${BASE_DIR} (Dockerfile, patches/, cfg/start.sh, run.sh).

Downloading the model locally

python3 -m venv "${BASE_DIR}/.venv"
source "${BASE_DIR}/.venv/bin/activate"
pip install "huggingface_hub[cli]"
hf download nvidia/Llama-4-Scout-17B-16E-Instruct-FP8 \
  --local-dir "${MODEL_DIR}"

Why the Dockerfile patches two files

Stock(Unpatched) vLLM pin_mmap_region registers the entire shared /dev/shm offload mmap in every tensor-parallel rank. At TP=8 with a 1600 GiB CPU tier that asks the driver for ~12.8 TiB of pinned mappings. On ROCm a failed cudaHostRegister leaves a sticky hipErrorInvalidValue; the next GPU op then crashes (often far from the pin site).

The files under patches/ change that to rank-local registration (page-aligned strided slots) and matching unregister in cleanup. Aggregate pinned bytes stay ~tier size. This is a source replacement in the image, not a runtime one (a layer that is applied in our Dockerfile).

Look for this line once per rank after engine start:

Rank-local host registration complete: rank=N rows=... ... GiB (shared region ... GiB)

 

Scripts

./cfg/start.sh

#!/usr/bin/env bash
set -euo pipefail

CPU_GB="${VLLM_OFFLOAD_CPU_GB:-160}"
CPU_BYTES=$((CPU_GB * 1024 * 1024 * 1024))
FS_ROOT="${VLLM_OFFLOAD_FS_ROOT:-/kvcache}"

KV_TRANSFER_CONFIG=$(cat <<EOF
{
  "kv_connector": "OffloadingConnector",
  "kv_role": "kv_both",
  "kv_connector_extra_config": {
    "spec_name": "TieringOffloadingSpec",
    "cpu_bytes_to_use": ${CPU_BYTES},
    "block_size": 8192,
    "eviction_policy": "lru",
    "offload_prompt_only": true,
    "secondary_tiers": [{
      "type": "fs",
      "root_dir": "${FS_ROOT}",
      "n_read_threads": 32,
      "n_write_threads": 16
    }]
  }
}
EOF
)

exec vllm serve /models/model \
  --host 0.0.0.0 --port 8000 \
  --served-model-name llama-4-scout \
  --tensor-parallel-size 2 \
  --max-model-len 131072 \
  --max-num-seqs 300 \
  --max-num-batched-tokens 8192 \
  --gpu-memory-utilization 0.92 \
  --enable-prefix-caching \
  --block-size=64 \
  --no-disable-hybrid-kv-cache-manager \
  --kv-transfer-config "${KV_TRANSFER_CONFIG}"

./run.sh

#!/usr/bin/env bash
set -euo pipefail

BASE_DIR="/root/vastdata"
MODEL_DIR="${BASE_DIR}/models/llama-4-scout"
CACHE_DIR="/mnt/kvcache-2ports"
CFG="${BASE_DIR}/cfg"

mkdir -p "${CACHE_DIR}"
chmod +x "${CFG}/start.sh"

# --ipc host uses the host /dev/shm. It must accommodate the 160 GiB
# host KV tier plus headroom.
OFFLOAD_GB=160
NEED_SHM_GB=$((OFFLOAD_GB + 32))
SHM_AVAIL_GB=$(df -BG /dev/shm | awk 'NR==2 {gsub(/G/,"",$4); print $4}')
if [[ "${SHM_AVAIL_GB}" -lt "${NEED_SHM_GB}" ]]; then
  sudo mount -o remount,size="${NEED_SHM_GB}G" /dev/shm
fi

docker run --name vllm-offloading-connector -d \
  --network host --ipc host --gpus all \
  --ulimit memlock=-1 --ulimit stack=67108864 \
  -v "${MODEL_DIR}:/models/model:ro" \
  -v "${CACHE_DIR}:/kvcache:rw" \
  -v "${CFG}:/cfg:ro" \
  -e VLLM_OFFLOAD_CPU_GB=160 \
  -e VLLM_OFFLOAD_FS_ROOT=/kvcache \
  -e SAFETENSORS_FAST_GPU=1 \
  --entrypoint bash \
  vllm-offloading:0.28.0-cu129 /cfg/start.sh

./Dockerfile

ARG VLLM_IMAGE=vllm/vllm-openai:v0.28.0-cu129
FROM ${VLLM_IMAGE}

ENV KV_CPU_DIR=/usr/local/lib/python3.12/dist-packages/vllm/v1/kv_offload/cpu

# Replace stock host-pin and unpin with rank-local registration for the
# OffloadingConnector shared CPU mmap.
COPY patches/gpu_worker.py \
     /usr/local/lib/python3.12/dist-packages/vllm/v1/kv_offload/cpu/gpu_worker.py
COPY patches/shared_offload_region.py \
     /usr/local/lib/python3.12/dist-packages/vllm/v1/kv_offload/cpu/shared_offload_region.py

# Fail the build if either replacement did not land.
RUN grep -q "Rank-local host registration complete" "$KV_CPU_DIR/gpu_worker.py" \
 && grep -q "_rank_local_registered_ptrs" "$KV_CPU_DIR/shared_offload_region.py" \
 && python3 -c "import vllm.v1.kv_offload.cpu.gpu_worker"

Build the image & Run it

docker build \
  --build-arg VLLM_IMAGE=vllm/vllm-openai:v0.28.0-cu129 \
  -t vllm-offloading:0.28.0-cu129 .

chmod +x cfg/start.sh run.sh
./run.sh
shared_offload_region
13.50 KB
gpu_worker
35.02 KB

Additional Resources

(*) While optimized, they are frequently not the latest vLLM builds

Final Note

As workloads and software stacks frequently change, performance opportunities and compatibility shifts may occur. VAST actively expands its capabilities in the KV cache space—contact our team directly to leverage our latest updates and maximize your performance.

Related articles