Deploying vLLM & LMCache In-Process on NVIDIA

vLLM & LMCache In-Process

As long-context LLM inference scales, GPU High Bandwidth Memory (HBM) rapidly becomes the primary bottleneck due to linear Key-Value (KV) cache expansion. Integrating vLLM, LMCache, and VAST Data creates an ideal disaggregated infrastructure that offloads KV cache to a scale-out storage fabric, bypassing GPU memory bounds and slashing Time-To-First-Token (TTFT) by up to 10x.

This KB guide details how to configure LMCache and vLLM to mount high-performance VAST storage volumes over RDMA-enabled fabrics for production inference.

Prerequisites

Please make sure you have completed the File KVCache Prerequisites before starting this guide, as it can drastically impact performance.

Motivation

GPUs: The testing environment utilizes 2x NVIDIA RTX PRO 6000 Blackwell

Network:
- The host is equipped with 2x Mellanox CX-7 200GbE Single-Port NICs
- The network configuration for VAST storage is NFSv3 over RDMA
Storage: The team tested two primary configurations for KV cache offloading:
- Local host: OffloadingConnector CPU DRAM tier (/dev/shm mmap)
- Remote VAST Data storage: FS secondary tier on a VAST partition via NFSv3 over RDMA, with NFS multipath
Software:
- vLLM: Version 0.28.0
- LMCache: Version 0.53
- NVIDIA CUDA: proprietary accelerated computing platform with programming models, compilers, libraries, runtimes, and deployment tools
Model: Llama-4-Scout-17B-16E-Instruct-FP8, TP-2

We evaluated a tp=2 configuration across concurrency levels, tuning the system according to the VAST Quick NFS Read Ahead Tuning guidelines to maximize performance:

nconnect: Set to 32
Read-ahead increased to 8192
vLLM OffloadingConnector optimized configurations, detailed below
Rank-local host pin baked into the image (required for large CPU tiers on ROCm)

Results

TTFT was measured with VAST storage backend (Layer 2) and without. Both setups utilize L0/L1 caching (GPU & CPU).

~35x Warm tok/s gain

~36x Warm mean TTFT gain

Software Stack

We’ll be using the following NVIDIA Stack:

vLLM(*) - vllm/vllm-openai:v0.28.0-cu129
(vllm/vllm-openai@sha256:ac259a0111c6cf462a72e449962b84f7a624b5cbec24bd7d9ec3b67d40ffd1bf)
LMCache 0.53 built from source
Model - nvidia/Llama-4-Scout-17B-16E-Instruct-FP8

Installation - NVIDIA

We’ll be working in the working directory /root/vastdata, and have cfg and models underneath it. We’ll also be using the /mnt/kvcache-2ports from earlier, so be sure you have this tree ready

BASE_DIR="/root/vastdata"
MODEL_DIR="${BASE_DIR}/models/llama-4-scout"
CACHE_DIR="/mnt/kvcache-2ports"
CFG="${BASE_DIR}/cfg"   # contains lmcache.yaml + start.sh
mkdir -p ${BASE_DIR} ${MODEL_DIR} ${CFG}

Downloading the model locally

python3 -m venv "${BASE_DIR}/.venv"
source "${BASE_DIR}/.venv/bin/activate"
pip install "huggingface_hub[cli]"
hf download nvidia/Llama-4-Scout-17B-16E-Instruct-FP8 \
  --local-dir "${MODEL_DIR}"

Scripts

`./cfg/start.sh`

#!/usr/bin/env bash
set -euo pipefail

exec vllm serve /models/model \
  --host 0.0.0.0 --port 8000 \
  --served-model-name llama-4-scout \
  --tensor-parallel-size 2 \
  --max-model-len 131072 \
  --max-num-seqs 300 \
  --gpu-memory-utilization 0.92 \
  --enable-prefix-caching \
  --block-size=64 \
  --no-disable-hybrid-kv-cache-manager \
  --kv-transfer-config \
  '{"kv_connector":"LMCacheConnectorV1","kv_role":"kv_both"}'

`./cfg/lmcache.yaml`

chunk_size: 8192
save_decode_cache: false
enable_async_loading: true
blocking_timeout_secs: 120
lookup_timeout_ms: 1800000
local_cpu: true
max_local_cpu_size: 80.0
save_unfull_chunk: false
remote_storage_plugins:
  - fs
extra_config:
  remote_storage_plugin.fs.base_path: /kvcache
  remote_storage_plugin.fs.use_odirect: true
  remote_storage_plugin.fs.read_ahead_size: 8192
  fs_connector_read_ahead_size: 8192
  fs_connector_use_odirect: true
  save_chunk_meta: false

`./cfg/l2-adapter.json`

{
  "type": "fs_native",
  "base_path": "/kvcache",
  "num_workers": 4,
  "use_odirect": true,
  "max_capacity_gb": 0,
  "read_ahead_size": 8192
}

`./run.sh`

#!/usr/bin/env bash
set -euo pipefail

BASE_DIR="/root/vastdata"
MODEL_DIR="${BASE_DIR}/models/llama-4-scout"
CACHE_DIR="/mnt/kvcache-2ports"
CFG="${BASE_DIR}/cfg"

mkdir -p "${CACHE_DIR}"
chmod +x "${CFG}/start.sh"

docker run --name vllm-lmcache-in-process -d \
  --network host --ipc host --gpus all \
  --ulimit memlock=-1 --ulimit stack=67108864 \
  -v "${MODEL_DIR}:/models/model:ro" \
  -v "${CACHE_DIR}:/kvcache:rw" \
  -v "${CFG}:/cfg:ro" \
  -e LMCACHE_CONFIG_FILE=/cfg/lmcache.yaml \
  -e PYTHONHASHSEED=0 \
  -e LMCACHE_DISABLE_BANNER=1 \
  -e SAFETENSORS_FAST_GPU=1 \
  --entrypoint bash \
  vllm-lmcache:0.28.0-0.5.3-cu129 /cfg/start.sh

`./Dockerfile`

ARG VLLM_IMAGE=vllm/vllm-openai:v0.28.0-cu129
FROM ${VLLM_IMAGE}

ARG LMCACHE_VERSION=0.5.3
ENV CUDA_HOME=/usr/local/cuda
ENV LMCACHE_CUDA_MAJOR=12

# Build the official LMCache source distribution against this image's
# PyTorch/CUDA ABI. No LMCache or vLLM source files are replaced.
RUN python3 -m pip install --no-cache-dir \
      "setuptools>=77" wheel ninja cmake packaging pybind11 \
 && CPATH="$(ls -d /usr/local/lib/python3.12/dist-packages/nvidia/*/include \
      | tr '\n' ':')${CUDA_HOME}/include" \
    MAX_JOBS="$(nproc)" CMAKE_BUILD_PARALLEL_LEVEL="$(nproc)" \
    python3 -m pip install --no-cache-dir --no-build-isolation --no-deps \
      --no-binary lmcache "lmcache==${LMCACHE_VERSION}" \
 && python3 -c "import lmcache; print(lmcache.__version__)"

Build the image & Run it

docker build -t vllm-lmcache:0.24.0-rocm-lmcache0.5.3 -f Dockerfile .
chmod +x run.sh
./run.sh

Additional Resources

https://hub.docker.com/r/vllm/vllm-openai/tags

(*) While optimized, they are frequently not the latest vLLM builds

Final Note

As workloads and software stacks frequently change, performance opportunities and compatibility shifts may occur. VAST actively expands its capabilities in the KV cache space—contact our team directly to leverage our latest updates and maximize your performance.