Press CTRL+K to open search

Deploying vLLM & LMCache Multi-Process on NVIDIA

  • Updated on Sep 14, 2026
  • Published on Sep 14, 2026
  • 3 minute(s) read
Prev Next

vLLM & LMCache Multi-Process

As long-context LLM inference scales, GPU High Bandwidth Memory (HBM) rapidly becomes the primary bottleneck due to linear Key-Value (KV) cache expansion. Integrating vLLM, LMCache, and VAST Data creates an ideal disaggregated infrastructure that offloads KV cache to a scale-out storage fabric, bypassing GPU memory bounds and slashing Time-To-First-Token (TTFT) by up to 10x.

This KB guide details how to configure LMCache and vLLM to mount high-performance VAST storage volumes over RDMA-enabled fabrics for production inference.

Prerequisites

Please make sure you have completed the File KVCache Prerequisites before starting this guide, as it can drastically impact performance.

Motivation

GPUs: The testing environment utilizes 2x NVIDIA RTX PRO 6000 Blackwell

  • Network:

    • The host is equipped with 2x Mellanox CX-7 200GbE Single-Port NICs

    • The network configuration for VAST storage is NFSv3 over RDMA

  • Storage: The team tested two primary configurations for KV cache offloading:

    • Local host: OffloadingConnector CPU DRAM tier (/dev/shm mmap)

    • Remote VAST Data storage: FS secondary tier on a VAST partition via NFSv3 over RDMA, with NFS multipath

  • Software:

    • vLLM: Version 0.28.0

    • LMCache: Version 0.53

    • NVIDIA CUDA: proprietary accelerated computing platform with programming models, compilers, libraries, runtimes, and deployment tools

  • Model: Llama-4-Scout-17B-16E-Instruct-FP8, TP-2

We evaluated a tp=2 configuration across concurrency levels, tuning the system according to the VAST Quick NFS Read Ahead Tuning guidelines to maximize performance:

  • nconnect: Set to 32

  • Read-ahead increased to 8192

  • vLLM OffloadingConnector optimized configurations, detailed below

  • Rank-local host pin baked into the image (required for large CPU tiers on ROCm)

Results

TTFT was measured with VAST storage backend (Layer 2) and without. Both setups utilize L0/L1 caching (GPU & CPU).

~35x Warm tok/s gain

~35x Warm tok/s gain

~40x Warm mean TTFT gain

~40x Warm mean TTFT gain

Software Stack

We’ll be using the following NVIDIA Stack:

  1. vLLM(*) - vllm/vllm-openai:v0.28.0-cu129

    (vllm/vllm-openai@sha256:ac259a0111c6cf462a72e449962b84f7a624b5cbec24bd7d9ec3b67d40ffd1bf)

  2. LMCache 0.53 built from source

  3. Model - nvidia/Llama-4-Scout-17B-16E-Instruct-FP8

Installation - NVIDIA

We’ll be working in the working directory /root/vastdata, and have cfg and models underneath it. We’ll also be using the /mnt/kvcache-2ports from earlier, so be sure you have this tree ready

BASE_DIR="/root/vastdata"
MODEL_DIR="${BASE_DIR}/models/llama-4-scout"
CACHE_DIR="/mnt/kvcache-2ports"
CFG="${BASE_DIR}/cfg"   # contains lmcache.yaml + start.sh
mkdir -p ${BASE_DIR} ${MODEL_DIR} ${CFG}

 

Downloading the model locally

python3 -m venv "${BASE_DIR}/.venv"
source "${BASE_DIR}/.venv/bin/activate"
pip install "huggingface_hub[cli]"
hf download nvidia/Llama-4-Scout-17B-16E-Instruct-FP8 \
  --local-dir "${MODEL_DIR}"

 Scripts

./cfg/start.sh

#!/usr/bin/env bash
set -euo pipefail

L2_ADAPTER=$(tr -d '\n' < /cfg/l2-adapter.json)

lmcache server \
  --host 127.0.0.1 --port 5555 \
  --l1-size-gb 160 \
  --eviction-policy LRU \
  --eviction-trigger-watermark 0.8 \
  --eviction-ratio 0.2 \
  --chunk-size 8192 \
  --l2-prefetch-max-in-flight 4 \
  --max-gpu-workers 8 \
  --max-cpu-workers 8 \
  --worker-reap-timeout-seconds 180 \
  --l2-adapter "${L2_ADAPTER}" \
  --l1-align-bytes 1048576 \
  --separate-object-groups &

for _ in $(seq 1 60); do
  (echo > /dev/tcp/127.0.0.1/5555) >/dev/null 2>&1 && break
  sleep 1
done
(echo > /dev/tcp/127.0.0.1/5555) >/dev/null 2>&1 \
  || { echo "LMCache server failed readiness on port 5555" >&2; exit 1; }

exec vllm serve /models/model \
  --host 0.0.0.0 --port 8000 \
  --served-model-name llama-4-scout \
  --tensor-parallel-size 2 \
  --max-model-len 131072 \
  --max-num-seqs 300 \
  --gpu-memory-utilization 0.92 \
  --enable-prefix-caching \
  --block-size=64 \
  --no-disable-hybrid-kv-cache-manager \
  --kv-transfer-config \
  '{"kv_connector":"LMCacheMPConnector","kv_connector_module_path":"lmcache.integration.vllm.lmcache_mp_connector","kv_role":"kv_both","kv_connector_extra_config":{"lmcache.mp.host":"tcp://127.0.0.1","lmcache.mp.port":5555,"lmcache.mp.heartbeat_interval":60.0}}'

./cfg/lmcache.yaml

chunk_size: 8192
save_decode_cache: false
enable_async_loading: true
blocking_timeout_secs: 120
lookup_timeout_ms: 1800000
mp_host: 127.0.0.1
mp_port: 5555

./cfg/l2-adapter.json

{
  "type": "fs_native",
  "base_path": "/kvcache",
  "num_workers": 4,
  "use_odirect": true,
  "max_capacity_gb": 0,
  "read_ahead_size": 8192
}

./run.sh

#!/usr/bin/env bash
set -euo pipefail

BASE_DIR="/root/vastdata"
MODEL_DIR="${BASE_DIR}/models/llama-4-scout"
CACHE_DIR="/mnt/kvcache-2ports"
CFG="${BASE_DIR}/cfg"

mkdir -p "${CACHE_DIR}"
chmod +x "${CFG}/start.sh"

docker run --name vllm-lmcache-multiprocess-hybrid -d \
  --network host --ipc host --gpus all \
  --ulimit memlock=-1 --ulimit stack=67108864 \
  -v "${MODEL_DIR}:/models/model:ro" \
  -v "${CACHE_DIR}:/kvcache:rw" \
  -v "${CFG}:/cfg:ro" \
  -e LMCACHE_CONFIG_FILE=/cfg/lmcache.yaml \
  -e PYTHONHASHSEED=0 \
  -e LMCACHE_DISABLE_BANNER=1 \
  -e SAFETENSORS_FAST_GPU=1 \
  --entrypoint bash \
  vllm-lmcache:0.28.0-0.5.3-cu129 /cfg/start.sh

./Dockerfile

ARG VLLM_IMAGE=vllm/vllm-openai:v0.28.0-cu129
FROM ${VLLM_IMAGE}

ARG LMCACHE_VERSION=0.5.3
ENV CUDA_HOME=/usr/local/cuda
ENV LMCACHE_CUDA_MAJOR=12

# Build the official LMCache source distribution against this image's
# PyTorch/CUDA ABI. No LMCache or vLLM source files are replaced.
RUN python3 -m pip install --no-cache-dir \
      "setuptools>=77" wheel ninja cmake packaging pybind11 \
 && CPATH="$(ls -d /usr/local/lib/python3.12/dist-packages/nvidia/*/include \
      | tr '\n' ':')${CUDA_HOME}/include" \
    MAX_JOBS="$(nproc)" CMAKE_BUILD_PARALLEL_LEVEL="$(nproc)" \
    python3 -m pip install --no-cache-dir --no-build-isolation --no-deps \
      --no-binary lmcache "lmcache==${LMCACHE_VERSION}" \
 && python3 -c "import lmcache; print(lmcache.__version__)"

 

Build the image & Run it

docker build -t vllm-lmcache:0.24.0-rocm-lmcache0.5.3 -f Dockerfile .
chmod +x run.sh
./run.sh

Additional Resources

(*) While optimized, they are frequently not the latest vLLM builds

Final Note

As workloads and software stacks frequently change, performance opportunities and compatibility shifts may occur. VAST actively expands its capabilities in the KV cache space—contact our team directly to leverage our latest updates and maximize your performance.

Related articles