ModelExpress with vLLM (NVIDIA)

ModelExpress (MX) is NVIDIA’s fast model loader for inference engines: it pulls safetensors from storage (InstantTensor / ModelStreamer / GDS) or from a ready peer over NIXL RDMA, and can optionally move JIT artifacts the same way. It is not a KV cache tier.

MX also supports SGLang (and potentially other LLM frameworks). Engine wiring differs per framework — this guide is the vLLM path; SGLang needs its own MX load-format / client setup.

This is an extension of the vLLM (NVIDIA) guide: same host stack (File Prerequisites, Docker, NVIDIA driver, NFSv3/RDMA, VAST mounts). For this guide, we’ll be using two servers to showcase model loading across multiple hosts, deploying gpt-oss-120b / TP=1, plus MX.

Use MX with OffloadingConnector — neither replaces the other.

Complete File Prerequisites and the base vLLM NVIDIA guide first.

Motivation

Cold model bring-up is slow: stock vLLM weight load plus first-time compile can dominate minutes of a rollout. At datacenter scale, you need to swap or scale models quickly as demand shifts — not wait on a full cold load every time a replica or model changes. MX shortens that path for both model weights and compile/JIT artifacts. Speedup (without NV-link) gets us as fast as 6x on weights alone, and with NV-link we should get even a bigger bump.

A production GPU fleet is typically much larger than the storage that serves it — for example, ~20 racks of GPUs against less than one rack of VAST. Aggregate GPU demand dwarfs that storage footprint, which can limit performance when many GPUs cold load from storage alone concurrently. ModelExpress addresses that: VAST feeds the first cold load(s); additional replicas take weights over NIXL RDMA from a READY peer (and can reuse JIT artifacts from a shared VAST root when configs match).

Setup

Lab used for the numbers below:

GPUs: 2× NVIDIA RTX PRO 6000 Blackwell — two servers, one GPU each, no NVLink (cross-node RoCE only for P2P)

• Network: 2× Mellanox CX-7 200GbE per host; VAST over NFSv3 over RDMA; RoCE used for MX NIXL P2P

• Storage: Model checkpoint + JIT scratch on VAST (/mnt/kvcache-2ports layout from the base guide)

• Software: vLLM 0.28.0 + ModelExpress 0.7.0 (pins below)

• Model: gpt-oss-120b, TP=1

Two components: weights and artifacts

MX (and engine-ready time) has two independent pieces:

You can optimize either without the other (fast weights + cold compile, or shared arts + slow weight path). Best bring-up hits both.

Technologies MX Uses

ModelExpress internally chooses among the below technologies for weights loading based on a set priority chain (see the chain below)

Upstream overview: ModelExpress GitHub, DEPLOYMENT.md.

Lab Results

Model: gpt-oss-120b in a TP=1 setting

• Lab scale doesn’t represent a fleet of GPUs — 2 GPUs across two servers, one model (gpt-oss-120b), short cold loads. This is a bring-up comparison, not a VAST cluster bandwidth benchmark.

• Cold paths (InstantTensor / ModelStreamer / GDS / default) measure loading safetensors from VAST into a GPU when no READY peer exists.

• NIXL P2P is warm peer GPU→GPU after someone already loaded — not “faster VAST.”

• Throughput note (InstantTensor cold) — model is multi-file safetensors; observed ~126–131 Gbit/s for those ~7–9s. Larger models and more concurrent clients are needed before claiming cluster line rate.

Weight Load by Technology

Artifacts / compile time by method

Other factors (typical)

Why ModelExpress at scale

Reading a model extremely fast onto a large GPU fleet (e.g. thousands of GPUs) implies very high aggregate bandwidth — TBps-class if every node cold-reads from storage. Doing that with local CPUs and SSDs on every node wastes power and hardware.

With ModelExpress: VAST supplies the first loader(s); peers fan out over NIXL; matching JIT can stay on shared VAST. Effective model delivery then scales with the GPU fabric rather than by sizing storage 1:1 with GPU racks.

At a realistic GPU∶CNode ratio, ModelExpress can look like ~100× better effective model-delivery than naïve all-nodes-from-storage — because one cold VAST load plus P2P fan-out replaces thousands of redundant cold reads. The lab microbenchmark (~2× InstantTensor vs default, ~2× NIXL vs InstantTensor) is only a small-scale illustration.

Software stack

Upstream project: ai-dynamo/modelexpress.

How MX chooses a load path

With --load-format modelexpress and default MX_LOAD_STRATEGY_CHAIN=INFERENCE, the client runs this priority chain (first available success wins; safe fall-through on miss):

1. NIXL RDMA P2P       ← peer already READY (weights; arts if MX_ARTIFACT_TRANSFER=1)
2. ServerCache         ← MODEL_EXPRESS_NO_SHARED_STORAGE=1
3. InstantTensor       ← default local fast path
4. ModelStreamer       ← only if MX_MODEL_URI is set
5. GDS                 ← cuFile / nvidia-fs; earlier paths skipped or failed
6. Default loader      ← stock vLLM safetensors

Optional overrides: MX_MODEL_URI=… for ModelStreamer; MX_INSTANT_TENSOR=0 to skip InstantTensor (e.g. force GDS). Start the cold engine first so the joiner’s RDMA step sees a READY peer.

Also set on every engine: MX_SERVER_ADDRESS, MX_REGISTER_LOADERS=1, MX_WORKER_HOST (routable RoCE IP), MX_METADATA_PORT / MX_WORKER_GRPC_PORT, MX_P2P_METADATA=1. Share JIT arts via the same RUN_ID / ARTIFACT_ROOT on VAST (MX_ARTIFACT_TRANSFER=0, default); set MX_ARTIFACT_TRANSFER=1 only when roots are not shared.

Installation

Reuse the base guide layout (/root/vastdata, VAST at /mnt/kvcache-2ports). Put the gpt-oss checkpoint under models/ and add an MX cfg / artifact leaf:

BASE_DIR="/root/vastdata"
MODEL_DIR="${BASE_DIR}/models/gpt-oss-120b"
CACHE_DIR="/mnt/kvcache-2ports"
CFG="${BASE_DIR}/cfg"
mkdir -p "${BASE_DIR}" "${MODEL_DIR}" "${CFG}" "${CACHE_DIR}/mx"

Engine image (vLLM + pinned MX client)

Dockerfile (build context = this directory):

ARG VLLM_IMAGE=vllm/vllm-openai:v0.28.0-cu129
FROM ${VLLM_IMAGE}

# Pin exact PyPI release (do not use unpinned "modelexpress").
ARG MODELEXPRESS_VERSION=0.7.0
RUN pip install --no-cache-dir "modelexpress==${MODELEXPRESS_VERSION}" \
 && python3 -c "import importlib.metadata as m; v=m.version('modelexpress'); assert v=='0.7.0', v"

ENV MX_REGISTER_LOADERS=1

docker build -t vllm-modelexpress:0.28.0-cu129-mx0.7.0 .

Optional stricter pin (copy the wheel next to the Dockerfile after pip download):

COPY modelexpress-0.7.0-cp312-cp312-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl /tmp/
RUN pip install --no-cache-dir /tmp/modelexpress-0.7.0-*.whl \
 && python3 -c "import importlib.metadata as m; assert m.version('modelexpress')=='0.7.0'"

MX server image (pinned digest)

# From a registry that already has this digest, or retag a known-good local copy:
docker pull <registry>/modelexpress-server@sha256:6ff0e82906bfc02672962c825836a5d2603ceae2a0423242bd090e43b25c61fc
docker tag <registry>/modelexpress-server@sha256:6ff0e82906bfc02672962c825836a5d2603ceae2a0423242bd090e43b25c61fc \
  modelexpress-server:0.7.0

Run

Both engines use the same launch script. Neither is configured as “publisher” or “consumer” — MX’s strategy chain picks NIXL P2P if a READY peer exists, otherwise InstantTensor / ModelStreamer / GDS / default. Per-node inputs: RoCE IP, MX server host, INSTANCE (docker name), shared RUN_ID.

Use host network, bind the VAST checkpoint at the same absolute path inside the container, and set MX_WORKER_HOST to that node’s RoCE address.

Redis + MX server (once; typically on node A)

docker run -d --name mx-redis --network host redis:7-alpine \
  redis-server --port 16379 --bind 0.0.0.0

docker run -d --name mx-server --network host \
  -e MX_METADATA_BACKEND=redis \
  -e REDIS_URL=redis://127.0.0.1:16379 \
  -e MODEL_EXPRESS_SERVER_HOST=0.0.0.0 \
  -e MODEL_EXPRESS_SERVER_PORT=18001 \
  -e MODEL_EXPRESS_LOG_LEVEL=info \
  -e MODEL_EXPRESS_CACHE_EVICTION_ENABLED=false \
  modelexpress-server:0.7.0

The server image reads MX_METADATA_BACKEND + REDIS_URL (not MODEL_EXPRESS_REDIS_URL). Confirm with docker logs mx-server that it connects to Redis and listens on 0.0.0.0:18001.

Shared engine script — ${CFG}/start-mx-engine.sh

JIT / compile caches do not all live under VLLM_CACHE_ROOT. Point every leaf at a VAST directory (and bind ninja-sensitive ones at a fixed container path so P2P extract does not force a rebuild when host paths differ per instance):

ARTIFACT_ROOT is keyed by RUN_ID (workload fingerprint, e.g. gpt-oss-120b-tp1), not by INSTANCE. Same RUN_ID on every node ⇒ shared VAST warm arts; set MX_ARTIFACT_TRANSFER=0. Use art P2P only when engines must keep separate roots (different RUN_ID / ARTIFACT_ROOT); keep the /opt/vast/jit/* binds either way so ninja paths stay stable.

vLLM sizing knobs below match the base guide’s cfg/start.sh (context length, GPU util, batching, block size, hybrid KV). Model/TP match the results table (gpt-oss-120b, TP=1). MX deltas: image, --load-format modelexpress, MX env, JIT binds. OffloadingConnector (--kv-transfer-config from the base start.sh) is unchanged if you still want KV tiering — append it the same way.

#!/usr/bin/env bash
set -euo pipefail

# Required from the environment (set by the per-node wrapper):
#   ROCE_IP            this node's RoCE address (MX_WORKER_HOST)
#   MX_SERVER_HOST     RoCE/host where mx-server listens (often node A)
#   INSTANCE           container name suffix only (e.g. replica-a / replica-b)
#   RUN_ID             shared artifact key for this workload (e.g. gpt-oss-120b-tp1)
# Optional overrides:
#   BASE_DIR MODEL_DIR CACHE_DIR ARTIFACT_ROOT PORT GPU

BASE_DIR="${BASE_DIR:-/root/vastdata}"
MODEL_DIR="${MODEL_DIR:-${BASE_DIR}/models/gpt-oss-120b}"
CACHE_DIR="${CACHE_DIR:-/mnt/kvcache-2ports}"
PORT="${PORT:-8000}"
GPU="${GPU:-0}"
NAME="vllm-mx-${INSTANCE}"
# Shared across nodes for the same workload; do not put INSTANCE here.
ARTIFACT_ROOT="${ARTIFACT_ROOT:-${CACHE_DIR}/mx/${RUN_ID}/artifacts}"

: "${ROCE_IP:?set ROCE_IP}"
: "${MX_SERVER_HOST:?set MX_SERVER_HOST}"
: "${INSTANCE:?set INSTANCE}"
: "${RUN_ID:?set RUN_ID}"

mkdir -p \
  "${ARTIFACT_ROOT}/vllm-cache" \
  "${ARTIFACT_ROOT}/flashinfer" \
  "${ARTIFACT_ROOT}/flashinfer_autotune" \
  "${ARTIFACT_ROOT}/triton" \
  "${ARTIFACT_ROOT}/deep_gemm" \
  "${ARTIFACT_ROOT}/tilelang" \
  "${ARTIFACT_ROOT}/cute_dsl"
chmod -R a+rwX "${ARTIFACT_ROOT}"   # VAST root_squash + Docker-as-root

docker rm -f "${NAME}" 2>/dev/null || true
docker run -d --name "${NAME}" --network host --gpus "device=${GPU}" \
  --ulimit memlock=-1 --ulimit stack=67108864 \
  -v "${MODEL_DIR}:/models/model:ro" \
  -v "${ARTIFACT_ROOT}/vllm-cache:${ARTIFACT_ROOT}/vllm-cache:rw" \
  -v "${ARTIFACT_ROOT}/flashinfer_autotune:${ARTIFACT_ROOT}/flashinfer_autotune:rw" \
  -v "${ARTIFACT_ROOT}/flashinfer:/opt/vast/jit/flashinfer:rw" \
  -v "${ARTIFACT_ROOT}/triton:/opt/vast/jit/triton:rw" \
  -v "${ARTIFACT_ROOT}/deep_gemm:/opt/vast/jit/deep_gemm:rw" \
  -v "${ARTIFACT_ROOT}/tilelang:/opt/vast/jit/tilelang:rw" \
  -v "${ARTIFACT_ROOT}/cute_dsl:/opt/vast/jit/cute_dsl:rw" \
  -e SAFETENSORS_FAST_GPU=1 \
  -e MX_SERVER_ADDRESS="${MX_SERVER_HOST}:18001" \
  -e MX_REGISTER_LOADERS=1 \
  -e MX_WORKER_HOST="${ROCE_IP}" \
  -e MX_METADATA_PORT=20000 \
  -e MX_WORKER_GRPC_PORT=21000 \
  -e MX_P2P_METADATA=1 \
  -e MX_ARTIFACT_TRANSFER="${MX_ARTIFACT_TRANSFER:-0}" \
  -e MX_ARTIFACT_READY_URL="http://127.0.0.1:${PORT}/health" \
  -e VLLM_CACHE_ROOT="${ARTIFACT_ROOT}/vllm-cache" \
  -e VLLM_FLASHINFER_AUTOTUNE_CACHE_DIR="${ARTIFACT_ROOT}/flashinfer_autotune" \
  -e FLASHINFER_WORKSPACE_BASE="/opt/vast/jit/flashinfer" \
  -e TRITON_CACHE_DIR="/opt/vast/jit/triton" \
  -e DG_JIT_CACHE_DIR="/opt/vast/jit/deep_gemm" \
  -e DEEP_GEMM_CACHE_DIR="/opt/vast/jit/deep_gemm" \
  -e TILELANG_CACHE_DIR="/opt/vast/jit/tilelang" \
  -e CUTE_DSL_CACHE_DIR="/opt/vast/jit/cute_dsl" \
  vllm-modelexpress:0.28.0-cu129-mx0.7.0 \
  /models/model \
    --host 0.0.0.0 --port "${PORT}" \
    --served-model-name gpt-oss-120b \
    --load-format modelexpress \
    --tensor-parallel-size 1 \
    --max-model-len 131072 \
    --max-num-seqs 300 \
    --max-num-batched-tokens 8192 \
    --gpu-memory-utilization 0.92 \
    --enable-prefix-caching \
    --block-size=64 \
    --no-disable-hybrid-kv-cache-manager

Image entrypoint is already ["vllm","serve"] (from vllm/vllm-openai). Pass only the model path and flags — do not prefix with vllm serve again.

chmod +x "${CFG}/start-mx-engine.sh"

Per-node wrappers (IPs only)

Node A (${CFG}/run-node-a.sh) — start this first if you want the second engine to have a READY peer for P2P:

#!/usr/bin/env bash
set -euo pipefail
export ROCE_IP="<node-A-roce-ip>"
export MX_SERVER_HOST="${ROCE_IP}"          # mx-server on A
export INSTANCE="replica-a"                 # docker name only
export RUN_ID="gpt-oss-120b-tp1"            # same on every node
exec "$(dirname "$0")/start-mx-engine.sh"

Node B (${CFG}/run-node-b.sh):

#!/usr/bin/env bash
set -euo pipefail
export ROCE_IP="<node-B-roce-ip>"
export MX_SERVER_HOST="<node-A-roce-ip>"    # same mx-server as A
export INSTANCE="replica-b"                 # docker name only
export RUN_ID="gpt-oss-120b-tp1"            # must match A
# Art P2P only if roots diverge:
#   export MX_ARTIFACT_TRANSFER=1
exec "$(dirname "$0")/start-mx-engine.sh"

# On A (after Redis + mx-server):
./cfg/run-node-a.sh
# Wait until http://127.0.0.1:8000/v1/models is up (optional; only affects who is READY first)
# On B:
./cfg/run-node-b.sh

Check logs for the path MX actually took (RDMA transfer complete, InstantTensor, GDS load complete, etc.). Starting B after A is READY biases B toward P2P; reverse start order or a missed READY peer falls through to storage.

Additional resources

• Base guide: vLLM (NVIDIA)

• File Prerequisites: File Prerequisites

• ModelExpress blog · GitHub · DEPLOYMENT.md

• NIXL · GPUDirect Storage · InstantTensor · ModelStreamer

• vLLM tags

Final note

Software and fabrics change. Pins above (modelexpress==0.7.0, server digest, vLLM 0.28.0) freeze the stack that produced the results table; bump deliberately and re-validate load-path logs after upgrades.