Inference

As long-context LLM inference scales, GPU High Bandwidth Memory (HBM) quickly becomes the primary bottleneck because the Key-Value (KV) cache grows linearly. Integrating vLLM’s native OffloadingConnector, host DRAM, and VAST Data creates a disaggregated path that spills KV from GPU → CPU → NFS, bypassing GPU memory bounds and cutting warm Time-To-First-Token (TTFT) when refixes hit the CPU/FS tiers.

KB Articles:

• File Prerequisites

• Dynamo

  • Dynamo VAST CSI Integration (Optional)

  • Dynamo using vLLM

• vLLM

  • vLLM (AMD)

  • vLLM (NVIDIA)

  • vLLM & LMCache In-Process (AMD)

  • vLLM & LMCache In-Process (NVIDIA)

  • vLLM & LMCache Multi-Process (AMD)

  • vLLM & LMCache Multi-Process (NVIDIA)

  • ModelExpress with vLLM (NVIDIA)

Use Case:

VAST Data for KV Cache Scale Context Windows, not GPU Costs