wikis / NVIDIA DGX Spark / wiki / concepts / vllm-on-spark.md view as markdown report a mistake

Definition

vLLM is a high-throughput LLM serving engine built around PagedAttention and continuous batching, exposed through an OpenAI-compatible API. General vLLM concepts (PagedAttention internals, scheduler design, the broader flag surface) are covered in depth by the dedicated vLLM wiki in this fleet. On DGX Spark the official playbook's Spark-specific content is: which prebuilt NGC/custom containers actually work on GB10's ARM64 + Blackwell combination, the NVFP4/FP8 quantized model catalog NVIDIA has validated for it, and how to cluster two or more Sparks over QSFP with Ray for models too large for one unit.

Note: a separate station-vllm playbook exists in the same source tree but targets DGX Station (GB300, 252GB HBM3e) — a different, larger product. It is not covered here; do not conflate its container tags (e.g. 26.01-py3, --cpu-offload-gb) with the Spark playbook's.

How It Works

The Spark playbook is Docker-first. After adding the user to the docker group, a model is served from the NVIDIA NGC container:

export HF_TOKEN="your_huggingface_token"
export LATEST_VLLM_VERSION=<latest_container_version>   ## e.g. 26.05.post1-py3
export HF_MODEL_HANDLE=<HF_HANDLE>                        ## e.g. openai/gpt-oss-20b

docker pull nvcr.io/nvidia/vllm:${LATEST_VLLM_VERSION}
docker run -d --gpus all -p 8000:8000 --name vllm-server \
  nvcr.io/nvidia/vllm:${LATEST_VLLM_VERSION} \
  vllm serve ${HF_MODEL_HANDLE}

Some model families need a different image: DiffusionGemma uses vllm/vllm-openai:gemma, the Gemma 4 family uses vllm/vllm-openai:gemma4-cu130. An "agent-ready" recipe serves nvidia/Qwen3.6-35B-A3B-NVFP4 with a long flag list tuned for tool-calling agents (MTP speculative decoding, prefix caching, --tool-call-parser qwen3_xml) — this is what the agents on spark Hermes playbook builds on.

For models too large for one Spark's 128GB, the playbook documents two- and multi-Spark Ray clusters over the QSFP interface:

wget https://raw.githubusercontent.com/vllm-project/vllm/51c1ee9b7c8acbba4899a8ebffd390685d171946/examples/ray_serving/run_cluster.sh
export MN_IF_NAME=enp1s0f1np1
export VLLM_HOST_IP=$(ip -4 addr show $MN_IF_NAME | grep -oP '(?<=inet\s)\d+(\.\d+){3}')
bash run_cluster.sh $VLLM_IMAGE $VLLM_HOST_IP --head ~/.cache/huggingface \
  -e VLLM_HOST_IP=$VLLM_HOST_IP -e UCX_NET_DEVICES=$MN_IF_NAME -e NCCL_SOCKET_IFNAME=$MN_IF_NAME \
  -e MASTER_ADDR=$VLLM_HOST_IP

A worker node joins with --worker and HEAD_NODE_IP pointed at the head's QSFP address; then vllm serve ... --tensor-parallel-size 2 --distributed-executor-backend ray splits the model. See clustering for the underlying QSFP/switch network setup this depends on.

Key Parameters

• --gpu-memory-utilization — fraction of GPU memory for weights + KV cache (0.4–0.95 range across the playbook's recipes depending on model size and headroom needed).
• --tensor-parallel-size — 1 for single-Spark, 2+ across a Ray cluster; must match node count.
• --kv-cache-dtype fp8 — halves KV-cache memory, common on agent-ready and multi-node recipes.
• --speculative-config '{"method":"mtp", ...}' — MTP speculative decoding for compatible checkpoints (e.g. Qwen3.6-35B-A3B).
• --trust-remote-code — required for Phi-4-multimodal-instruct and other models shipping custom modeling code.
• --enable-auto-tool-choice + --tool-call-parser <model> — OpenAI-style function calling, needed for agentic use (see agents on spark).

When To Use

Reach for vLLM on Spark when throughput/concurrency matters more than build simplicity, when a model needs multi-Spark tensor parallelism to fit at all, or when the workload is agentic and needs vLLM's mature tool-calling parser ecosystem — NVIDIA's own Hermes Agent playbook standardizes on it. For a simpler single-process server, prefer llama cpp on spark; for zero-fuss model management, ollama on spark. See stack picker.

Risks & Pitfalls

• NVFP4 checkpoints are not a free lunch on Blackwell. A confidence-medium (single-issue, non-Spark but same Blackwell/NVFP4 ecosystem) report found nvidia/Qwen3.5-397B-A17B-NVFP4 scoring ~0.35 on GSM8K vs. 0.90 for the FP8 original — a large accuracy regression that persisted across TRITON_ATTN/FLASHINFER backends and --kv-cache-dtype settings. Validate accuracy on your own eval before trusting an NVFP4 quant's throughput/memory wins at face value, especially for large MoE models.
• Consumer/workstation Blackwell GPUs (sm_120, not Spark's sm_121a) have hit outright failures loading NVFP4 checkpoints and CUBLAS errors during GEMM calls (confidence medium, reported on RTX 5090 and RTX PRO 6000 Blackwell, not confirmed on GB10) — if a CUBLAS_STATUS_INVALID_VALUE or NVFP4 load failure appears, check the vLLM/CUDA/driver version triple against what NVIDIA has validated for GB10 before assuming the model itself is broken.
• Streaming responses can arrive missing or scrambled tokens with some large MoE models (reported with GPT-OSS 120B, confidence medium, non-Spark hardware) while non-streaming and --enforce-eager both produce correct output at a speed cost — if a served agent sees garbled streamed output, try disabling streaming or --enforce-eager as a diagnostic step before filing it as a model problem.
• Tool-calling reliability with Gemma 4 + coding agents has been reported as inconsistent (confidence medium, single issue, Blackwell workstation GPUs) — worth smoke-testing --tool-call-parser gemma4 end-to-end with your actual agent client before depending on it for unattended agentic workflows (see agents on spark).
• Gated models (Llama 3.3 70B, FLUX, etc.) require accepting the license on Hugging Face and authenticating with hf auth login inside the container so the token lands in the mounted cache — a common source of "cannot access gated repo" errors that look like network problems.
• Multi-Spark clusters: closing the terminal running run_cluster.sh tears down the container (it has an EXIT trap) — always run cluster nodes inside tmux/screen.
• Standard UMA caveat applies (sudo sh -c 'sync; echo 3 > /proc/sys/vm/drop_caches') — see unified memory and arm64.

Related Concepts

• clustering — QSFP/switch network setup underlying multi-Spark vLLM
• unified memory and arm64 — memory model implications for large-model serving
• agents on spark — Hermes Agent's vLLM-backed "agent-ready" recipe
• llama cpp on spark / ollama on spark — simpler alternatives
• troubleshooting casebook — cross-stack NVFP4/Blackwell failure patterns
• stack picker — choosing among inference stacks on Spark
• playbooks catalog — full list of official Spark playbooks

Sources

• raw/github_doc-nvidia-vllm-readme-md.md
• raw/github_issue-vllm-bug-5090-cannot-run-qwen3-30b-a3b-nvfp4.md
• raw/github_issue-vllm-bug-qwen3-5-nvfp4-checkpoint-has-poor-accuracy.md
• raw/github_issue-vllm-bug-qwen3-vl-235b-a22b-instruct-grounding-accuracy-issue-in-.md
• raw/github_issue-vllm-bug-runtimeerror-cuda-error-cublas-status-invalid-value-when.md
• raw/github_issue-vllm-bug-streaming-true-causes-missing-or-scrambled-tokens-with-g.md
• raw/github_issue-vllm-bug-vllm-gemma-4-claude-code-tool-calling-problems.md