--- title: "vLLM on Spark" type: concept tags: [inference, vllm, arm64, clustering, foundational, well-established] created: 2026-07-18 updated: 2026-07-18 sources: ["raw/github_doc-nvidia-vllm-readme-md.md", "raw/github_issue-vllm-bug-5090-cannot-run-qwen3-30b-a3b-nvfp4.md", "raw/github_issue-vllm-bug-qwen3-5-nvfp4-checkpoint-has-poor-accuracy.md", "raw/github_issue-vllm-bug-qwen3-vl-235b-a22b-instruct-grounding-accuracy-issue-in-.md", "raw/github_issue-vllm-bug-runtimeerror-cuda-error-cublas-status-invalid-value-when.md", "raw/github_issue-vllm-bug-streaming-true-causes-missing-or-scrambled-tokens-with-g.md", "raw/github_issue-vllm-bug-vllm-gemma-4-claude-code-tool-calling-problems.md"] confidence: high dgx_os_version: 7.5.0 --- ## Definition vLLM is a high-throughput LLM serving engine built around PagedAttention and continuous batching, exposed through an OpenAI-compatible API. General vLLM concepts (PagedAttention internals, scheduler design, the broader flag surface) are covered in depth by the dedicated vLLM wiki in this fleet. On DGX Spark the official playbook's Spark-specific content is: which prebuilt NGC/custom containers actually work on GB10's ARM64 + Blackwell combination, the NVFP4/FP8 quantized model catalog NVIDIA has validated for it, and how to cluster two or more Sparks over QSFP with Ray for models too large for one unit. Note: a separate `station-vllm` playbook exists in the same source tree but targets **DGX Station** (GB300, 252GB HBM3e) — a different, larger product. It is not covered here; do not conflate its container tags (e.g. `26.01-py3`, `--cpu-offload-gb`) with the Spark playbook's. ## How It Works The Spark playbook is Docker-first. After adding the user to the `docker` group, a model is served from the NVIDIA NGC container: ```bash export HF_TOKEN="your_huggingface_token" export LATEST_VLLM_VERSION= ## e.g. 26.05.post1-py3 export HF_MODEL_HANDLE= ## e.g. openai/gpt-oss-20b docker pull nvcr.io/nvidia/vllm:${LATEST_VLLM_VERSION} docker run -d --gpus all -p 8000:8000 --name vllm-server \ nvcr.io/nvidia/vllm:${LATEST_VLLM_VERSION} \ vllm serve ${HF_MODEL_HANDLE} ``` Some model families need a different image: DiffusionGemma uses `vllm/vllm-openai:gemma`, the Gemma 4 family uses `vllm/vllm-openai:gemma4-cu130`. An "agent-ready" recipe serves `nvidia/Qwen3.6-35B-A3B-NVFP4` with a long flag list tuned for tool-calling agents (MTP speculative decoding, prefix caching, `--tool-call-parser qwen3_xml`) — this is what the [[concepts/agents-on-spark]] Hermes playbook builds on. For models too large for one Spark's 128GB, the playbook documents **two- and multi-Spark Ray clusters** over the QSFP interface: ```bash wget https://raw.githubusercontent.com/vllm-project/vllm/51c1ee9b7c8acbba4899a8ebffd390685d171946/examples/ray_serving/run_cluster.sh export MN_IF_NAME=enp1s0f1np1 export VLLM_HOST_IP=$(ip -4 addr show $MN_IF_NAME | grep -oP '(?<=inet\s)\d+(\.\d+){3}') bash run_cluster.sh $VLLM_IMAGE $VLLM_HOST_IP --head ~/.cache/huggingface \ -e VLLM_HOST_IP=$VLLM_HOST_IP -e UCX_NET_DEVICES=$MN_IF_NAME -e NCCL_SOCKET_IFNAME=$MN_IF_NAME \ -e MASTER_ADDR=$VLLM_HOST_IP ``` A worker node joins with `--worker` and `HEAD_NODE_IP` pointed at the head's QSFP address; then `vllm serve ... --tensor-parallel-size 2 --distributed-executor-backend ray` splits the model. See [[concepts/clustering]] for the underlying QSFP/switch network setup this depends on. ## Key Parameters - `--gpu-memory-utilization` — fraction of GPU memory for weights + KV cache (0.4–0.95 range across the playbook's recipes depending on model size and headroom needed). - `--tensor-parallel-size` — 1 for single-Spark, 2+ across a Ray cluster; must match node count. - `--kv-cache-dtype fp8` — halves KV-cache memory, common on agent-ready and multi-node recipes. - `--speculative-config '{"method":"mtp", ...}'` — MTP speculative decoding for compatible checkpoints (e.g. Qwen3.6-35B-A3B). - `--trust-remote-code` — required for Phi-4-multimodal-instruct and other models shipping custom modeling code. - `--enable-auto-tool-choice` + `--tool-call-parser ` — OpenAI-style function calling, needed for agentic use (see [[concepts/agents-on-spark]]). ## When To Use Reach for vLLM on Spark when throughput/concurrency matters more than build simplicity, when a model needs multi-Spark tensor parallelism to fit at all, or when the workload is agentic and needs vLLM's mature tool-calling parser ecosystem — NVIDIA's own [[concepts/agents-on-spark|Hermes Agent playbook]] standardizes on it. For a simpler single-process server, prefer [[concepts/llama-cpp-on-spark]]; for zero-fuss model management, [[concepts/ollama-on-spark]]. See [[syntheses/stack-picker]]. ## Risks & Pitfalls - **NVFP4 checkpoints are not a free lunch on Blackwell.** A confidence-medium (single-issue, non-Spark but same Blackwell/NVFP4 ecosystem) report found `nvidia/Qwen3.5-397B-A17B-NVFP4` scoring ~0.35 on GSM8K vs. 0.90 for the FP8 original — a large accuracy regression that persisted across `TRITON_ATTN`/`FLASHINFER` backends and `--kv-cache-dtype` settings. Validate accuracy on your own eval before trusting an NVFP4 quant's throughput/memory wins at face value, especially for large MoE models. - **Consumer/workstation Blackwell GPUs (sm_120, not Spark's sm_121a) have hit outright failures loading NVFP4 checkpoints** and CUBLAS errors during GEMM calls (confidence medium, reported on RTX 5090 and RTX PRO 6000 Blackwell, not confirmed on GB10) — if a `CUBLAS_STATUS_INVALID_VALUE` or NVFP4 load failure appears, check the vLLM/CUDA/driver version triple against what NVIDIA has validated for GB10 before assuming the model itself is broken. - **Streaming responses can arrive missing or scrambled tokens** with some large MoE models (reported with GPT-OSS 120B, confidence medium, non-Spark hardware) while non-streaming and `--enforce-eager` both produce correct output at a speed cost — if a served agent sees garbled streamed output, try disabling streaming or `--enforce-eager` as a diagnostic step before filing it as a model problem. - **Tool-calling reliability with Gemma 4 + coding agents has been reported as inconsistent** (confidence medium, single issue, Blackwell workstation GPUs) — worth smoke-testing `--tool-call-parser gemma4` end-to-end with your actual agent client before depending on it for unattended agentic workflows (see [[concepts/agents-on-spark]]). - Gated models (Llama 3.3 70B, FLUX, etc.) require accepting the license on Hugging Face and authenticating with `hf auth login` **inside the container** so the token lands in the mounted cache — a common source of "cannot access gated repo" errors that look like network problems. - Multi-Spark clusters: closing the terminal running `run_cluster.sh` tears down the container (it has an `EXIT` trap) — always run cluster nodes inside `tmux`/`screen`. - Standard UMA caveat applies (`sudo sh -c 'sync; echo 3 > /proc/sys/vm/drop_caches'`) — see [[concepts/unified-memory-and-arm64]]. ## Related Concepts - [[concepts/clustering]] — QSFP/switch network setup underlying multi-Spark vLLM - [[concepts/unified-memory-and-arm64]] — memory model implications for large-model serving - [[concepts/agents-on-spark]] — Hermes Agent's vLLM-backed "agent-ready" recipe - [[concepts/llama-cpp-on-spark]] / [[concepts/ollama-on-spark]] — simpler alternatives - [[syntheses/troubleshooting-casebook]] — cross-stack NVFP4/Blackwell failure patterns - [[syntheses/stack-picker]] — choosing among inference stacks on Spark - [[entities/playbooks-catalog]] — full list of official Spark playbooks ## Sources - raw/github_doc-nvidia-vllm-readme-md.md - raw/github_issue-vllm-bug-5090-cannot-run-qwen3-30b-a3b-nvfp4.md - raw/github_issue-vllm-bug-qwen3-5-nvfp4-checkpoint-has-poor-accuracy.md - raw/github_issue-vllm-bug-qwen3-vl-235b-a22b-instruct-grounding-accuracy-issue-in-.md - raw/github_issue-vllm-bug-runtimeerror-cuda-error-cublas-status-invalid-value-when.md - raw/github_issue-vllm-bug-streaming-true-causes-missing-or-scrambled-tokens-with-g.md - raw/github_issue-vllm-bug-vllm-gemma-4-claude-code-tool-calling-problems.md