# NVIDIA DGX Spark — full corpus # LLM Wiki An open-source template for building LLM-powered knowledge bases, following [Andrej Karpathy's "LLM Wiki" pattern](https://gist.github.com/karpathy/442a6bf555914893e9891c11519de94f). You provide raw sources. The LLM reads them, writes structured wiki pages, cross-links everything, and maintains it over time. You never edit the wiki directly — you curate sources and ask questions. ## How It Works The system has three layers: ``` raw/ Sources you collect (articles, transcripts, notes, PDFs) wiki/ LLM-written & maintained pages (summaries, concepts, entities, syntheses) CLAUDE.md Schema that tells the LLM how to structure everything ``` Three operations drive the workflow: | Operation | Trigger | What happens | |-----------|---------|--------------| | **Ingest** | "ingest raw/my-source.txt" | LLM reads the source, creates a summary page, creates/updates concept and entity pages, adds cross-links, updates the index and log | | **Query** | Ask any question | LLM searches the wiki, synthesizes an answer with citations, optionally creates a synthesis page for novel insights | | **Lint** | "lint" or "health check" | LLM audits all pages for orphans, contradictions, missing links, incomplete sections, and low-confidence claims — fixes what it can, reports the rest | ## Quick Start 1. **Clone this repo** ```bash git clone https://github.com/YOUR_USERNAME/llm-wiki.git my-knowledge-base cd my-knowledge-base ``` 2. **Customize CLAUDE.md** for your domain - Update the Purpose section with your topic - Replace the placeholder tagging taxonomy with your own categories - Adjust confidence level descriptions if needed - Everything else (workflows, page formats, linking rules) works as-is 3. **Drop sources into `raw/`** - Text files, transcripts, articles, notes — any plain text - These are immutable once added; the LLM never modifies them 4. **Tell the LLM to ingest** ``` ingest raw/my-first-source.txt ``` The LLM will create summary pages, concept pages, entity pages, cross-links, and update the index. 5. **Ask questions** ``` What are the key differences between X and Y? ``` The LLM answers from the wiki, citing specific pages. 6. **Run health checks** ``` lint ``` The LLM audits the wiki and fixes issues. ## Directory Structure ``` . ├── CLAUDE.md # Schema — the LLM's instructions ├── raw/ # Your source documents (immutable) └── wiki/ ├── index.md # Master catalog of all pages ├── log.md # Append-only activity log ├── dashboard.md # Dataview dashboard (Obsidian) ├── analytics.md # Charts View analytics (Obsidian) ├── flashcards.md # Spaced repetition cards ├── summaries/ # One page per source document ├── concepts/ # Concept and framework pages ├── entities/ # People, tools, organizations, etc. ├── syntheses/ # Cross-cutting analyses and comparisons ├── journal/ # Research/session journal entries │ └── template.md # Journal entry template └── presentations/ # Marp slide decks ``` ## Enhancements This template includes several extras beyond the core wiki pattern: ### Dataview Dashboard (`wiki/dashboard.md`) Live queries that surface low-confidence pages, recent updates, concepts by tag, and pages with the most sources. Requires the [Dataview](https://github.com/blacksmithgu/obsidian-dataview) Obsidian plugin. ### Charts View Analytics (`wiki/analytics.md`) Visual analytics with pie charts, bar charts, and word clouds. Requires the [Charts View](https://github.com/caronchen/obsidian-chartsview-plugin) Obsidian plugin. ### Mermaid Diagrams Use Mermaid code blocks in any wiki page to create flowcharts, sequence diagrams, or concept maps. Native support in Obsidian and GitHub. ### Marp Slides (`wiki/presentations/`) Create slide decks from markdown using [Marp](https://marp.app/). Drop presentation files in this directory. ### Research Journal (`wiki/journal/`) Track your research sessions, experiments, or applied work with the included template. The LLM can reference journal entries when answering queries. ### Spaced Repetition (`wiki/flashcards.md`) Flashcards in the format used by the [Spaced Repetition](https://github.com/st3v3nmw/obsidian-spaced-repetition) Obsidian plugin. Ask the LLM to generate flashcards from any wiki page. ### MCP Server This repo works with Claude Code's MCP server capabilities. Point an MCP-compatible client at this repo and the LLM can read/write the wiki programmatically. ## Customizing for Your Domain The schema in `CLAUDE.md` is domain-agnostic. To adapt it: 1. **Purpose** — Describe your knowledge domain in one paragraph 2. **Tagging taxonomy** — Replace placeholder categories with your own (e.g., for a cooking KB: `cuisine`, `technique`, `ingredient`, `equipment`) 3. **Confidence levels** — Adjust the descriptions to match your domain's evidence standards 4. **Entity types** — Update the entity page description to match what entities mean in your domain (people, tools, companies, etc.) 5. **Journal template** — Customize `wiki/journal/template.md` for your workflow Everything else — page format, linking conventions, workflows, rules — is universal and works across domains. ## Example Domains This template works for any knowledge-intensive topic: - **Research notes** — papers, experiments, methodologies - **Book analysis** — themes, characters, author techniques - **Competitive analysis** — companies, products, market trends - **Course notes** — lectures, readings, key concepts - **Personal development** — frameworks, habits, book summaries - **Technical documentation** — APIs, architectures, design patterns - **Hobby deep-dives** — any subject you want to master ## License MIT --- title: "Knowledge Base Index" type: index updated: 2026-07-18 dgx_os_version: "7.5.0" --- # Knowledge Base Index Master catalog of all wiki pages. Every page in the wiki must have an entry here. ## Concepts | Page | Tags | Confidence | Updated | |------|------|------------|---------| | [[concepts/agents-on-spark]] | agents, inference, arm64, foundational, emerging | high | 2026-07-18 | | [[concepts/clustering]] | clustering, advanced, well-established | high | 2026-07-18 | | [[concepts/comfyui-and-image-gen]] | inference, comfyui, arm64, fine-tuning, foundational, well-established | medium | 2026-07-18 | | [[concepts/containers-and-ngc]] | containers, setup, well-established | high | 2026-07-18 | | [[concepts/dgx-os-and-updates]] | dgx-os, setup, well-established | high | 2026-07-18 | | [[concepts/fine-tuning-and-quantization]] | fine-tuning, unified-memory, arm64, advanced, well-established | high | 2026-07-18 | | [[concepts/first-boot-and-setup]] | setup, foundational, well-established | high | 2026-07-18 | | [[concepts/llama-cpp-on-spark]] | inference, llama-cpp, arm64, foundational, well-established | high | 2026-07-18 | | [[concepts/lm-studio-on-spark]] | inference, lm-studio, arm64, foundational, well-established | medium | 2026-07-18 | | [[concepts/ollama-on-spark]] | inference, ollama, arm64, foundational, well-established | high | 2026-07-18 | | [[concepts/overview-and-hardware]] | setup, foundational, well-established | high | 2026-07-18 | | [[concepts/system-operations]] | setup, dgx-os, well-established | high | 2026-07-18 | | [[concepts/unified-memory-and-arm64]] | unified-memory, arm64, foundational, well-established | high | 2026-07-18 | | [[concepts/vllm-on-spark]] | inference, vllm, arm64, clustering, foundational, well-established | high | 2026-07-18 | ## Entities | Page | Tags | Updated | |------|------|---------| | [[entities/playbooks-catalog]] | setup, inference, agents, fine-tuning, foundational, well-established | 2026-07-18 | ## Summaries | Page | Source | Key Topics | Created | |------|--------|------------|---------| | [[summaries/release-digest]] | `raw/web_community-dgx-spark-release-notes-dgx-spark-user-guide.md` | DGX Spark Release Notes Digest | 2026-07-18 | ## Syntheses | Page | Pages Compared | Created | |------|----------------|---------| | [[syntheses/stack-picker]] | inference, llama-cpp, ollama, vllm, lm-studio, nim, unified-memory | 2026-07-18 | | [[syntheses/troubleshooting-casebook]] | troubleshooting, unified-memory, arm64, llama-cpp, ollama, vllm | 2026-07-18 | ## XL Edition (Pro) Per-item reference depth beyond this edition, in the gated XL layer (34 pages): one operational reference per official playbook with full verbatim command sequences (30 — inference, agents, connectivity, fine-tuning/data), the benchmarking methodology with measured numbers, and three deep per-project troubleshooting casebooks (llama.cpp / Ollama / vLLM) mined from upstream issue trackers with hardware-provenance labels. Agents without Pro access should treat those areas as "covered in the XL edition" rather than out of scope. ## Statistics - **Total pages**: 18 - **Concepts**: 14 - **Entities**: 1 - **Summaries**: 1 - **Syntheses**: 2 - **Sources ingested**: 0 - **High confidence**: 14 - **Medium confidence**: 4 - **Low confidence**: 0 --- title: "Agents on Spark" type: concept tags: [agents, inference, arm64, foundational, emerging] created: 2026-07-18 updated: 2026-07-18 sources: ["raw/github_doc-nvidia-hermes-agent-readme-md.md", "raw/github_doc-nvidia-cli-coding-agent-readme-md.md", "raw/github_doc-nvidia-multi-agent-chatbot-readme-md.md", "raw/github_doc-nvidia-open-webui-readme-md.md", "raw/github_doc-nvidia-live-vlm-webui-readme-md.md"] confidence: high dgx_os_version: 7.5.0 --- ## Definition "Agents on Spark" covers the family of official playbooks that run an autonomous or semi-autonomous AI agent — a persistent assistant, a CLI coding agent, a multi-agent orchestration system, or a chat/vision front end — fully on-device, using Spark's 128GB unified memory to serve the underlying model locally rather than calling a cloud API. The common thread across all of them is a local OpenAI-compatible model server ([[concepts/ollama-on-spark]], [[concepts/vllm-on-spark]], or [[concepts/llama-cpp-on-spark]]) plus an agent framework or UI layered on top. ## How It Works **Hermes Agent** (Nous Research) is the most complete example: a self-improving TUI agent with a Telegram/Discord/Slack messaging gateway, persistent skills, and built-in cron. The playbook serves the model with vLLM's "agent-ready" `nvidia/Qwen3.6-35B-A3B-NVFP4` recipe (see [[concepts/vllm-on-spark]]) on `http://localhost:8000/v1`, then installs Hermes and points it at that endpoint: ```bash curl -fsSL https://raw.githubusercontent.com/NousResearch/hermes-agent/main/scripts/install.sh | bash hermes config set model.provider custom hermes config set model.base_url http://localhost:8000/v1 hermes config set model.default nvidia/Qwen3.6-35B-A3B-NVFP4 ``` The gateway installs as a systemd service (`sudo "$(which hermes)" gateway install --system --run-as-user "$USER"`) so it survives reboots and answers Telegram messages without an interactive session. Switching models later is just re-serving vLLM with a new handle and re-running `hermes model` (or the equivalent `hermes config set` commands). **CLI Coding Agent** wires Claude Code, OpenCode, or Codex CLI to a local Ollama model via Ollama's built-in launch method — no environment variables or provider config files: ```bash ollama pull qwen3.6 ollama launch claude --model qwen3.6 ## or: ollama launch opencode / ollama launch codex ``` **Multi-Agent Chatbot** goes further: a supervisor agent (gpt-oss-120B) orchestrates coding, RAG, and image-understanding sub-agents, each served via llama.cpp/TensorRT-LLM containers and wired together with MCP servers, all launched with `docker compose -f docker-compose.yml -f docker-compose-models.yml up -d --build`. It uses ~120 of Spark's 128GB by default. **Open WebUI** and **Live VLM WebUI** are the chat/vision front ends most of the other agent stacks plug into — Open WebUI bundles Ollama and a browser chat UI (optionally tunneled through NVIDIA Sync, mirroring the [[concepts/ollama-on-spark]] pattern); Live VLM WebUI streams a webcam over WebRTC to any OpenAI-compatible VLM backend (Ollama by default) for real-time scene description, OCR, or safety monitoring at 1–2s/frame on GB10. ## Key Parameters - **Allowed user IDs** (Hermes) — leaving this blank during Telegram setup opens the bot to anyone who finds it; always restrict it for anything beyond a demo. - **Terminal backend = local** (Hermes) — runs shell commands directly on the Spark; the security tradeoff of an agent with local command execution is the main risk surface for this whole category. - `ollama launch --model ` — the one-command wiring point for CLI Coding Agent; requires Ollama v0.15+. - Supervisor model size (Multi-Agent Chatbot) — gpt-oss-120B by default; drop to gpt-oss-20B if other workloads need the remaining ~8GB headroom. - Live VLM WebUI's **Frame Processing Interval** and **Max Tokens** — the two knobs that trade responsiveness for GPU load in real-time vision loops. ## When To Use Reach for Hermes Agent when the goal is a persistent, remotely-reachable personal assistant with memory and scheduled tasks. Use CLI Coding Agent when the goal is simply "point my existing coding-agent CLI at a local model" with minimal setup. Use Multi-Agent Chatbot as a reference architecture for building supervisor/sub-agent systems with MCP tools. Use Open WebUI as a general-purpose chat front end, and Live VLM WebUI specifically for real-time vision/webcam use cases. See [[syntheses/stack-picker]] for how the underlying model-serving choice (Ollama vs vLLM vs llama.cpp) interacts with each of these. ## Risks & Pitfalls - **Agents that execute commands and reach external services are a real security surface, not a theoretical one.** The Hermes playbook is explicit: unrestricted Telegram bots can be misused by anyone who finds them, and a model endpoint bound beyond `localhost` can be abused. Restrict allowed user IDs, keep model endpoints local-only, and periodically review gateway logs (`sudo journalctl -u -e`). - **Outbound HTTPS to messaging platforms is sometimes blocked on corporate/lab networks**, producing a working local install with a bot that silently never responds — verify with `curl -sS --connect-timeout 10 -o /dev/null -w "HTTP %{http_code}\n" https://api.telegram.org/` before debugging further. - **`sudo` does not inherit a user's `PATH`**, so `sudo hermes ...` commands routinely fail with "command not found" during gateway install/uninstall — invoke the full path (`sudo "$(which hermes)" ...`) instead. - **Tool-calling reliability varies by model/server combination.** The CLI Coding Agent playbook notes that a direct Claude Code setup against a raw Anthropic-compatible Ollama endpoint can produce prose without ever editing files — `ollama launch claude` with Qwen3.6 is the path verified to actually emit tool calls. See also vLLM's Gemma-4 tool-calling caveat in [[concepts/vllm-on-spark]]. - **CUDA multi-GPU topology issues can make Ollama crash with `context is destroyed`** on systems it perceives as mixed-GPU; pinning with `CUDA_VISIBLE_DEVICES=0` resolves it. - Multi-Agent Chatbot's ~120GB default footprint leaves little headroom — check `nvidia-smi` for other workloads before launching, or drop to the 20B supervisor variant. - Live VLM WebUI over SSH port forwarding does not work for the webcam feed itself — WebRTC needs direct UDP connectivity that SSH tunnels (TCP-only) cannot provide; access the web UI directly from a browser on the same network instead. - Standard UMA caveat applies across all of these: flush the page cache (`sudo sh -c 'sync; echo 3 > /proc/sys/vm/drop_caches'`) under unexpected memory pressure — see [[concepts/unified-memory-and-arm64]]. ## Related Concepts - [[concepts/vllm-on-spark]] — the model server behind Hermes Agent's agent-ready recipe - [[concepts/ollama-on-spark]] — the model server behind CLI Coding Agent, Open WebUI, and Live VLM WebUI - [[concepts/llama-cpp-on-spark]] — one of the servers behind Multi-Agent Chatbot's sub-agents - [[concepts/unified-memory-and-arm64]] — why Spark can run a 120B supervisor plus sub-agents at once - [[syntheses/troubleshooting-casebook]] — cross-stack gateway/PATH/tool-calling failure patterns - [[syntheses/stack-picker]] — choosing among inference stacks that these agent playbooks sit on top of - [[entities/playbooks-catalog]] — full list of official Spark playbooks ## Sources - raw/github_doc-nvidia-hermes-agent-readme-md.md - raw/github_doc-nvidia-cli-coding-agent-readme-md.md - raw/github_doc-nvidia-multi-agent-chatbot-readme-md.md - raw/github_doc-nvidia-open-webui-readme-md.md - raw/github_doc-nvidia-live-vlm-webui-readme-md.md --- title: "Clustering Multiple DGX Sparks" type: concept tags: [clustering, advanced, well-established] created: 2026-07-18 updated: 2026-07-18 sources: ["raw/web_community-spark-stacking-dgx-spark-user-guide.md", "raw/github_doc-nvidia-connect-two-sparks-readme-md.md", "raw/github_doc-nvidia-connect-three-sparks-readme-md.md", "raw/github_doc-nvidia-multi-sparks-through-switch-readme-md.md", "raw/github_doc-nvidia-connect-two-sparks-assets-performance-benchmarking-gu.md", "raw/github_doc-nvidia-nccl-readme-md.md"] confidence: high dgx_os_version: 7.5.0 --- ## Definition Clustering (NVIDIA's docs call it "Spark Stacking") is connecting two or more DGX Spark units over their rear ConnectX-7 QSFP ports into a high-speed compute cluster, enabling distributed workloads — larger models (up to 405B parameters across two units, versus 200B on one — see [[concepts/overview-and-hardware]]), faster training, and multi-node inference — using MPI for CPU-side coordination and NCCL for GPU-accelerated collective communication. ## How It Works **Physical connection.** Each Spark has two ConnectX-7 (QSFP/CX7) ports on the rear panel. Cables connect matching ports between units (e.g., both "left" ports); the connector's pull-tab must face up, and it should seat without force — forcing a misaligned connector can damage the port. Two units need one cable (direct 200GbE); three units use three cables in a ring topology (Node1 Port0→Node2 Port1, Node2 Port0→Node3 Port1, Node3 Port0→Node1 Port1); four or more units go through a managed QSFP switch (at least 4x QSFP56-DD ports, 200Gbps each), with all switch ports placed in the same layer-2 bridge. **Network configuration.** After cabling, `ibdev2netdev` shows which logical interfaces (e.g. `enp1s0f1np1`) are "Up" — only the `enp1<...>` interfaces are used; `enP2p1<...>` counterparts for the same physical port are disregarded per the two/three-Spark playbooks. Netplan then assigns IPs, either automatically via link-local addressing: ```yaml network: version: 2 ethernets: enp1s0f0np0: link-local: [ ipv4 ] enp1s0f1np1: link-local: [ ipv4 ] ``` or with explicit static addresses per node (each playbook gives node-specific YAML). For switch-connected clusters, a `spark_cluster_setup` script automates interface config, SSH, connectivity verification, and NCCL bandwidth testing (`--run-setup` / `--run-nccl-test` arguments) — same-username prerequisite across all nodes applies everywhere (`whoami`; create a shared `nvidia` user with `sudo useradd -m nvidia && sudo usermod -aG sudo nvidia` if needed). **Passwordless SSH.** The `discover-sparks` script automates bidirectional key distribution across all discovered nodes; a manual fallback uses `ssh-copy-id -i ~/.ssh/id_rsa.pub @` per node pair. Basic connectivity is checked with `ssh hostname` across all nodes. **NCCL.** Built from source with Blackwell arch support: ```bash sudo apt-get update && sudo apt-get install -y libopenmpi-dev git clone -b v2.28.9-1 https://github.com/NVIDIA/nccl.git ~/nccl/ cd ~/nccl/ make -j src.build NVCC_GENCODE="-gencode=arch=compute_121,code=sm_121" export CUDA_HOME="/usr/local/cuda" export MPI_HOME="/usr/lib/aarch64-linux-gnu/openmpi" export NCCL_HOME="$HOME/nccl/build/" export LD_LIBRARY_PATH="$NCCL_HOME/lib:$CUDA_HOME/lib64/:$MPI_HOME/lib:$LD_LIBRARY_PATH" ``` NCCL tests are built with `git clone https://github.com/NVIDIA/nccl-tests.git ~/nccl-tests/ && cd ~/nccl-tests/ && make MPI=1`, then run across nodes with mpirun, e.g.: ```bash export UCX_NET_DEVICES=enp1s0f1np1 export NCCL_SOCKET_IFNAME=enp1s0f1np1 export OMPI_MCA_btl_tcp_if_include=enp1s0f1np1 mpirun -np 2 -H :1,:1 \ --mca plm_rsh_agent "ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no" \ -x LD_LIBRARY_PATH=$LD_LIBRARY_PATH \ $HOME/nccl-tests/build/all_gather_perf ``` The release notes note NCCL v2.30u1 added ring-topology support for three-Spark clusters, and NVIDIA Sync's Cluster Assistant (Settings page) can now help connect up to three devices without a switch, or up to four with one (see [[concepts/system-operations]]). **Performance benchmarking.** The dual-Spark performance benchmarking guide validates the fabric at two layers: GPU collective bandwidth via NCCL, and raw RDMA fabric bandwidth using `perftest` tools (`sudo apt install perftest`) over RoCE — `ib_write_bw -d -i 1 -p -F --report_gbits --run_infinitely` on both ends, yielding per-link throughput (observed example: ~92.6 + ~97.3 Gbps = ~189.85 Gbps aggregate across two links) and `ib_write_lat` / `ib_read_lat` for one-way RDMA latency (measured per-link, not aggregated). The same guide covers single-Spark benchmarking of TensorRT-LLM, vLLM, SGLang, and Llama.cpp for comparison — see [[concepts/vllm-on-spark]] and [[concepts/llama-cpp-on-spark]]. ## Key Parameters - Interconnect: ConnectX-7 QSFP, 200GbE per port (link speed must be manually verified/forced to 200000Mb/s on switch setups via `ethtool | grep Speed`) - NCCL version referenced: v2.28.3 (Spark Stacking overview) / v2.28.9-1 (playbook) / v2.30u1 (three-Spark ring, per release notes) - NCCL build arch flag: `arch=compute_121,code=sm_121` - Netplan config file: `/etc/netplan/40-cx7.yaml`, `chmod 600` - Max scale demonstrated: 2 (direct), 3 (ring), 4+ (via switch, extensible) - Next-step orchestration: Slurm or Kubernetes for job scheduling, Singularity or Docker for containerized execution ## When To Use Cluster multiple Sparks when a single unit's 128 GB unified memory or compute is insufficient — e.g., serving/fine-tuning models beyond ~200B parameters (405B needs two units), or when NCCL-accelerated distributed training/inference throughput is required. Use direct two/three-unit cabling for small, dedicated clusters; use a switch for four or more units or when future expansion is expected. ## Risks & Pitfalls - Forcing a QSFP connector that doesn't seat smoothly can damage the port — verify tab orientation first. - Assigning two CX7 logical interfaces to the same subnet causes routing ambiguity and NCCL communication failures — the switch playbook explicitly requires distinct subnets per interface (e.g. `192.168.100.x/24` and `192.168.101.x/24`). - Auto-negotiated switch link speed can silently fall back to 100Gbps instead of 200Gbps — always verify with `ethtool`. - All nodes must share a username, and all must be updated to the latest OS/firmware before clustering (stale nodes are a common cause of NCCL test failures with `libnccl.so.2: cannot open shared object file` errors). - mpirun hangs are usually SSH connectivity problems, not NCCL problems — test with a plain `mpirun -np 2 -H :1,:1 hostname` first. - Disregard the `enP2p1<...>` interface names in two/three-Spark direct setups (they duplicate the `enp1<...>` physical ports); switch setups use the opposite convention in one example — always confirm with `ibdev2netdev` rather than assuming interface names. ## Related Concepts - [[concepts/overview-and-hardware]] — the 405B dual-Spark model-size ceiling and QSFP ports - [[concepts/unified-memory-and-arm64]] — per-node memory model that clustering extends - [[concepts/system-operations]] — NVIDIA Sync Cluster Assistant for GUI-based clustering - [[concepts/vllm-on-spark]], [[concepts/llama-cpp-on-spark]] — inference stacks benchmarked in the performance guide - [[entities/playbooks-catalog]] — connect-two-sparks, connect-three-sparks, multi-sparks-through-switch, nccl playbooks ## Sources - raw/web_community-spark-stacking-dgx-spark-user-guide.md - raw/github_doc-nvidia-connect-two-sparks-readme-md.md - raw/github_doc-nvidia-connect-three-sparks-readme-md.md - raw/github_doc-nvidia-multi-sparks-through-switch-readme-md.md - raw/github_doc-nvidia-connect-two-sparks-assets-performance-benchmarking-gu.md - raw/github_doc-nvidia-nccl-readme-md.md --- title: "ComfyUI and Image Generation on Spark" type: concept tags: [inference, comfyui, arm64, fine-tuning, foundational, well-established] created: 2026-07-18 updated: 2026-07-18 sources: ["raw/github_doc-nvidia-comfy-ui-readme-md.md", "raw/github_doc-nvidia-flux-finetuning-readme-md.md", "raw/github_doc-nvidia-station-comfyui-readme-md.md"] confidence: medium dgx_os_version: 7.5.0 --- ## Definition ComfyUI is a node-based web application for diffusion-model image (and video) generation, run directly on the Spark's GPU with a browser UI at `:8188`. General ComfyUI usage (node graph mechanics, workflow JSON format) is covered by the dedicated ComfyUI wiki in this fleet. The Spark-specific content here is: installing it natively (not containerized) on aarch64 with CUDA 13.0 PyTorch, and the companion FLUX.1 Dreambooth LoRA fine-tuning playbook that trains custom concepts and feeds the LoRA back into ComfyUI — both exploiting Spark's 128GB unified memory to hold multiple large diffusion-model components (transformer, CLIP, T5, autoencoder) simultaneously. Note: a separate `station-comfyui` playbook targets **DGX Station** (GB300, 252GB HBM3e) with a heavier Docker-based, multi-model (FLUX/Wan/HunyuanVideo/Cosmos) video-generation setup — a different product with roughly double the memory. It is not the subject of this page, but its Dockerfile surfaced two aarch64 Python-packaging gotchas (below) that apply equally to any ARM64 ComfyUI install, including Spark's. ## How It Works The Spark ComfyUI playbook installs natively into a venv rather than a container: ```bash python3 -m venv comfyui-env source comfyui-env/bin/activate pip3 install torch torchvision --index-url https://download.pytorch.org/whl/cu130 git clone https://github.com/comfyanonymous/ComfyUI.git cd ComfyUI/ pip install -r requirements.txt cd models/checkpoints/ wget https://huggingface.co/Comfy-Org/stable-diffusion-v1-5-archive/resolve/main/v1-5-pruned-emaonly-fp16.safetensors cd ../../ python main.py --listen 0.0.0.0 ``` `--listen 0.0.0.0` is required for LAN access to the 8188 web UI; the default checkpoint (SD 1.5, ~2GB) is a bare minimum smoke test — real workloads load FLUX-class 12B+ models, which is where Spark's unified memory pool earns its keep versus a discrete-VRAM card. The **FLUX.1 Dreambooth LoRA fine-tuning** playbook builds two Docker images (inference and training) around the same idea: generate with the base FLUX.1-dev 12B model first (`Dockerfile.inference`, `launch_comfyui.sh`), then fine-tune multi-concept LoRAs on a small custom dataset (5–10 images per concept, defined in `flux_data/data.toml`) with `Dockerfile.train` / `launch_train.sh`, and finally load the trained LoRA weights back into a `finetuned_flux.json` ComfyUI workflow for inference combining multiple learned concepts in one prompt. A full training run for two concepts (a toy figure and a GPU) takes roughly 90 minutes; base-model 1024px generations take about 3 minutes each. ## Key Parameters - `--index-url https://download.pytorch.org/whl/cu130` — CUDA 13.0 wheel index; using the wrong CUDA-tagged wheel is the most common "torch not using GPU" failure on Spark. - `--listen 0.0.0.0` — binds ComfyUI's web server to all interfaces for remote browser access. - LoRA training dataset size: 5–10 images per concept is the playbook's working minimum; each concept gets its own `image_dir` and `class_tokens` entry in `data.toml`. - Trigger phrases (e.g. `tjtoy toy`, `sparkgpu gpu`) — appending a generic class token to a custom concept name measurably improves fine-tuning quality per the playbook's own guidance. ## When To Use Use ComfyUI on Spark for local, private image generation and workflow prototyping where the 128GB unified pool lets you load high-resolution (1024px+) diffusion pipelines with multiple text encoders without VRAM juggling. Add the FLUX fine-tuning playbook when the goal is teaching the model new characters/styles/objects via LoRA rather than just running inference. Neither is a general-purpose LLM serving stack — for that see [[concepts/vllm-on-spark]], [[concepts/ollama-on-spark]], or [[concepts/llama-cpp-on-spark]]. ## Risks & Pitfalls - **`onnxruntime-gpu` has no aarch64 wheel on PyPI** (confidence medium, sourced from the DGX Station Dockerfile but applicable to any ARM64 ComfyUI install including Spark) — custom nodes that depend on it (e.g. DWPose preprocessing) must fall back to CPU `onnxruntime`; expect a `DWPose: Onnxruntime not found ... switch to OpenCV with CPU device` message, which is informational, not an error — preprocessing still works, just slower. - **A bare `pip install torchaudio` on top of an NGC-optimized PyTorch build breaks with `undefined symbol: torch_dtype_float4_e2m1fn_x2`** (confidence medium, same Station-Dockerfile source) because NGC's custom NVFP4 ABI is incompatible with the PyPI torchaudio wheel — if a containerized ComfyUI build needs torchaudio, use an import-only stub instead of installing the real package; no core image-generation workflow needs it. - On the native (non-Docker) Spark playbook, PyTorch CUDA unavailability is almost always a CUDA-toolkit/wheel mismatch — verify `nvcc --version` matches the `cu130` index used for the pip install before assuming a driver problem. - FLUX.1-dev and other gated Hugging Face models require accepting the license and a valid `HF_TOKEN` — "cannot access gated repo" errors are a licensing/token issue, not a download failure. - Standard UMA caveat: flush the page cache (`sudo sh -c 'sync; echo 3 > /proc/sys/vm/drop_caches'`) between the inference and training containers if memory pressure appears despite apparent headroom — see [[concepts/unified-memory-and-arm64]]. ## Related Concepts - [[concepts/unified-memory-and-arm64]] — why Spark can hold multi-encoder diffusion pipelines that need VRAM juggling elsewhere - [[concepts/agents-on-spark]] — the multi-agent chatbot playbook's use of local model serving alongside image/vision workloads - [[concepts/vllm-on-spark]] / [[concepts/ollama-on-spark]] / [[concepts/llama-cpp-on-spark]] — LLM-side serving stacks, not image generation - [[syntheses/troubleshooting-casebook]] — cross-stack aarch64 packaging failures - [[entities/playbooks-catalog]] — full list of official Spark playbooks ## Sources - raw/github_doc-nvidia-comfy-ui-readme-md.md - raw/github_doc-nvidia-flux-finetuning-readme-md.md - raw/github_doc-nvidia-station-comfyui-readme-md.md --- title: "Containers and NGC" type: concept tags: [containers, setup, well-established] created: 2026-07-18 updated: 2026-07-18 sources: ["raw/web_community-nvidia-container-runtime-for-docker-dgx-spark-user-guide.md", "raw/web_community-ngc-dgx-spark-user-guide.md", "raw/web_community-nvidia-ai-enterprise-dgx-spark-quick-start-guide-dgx-spark-u.md"] confidence: high dgx_os_version: 7.5.0 --- ## Definition DGX Spark ships with the NVIDIA Container Toolkit preinstalled, which lets Docker containers access the GB10 GPU through OCI-spec hooks triggered by the `--gpus` flag. NGC (NVIDIA GPU Cloud) is NVIDIA's registry of GPU-optimized containers, pre-trained models, and NIM microservices; NVIDIA AI Enterprise (NVAIE) is the separate, purchasable enterprise entitlement layered on top for production support. ## How It Works The NVIDIA Container Runtime bridges Docker and the NVIDIA driver stack: passing `--gpus=all` (or a scoped `--gpus '"capabilities=compute,utility"'`) exposes GPU devices and CUDA libraries inside the container without any host-level CUDA install being required for the container itself. Because the toolkit is preinstalled and configured on DGX Spark, containers work out of the box: ``` docker run -it --gpus=all nvcr.io/nvidia/cuda:13.0.1-devel-ubuntu24.04 nvidia-smi ``` Docker requires `sudo` by default; adding the user to the docker group removes that friction: ``` sudo usermod -aG docker $USER newgrp docker ``` NGC access for pulling containers/models requires an NGC Personal API Key (generated on the Setup API Keys page after joining the free NVIDIA Developer Program, with at least the "NGC Catalog" service selected — legacy API keys are not supported for NIM-for-LLMs). Authenticate Docker with: ``` docker login nvcr.io # Username: $oauthtoken # Password: ``` DGX Spark and GB10 OEM systems do **not** use device serial-number registration to unlock NIM pulls — that's an important distinction from other DGX/enterprise flows. Before pulling any NIM container, confirm it has a DGX Spark-compatible image/profile in the DGX Spark Collection or the NIM-for-LLMs support matrix, since not every NIM has a Spark variant. The ARM64 NGC CLI (needed because of the Spark's Arm64 CPU — see [[concepts/unified-memory-and-arm64]]) is installed from the "ARM64 Linux" tab of the NGC CLI installer page. Typical workflow: pull an optimized framework container (e.g. `docker pull nvcr.io/nvidia/pytorch:24.08-py3`), run it with `--gpus=all` and a volume mount for persistent project data (`-v /path/to/your/project:/workspace`), and use `ngc registry model download-version "nvidia/nemo/bertbaseuncased:1.0.0rc1"` to pull models directly via the NGC CLI. **NVIDIA AI Enterprise (NVAIE)** is a separate, licensed layer providing enterprise support, guardrailing, and observability for production use. NIM container access and an NVAIE entitlement are related but distinct — NIM containers can be pulled with just an NGC Personal API key, while NVAIE requires a purchased/evaluated entitlement (a 90-day trial is available) tied to an NVIDIA Entitlement Certificate and NVIDIA Cloud Account, accessed through the NVIDIA Application Hub and the DGX Spark filter/collection in the NGC Catalog. If an NVAIE activation shows "on hold" or "already claimed," contact NVIDIA Enterprise Support with the activation reference ID rather than using the generic GPU serial-number activation workflow (which does not apply to DGX Spark). ## Key Parameters - GPU access flag: `--gpus=all` or scoped `--gpus '"capabilities=compute,utility"'` - NGC Docker login: username `$oauthtoken`, password = NGC Personal API Key - Required API key scope: at least "NGC Catalog" under Services Included - CLI architecture: ARM64 Linux build only - Container toolkit version check: `nvidia-ctk --version` - Docker daemon config: `/etc/docker/daemon.json` ## When To Use Use containers for reproducible, GPU-accelerated environments for any inference or training workload (see [[concepts/llama-cpp-on-spark]], [[concepts/vllm-on-spark]], [[concepts/ollama-on-spark]], [[concepts/comfyui-and-image-gen]]). Use an NGC Personal API key for research/development NIM access; pursue an NVAIE entitlement only for production support, compliance, or guardrailing needs. ## Risks & Pitfalls - "Runtime not found" errors typically mean the container toolkit or Docker daemon config is misconfigured — check with `nvidia-ctk --version`, inspect `/etc/docker/daemon.json`, and `sudo systemctl restart docker`. - CUDA version mismatches between host driver and container image show up as runtime errors; check host CUDA version with `nvidia-smi` and pick a compatible container tag. - Legacy NGC API keys do not work for NIM-for-LLMs — only Personal API Keys with NGC Catalog scope. - Not every NIM microservice has a DGX Spark-compatible profile; always check the DGX Spark Collection or support matrix before pulling. - Confusing NIM container access (free, via NGC API key) with an NVAIE entitlement (paid/evaluated) leads to wasted troubleshooting on the wrong activation workflow. ## Related Concepts - [[concepts/overview-and-hardware]] — GB10 hardware the container runtime targets - [[concepts/unified-memory-and-arm64]] — why ARM64-specific CLI/container builds matter - [[concepts/llama-cpp-on-spark]], [[concepts/vllm-on-spark]], [[concepts/ollama-on-spark]] — inference stacks typically run via these containers - [[concepts/comfyui-and-image-gen]] — image-generation workloads also delivered via NGC containers - [[syntheses/stack-picker]] — choosing among containerized inference stacks ## Sources - raw/web_community-nvidia-container-runtime-for-docker-dgx-spark-user-guide.md - raw/web_community-ngc-dgx-spark-user-guide.md - raw/web_community-nvidia-ai-enterprise-dgx-spark-quick-start-guide-dgx-spark-u.md --- title: "DGX OS and Updates" type: concept tags: [dgx-os, setup, well-established] created: 2026-07-18 updated: 2026-07-18 sources: ["raw/web_community-dgx-os-dgx-spark-user-guide.md", "raw/web_community-software-dgx-spark-user-guide.md", "raw/web_community-os-and-component-update-guide-dgx-spark-user-guide.md", "raw/web_community-uefi-settings-dgx-spark-user-guide.md", "raw/web_community-pxe-boot-setup-dgx-spark-user-guide.md", "raw/web_community-system-recovery-dgx-spark-user-guide.md", "raw/web_community-dgx-spark-release-notes-dgx-spark-user-guide.md"] confidence: high dgx_os_version: 7.5.0 --- ## Definition NVIDIA DGX OS is the customized, Ubuntu-based Linux distribution that underlies the DGX Spark — it bundles NVIDIA-specific drivers, system settings, and diagnostic tools tailored to GB10 hardware, plus security updates and maintenance capabilities. As of the July 2026 release notes, the current DGX Spark software stack is DGX OS **7.5.0**, NVIDIA GPU Driver 580.159.03, CUDA Toolkit 13.0.2, Canonical Kernel 6.17, UEFI 1.110.13, EC 3.5.8, USB PD 0.5.22, TPM 7.516.1, and SoC 2.155.11. These versions apply to the Founders Edition only — GB10 partner systems may lag. ## How It Works DGX OS follows a semiannual release cadence (roughly February and August) for the first two years after a platform's initial release, with additional patches between majors. Ubuntu Pro licensing on DGX Spark provides 10-year OS support from Canonical, and Ubuntu's standard compliance posture (FIPS, CIS, DISA-STIG) carries over. **Updates.** The strongly recommended path is the DGX Dashboard (see [[concepts/system-operations]]), which centralizes update visibility, driver/firmware updates, and progress monitoring. Manual updates are also possible from a terminal: ``` sudo apt update sudo apt dist-upgrade sudo fwupdmgr refresh sudo fwupdmgr upgrade sudo reboot ``` Best practice is stable power, a maintenance window, and a backup before major changes. **UEFI.** Accessed by holding `Esc` or `Del` immediately at power-on (Mac keyboards: use `Esc` only — `Del` isn't recognized), using a keyboard physically connected to the device. Notable settings: Wi-Fi/Bluetooth can be jointly disabled under Advanced → Advanced Menu → IO Port Access for security-hardened environments; boot order and one-time Boot Override live under Boot / Save & Exit; Secure Boot toggles under Security → Secure Boot (enabled by default, requires enrolled Platform Key + User mode to be active); PXE requires Advanced → Network Stack Configuration with Network Stack and IPv4/IPv6 PXE Support enabled per-NIC, plus either disabling Secure Boot or enrolling the signed PXE bootloader (`grubnetaa64.efi.signed`). **PXE boot.** For network-based install/recovery, an admin sets up TFTP (serves kernel/initrd), HTTP (serves the ISO/tarball, larger files), and DHCP (points at the TFTP server via `next-server` and the bootloader via `filename`, using the `pxeclients` vendor-class match). Example DGX OS PXE grub entry: ``` menuentry 'Install BaseOS 7.0.0' { linux /baseos/vmlinuz fsck.mode=skip autoinstall ip=dhcp url=http:///base_os_7.0.0/base_os_7.0.0.iso nvme-core.multipath=n nouveau.modeset=0 initrd /baseos/initrd } ``` Installer-specific kernel parameters include `usb.skipfw` (skip firmware update), `usb.shell` (shell instead of install), `noui`, and `autoinstall`. **System Recovery.** DGX Spark Founders Edition uses its own recovery flow — not the Enterprise DGX ISO/Download Center. Recovery media (a tar.gz, not an ISO) comes from developer.nvidia.com and needs no enterprise account. The flow: download and unzip the archive, run the platform-specific `CreateUSBKey.cmd`/`.sh`/`CreateUSBKeyMacOS.sh` as admin to build a 16GB+ USB drive (erases the drive), then boot into UEFI three separate times in sequence — first to Restore Defaults, second to confirm Secure Boot is Enabled and Restore Factory Keys, third to Boot Override to the USB drive — before entering the recovery environment and selecting START RECOVERY, which completely erases and reflashes the internal SSD. A directly wired USB keyboard is required; Bluetooth keyboards may not work during this process. ## Key Parameters - Current versions (July 2026): DGX OS 7.5.0, Driver 580.159.03, CUDA 13.0.2, Kernel 6.17, UEFI 1.110.13 - Release cadence: ~twice yearly (Feb/Aug) for first two years - Manual update commands: `apt update && apt dist-upgrade`, `fwupdmgr refresh && fwupdmgr upgrade` - UEFI entry keys: `Esc` or `Del` (Mac keyboards: `Esc` only) - Recovery USB minimum size: 16 GB - Recovery scripts: `CreateUSBKey.cmd` (Windows), `CreateUSBKey.sh` (Linux), `CreateUSBKeyMacOS.sh` (macOS) ## When To Use Use the DGX Dashboard update path for routine maintenance; fall back to manual `apt`/`fwupdmgr` commands only when the dashboard is unavailable. Use UEFI configuration when disabling wireless for security policy, enabling PXE, or troubleshooting boot devices. Use PXE boot setup for fleet imaging/reimaging across many units (see also [[concepts/system-operations]]'s enterprise manageability note). Use System Recovery only when the OS is corrupted or fatally misconfigured — it is destructive. ## Risks & Pitfalls - Recovery completely erases the internal SSD; there is no partial/selective recovery. - Bluetooth keyboards frequently fail during UEFI and recovery boot steps — always keep a wired USB keyboard available for these operations. - PXE requires Secure Boot to be disabled or the PXE bootloader specifically enrolled; skipping this step causes PXE boot to silently fail. - The July 2026 release notes call out improved OOM handling for the unified-memory architecture as a recent fix — older DGX OS versions had rougher memory-pressure behavior (see [[concepts/unified-memory-and-arm64]]). - DGX OS instructions in this KB apply to Founders Edition only; OEM/partner GB10 systems may differ in recovery media and update cadence. ## Related Concepts - [[concepts/overview-and-hardware]] — the hardware DGX OS runs on - [[concepts/first-boot-and-setup]] — initial installation, which precedes any update/recovery workflow - [[concepts/system-operations]] — DGX Dashboard as the primary update interface - [[concepts/unified-memory-and-arm64]] — memory-management behavior referenced in recent OS fixes - [[summaries/release-digest]] — full changelog across releases ## Sources - raw/web_community-dgx-os-dgx-spark-user-guide.md - raw/web_community-software-dgx-spark-user-guide.md - raw/web_community-os-and-component-update-guide-dgx-spark-user-guide.md - raw/web_community-uefi-settings-dgx-spark-user-guide.md - raw/web_community-pxe-boot-setup-dgx-spark-user-guide.md - raw/web_community-system-recovery-dgx-spark-user-guide.md - raw/web_community-dgx-spark-release-notes-dgx-spark-user-guide.md --- title: "Fine-Tuning and Quantization on Spark" type: concept tags: [fine-tuning, unified-memory, arm64, advanced, well-established] created: 2026-07-18 updated: 2026-07-18 sources: ["raw/github_doc-nvidia-nemo-fine-tune-readme-md.md", "raw/github_doc-nvidia-llama-factory-readme-md.md", "raw/github_doc-nvidia-pytorch-fine-tune-readme-md.md", "raw/github_doc-nvidia-unsloth-readme-md.md", "raw/github_doc-nvidia-flux-finetuning-readme-md.md", "raw/github_doc-nvidia-nvfp4-quantization-readme-md.md", "raw/github_doc-nvidia-cutile-kernels-readme-md.md", "raw/github_doc-nvidia-station-kernel-dev-ft-readme-md.md", "raw/github_doc-nvidia-station-nvfp4-pretraining-readme-md.md", "raw/github_doc-nvidia-station-nvfp4-quantization-readme-md.md"] confidence: high dgx_os_version: 7.5.0 --- ## Definition DGX Spark's 128 GB unified memory (see [[concepts/unified-memory-and-arm64]]) makes it usable not just for inference but for local fine-tuning (1–70B-parameter LLMs, plus image models like FLUX.1) and for NVFP4 quantization — a 4-bit floating-point format native to Blackwell Tensor Cores. NVIDIA's official playbook catalog (see [[entities/playbooks-catalog]]) covers this space through five overlapping tools: **NeMo AutoModel** (nemo-fine-tune), **LLaMA Factory**, plain **PyTorch** scripts (pytorch-fine-tune), **Unsloth** (kernel-optimized fine-tuning), **FLUX.1 Dreambooth LoRA** (image model fine-tuning), **TensorRT Model Optimizer NVFP4 quantization**, and **cuTile kernels** (writing custom GPU kernels, including for training bottlenecks). ## How It Works **NeMo AutoModel** (`nemo-fine-tune`) is NVIDIA's own recommended path for LLM/VLM fine-tuning: a Docker container (`nvcr.io/nvidia/nemo-automodel:26.02`) with pre-built recipes for full SFT, LoRA, and QLoRA, targeting 1–70B models with native Hugging Face compatibility. A LoRA run on Llama-3.1-8B, and a QLoRA run scaling to a 70B model, both use the same `finetune.py` entry point with YAML recipe overrides: ```bash python3 examples/llm_finetune/finetune.py \ -c examples/llm_finetune/llama3_2/llama3_2_1b_squad_peft.yaml \ --model.pretrained_model_name_or_path meta-llama/Llama-3.1-8B \ --packed_sequence.packed_sequence_size 1024 \ --step_scheduler.max_steps 20 ``` **LLaMA Factory** takes a venv-based (no Docker) approach: install PyTorch from the CUDA-13 ARM64 wheel index, clone the upstream `hiyouga/LLaMA-Factory` repo, then drive everything through its `llamafactory-cli` (`train`, `chat`, `export`) against YAML configs (SFT, RLHF, QLoRA). Its explicit prerequisite is CUDA 12.9+ and the ARM64 PyTorch wheel — a direct instance of the [[concepts/unified-memory-and-arm64]] wheel-compatibility issue. **Plain PyTorch** (`pytorch-fine-tune`) ships ready-made scripts for full SFT (Llama 3.2 3B), LoRA (Llama 3.1 8B), and LoRA+FSDP / QLoRA (Llama 3.1 70B), plus a **two-Spark distributed** path built on Docker Swarm: both nodes join a swarm over the QSFP interconnect (see [[concepts/clustering]]), and `accelerate launch` with a `config_fsdp_lora.yaml` splits the 70B LoRA+FSDP run across them — `machine_rank`, `main_process_ip`, and `main_process_port` are set per node. **Unsloth** targets fine-tuning speed and memory, not new capability: it claims 2x faster single-GPU training (up to 30x multi-GPU) via custom Triton kernels, dynamic 4-bit/16-bit quantization, and GGUF export for direct use with [[concepts/llama-cpp-on-spark]]/[[concepts/ollama-on-spark]]/[[concepts/vllm-on-spark]]. Installed with `pip install --no-deps unsloth unsloth_zoo bitsandbytes` inside the standard `nvcr.io/nvidia/pytorch:25.11-py3` container. **FLUX.1 Dreambooth LoRA fine-tuning** is the image-generation analog: multi-concept Dreambooth LoRA training of the FLUX.1-dev 12B diffusion model, explicitly designed around Spark's ability to hold the Diffusion Transformer, CLIP encoder, T5 encoder, and autoencoder simultaneously in the unified pool while training at 1024px+ resolution. Trained LoRAs drop straight into ComfyUI workflows (see [[concepts/comfyui-and-image-gen]]). **NVFP4 quantization** uses NVIDIA's TensorRT Model Optimizer, run inside a TensorRT-LLM or vLLM container, to post-training-quantize a model (the playbook's example: DeepSeek-R1-Distill-Llama-8B) to NVFP4: ```bash docker run --rm -it --gpus all --ipc=host --ulimit memlock=-1 --ulimit stack=67108864 \ -v "./output_models:/workspace/output_models" -e HF_TOKEN=$HF_TOKEN \ nvcr.io/nvidia/tensorrt-llm/release:spark-single-gpu-dev \ bash -c "... huggingface_example.sh --model deepseek-ai/DeepSeek-R1-Distill-Llama-8B --quant nvfp4 --tp 1 --export_fmt hf" ``` NVFP4 cuts memory ~3.5x vs FP16 and ~1.8x vs FP8 while claiming <1% accuracy loss versus FP8 — a claim that a real vLLM-ecosystem report contradicts for at least one large MoE model (see Risks below and [[syntheses/troubleshooting-casebook]]). A parallel **NVFP4 pretraining** playbook (`station-nvfp4-pretraining`, DGX Station/GB300 not Spark) uses Megatron-Bridge's `bf16_with_nvfp4_mixed` recipe and measured 1.68x faster steps with ~6% less peak VRAM than a BF16 baseline on Llama 3.1 8B — useful as a directional signal even though it runs on the larger Station product. **cuTile kernels** is the lowest-level entry: NVIDIA's TileGym benchmark suite and a from-scratch Flash Multi-Head Attention (FMHA) tutorial written in the cuTile Python DSL, which JIT-compiles to both Spark's `sm_121` and B300's `sm_103` from the same source. On Spark it measured 18.52 tok/s for Qwen2-7B with cuTile-optimized kernels vs. a naive PyTorch baseline, and demonstrates `ct.ByTarget`/`ct.autotune` for per-architecture tuning (`TILE_M=64` on Spark's 48 SMs vs. `TILE_M=256` on B300) — directly illustrating the [[concepts/unified-memory-and-arm64]] point that Blackwell code portability does not imply tuning portability. A related DGX Station playbook (`station-kernel-dev-ft`) profiles a Llama 3.1 8B fine-tune with `torch.profiler` and writes custom Triton kernels (fused RMSNorm, fused online-softmax cross-entropy) that cut cross-entropy peak memory ~6x and speed up RMSNorm ~1.5–6.8x — the same kernel-fusion principle Unsloth applies pre-built. ## Key Parameters - **NeMo AutoModel**: `--model.pretrained_model_name_or_path`, `--step_scheduler.max_steps`, `--packed_sequence.packed_sequence_size`, `--loss_fn._target_` (TE-parallel CE for 70B QLoRA) - **LLaMA Factory**: `per_device_train_batch_size`, `gradient_accumulation_steps`, CUDA-13 ARM64 wheel index required - **PyTorch multi-Spark**: `machine_rank`, `main_process_ip`, `main_process_port` in `config_fsdp_lora.yaml`; Docker Swarm + `accelerate launch` - **NVFP4 quantization**: `--quant nvfp4`, `--tp `, `--export_fmt hf`; served after with `trtllm-serve` or `vllm serve` - **NVFP4 pretraining knobs**: `first_last_layers_bf16`, `num_layers_at_end_in_bf16` — pin unstable layers to BF16 for training stability - **cuTile tuning**: `ct.ByTarget({"sm_121": ..., "sm_103": ..., "default": ...})`, `ct.autotune([...])`, `occupancy`, `num_ctas` ## When To Use Use **NeMo AutoModel** as the default for LLM/VLM fine-tuning when you want NVIDIA-maintained recipes and Docker isolation. Use **LLaMA Factory** for a CLI-first workflow with broad architecture support (LLaMA, Mistral, Qwen) and built-in RLHF. Use plain **PyTorch scripts** when you want minimal abstraction or need the two-Spark distributed path. Use **Unsloth** specifically for speed/memory-constrained fine-tuning runs, especially when the output needs to land in GGUF for [[concepts/llama-cpp-on-spark]]. Use **FLUX.1 Dreambooth LoRA** for image-model personalization feeding [[concepts/comfyui-and-image-gen]]. Use **NVFP4 quantization** when serving throughput/memory matters more than the last percent of accuracy — but validate accuracy yourself first (see Risks). Use **cuTile** only when profiling has already identified a specific memory-bound or compute-bound kernel bottleneck that off-the-shelf fused kernels (FlashAttention, cuBLAS) don't already solve. ## Risks & Pitfalls - **UMA memory pressure hits fine-tuning harder than inference.** Every fine-tuning playbook in this catalog repeats the same note: "you may encounter memory issues even when within the memory capacity of DGX Spark," with the fix being `sudo sh -c 'sync; echo 3 > /proc/sys/vm/drop_caches'` — see [[concepts/unified-memory-and-arm64]] for why `cudaMemGetInfo` under-reports on UMA. - **NVFP4 accuracy is not guaranteed close to FP8**, despite marketing claims of "<1% loss." A vLLM-ecosystem report (confidence medium, non-Spark hardware but same NVFP4/Blackwell mechanism) found `nvidia/Qwen3.5-397B-A17B-NVFP4` scoring ~0.35 on GSM8K vs. 0.90 for the FP8 original, a regression that persisted across attention backends and `--kv-cache-dtype` settings — always run your own eval before trusting an NVFP4 quant's accuracy, especially for large MoE models. See [[syntheses/troubleshooting-casebook]]. - **Gated Hugging Face models require accepting the license and authenticating inside the container**, not just on the host — "cannot access gated repo" is the single most repeated Troubleshooting entry across every fine-tuning playbook (NeMo, LLaMA Factory, PyTorch, NVFP4 quantization, FLUX). - **ARM64 wheel mismatches** are called out explicitly by name in NeMo AutoModel's and LLaMA Factory's own Troubleshooting tables ("ARM64 package compatibility issues may require troubleshooting") — see [[concepts/unified-memory-and-arm64]]. - **Multi-Spark Docker Swarm setups are fragile to Docker daemon crashes** from stale link-local IP bindings; the PyTorch fine-tune playbook's documented fix is stopping Docker, removing `/var/lib/docker/swarm`, and re-initializing with a valid advertise address. - **Output directories written by root-owned containers** (NVFP4 quantization) require `sudo rm -rf` for cleanup, not a plain `rm -rf` — a frequent "permission denied" surprise. - **`nvfp4` container quantization output can silently fail its final step via an unrelated MPI error** (`ModuleNotFoundError: No module named 'mpi4py'`) even though the quantization itself succeeded — check for the actual output artifacts before assuming the whole run failed. - **cuTile/kernel tuning parameters do not transfer between Blackwell members** — a `TILE_M`/`occupancy` combination tuned for Spark's 48 SMs is not automatically right for B300's 132 SMs; use `ct.ByTarget` or `ct.autotune` rather than hardcoding one value. ## Related Concepts - [[concepts/unified-memory-and-arm64]] — the UMA memory-pressure and ARM64 wheel issues underlying every fine-tuning workflow here - [[concepts/llama-cpp-on-spark]], [[concepts/ollama-on-spark]], [[concepts/vllm-on-spark]] — where fine-tuned/quantized outputs (GGUF, NVFP4) get served - [[concepts/comfyui-and-image-gen]] — where FLUX.1 Dreambooth LoRA outputs are used - [[concepts/clustering]] — the two-Spark distributed fine-tuning network setup - [[syntheses/troubleshooting-casebook]] — NVFP4 accuracy and memory-pressure cases in full - [[entities/playbooks-catalog]] — nemo-fine-tune, llama-factory, pytorch-fine-tune, unsloth, flux-finetuning, nvfp4-quantization, cutile-kernels, and their DGX Station counterparts ## Sources - raw/github_doc-nvidia-nemo-fine-tune-readme-md.md - raw/github_doc-nvidia-llama-factory-readme-md.md - raw/github_doc-nvidia-pytorch-fine-tune-readme-md.md - raw/github_doc-nvidia-unsloth-readme-md.md - raw/github_doc-nvidia-flux-finetuning-readme-md.md - raw/github_doc-nvidia-nvfp4-quantization-readme-md.md - raw/github_doc-nvidia-cutile-kernels-readme-md.md - raw/github_doc-nvidia-station-kernel-dev-ft-readme-md.md - raw/github_doc-nvidia-station-nvfp4-pretraining-readme-md.md - raw/github_doc-nvidia-station-nvfp4-quantization-readme-md.md - raw/github_issue-vllm-bug-qwen3-5-nvfp4-checkpoint-has-poor-accuracy.md --- title: "First Boot and Setup" type: concept tags: [setup, foundational, well-established] created: 2026-07-18 updated: 2026-07-18 sources: ["raw/web_community-initial-setup-first-boot-dgx-spark-user-guide.md", "raw/github_doc-nvidia-connect-to-your-spark-readme-md.md"] confidence: high dgx_os_version: 7.5.0 --- ## Definition First boot is the guided, one-time process that turns a freshly unboxed DGX Spark into a usable system: choosing an access method, running the first-time setup utility, and completing initial account/network configuration. Once finished, the device can be accessed in any combination of local (monitor/keyboard) or network access for the rest of its life — the choice made during setup is not permanent. ## How It Works Setup can be completed two ways. **With a Display (Local Setup)**: connect keyboard, mouse, and a display, then follow the on-screen wizard. **Over the Network (as a Network Appliance)**: power on the unit, which creates a Wi-Fi hotspot (SSID/password printed on the Quick Start Guide sticker); connect to that hotspot from another computer, and a captive portal opens the setup page in a browser. Peripherals must be attached *before* power is connected, since the DGX Spark boots immediately when power is applied. The first-time setup utility walks through: language/time zone, keyboard layout (local setup only), terms and conditions, user account creation, optional analytics/crash-reporting preferences, Wi-Fi network selection (skipped if Ethernet is already providing internet), and joining the network. During network setup, the Spark's hotspot tears down once it joins the home network, and the setup client reconnects automatically — if it can't, a display/keyboard/mouse is required to continue (common causes: device isolation on the router, the laptop not joining the same network, or mDNS being blocked on a complex corporate network). After networking is established, the system downloads and installs the full software image, which can take up to ~10 minutes and may involve multiple reboots; this step must not be interrupted. After first boot, the recommended way to reach the device day-to-day is the **connect-to-your-spark** playbook, which offers two paths: 1. **NVIDIA Sync** (recommended) — a desktop app installed on your laptop that discovers the Spark, sets up SSH key-based auth automatically, and gives one-click launch of Terminal, DGX Dashboard, VS Code, Cursor, and other tools. Setup: install Sync, add the device by hostname (e.g. `spark-abcd.local`, found on the Quick Start Guide or the DGX Dashboard Settings page) or IP, plus username/password — Sync then generates an SSH key pair locally, copies the public key to the Spark's `~/.ssh/authorized_keys`, and discards the password. 2. **Manual SSH** — direct command-line control, verified with `ssh @.local`, with port forwarding for web apps, e.g. `ssh -L 11000:localhost:11000 @.local` to reach the DGX Dashboard (see [[concepts/system-operations]]). mDNS (`.local` hostnames) is the default discovery mechanism; if it doesn't resolve (`ssh: Could not resolve hostname spark-abcd.local`), test with `ping spark-abcd.local` and fall back to the device's IP address, found via router admin panel or a directly connected display. ## Key Parameters - SSID/password for initial Wi-Fi hotspot: printed on Quick Start Guide sticker - Default mDNS hostname pattern: `spark-abcd.local` - Full software image download/install: up to ~10 minutes, multiple reboots possible - DGX Dashboard default port: 11000 (`http://localhost:11000`) - SSH client check: `ssh -V` - SSH test commands on-device: `hostname`, `whoami`, `uname -a` ## When To Use Follow this process for any brand-new or factory-reset DGX Spark. Choose local setup when a monitor/keyboard is convenient; choose network setup for headless/appliance deployment. Use NVIDIA Sync afterward for ongoing remote access unless a minimal, script-friendly manual SSH setup is preferred. If setup fails outright due to an unreliable connection, consider [[concepts/dgx-os-and-updates]]'s system recovery path instead of retrying network setup repeatedly. ## Risks & Pitfalls - Using captive-portal Wi-Fi (hotel/airport) or unstable connections (phone hotspots) during setup is explicitly discouraged — the update download cannot be safely interrupted. - Shutting down or rebooting during the critical update/install phase can damage the system; this is called out as a hard warning in the source docs. - If your computer auto-reconnects to the same network as the Spark after the hotspot drops, setup continues seamlessly; if not, you must manually rejoin the same network — and if mDNS is blocked (common on corporate networks), you'll need a physical display/keyboard/mouse to finish. - NVIDIA Sync's password prompt is used only once, to install the SSH key — if a connection fails right after setup, wait 3-4 minutes for the device to finish becoming available on the network before retrying. - Bluetooth keyboards, even USB-capable ones, can fail during low-level operations like UEFI/recovery access — always keep a wired USB keyboard on hand (see [[concepts/dgx-os-and-updates]]). ## Related Concepts - [[concepts/overview-and-hardware]] — the hardware being set up - [[concepts/dgx-os-and-updates]] — recovery path when setup can't complete normally - [[concepts/system-operations]] — DGX Dashboard and NVIDIA Sync, used immediately after setup - [[entities/playbooks-catalog]] — the connect-to-your-spark playbook and others ## Sources - raw/web_community-initial-setup-first-boot-dgx-spark-user-guide.md - raw/github_doc-nvidia-connect-to-your-spark-readme-md.md --- title: "llama.cpp on Spark" type: concept tags: [inference, llama-cpp, arm64, foundational, well-established] created: 2026-07-18 updated: 2026-07-18 sources: ["raw/github_doc-nvidia-llama-cpp-readme-md.md", "raw/github_issue-llamacpp-compile-bug-cuda-build-for-mmq-breaks-for-compute-capability.md", "raw/github_issue-llamacpp-eval-bug-data-corruption-on-cuda-experts-offload.md", "raw/github_issue-llamacpp-eval-bug-gemma-4-generates-unused24-tokens.md", "raw/github_issue-llamacpp-misc-bug-many-grammar-errors-when-called-by-openclaw.md", "raw/github_issue-llamacpp-model-request-deepseek-v4-series.md"] confidence: high dgx_os_version: 7.5.0 --- ## Definition llama.cpp is a lightweight C/C++ inference engine for GGUF-format language models. On DGX Spark it is built from source with CUDA enabled and targeted at the GB10 GPU's **sm_121a** compute capability, then served through `llama-server`'s OpenAI-compatible HTTP API. General llama.cpp usage (GGUF format, quantization schemes, the broader flag surface) is covered in depth by the dedicated llama.cpp wiki in this fleet — this page covers what is different when the target is Spark's GB10/aarch64 combination. ## How It Works The official [[entities/playbooks-catalog|NVIDIA playbook]] builds llama.cpp against the system CUDA Toolkit with a CMake invocation that pins the architecture flag to the GB10: ```shell sudo apt update sudo apt install -y git clang cmake libcurl4-openssl-dev libssl-dev git clone https://github.com/ggml-org/llama.cpp ~/llama.cpp cd ~/llama.cpp cmake -B build -DGGML_NATIVE=ON -DGGML_CUDA=ON -DGGML_CURL=ON -DGGML_RPC=ON -DCMAKE_CUDA_ARCHITECTURES=121a-real cmake --build build --config Release --target llama-server -j ``` The build takes roughly 5–10 minutes and produces `llama-server` under `build/bin/`. Models are pulled straight from Hugging Face with `-hf` (cached under `~/.cache/huggingface/hub`), or loaded locally with `--model` (plus `--mmproj` for multimodal GGUFs if not using `-hf`, which fetches the mmproj automatically). A representative launch, using the playbook's MTP-enabled Qwen3.6-35B-A3B example: ```shell ./bin/llama-server \ -hf unsloth/Qwen3.6-35B-A3B-MTP-GGUF:UD-Q4_K_XL \ --host 0.0.0.0 \ --port 30000 \ --chat-template-kwargs '{"preserve_thinking": true}' \ --spec-type draft-mtp \ --spec-draft-n-max 3 ``` `--spec-type draft-mtp` and `--spec-draft-n-max 3` enable Multi-Token Prediction speculative decoding for MTP-compatible checkpoints; `--chat-template-kwargs` passes a JSON blob into the chat template (here, preserving Qwen's interleaved-thinking blocks for agentic use). By default llama-server tries to fit the model's full context with 4 concurrent slots and adjusts automatically if that does not fit. Readiness is signaled by `server is listening on http://0.0.0.0:30000` in the logs; poll with `timeout 900 bash -c 'until curl -sf http://127.0.0.1:30000/health > /dev/null 2>&1; do sleep 5; done'` before sending requests. ## Key Parameters - `-DCMAKE_CUDA_ARCHITECTURES=121a-real` — matches GGML's CUDA backend to GB10's compute capability; the single most Spark-specific build flag in this playbook. - `-hf :` vs `--model ` — automatic HF-managed download/update vs. a local GGUF path (multimodal models need `--mmproj` only with `--model`). - `--ctx-size` / `-c` — manual context override; agentic/coding use cases want at least 32768, preferably 100000+. - `--spec-type draft-mtp` + `--spec-draft-n-max` — MTP speculative decoding for compatible checkpoints (check the model card first). - `--n-gpu-layers` — must be high enough to actually offload to the GPU; low values silently degrade to CPU-bound latency. ## When To Use Choose llama.cpp on Spark when the goal is a single-process, no-Python-framework GGUF server — it needs no separate inference framework, exposes an OpenAI-compatible endpoint immediately usable by Open WebUI, Continue.dev, or custom clients (see [[concepts/agents-on-spark]]), and its quantization ecosystem (GGUF, various K-quants) tends to have the widest day-one community coverage for new model releases. It is a reasonable default before reaching for [[concepts/vllm-on-spark]] (which trades simplicity for throughput/serving features) or [[concepts/ollama-on-spark]] (which trades control for convenience). See [[syntheses/stack-picker]] for the full tradeoff comparison. ## Risks & Pitfalls - **CUDA architecture mismatches break the build outright, not just performance.** A community report (confidence medium, not confirmed on GB10 specifically) shows that CUDA compute-capability 120 (Blackwell consumer, `sm_120`) fails to compile `mmq` kernels with `ptxas` errors like `Instruction 'mma with block scale' not supported on .target 'sm_120'` — a reminder that Blackwell sub-variants are not interchangeable and the playbook's exact `121a-real` flag (GB10's own capability) matters more than "any Blackwell arch string will do." - **Gemma 4 on GB10 can degenerate into runaway `` token spam** mid-response (confidence medium, single issue, but explicitly reproduced on DGX Spark hardware with CUDA compute capability 12.1) — a sampling/vocabulary issue rather than a hardware fault; if you hit it, stop generation and consider excluding unused tokens from sampling or trying a different quant/build. - **MoE expert-offload configurations (`-ot ... =CPU, ... =CUDA0`) have been reported to produce data corruption in long-prompt generations** on multi-GPU CUDA setups (confidence medium, reported on discrete RTX 5070 Ti, not confirmed on Spark's single-GPU unified-memory layout) — worth extra scrutiny before relying on manual per-tensor CPU/GPU splits for large MoE GGUFs on Spark. - Agentic callers (e.g. OpenClaw-style tool orchestrators) can trigger excessive grammar-constraint output/errors in some configurations (confidence medium, reported on a CPU-only Windows build, not Spark-specific) — if a tool-calling agent seems to spam grammar noise, suspect the grammar/tool-schema interaction before the model itself. - Standard UMA gotcha applies: flush the page cache with `sudo sh -c 'sync; echo 3 > /proc/sys/vm/drop_caches'` if you hit unexpected memory pressure — see [[concepts/unified-memory-and-arm64]]. - Model requests for very new architectures (e.g. DeepSeek-V4) routinely outrun llama.cpp's support — check the model's `general.architecture` GGUF key against your llama.cpp build before assuming an unsupported model is a Spark-specific problem. ## Related Concepts - [[concepts/unified-memory-and-arm64]] — why UMA behavior and aarch64 builds matter for this playbook - [[concepts/vllm-on-spark]] — the higher-throughput alternative for served inference - [[concepts/ollama-on-spark]] — the friendlier-but-less-tunable alternative - [[concepts/agents-on-spark]] — pointing coding/chat agents at a local llama-server endpoint - [[syntheses/troubleshooting-casebook]] — cross-stack build/runtime failure patterns - [[syntheses/stack-picker]] — choosing among llama.cpp/Ollama/vLLM/LM Studio on Spark - [[entities/playbooks-catalog]] — full list of official Spark playbooks ## Sources - raw/github_doc-nvidia-llama-cpp-readme-md.md - raw/github_issue-llamacpp-compile-bug-cuda-build-for-mmq-breaks-for-compute-capability.md - raw/github_issue-llamacpp-eval-bug-data-corruption-on-cuda-experts-offload.md - raw/github_issue-llamacpp-eval-bug-gemma-4-generates-unused24-tokens.md - raw/github_issue-llamacpp-misc-bug-many-grammar-errors-when-called-by-openclaw.md - raw/github_issue-llamacpp-model-request-deepseek-v4-series.md --- title: "LM Studio on Spark" type: concept tags: [inference, lm-studio, arm64, foundational, well-established] created: 2026-07-18 updated: 2026-07-18 sources: ["raw/github_doc-nvidia-lm-studio-readme-md.md"] confidence: medium dgx_os_version: 7.5.0 --- ## Definition LM Studio is an application for discovering, running, and serving local LLMs, usable headlessly via **llmster** (its terminal-native daemon) and its `lms` CLI/SDK/REST API. General LM Studio usage is covered by the dedicated LM Studio wiki in this fleet. The Spark-specific angle is running it **headless** on the Spark as a private LLM server and reaching it from a laptop — either over the LAN or through **LM Link**, LM Studio's Tailscale-based zero-config remote-access mesh. ## How It Works Install `llmster` (the headless daemon) on the Spark and start the API server bound to all interfaces: ```bash curl -fsSL https://lmstudio.ai/install.sh | bash lms server start --bind 0.0.0.0 --port 1234 ``` Download and load a model directly on the Spark, sized to take advantage of its unified memory (the playbook's example, Nemotron 3 Nano Omni, needs 65GB+ GPU memory): ```bash lms get nvidia/nemotron-3-nano-omni lms load nvidia/nemotron-3-nano-omni ``` From a laptop on the same LAN, hit the Spark directly: `curl http://:1234/api/v1/models`. Alternatively, **LM Link** (in Preview, free for up to 2 users/5 devices each) links the Spark and laptop over an end-to-end encrypted Tailscale mesh — once both are joined to the same Link, the laptop points at `localhost:1234` and the Spark's loaded models appear as if local, with no LAN adjacency or `--bind 0.0.0.0` needed. Any tool that talks to LM Studio's local API — including Codex, Claude Code, and OpenCode — works unmodified through a Link (see [[concepts/agents-on-spark]]). ## Key Parameters - `lms server start --bind 0.0.0.0 --port 1234` — required to expose the API beyond localhost for LAN access; skip this if using LM Link only. - Minimum **65GB GPU memory / 70GB+ recommended** and matching storage — LM Studio on Spark is sized for large models, not an afterthought. - LM Link device limits: free tier caps at 2 users, 5 devices each (10 total). ## When To Use Prefer LM Studio on Spark when the priority is a GUI-adjacent, low-friction model catalog (`lmstudio.ai/models`) with headless server mode, or when LM Link's zero-config remote access is attractive over manually configuring SSH tunnels (contrast with the [[concepts/ollama-on-spark]] playbook, which tunnels through NVIDIA Sync instead). For scripted/production serving with more tuning knobs, prefer [[concepts/vllm-on-spark]] or [[concepts/llama-cpp-on-spark]]. See [[syntheses/stack-picker]]. ## Risks & Pitfalls - No community issue mirrors were available for this playbook at ingest time — Risks below are drawn solely from the official playbook's own Troubleshooting table and general Spark UMA notes (confidence medium, single-source). - `lms` not found immediately after install: the shell needs a reload (`source ~/.bashrc`) for the PATH change to take effect. - Model load failing with CUDA OOM despite Spark's large unified pool usually means the specific model+quant combination is genuinely too big for available VRAM headroom — switch quantization rather than assuming a Spark-specific limitation. - LM Link connectivity issues (devices not seeing each other's models) are almost always a "both devices signed into the same Link" problem — restart LM Studio/llmster after joining, on both ends. - Standard UMA caveat: flush the page cache (`sudo sh -c 'sync; echo 3 > /proc/sys/vm/drop_caches'`) under unexpected memory pressure — see [[concepts/unified-memory-and-arm64]]. ## Related Concepts - [[concepts/unified-memory-and-arm64]] — why Spark can host unusually large models for a desktop box - [[concepts/ollama-on-spark]] — the NVIDIA-Sync-tunnel alternative for remote API access - [[concepts/agents-on-spark]] — coding agents (Claude Code, OpenCode) consuming LM Studio's local API through LM Link - [[syntheses/stack-picker]] — choosing among inference stacks on Spark - [[entities/playbooks-catalog]] — full list of official Spark playbooks ## Sources - raw/github_doc-nvidia-lm-studio-readme-md.md --- title: "Ollama on Spark" type: concept tags: [inference, ollama, arm64, foundational, well-established] created: 2026-07-18 updated: 2026-07-18 sources: ["raw/github_doc-nvidia-ollama-readme-md.md", "raw/github_issue-ollama-can-t-pull-nvfp4.md", "raw/github_issue-ollama-i-can-t-run-gemma4-31b.md", "raw/github_issue-ollama-ollama-slow-performance.md", "raw/github_issue-ollama-overrun-of-a-stack-based-buffer-error-when-running-phi4.md", "raw/github_issue-ollama-qwen3-embedding-8b-model-does-not-support-embeddings.md", "raw/github_issue-ollama-version-11-bombing-out-and-responds-with-ggggggggggggggg.md"] confidence: high dgx_os_version: 7.5.0 --- ## Definition Ollama is a model-management-plus-server tool that wraps a llama.cpp-derived engine behind a simple pull/run CLI and a REST API on port 11434. General Ollama usage — the model library, Modelfile format, API shape — is covered in depth by the dedicated Ollama wiki in this fleet. On DGX Spark the official playbook's specific angle is **remote access**: installing Ollama locally on the Spark and reaching its API from a laptop through an NVIDIA Sync SSH tunnel, rather than exposing ports directly on the network. ## How It Works Install and pull models the standard way: ```bash curl -fsSL https://ollama.com/install.sh | sh ollama pull qwen2.5:32b ``` The Spark-specific piece is remote access via **NVIDIA Sync's Custom Apps** feature: in Sync's Settings → Custom tab, add an entry named e.g. `Ollama Server` with Port `11434` and no start script, then activate it to create an SSH tunnel from `localhost:11434` on the laptop to the Spark's Ollama service — no LAN port forwarding required. Once tunneled, the API is used exactly as if Ollama were running locally: ```bash curl http://localhost:11434/api/chat -d '{ "model": "qwen2.5:32b", "messages": [{"role": "user", "content": "Write me a haiku about GPUs and AI."}], "stream": false }' ``` Full teardown removes both the tunnel (deactivate in Sync) and, if desired, the Ollama install itself (`sudo systemctl stop/disable ollama`, remove the binary and `/usr/share/ollama`, `userdel ollama`). ## Key Parameters - **NVIDIA Sync Custom App** — Name/Port/Start-Script triplet that creates the SSH tunnel; this is the Spark-specific integration point, not a generic Ollama concept. - Model choice matters more than usual given Spark's 128GB unified pool — the playbook uses `qwen2.5:32b` as "optimized for Blackwell GPUs," but nothing stops loading much larger models within the unified-memory ceiling (see [[concepts/unified-memory-and-arm64]]). - `OLLAMA_FLASH_ATTENTION` and `OLLAMA_CONTEXT_LENGTH` env vars are worth checking explicitly on Spark — see Risks below. ## When To Use Use Ollama on Spark when convenience matters more than fine-grained control: single-command pulls from the Ollama library, painless model switching, and (per this playbook) API access from a laptop without exposing the Spark on the LAN. It underlies several other Spark playbooks — [[concepts/agents-on-spark]] (CLI coding agents via `ollama launch`, Live VLM WebUI, Open WebUI) all default to Ollama as the easiest local backend. For raw throughput or fine-grained serving flags, see [[concepts/vllm-on-spark]]; for a from-source, framework-free server, see [[concepts/llama-cpp-on-spark]]. Comparison in [[syntheses/stack-picker]]. ## Risks & Pitfalls - **Slow inference reported specifically on 2x DGX Spark hardware** (confidence medium, single issue but explicitly on-device): a user running gpt-oss:20b, phi4:14b, gemma3:12b, qwen3:8b, and granite3.3:8b via Dockerized Ollama saw response times degrade from fast to 10+ minutes after 20–30 minutes of uptime, with `nvidia-smi` showing the GB10 iGPU at 92% utilization. The captured server config showed `OLLAMA_FLASH_ATTENTION:false` and `OLLAMA_CONTEXT_LENGTH:4096` — worth checking both settings explicitly rather than trusting Docker defaults, and worth watching whether GPU utilization staying pegged near 100% while wall-clock time balloons indicates thermal throttling or a stuck/looping generation rather than genuine load. - **NVFP4-quantized model tags may refuse to pull outside macOS** (confidence medium, single issue): `ollama pull qwen3.5:27b-nvfp4` on Windows returned `Error: pull model manifest: 412: this model requires macOS`, i.e. some NVFP4 tags in the Ollama library are gated to a specific OS regardless of the actual GPU/quantization compatibility. If an NVFP4 pull fails with a platform-requirement error on Linux (DGX OS), treat it as a library-side tag restriction, not a Spark hardware limitation. - Embedding models can silently lose embedding support after an Ollama upgrade (confidence low, single issue, non-Spark hardware) — if `model does not support embeddings` appears after an update, check whether the specific tag/quant regressed rather than assuming embeddings are broken generally. - Some model+version combinations have historically produced runaway garbage output (e.g. repeated `G` characters) or crashed with low-level buffer errors on non-Spark hardware (confidence low, Windows/Jetson reports) — not confirmed on Spark, but a reminder to pin Ollama versions when a model that worked yesterday starts behaving strangely after an auto-update. - Standard UMA caveat: flush the page cache (`sudo sh -c 'sync; echo 3 > /proc/sys/vm/drop_caches'`) if memory pressure appears despite headroom — see [[concepts/unified-memory-and-arm64]]. ## Related Concepts - [[concepts/unified-memory-and-arm64]] — why model sizing and memory pressure behave differently on Spark - [[concepts/vllm-on-spark]] — higher-throughput alternative - [[concepts/llama-cpp-on-spark]] — framework-free alternative - [[concepts/agents-on-spark]] — CLI coding agents, Live VLM WebUI, and Open WebUI built on Ollama - [[syntheses/troubleshooting-casebook]] — cross-stack failure patterns including the Spark slow-performance report - [[syntheses/stack-picker]] — choosing among inference stacks on Spark - [[entities/playbooks-catalog]] — full list of official Spark playbooks ## Sources - raw/github_doc-nvidia-ollama-readme-md.md - raw/github_issue-ollama-can-t-pull-nvfp4.md - raw/github_issue-ollama-i-can-t-run-gemma4-31b.md - raw/github_issue-ollama-ollama-slow-performance.md - raw/github_issue-ollama-overrun-of-a-stack-based-buffer-error-when-running-phi4.md - raw/github_issue-ollama-qwen3-embedding-8b-model-does-not-support-embeddings.md - raw/github_issue-ollama-version-11-bombing-out-and-responds-with-ggggggggggggggg.md --- title: "Overview and Hardware" type: concept tags: [setup, foundational, well-established] created: 2026-07-18 updated: 2026-07-18 sources: ["raw/web_community-system-overview-dgx-spark-user-guide.md", "raw/web_community-hardware-overview-dgx-spark-user-guide.md", "raw/web_community-dgx-spark-user-guide-dgx-spark-user-guide.md"] confidence: high dgx_os_version: 7.5.0 --- ## Definition The NVIDIA DGX Spark is a compact desktop AI computer built around the NVIDIA GB10 Grace Blackwell Superchip — a single package that combines a 20-core Arm CPU (10 Cortex-X925 + 10 Cortex-A725) with an integrated Blackwell-architecture GPU, sharing 128 GB of unified LPDDR5x system memory. It is designed for developers, researchers, and data scientists who need to prototype, deploy, and fine-tune large AI models on their desktop rather than in the cloud. See [[concepts/dgx-os-and-updates]] for the software layer that runs on top of this hardware. ## How It Works The defining architectural feature is the CPU and GPU sharing one 128 GB pool of unified memory over a 256-bit LPDDR5x interface (4266 MHz, 273 GB/s bandwidth, 16 channels), rather than the CPU having its own RAM and the GPU having separate, smaller VRAM. This means model weights, KV caches, and activations all draw from the same large pool, which is what allows a desktop machine this size to load models up to 200 billion parameters (or 405B in a dual-Spark configuration — see [[concepts/clustering]]). For the practical implications of this design, see [[concepts/unified-memory-and-arm64]]. The GPU itself uses 5th-generation Tensor Cores with FP4 support and 4th-generation RT Cores, delivering up to 1,000 TOPS of AI inference throughput and up to 1 PFLOP at FP4 precision with sparsity, across 6,144 CUDA cores. Two on-die copy engines allow simultaneous data transfer to and from GPU memory, improving throughput for AI workloads that overlap compute with data movement. Networking is built for both everyday use and clustering: Wi-Fi 7, Bluetooth 5.4, 10 GbE (1x RJ-45), and — critically for multi-Spark setups — a ConnectX-7 Smart NIC exposed as two rear QSFP ports. Storage is 1 TB or 4 TB self-encrypting NVMe M.2. The whole system fits in a 150mm x 150mm x 50.5mm small-form-factor chassis weighing 1.2 kg, drawing power through an included 240W external supply (required for full performance — the GB10 SoC alone has a 140W thermal design power). ## Key Parameters - **Unified memory**: 128 GB LPDDR5x, 256-bit interface, 4266 MHz, 273 GB/s bandwidth, 16 channels - **CPU**: 20-core Arm (10 Cortex-X925 + 10 Cortex-A725) - **GPU**: Blackwell architecture, 6,144 CUDA cores, 5th-gen Tensor Cores (FP4), 4th-gen RT Cores - **AI compute**: up to 1,000 TOPS inference, up to 1 PFLOP FP4 (with sparsity) - **Storage**: 1 TB or 4 TB NVMe M.2 (self-encrypting) - **Networking**: Wi-Fi 7, Bluetooth 5.4, 10 GbE RJ-45, ConnectX-7 (2x QSFP) - **I/O**: 4x USB Type-C (one for power delivery), 1x HDMI 2.1a, 1x NVENC, 1x NVDEC - **Power**: 240W external supply required; GB10 SoC TDP 140W; 100W available for other components - **Model size support**: up to 200B parameters single-unit, 405B in dual-Spark configuration - **Operating temperature**: 5°C to 30°C (41°F to 86°F), 10-90% non-condensing humidity ## When To Use DGX Spark is the right hardware choice when the goal is local inference, fine-tuning, or experimentation with large models without cloud compute costs — particularly workloads that are memory-bound rather than pure-throughput-bound, since the 128 GB unified pool can hold models that would need multiple discrete-VRAM GPUs elsewhere. It suits desktop or "network appliance" deployment (see [[concepts/first-boot-and-setup]]) and scales via [[concepts/clustering]] when a single unit's memory or compute is insufficient (e.g., 200B+ parameter models, faster multi-GPU training). ## Risks & Pitfalls - Using a power supply other than the included 240W unit can cause reduced performance, boot failure, or unexpected shutdowns — this is an explicit requirement, not a recommendation. - Unified memory means `nvidia-smi` and similar tools report memory differently than on discrete-GPU systems; expect Out-of-Memory behavior and reporting to differ from traditional CUDA systems (see [[concepts/unified-memory-and-arm64]]). - The Arm64 (aarch64) CPU architecture means x86-only binaries, containers, and prebuilt wheels will not run without an Arm-compatible build — a frequent source of setup friction (see [[concepts/unified-memory-and-arm64]] and [[syntheses/troubleshooting-casebook]]). - Operating outside the 5-30°C ideal range can affect thermal performance in the compact chassis. ## Related Concepts - [[concepts/unified-memory-and-arm64]] — deep dive on the memory model and ARM64 implications - [[concepts/first-boot-and-setup]] — getting a new unit running - [[concepts/dgx-os-and-updates]] — the OS and software stack this hardware runs - [[concepts/clustering]] — connecting multiple Sparks to exceed single-unit limits - [[entities/playbooks-catalog]] — catalog of official setup and workload playbooks ## Sources - raw/web_community-system-overview-dgx-spark-user-guide.md - raw/web_community-hardware-overview-dgx-spark-user-guide.md - raw/web_community-dgx-spark-user-guide-dgx-spark-user-guide.md --- title: "System Operations" type: concept tags: [setup, dgx-os, well-established] created: 2026-07-18 updated: 2026-07-18 sources: ["raw/web_community-dgx-dashboard-dgx-spark-user-guide.md", "raw/github_doc-nvidia-dgx-dashboard-readme-md.md", "raw/web_community-nvidia-sync-dgx-spark-user-guide.md", "raw/web_community-nvidia-nsight-dgx-spark-user-guide.md", "raw/web_community-system-configuration-and-operation-dgx-spark-user-guide.md", "raw/web_community-common-use-cases-dgx-spark-user-guide.md", "raw/web_community-enterprise-manageability-dgx-spark-user-guide.md"] confidence: high dgx_os_version: 7.5.0 --- ## Definition System operations covers the day-to-day tools for monitoring, updating, developing on, and profiling a DGX Spark: the built-in **DGX Dashboard**, the companion desktop app **NVIDIA Sync**, and the **NVIDIA Nsight** profiling suite. Together with UEFI settings and clustering (covered in [[concepts/dgx-os-and-updates]] and [[concepts/clustering]]), these are the "System Configuration and Operation" surface described in the official user guide. ## How It Works **DGX Dashboard** is a web app running locally on the Spark (port 11000) that shows real-time system telemetry (GPU utilization visible while workloads run), applies system/firmware updates, lets you change the device name, and hosts an integrated JupyterLab instance. Sudo access (granted to the account created at first boot — see [[concepts/first-boot-and-setup]]) is required to run updates or rename the device. Locally, it's opened via the "Show Apps" grid → "DGX Dashboard" shortcut; remotely, NVIDIA Sync's "DGX Dashboard" button opens it automatically at `http://localhost:11000`, or a manual tunnel works: `ssh -L 11000:localhost:11000 @`. JupyterLab inside the Dashboard creates a virtual environment in a chosen working directory (default `/home//jupyterlab`) and auto-installs recommended packages (visible in the generated `requirements.txt`); changing the working directory and restarting creates a fresh isolated environment. Each user is assigned a distinct JupyterLab port, recorded in `/opt/nvidia/dgx-dashboard-service/jupyterlab_ports.yaml` — check it with `cat /opt/nvidia/dgx-dashboard-service/jupyterlab_ports.yaml` before building a manual tunnel that includes both the dashboard and JupyterLab ports: ``` ssh -L 11000:localhost:11000 -L :localhost: @ ``` A worked example workload is running Stable Diffusion XL from a JupyterLab cell using `diffusers.DiffusionPipeline` while watching GPU telemetry update live in the Dashboard's monitoring panel (see [[concepts/comfyui-and-image-gen]] for a dedicated image-generation stack). Updates are applied from Settings → Updates → Update → Update Now, which triggers a reboot — the release notes note that DGX Dashboard now also surfaces release highlights so users can judge update urgency. **NVIDIA Sync** is the system-tray utility (Windows/macOS/Ubuntu) that manages SSH connections, port forwarding, and tunnels to one or more DGX Sparks from a laptop or workstation, discovered via mDNS. Beyond DGX Dashboard access, it offers one-click launch of configured apps (VS Code, Cursor, NVIDIA AI Workbench), a settings-defined working directory apps open into, custom port forwarding for arbitrary web apps/APIs, and optional Tailscale integration for access when off the local network. As of the June 2026 release, Sync's Settings page gained a **Cluster Assistant** that helps connect up to three devices without a switch or up to four with one (see [[concepts/clustering]]). **NVIDIA Nsight** is NVIDIA's GPU profiling/debugging family: Nsight Systems (timeline profiling of CPU/GPU/system activity), Nsight Graphics (frame capture for graphics APIs), Nsight DL Designer (deep-learning model visualization), and Nsight Compute (CUDA kernel profiling and guided performance reports). DGX Spark follows NVIDIA's broader release model for these tools rather than shipping fixed versions — current installers and docs live on the NVIDIA Developer website per tool. **Common Use Cases and Enterprise Manageability.** The official guide defers "Common Use Cases" almost entirely to the community playbook site at build.nvidia.com/spark, calling it the single source of truth for practical guides. Enterprise Manageability adds fleet-scale tooling — lifecycle integration scripts (asset inventory, controlled update/firmware tools, remote ops/remediation, invoked over agentless SSH from a management platform) and cloud-init-based custom installation for air-gapped or on-prem-mirror deployments — aimed at IT administrators managing many units rather than a single desktop. ## Key Parameters - DGX Dashboard port: 11000 (`http://localhost:11000`) - JupyterLab port registry: `/opt/nvidia/dgx-dashboard-service/jupyterlab_ports.yaml` - Default JupyterLab working directory: `/home//jupyterlab` - NVIDIA Sync Cluster Assistant: up to 3 devices without a switch, up to 4 with one (added June 2026) - Nsight tool family: Systems, Graphics, DL Designer, Compute (versioned independently on NVIDIA Developer site) ## When To Use Use the DGX Dashboard for routine monitoring, updates, and quick JupyterLab experimentation. Use NVIDIA Sync as the default remote-access and multi-device-management tool, especially once clustering multiple units (see [[concepts/clustering]]). Reach for Nsight when profiling or debugging GPU kernel performance beyond what basic telemetry shows. Reach for the enterprise manageability tooling only at fleet scale (multiple DGX Sparks under central IT control), not for single-desktop use. ## Risks & Pitfalls - Running updates or renaming the device requires sudo — a non-sudo user will see the update controls but cannot use them; add the user with `sudo usermod -aG sudo `. - System updates trigger a reboot and cannot be rolled back except via full system recovery (see [[concepts/dgx-os-and-updates]]) — save work first. - JupyterLab environment issues are usually best resolved by changing the working directory (forcing a fresh virtual environment) rather than debugging the existing one. - Enterprise lifecycle integration scripts are reference material only — they are not preinstalled and must be downloaded and adapted, and their custom-installation workflow explicitly bypasses individual end-user license prompts, which carries legal/administrative responsibility for the person enabling it. - Nsight tooling versions are not pinned to a DGX OS release — always check the NVIDIA Developer site for the current build rather than assuming compatibility. ## Related Concepts - [[concepts/first-boot-and-setup]] — the connect-to-your-spark playbook that Sync/SSH access builds on - [[concepts/dgx-os-and-updates]] — the update mechanism the Dashboard drives - [[concepts/clustering]] — Sync's Cluster Assistant and multi-device management - [[concepts/comfyui-and-image-gen]] — an example workload run through JupyterLab/Dashboard - [[entities/playbooks-catalog]] — the dgx-dashboard playbook ## Sources - raw/web_community-dgx-dashboard-dgx-spark-user-guide.md - raw/github_doc-nvidia-dgx-dashboard-readme-md.md - raw/web_community-nvidia-sync-dgx-spark-user-guide.md - raw/web_community-nvidia-nsight-dgx-spark-user-guide.md - raw/web_community-system-configuration-and-operation-dgx-spark-user-guide.md - raw/web_community-common-use-cases-dgx-spark-user-guide.md - raw/web_community-enterprise-manageability-dgx-spark-user-guide.md --- title: "Unified Memory and ARM64 on DGX Spark" type: concept tags: [unified-memory, arm64, foundational, well-established] created: 2026-07-18 updated: 2026-07-18 sources: ["raw/web_community-system-overview-dgx-spark-user-guide.md", "raw/web_community-hardware-overview-dgx-spark-user-guide.md", "raw/web_community-known-issues-dgx-spark-user-guide.md", "raw/github_issue-llamacpp-eval-bug-gemma-4-generates-unused24-tokens.md", "raw/github_issue-vllm-bug-5090-cannot-run-qwen3-30b-a3b-nvfp4.md", "raw/github_issue-vllm-bug-runtimeerror-cuda-error-cublas-status-invalid-value-when.md", "raw/github_doc-nvidia-nemo-fine-tune-readme-md.md", "raw/github_doc-nvidia-llama-factory-readme-md.md", "raw/github_doc-nvidia-flux-finetuning-readme-md.md"] confidence: high dgx_os_version: 7.5.0 --- ## Definition DGX Spark's GB10 Grace Blackwell Superchip uses a **Unified Memory Architecture (UMA)**: there is one 128 GB pool of LPDDR5x DRAM shared by the CPU and the integrated Blackwell GPU, rather than the CPU having its own RAM and the GPU having a separate, smaller VRAM pool. Combined with a 20-core **ARM64 (aarch64)** CPU, this is the single fact that shapes almost every other page in this wiki — what models fit, how memory errors look, and which prebuilt software just works versus needs a rebuild. See [[concepts/overview-and-hardware]] for the surrounding hardware spec. ## How It Works **Unified memory.** In a conventional discrete-GPU machine, the GPU has its own VRAM (say, 24 GB) and the host has separate system RAM; anything the GPU needs must be copied across PCIe, and the VRAM figure is a hard ceiling on model size. On DGX Spark, the GPU and CPU both read and write the same 128 GB LPDDR5x pool (256-bit interface, 4266 MHz, 273 GB/s bandwidth) — there is no separate VRAM to run out of in the traditional sense, and no PCIe copy required to hand data between CPU and GPU code. This is what lets a single small-form-factor desktop box load models up to 200B parameters (405B across two clustered units — see [[concepts/clustering]]). The practical consequence for **model sizing and KV cache**: instead of asking "does the model fit in 24 GB of VRAM," the question becomes "does the model plus its KV cache plus activations fit in the 128 GB pool, alongside whatever else the CPU is using." A 70B model at Q4 quantization (~40 GB) plus a large KV cache for a long-context agentic session might still comfortably fit; the old VRAM-ceiling mental model over- or under-estimates capacity because it assumes a hard split that doesn't exist here. **The catch: `cudaMemGetInfo` lies (a little) on UMA.** The official Known Issues documentation is explicit about this: the CPU can dynamically manage DRAM contents on UMA systems, including freeing memory by swapping pages to the system's SWAP area. The `cudaMemGetInfo` CUDA API does **not** account for memory that could be reclaimed from SWAP, so it can under-report how much memory is actually allocatable — an application may refuse to load a model that would, in practice, fit once the kernel reclaims swappable pages. NVIDIA's own workaround (documented in the known-issues page and repeated verbatim in nearly every fine-tuning/quantization playbook's Troubleshooting section) is a manual buffer-cache flush: ```bash sudo sh -c 'sync; echo 3 > /proc/sys/vm/drop_caches' ``` The July 2026 DGX OS release (7.5.0) shipped an improved-OOM-handling driver specifically aimed at this: "User feedback is now available when the system encounters memory pressure, which improves robustness and reliability when operating larger models" (see [[summaries/release-digest]]). Older stacks are more likely to need the manual flush. **`nvidia-smi` reporting is also different.** On the GB10 iGPU, `nvidia-smi` shows `Memory-Usage: Not Supported` even though per-process GPU memory is listed — expected behavior because integrated GPUs have no dedicated framebuffer memory to report against, not a bug. **ARM64 / aarch64 realities.** The GB10's CPU is 20 Arm cores (10 Cortex-X925 + 10 Cortex-A725), meaning DGX Spark runs `aarch64` Linux, not `x86_64`. Every "official" playbook in this wiki (see [[entities/playbooks-catalog]]) either pins an ARM64-native container image (`nvcr.io/nvidia/pytorch:25.11-py3`, `nvcr.io/nvidia/nemo-automodel:26.02`, `nvcr.io/nvidia/tensorrt-llm/release:spark-single-gpu-dev`, etc.) or builds from source against the system CUDA toolkit — this is not incidental, it's because the ARM64 + Blackwell (`sm_121`/`sm_121a`) combination is narrower than the x86_64 + CUDA ecosystem most tooling assumes. Concretely: - **Compute-capability flags matter more than "any CUDA GPU."** llama.cpp's official Spark build pins `-DCMAKE_CUDA_ARCHITECTURES=121a-real` specifically for GB10 (see [[concepts/llama-cpp-on-spark]]); a generic CUDA build targeting `sm_120` (Blackwell consumer, a different sub-variant) has been reported to fail to compile MMQ kernels outright with `ptxas` errors like `Feature '.block_scale' not supported on .target 'sm_120'` — Blackwell sub-variants are not interchangeable, and neither are their compute-capability strings. - **Wheel availability is the most common source of setup friction.** PyTorch, Triton, and related packages generally need the CUDA-13-tagged ARM64 wheel index (`pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu130`) rather than a default PyPI install, which frequently resolves to an x86_64 or CUDA-12 build that either fails to import or silently runs on CPU. The NeMo AutoModel and LLaMA Factory playbooks both call this out in their own Troubleshooting tables as "ARM64 package compatibility issues may require troubleshooting." - **cuTile/kernel code is portable across Blackwell members, but tuning is not.** The cuTile Kernels playbook shows the *same* Python DSL kernel JIT-compiling for both DGX Spark (`sm_121`, 48 SMs) and B300 (`sm_103`, 132 SMs, HBM3e) — but the tile sizes and occupancy hints that are optimal on one are not optimal on the other (e.g. `TILE_M=64` for Spark vs `TILE_M=256` for B300); NVIDIA's own kernels use `ct.ByTarget({...})` to select per-architecture launch parameters rather than assuming portability of the numbers, only the code. - **GGUF/safetensors loading is otherwise architecture-agnostic**, but multi-GB model downloads plus a UMA memory model compound: a model that "fits" on paper can still trip the `cudaMemGetInfo` under-reporting issue above during load, which reads like an aarch64/Spark-specific bug but is actually the same UMA reporting gap independent of CPU architecture. ## Key Parameters - **Unified pool size**: 128 GB LPDDR5x, shared CPU+GPU, 273 GB/s bandwidth - **`cudaMemGetInfo`**: under-reports allocatable memory on UMA because it ignores SWAP-reclaimable DRAM — do not treat its output as a hard ceiling - **Manual cache flush**: `sudo sh -c 'sync; echo 3 > /proc/sys/vm/drop_caches'` — the standard workaround, repeated across nearly every official playbook - **`nvidia-smi` Memory-Usage: Not Supported`** — expected on the GB10 iGPU, not an error - **CPU architecture**: aarch64 (20-core Arm: 10 Cortex-X925 + 10 Cortex-A725) - **GPU compute capability**: `sm_121` / `121a` (GB10) — distinct from `sm_120` (Blackwell consumer/workstation) and `sm_103`/`sm_100` (B300/B300 Ultra) - **PyTorch/Triton install channel**: CUDA-13 ARM64 wheels via `--index-url https://download.pytorch.org/whl/cu130`, not default PyPI ## When To Use Reach for this concept whenever sizing a model or diagnosing a memory error on Spark: the question is never "does this fit in VRAM" (there is no separate VRAM), it's "does the process see enough allocatable unified memory, and if not, is that a real capacity problem or a `cudaMemGetInfo`/cache-pressure reporting artifact." Reach for the ARM64 half whenever a pip install, Docker image pull, or CUDA build fails in a way that looks unrelated to Spark — check architecture and compute-capability compatibility before assuming a model-specific or Spark-specific bug. This concept underlies every inference-stack page ([[concepts/llama-cpp-on-spark]], [[concepts/ollama-on-spark]], [[concepts/vllm-on-spark]], [[concepts/lm-studio-on-spark]]) and every fine-tuning workflow ([[concepts/fine-tuning-and-quantization]]). ## Risks & Pitfalls - **Trusting `cudaMemGetInfo` (or a memory API built on it) as a hard capacity check** will cause false "insufficient memory" failures on Spark that a manual cache flush or retry resolves — don't debug the model before ruling this out. - **Assuming any Blackwell CUDA build works on GB10.** The `sm_120` MMQ compile failure (confidence medium — reported on a discrete RTX card, not confirmed on GB10 itself, but the underlying compute-capability mismatch mechanism is architecture-general) shows that "Blackwell" is not one target; always match the exact compute-capability string (`121a` for Spark) rather than copying a flag from an unrelated Blackwell guide. - **Gemma 4 on GB10 generating runaway `` tokens** was reported explicitly on DGX Spark hardware (compute capability 12.1) — while this reads as a UMA/aarch64 issue at first glance, it is a sampling/vocabulary bug independent of the memory architecture; worth knowing so it isn't mis-triaged as a memory problem. See [[syntheses/troubleshooting-casebook]]. - **NVFP4 checkpoint failures on non-Spark Blackwell cards** (RTX 5090, RTX PRO 6000 — confidence medium, not confirmed on GB10) show `CUBLAS_STATUS_INVALID_VALUE` and outright load failures; because NVFP4 is a Blackwell-native format, these are a useful early warning that quantization-format support can lag hardware support even within the same GPU generation — verify NVFP4 compatibility per-stack before assuming it "just works" because the hardware is Blackwell (see [[concepts/fine-tuning-and-quantization]]). - **Default `pip install torch`** (no explicit index) frequently resolves to a build that doesn't match Spark's ARM64+CUDA13 stack — several official playbooks (NeMo AutoModel, LLaMA Factory) list this as their top Troubleshooting entry. - **Model downloads plus UMA memory pressure compound during large-model loads** — safetensors/GGUF loading itself is architecture-agnostic, but watch for OOM-looking failures during checkpoint loading that are actually the `cudaMemGetInfo` reporting gap rather than genuinely insufficient memory. ## Related Concepts - [[concepts/overview-and-hardware]] — the GB10 hardware spec this memory model runs on - [[concepts/clustering]] — extending the unified-memory ceiling across multiple Sparks - [[concepts/fine-tuning-and-quantization]] — where UMA memory pressure and NVFP4/aarch64 compatibility most directly bite - [[concepts/llama-cpp-on-spark]], [[concepts/ollama-on-spark]], [[concepts/vllm-on-spark]], [[concepts/lm-studio-on-spark]] — each stack's specific UMA/aarch64 notes - [[syntheses/troubleshooting-casebook]] — the memory/wheels/build failure patterns catalogued in one place - [[summaries/release-digest]] — DGX OS 7.5.0's improved OOM handling for UMA ## Sources - raw/web_community-system-overview-dgx-spark-user-guide.md - raw/web_community-hardware-overview-dgx-spark-user-guide.md - raw/web_community-known-issues-dgx-spark-user-guide.md - raw/github_issue-llamacpp-eval-bug-gemma-4-generates-unused24-tokens.md - raw/github_issue-llamacpp-compile-bug-cuda-build-for-mmq-breaks-for-compute-capability.md - raw/github_issue-vllm-bug-5090-cannot-run-qwen3-30b-a3b-nvfp4.md - raw/github_issue-vllm-bug-runtimeerror-cuda-error-cublas-status-invalid-value-when.md - raw/github_doc-nvidia-nemo-fine-tune-readme-md.md - raw/github_doc-nvidia-llama-factory-readme-md.md - raw/github_doc-nvidia-flux-finetuning-readme-md.md - raw/github_doc-nvidia-cutile-kernels-readme-md.md --- title: "vLLM on Spark" type: concept tags: [inference, vllm, arm64, clustering, foundational, well-established] created: 2026-07-18 updated: 2026-07-18 sources: ["raw/github_doc-nvidia-vllm-readme-md.md", "raw/github_issue-vllm-bug-5090-cannot-run-qwen3-30b-a3b-nvfp4.md", "raw/github_issue-vllm-bug-qwen3-5-nvfp4-checkpoint-has-poor-accuracy.md", "raw/github_issue-vllm-bug-qwen3-vl-235b-a22b-instruct-grounding-accuracy-issue-in-.md", "raw/github_issue-vllm-bug-runtimeerror-cuda-error-cublas-status-invalid-value-when.md", "raw/github_issue-vllm-bug-streaming-true-causes-missing-or-scrambled-tokens-with-g.md", "raw/github_issue-vllm-bug-vllm-gemma-4-claude-code-tool-calling-problems.md"] confidence: high dgx_os_version: 7.5.0 --- ## Definition vLLM is a high-throughput LLM serving engine built around PagedAttention and continuous batching, exposed through an OpenAI-compatible API. General vLLM concepts (PagedAttention internals, scheduler design, the broader flag surface) are covered in depth by the dedicated vLLM wiki in this fleet. On DGX Spark the official playbook's Spark-specific content is: which prebuilt NGC/custom containers actually work on GB10's ARM64 + Blackwell combination, the NVFP4/FP8 quantized model catalog NVIDIA has validated for it, and how to cluster two or more Sparks over QSFP with Ray for models too large for one unit. Note: a separate `station-vllm` playbook exists in the same source tree but targets **DGX Station** (GB300, 252GB HBM3e) — a different, larger product. It is not covered here; do not conflate its container tags (e.g. `26.01-py3`, `--cpu-offload-gb`) with the Spark playbook's. ## How It Works The Spark playbook is Docker-first. After adding the user to the `docker` group, a model is served from the NVIDIA NGC container: ```bash export HF_TOKEN="your_huggingface_token" export LATEST_VLLM_VERSION= ## e.g. 26.05.post1-py3 export HF_MODEL_HANDLE= ## e.g. openai/gpt-oss-20b docker pull nvcr.io/nvidia/vllm:${LATEST_VLLM_VERSION} docker run -d --gpus all -p 8000:8000 --name vllm-server \ nvcr.io/nvidia/vllm:${LATEST_VLLM_VERSION} \ vllm serve ${HF_MODEL_HANDLE} ``` Some model families need a different image: DiffusionGemma uses `vllm/vllm-openai:gemma`, the Gemma 4 family uses `vllm/vllm-openai:gemma4-cu130`. An "agent-ready" recipe serves `nvidia/Qwen3.6-35B-A3B-NVFP4` with a long flag list tuned for tool-calling agents (MTP speculative decoding, prefix caching, `--tool-call-parser qwen3_xml`) — this is what the [[concepts/agents-on-spark]] Hermes playbook builds on. For models too large for one Spark's 128GB, the playbook documents **two- and multi-Spark Ray clusters** over the QSFP interface: ```bash wget https://raw.githubusercontent.com/vllm-project/vllm/51c1ee9b7c8acbba4899a8ebffd390685d171946/examples/ray_serving/run_cluster.sh export MN_IF_NAME=enp1s0f1np1 export VLLM_HOST_IP=$(ip -4 addr show $MN_IF_NAME | grep -oP '(?<=inet\s)\d+(\.\d+){3}') bash run_cluster.sh $VLLM_IMAGE $VLLM_HOST_IP --head ~/.cache/huggingface \ -e VLLM_HOST_IP=$VLLM_HOST_IP -e UCX_NET_DEVICES=$MN_IF_NAME -e NCCL_SOCKET_IFNAME=$MN_IF_NAME \ -e MASTER_ADDR=$VLLM_HOST_IP ``` A worker node joins with `--worker` and `HEAD_NODE_IP` pointed at the head's QSFP address; then `vllm serve ... --tensor-parallel-size 2 --distributed-executor-backend ray` splits the model. See [[concepts/clustering]] for the underlying QSFP/switch network setup this depends on. ## Key Parameters - `--gpu-memory-utilization` — fraction of GPU memory for weights + KV cache (0.4–0.95 range across the playbook's recipes depending on model size and headroom needed). - `--tensor-parallel-size` — 1 for single-Spark, 2+ across a Ray cluster; must match node count. - `--kv-cache-dtype fp8` — halves KV-cache memory, common on agent-ready and multi-node recipes. - `--speculative-config '{"method":"mtp", ...}'` — MTP speculative decoding for compatible checkpoints (e.g. Qwen3.6-35B-A3B). - `--trust-remote-code` — required for Phi-4-multimodal-instruct and other models shipping custom modeling code. - `--enable-auto-tool-choice` + `--tool-call-parser ` — OpenAI-style function calling, needed for agentic use (see [[concepts/agents-on-spark]]). ## When To Use Reach for vLLM on Spark when throughput/concurrency matters more than build simplicity, when a model needs multi-Spark tensor parallelism to fit at all, or when the workload is agentic and needs vLLM's mature tool-calling parser ecosystem — NVIDIA's own [[concepts/agents-on-spark|Hermes Agent playbook]] standardizes on it. For a simpler single-process server, prefer [[concepts/llama-cpp-on-spark]]; for zero-fuss model management, [[concepts/ollama-on-spark]]. See [[syntheses/stack-picker]]. ## Risks & Pitfalls - **NVFP4 checkpoints are not a free lunch on Blackwell.** A confidence-medium (single-issue, non-Spark but same Blackwell/NVFP4 ecosystem) report found `nvidia/Qwen3.5-397B-A17B-NVFP4` scoring ~0.35 on GSM8K vs. 0.90 for the FP8 original — a large accuracy regression that persisted across `TRITON_ATTN`/`FLASHINFER` backends and `--kv-cache-dtype` settings. Validate accuracy on your own eval before trusting an NVFP4 quant's throughput/memory wins at face value, especially for large MoE models. - **Consumer/workstation Blackwell GPUs (sm_120, not Spark's sm_121a) have hit outright failures loading NVFP4 checkpoints** and CUBLAS errors during GEMM calls (confidence medium, reported on RTX 5090 and RTX PRO 6000 Blackwell, not confirmed on GB10) — if a `CUBLAS_STATUS_INVALID_VALUE` or NVFP4 load failure appears, check the vLLM/CUDA/driver version triple against what NVIDIA has validated for GB10 before assuming the model itself is broken. - **Streaming responses can arrive missing or scrambled tokens** with some large MoE models (reported with GPT-OSS 120B, confidence medium, non-Spark hardware) while non-streaming and `--enforce-eager` both produce correct output at a speed cost — if a served agent sees garbled streamed output, try disabling streaming or `--enforce-eager` as a diagnostic step before filing it as a model problem. - **Tool-calling reliability with Gemma 4 + coding agents has been reported as inconsistent** (confidence medium, single issue, Blackwell workstation GPUs) — worth smoke-testing `--tool-call-parser gemma4` end-to-end with your actual agent client before depending on it for unattended agentic workflows (see [[concepts/agents-on-spark]]). - Gated models (Llama 3.3 70B, FLUX, etc.) require accepting the license on Hugging Face and authenticating with `hf auth login` **inside the container** so the token lands in the mounted cache — a common source of "cannot access gated repo" errors that look like network problems. - Multi-Spark clusters: closing the terminal running `run_cluster.sh` tears down the container (it has an `EXIT` trap) — always run cluster nodes inside `tmux`/`screen`. - Standard UMA caveat applies (`sudo sh -c 'sync; echo 3 > /proc/sys/vm/drop_caches'`) — see [[concepts/unified-memory-and-arm64]]. ## Related Concepts - [[concepts/clustering]] — QSFP/switch network setup underlying multi-Spark vLLM - [[concepts/unified-memory-and-arm64]] — memory model implications for large-model serving - [[concepts/agents-on-spark]] — Hermes Agent's vLLM-backed "agent-ready" recipe - [[concepts/llama-cpp-on-spark]] / [[concepts/ollama-on-spark]] — simpler alternatives - [[syntheses/troubleshooting-casebook]] — cross-stack NVFP4/Blackwell failure patterns - [[syntheses/stack-picker]] — choosing among inference stacks on Spark - [[entities/playbooks-catalog]] — full list of official Spark playbooks ## Sources - raw/github_doc-nvidia-vllm-readme-md.md - raw/github_issue-vllm-bug-5090-cannot-run-qwen3-30b-a3b-nvfp4.md - raw/github_issue-vllm-bug-qwen3-5-nvfp4-checkpoint-has-poor-accuracy.md - raw/github_issue-vllm-bug-qwen3-vl-235b-a22b-instruct-grounding-accuracy-issue-in-.md - raw/github_issue-vllm-bug-runtimeerror-cuda-error-cublas-status-invalid-value-when.md - raw/github_issue-vllm-bug-streaming-true-causes-missing-or-scrambled-tokens-with-g.md - raw/github_issue-vllm-bug-vllm-gemma-4-claude-code-tool-calling-problems.md --- title: "DGX Spark Playbooks Catalog" type: entity tags: [setup, inference, agents, fine-tuning, foundational, well-established] created: 2026-07-18 updated: 2026-07-18 sources: ["raw/github_doc-readme-md.md", "raw/github_doc-nvidia-cli-coding-agent-readme-md.md", "raw/github_doc-nvidia-comfy-ui-readme-md.md", "raw/github_doc-nvidia-connect-three-sparks-readme-md.md", "raw/github_doc-nvidia-connect-to-your-spark-readme-md.md", "raw/github_doc-nvidia-connect-two-sparks-assets-performance-benchmarking-gu.md", "raw/github_doc-nvidia-connect-two-sparks-readme-md.md", "raw/github_doc-nvidia-cuda-x-data-science-readme-md.md", "raw/github_doc-nvidia-cutile-kernels-readme-md.md", "raw/github_doc-nvidia-dgx-dashboard-readme-md.md", "raw/github_doc-nvidia-flux-finetuning-readme-md.md", "raw/github_doc-nvidia-hermes-agent-readme-md.md", "raw/github_doc-nvidia-isaac-readme-md.md", "raw/github_doc-nvidia-jax-readme-md.md", "raw/github_doc-nvidia-live-vlm-webui-readme-md.md", "raw/github_doc-nvidia-llama-cpp-readme-md.md", "raw/github_doc-nvidia-llama-factory-readme-md.md", "raw/github_doc-nvidia-lm-studio-readme-md.md", "raw/github_doc-nvidia-multi-agent-chatbot-readme-md.md", "raw/github_doc-nvidia-multi-modal-inference-readme-md.md", "raw/github_doc-nvidia-multi-sparks-through-switch-readme-md.md", "raw/github_doc-nvidia-nccl-readme-md.md", "raw/github_doc-nvidia-nemo-fine-tune-readme-md.md", "raw/github_doc-nvidia-nemoclaw-applications-readme-md.md", "raw/github_doc-nvidia-nemoclaw-readme-md.md", "raw/github_doc-nvidia-nemotron-readme-md.md", "raw/github_doc-nvidia-nim-llm-readme-md.md", "raw/github_doc-nvidia-nvfp4-quantization-readme-md.md", "raw/github_doc-nvidia-ollama-readme-md.md", "raw/github_doc-nvidia-open-webui-readme-md.md", "raw/github_doc-nvidia-openclaw-readme-md.md", "raw/github_doc-nvidia-openshell-readme-md.md", "raw/github_doc-nvidia-portfolio-optimization-readme-md.md", "raw/github_doc-nvidia-pytorch-fine-tune-readme-md.md", "raw/github_doc-nvidia-rag-ai-workbench-readme-md.md", "raw/github_doc-nvidia-reachy-photo-booth-readme-md.md", "raw/github_doc-nvidia-register-to-brev-readme-md.md", "raw/github_doc-nvidia-sglang-readme-md.md", "raw/github_doc-nvidia-single-cell-readme-md.md", "raw/github_doc-nvidia-spark-reachy-photo-booth-readme-md.md", "raw/github_doc-nvidia-speculative-decoding-readme-md.md", "raw/github_doc-nvidia-station-ai-skills-readme-md.md", "raw/github_doc-nvidia-station-brev-readme-md.md", "raw/github_doc-nvidia-station-comfyui-readme-md.md", "raw/github_doc-nvidia-station-connect-two-stations-readme-md.md", "raw/github_doc-nvidia-station-gr00t-readme-md.md", "raw/github_doc-nvidia-station-healthcare-agent-readme-md.md", "raw/github_doc-nvidia-station-kernel-dev-ft-readme-md.md", "raw/github_doc-nvidia-station-local-coding-agent-readme-md.md", "raw/github_doc-nvidia-station-mig-readme-md.md", "raw/github_doc-nvidia-station-nanochat-readme-md.md", "raw/github_doc-nvidia-station-nemoclaw-readme-md.md", "raw/github_doc-nvidia-station-nvfp4-pretraining-readme-md.md", "raw/github_doc-nvidia-station-nvfp4-quantization-readme-md.md", "raw/github_doc-nvidia-station-openshell-readme-md.md", "raw/github_doc-nvidia-station-rec-sys-readme-md.md", "raw/github_doc-nvidia-station-sglang-inference-readme-md.md", "raw/github_doc-nvidia-station-topic-modeling-readme-md.md", "raw/github_doc-nvidia-station-txt2kg-readme-md.md", "raw/github_doc-nvidia-station-vllm-readme-md.md", "raw/github_doc-nvidia-tailscale-readme-md.md", "raw/github_doc-nvidia-trt-llm-readme-md.md", "raw/github_doc-nvidia-txt2kg-readme-md.md", "raw/github_doc-nvidia-unsloth-readme-md.md", "raw/github_doc-nvidia-vibe-coding-readme-md.md", "raw/github_doc-nvidia-vllm-readme-md.md", "raw/github_doc-nvidia-vscode-readme-md.md", "raw/github_doc-nvidia-vss-readme-md.md"] confidence: high dgx_os_version: 7.5.0 --- ## Overview The `nvidia/dgx-spark-playbooks` GitHub repository (`https://github.com/NVIDIA/dgx-spark-playbooks`) is NVIDIA's official collection of step-by-step guides for doing specific things on a DGX Spark or DGX Station. Every playbook lives at `https://github.com/NVIDIA/dgx-spark-playbooks/blob/main/nvidia//README.md` and follows the same template (Overview → Instructions → Troubleshooting, with "What you'll accomplish," Prerequisites, and "Time & risk" sections). This wiki mirrors 66 of these playbook READMEs; this page is the catalog that lets an agent answer "is there a playbook for X" without reading all 66. The top-level repo README is itself a stale/incomplete index (it omits most `station-*` playbooks) — treat this catalog, not that README, as the authoritative list. Playbooks come in **Spark** and **Station** flavors for several topics: DGX Station (GB300 Blackwell Ultra, larger GPU/memory) gets a parallel playbook with a `station-` prefix covering the same workflow at larger scale (e.g. `vllm` / `station-vllm`, `nvfp4-quantization` / `station-nvfp4-quantization`). This is intentional duplication for a different product line, not redundant content. ## Characteristics **Setup & Connectivity (11)** - `connect-to-your-spark` — local network access to a single Spark via NVIDIA Sync or SSH - `connect-two-sparks` — direct 200GbE QSFP link between two Sparks (see [[concepts/clustering]]) - `connect-three-sparks` — 200GbE ring topology for three Sparks - `multi-sparks-through-switch` — cluster of four+ Sparks via a managed QSFP switch - `station-connect-two-stations` — dual ConnectX-8 400Gb/s RoCEv2 link between two DGX Stations - `register-to-brev` / `station-brev` — register a Spark/Station to NVIDIA Brev for remote/shared access - `tailscale` — VPN-based remote access to a Spark - `vscode` — VS Code on-device or remote via NVIDIA Sync - `nccl` — build/test NCCL for GPU-to-GPU communication across two Sparks - `dgx-dashboard` — web dashboard for monitoring, updates, and JupyterLab launch (see [[concepts/system-operations]]) **Inference & Model Serving (14)** - `llama-cpp` — build llama.cpp with CUDA, serve GGUF via OpenAI-compatible API (see [[concepts/llama-cpp-on-spark]]) - `ollama` — install Ollama, tunnel remote access via NVIDIA Sync (see [[concepts/ollama-on-spark]]) - `lm-studio` — headless LM Studio + LM Link remote access (see [[concepts/lm-studio-on-spark]]) - `vllm` / `station-vllm` — high-throughput serving, single/multi-Spark or DGX Station (see [[concepts/vllm-on-spark]]) - `sglang` / `station-sglang-inference` — RadixAttention serving framework, prefix caching, structured output - `trt-llm` — TensorRT-LLM inference, single or multi-Spark - `nim-llm` — deploy an NVIDIA NIM microservice (e.g. Llama 3.1 8B, Qwen3-32B) via Docker + NGC auth - `nemotron` — deploy the Nemotron 3 model family (Nano or Super) - `speculative-decoding` — EAGLE-3 / draft-target speculative decoding, including two-Spark - `multi-modal-inference` — text/image/audio multi-modal inference via TensorRT - `live-vlm-webui` — real-time VLM webcam streaming/benchmarking web UI - `open-webui` — self-hosted chat UI backed by Ollama **Agents & Coding Tools (13)** - `cli-coding-agent` / `station-local-coding-agent` — local CLI coding agents (Claude Code, OpenCode, Codex CLI) via Ollama - `vibe-coding` — Ollama + Continue.dev in VS Code as a local coding assistant - `hermes-agent` — Hermes self-improving AI agent, optional Telegram integration - `nemoclaw` / `station-nemoclaw` — local AI assistant with NemoClaw + vLLM inside an OpenShell sandbox - `nemoclaw-applications` — example NemoClaw apps (news digest, dev agent, deck reviewer) - `openclaw` — persistent local-first AI agent backed by vLLM - `openshell` / `station-openshell` — kernel-level sandbox isolation for OpenClaw - `station-ai-skills` — AGENTS.md + on-demand Agent Skills for coding agents (Claude Code/Codex/Gemini CLI/Cursor) - `station-healthcare-agent` — six-agent healthcare system (patient data + OpenFold3), OpenShell-sandboxed - `multi-agent-chatbot` — supervisor-led multi-agent chatbot (coding/RAG/vision sub-agents) **Fine-Tuning, Quantization & Kernel Dev (11)** — see [[concepts/fine-tuning-and-quantization]] for the full concept treatment - `nemo-fine-tune` — NeMo AutoModel PEFT/SFT fine-tuning - `pytorch-fine-tune` — PyTorch fine-tuning, including multi-Spark Docker Swarm - `llama-factory` — LLaMA Factory CLI (SFT/RLHF/QLoRA) - `unsloth` — faster/memory-efficient fine-tuning via custom kernels - `flux-finetuning` — FLUX.1-dev 12B Dreambooth LoRA image fine-tuning - `nvfp4-quantization` / `station-nvfp4-quantization` — TensorRT Model Optimizer NVFP4 quantization - `station-nvfp4-pretraining` — NVFP4 mixed-precision pretraining (Megatron Bridge, DGX Station) - `cutile-kernels` — cuTile kernel benchmarks, FMHA from scratch, LLM inference - `station-kernel-dev-ft` — profiler-driven Triton kernel optimization for fine-tuning - `station-nanochat` — train a small ChatGPT-style LLM end-to-end (tokenizer → pretrain → SFT) **Data Science & Analytics (9)** - `cuda-x-data-science` — cuDF/cuML acceleration of pandas/scikit-learn/UMAP/HDBSCAN - `jax` — optimizing JAX on Spark's GPU - `single-cell` — GPU-accelerated scRNA-seq via RAPIDS-singlecell/Scanpy - `portfolio-optimization` — Mean-CVaR portfolio optimization via cuOpt/cuML - `station-rec-sys` — HLLM+LightGBM fashion recommender with LoRA, FAISS, PPO pricing agent - `station-topic-modeling` — BERTopic over 40M Amazon reviews via cuML - `txt2kg` / `station-txt2kg` — text-to-knowledge-graph pipelines (Station variant uses Llama 3.1 405B) - `rag-ai-workbench` — agentic RAG app via NVIDIA AI Workbench **Image / Video Generation (2)** — see [[concepts/comfyui-and-image-gen]] - `comfy-ui` — node-based image generation (SDXL, Flux) - `station-comfyui` — FLUX, Wan 2.1, HunyuanVideo, Cosmos-Predict2 generation **Robotics / Hardware Demos (4)** - `isaac` — Isaac Sim + Isaac Lab built from source - `reachy-photo-booth` / `spark-reachy-photo-booth` — Reachy Mini robot photo booth (likely duplicate/renamed pair — see Risks) - `station-gr00t` — fine-tune/benchmark Isaac GR00T N1.6 robotics foundation model **Monitoring & Misc Infra (2)** - `station-mig` — Multi-Instance GPU partitioning on DGX Station's B300 - `vss` — NVIDIA Video Search and Summarization Blueprint (VLM+LLM+RAG video analytics) **Total: 66 playbooks catalogued** (plus the top-level org README for context). ## How to Use To check whether a workflow has an official playbook, scan the category lists above for the closest match, then read the corresponding `raw/github_doc-nvidia--readme-md.md` mirror (or `raw/github_doc-nvidia-station--readme-md.md` for the Station variant) for exact commands. Every playbook follows the same structure, so once one has been read (e.g. [[concepts/llama-cpp-on-spark]]'s source), the shape of any other is predictable: Overview → Prerequisites → numbered Instructions → Troubleshooting table → a standard UMA cache-flush note (see [[concepts/unified-memory-and-arm64]]). To clone the actual repo and run a playbook locally: `git clone https://github.com/NVIDIA/dgx-spark-playbooks`, then `cd dgx-spark-playbooks/nvidia/` and follow that directory's README. ## Related Entities - [[concepts/unified-memory-and-arm64]] — the UMA cache-flush note and ARM64 wheel caveat that recur across nearly every playbook - [[concepts/fine-tuning-and-quantization]] — deep dive on the fine-tuning/quantization/kernel-dev cluster - [[concepts/llama-cpp-on-spark]], [[concepts/ollama-on-spark]], [[concepts/vllm-on-spark]], [[concepts/lm-studio-on-spark]] — deep dives on the individual inference-stack playbooks - [[concepts/clustering]] — connect-two-sparks/connect-three-sparks/multi-sparks-through-switch/nccl playbooks - [[concepts/comfyui-and-image-gen]] — comfy-ui/station-comfyui/flux-finetuning - [[syntheses/stack-picker]] — choosing among the inference-serving playbooks - [[syntheses/troubleshooting-casebook]] — cross-playbook failure patterns ## Risks & Notes - `reachy-photo-booth` and `spark-reachy-photo-booth` share an identical title and tagline in their README mirrors — very likely a duplicate or a renamed/superseded slug for the same playbook rather than two distinct workflows; verify against the live repo before treating them as separate. - `multi-sparks-through-switch` and `nemotron` had sparser "Basic idea" prose than most other playbooks in this set — their one-line descriptions above lean more on the title/tagline than on rich body text (confidence medium for those two entries specifically; high for the rest). - The repo's own top-level README undercounts the playbook set (missing most `station-*` entries) — do not use it as a completeness check. ## Sources See frontmatter `sources` list — all 66 `raw/github_doc-nvidia-*-readme-md.md` mirrors plus `raw/github_doc-readme-md.md` (top-level repo README). --- title: "Activity Log" type: log --- # Activity Log Append-only record of all wiki changes. ## Format Each entry follows this format: ``` ### YYYY-MM-DD HH:MM — [Action Type] - **Source/Trigger**: what initiated the action - **Pages created**: list of new pages - **Pages updated**: list of updated pages - **Notes**: any contradictions flagged, decisions made ``` --- ### 2026-04-08 00:00 — Setup - **Source/Trigger**: Repository initialized - **Pages created**: index.md, log.md, dashboard.md, analytics.md, flashcards.md - **Pages updated**: none - **Notes**: Empty knowledge base ready for first source ingestion --- title: "DGX Spark Release Notes Digest" type: summary tags: [dgx-os, unified-memory, foundational, well-established] created: 2026-07-18 updated: 2026-07-18 sources: ["raw/web_community-dgx-spark-release-notes-dgx-spark-user-guide.md"] confidence: high dgx_os_version: 7.5.0 --- ## Key Points - **Current software versions (as of the July 2026 release, DGX Spark Founders Edition only — partner GB10 systems may lag):** NVIDIA DGX OS **7.5.0**, NVIDIA GPU Driver 580.159.03, NVIDIA CUDA Toolkit 13.0.2, Canonical Kernel 6.17, UEFI 1.110.13, Embedded Controller (EC) 3.5.8, USB Power Delivery 0.5.22, TPM 7.516.1, System on Chip (SoC) 2.155.11. - **July 2026 release — What's New:** Improved Memory Management (the driver enhances OOM handling for GB10's unified memory architecture, giving user-visible feedback under memory pressure — see [[concepts/unified-memory-and-arm64]] for why this matters); Adjustable Display Reserved Memory (toggle between 2GB default and 4GB via BIOS, for workflows with many open windows); Enterprise Management enhancements (cloud-init now supports DHCP/HTTP/TFTP network protocols in addition to direct USB, and FastOS recovery image customization). - **July 2026 — Fixed Issues:** system/display instability when hot-plugging displays. - **June 2026 release:** OOBE (Out-of-Box-Experience) enhancements — OTA updates no longer install by default during initial setup (users can update after setup completes); the DGX Spark playbook site now opens after first boot with the NemoClaw playbook prominently displayed for sandboxed local-agent setup (see [[entities/playbooks-catalog]]); NVIDIA Sync's Settings page gained a **Cluster Assistant** that helps connect up to three Sparks without a switch or four with one (see [[concepts/clustering]]); NCCL updated to v2.30u1 with ring-topology support for three-Spark clusters. - **April 2026 release:** Enterprise Features for fleet management — an Enterprise Manageability guide, ability to skip the automated OOBE for IT-administrator provisioning, USB/local-repository-based installs and updates (no cloud dependency), air-gapped deployment support, and cloud-init-based customized enterprise ISOs. DGX Dashboard gained release highlights so admins can gauge update urgency. Fixed: Bluetooth keyboard pairing modal not appearing in some cases; Wi-Fi/Ethernet connectivity issues during system setup. - **March 2026 release:** firmware update to facilitate GB10 partner-factory updates (no user-facing changes). - **February 2026 release:** fixed a performance regression affecting multiple connected DGX Spark systems that was introduced by DGX OS 7.4.0 — a reminder that multi-Spark cluster performance has regressed at least once historically after an OS update, worth checking after any future upgrade (see [[concepts/clustering]]). - **January 2026 release:** hot-plug power management for the ConnectX-7 adapter (saves up to 18W when unused); Bluetooth audio device support; Wi-Fi/Bluetooth can now be disabled in UEFI for security-sensitive environments. Fixed: monitor/TV compatibility (multi-monitor, non-native resolutions); smoother OOBE peripheral setup. - **November 2025 release:** DGX OS moved to the Ubuntu 6.14 HWE kernel stack for broader hardware compatibility and security updates; JupyterLab updated to CUDA 13.0.2 and the latest PyTorch. Fixed: memory-reporting differences in DGX Dashboard for unified memory architecture (readout now consistent with CUDA's UMA guidance — directly related to the `cudaMemGetInfo` behavior documented in [[concepts/unified-memory-and-arm64]]); Stable Diffusion XL JupyterLab playbook image generation restored; improved USB-C/monitor/Bluetooth/Wi-Fi peripheral interoperability; more reliable recovery-image installation on macOS and with multiple external USB-C drives; keyboard-only OOBE completion. - **Known-issue pattern, current as of this digest:** the official Known Issues page (a separate document from the release notes, but directly related) centers on the same unified-memory-architecture theme that recurs across nearly every release above — `nvidia-smi` reporting `Memory-Usage: Not Supported` on the iGPU (expected, not a bug), and `cudaMemGetInfo` under-reporting allocatable memory because it ignores DRAM reclaimable from SWAP (workaround: `sudo sh -c 'sync; echo 3 > /proc/sys/vm/drop_caches'`). A secondary known issue: attached HDMI displays can enter deep sleep after long inactivity and require a physical button press to wake. ## Relevant Concepts - [[concepts/unified-memory-and-arm64]] — the memory-reporting fix (Nov 2025) and OOM-handling improvement (July 2026) both target the exact UMA behavior this concept page explains in depth - [[concepts/clustering]] — the Cluster Assistant (June 2026), NCCL ring-topology support (June 2026), and the Feb 2026 multi-Spark performance-regression fix all land here - [[concepts/dgx-os-and-updates]] — release cadence, cloud-init, and enterprise-fleet-management mechanics this digest's version history feeds into - [[concepts/system-operations]] — DGX Dashboard release-highlights feature and NVIDIA Sync's Cluster Assistant UI - [[entities/playbooks-catalog]] — the NemoClaw playbook surfaced in the OOBE as of June 2026 ## Source Metadata - **Type:** Official web documentation (`web_community`) - **Title:** "DGX Spark Release Notes — DGX Spark User Guide" - **Source URL:** https://docs.nvidia.com/dgx/dgx-spark/release-notes.html - **Fetched:** 2026-07-19 - **Scope:** Release notes from November 2025 through July 2026 (current at time of mirroring), covering DGX OS versions up to and including 7.5.0. Applies to DGX Spark Founders Edition; partner GB10-based systems may not receive updates on the same schedule. --- title: "Stack Picker: Choosing an Inference Stack on Spark" type: synthesis tags: [inference, llama-cpp, ollama, vllm, lm-studio, nim, unified-memory] created: 2026-07-18 updated: 2026-07-18 sources: ["raw/github_doc-nvidia-llama-cpp-readme-md.md", "raw/github_doc-nvidia-ollama-readme-md.md", "raw/github_doc-nvidia-vllm-readme-md.md", "raw/github_doc-nvidia-lm-studio-readme-md.md", "raw/github_doc-nvidia-nim-llm-readme-md.md", "raw/github_doc-nvidia-sglang-readme-md.md", "raw/github_doc-nvidia-trt-llm-readme-md.md", "raw/github_doc-nvidia-speculative-decoding-readme-md.md"] confidence: medium dgx_os_version: 7.5.0 --- ## Comparison | Stack | Install model | Serving API | Model management | Multi-Spark | Best-fit workload | |---|---|---|---|---|---| | **llama.cpp** ([[concepts/llama-cpp-on-spark]]) | Build from source, CMake pinned to `sm_121a` | `llama-server`, OpenAI-compatible | Manual GGUF (`-hf` auto-pull or `--model` local path) | Not built-in (single-process) | Simple, no-framework GGUF serving; widest day-one GGUF quant coverage for new model releases | | **Ollama** ([[concepts/ollama-on-spark]]) | `curl install.sh` (prebuilt binary) | REST API on :11434 | `ollama pull`/library-managed | Not built-in per playbook (Docker-based multi-instance possible) | Convenience-first single-command model switching; backend for [[concepts/agents-on-spark]] CLI coding agents, Open WebUI | | **vLLM** ([[concepts/vllm-on-spark]]) | Docker (NGC container, model-family-specific images for Gemma/DiffusionGemma) | OpenAI-compatible, PagedAttention + continuous batching | HF handle at container launch | Yes — Ray cluster over QSFP, `--tensor-parallel-size` | High-throughput/high-concurrency serving; models too large for one Spark; agentic tool-calling (mature `--tool-call-parser` ecosystem) | | **LM Studio** ([[concepts/lm-studio-on-spark]]) | `llmster` headless daemon install script | REST API on :1234 (`lms` CLI/SDK) | `lms get`/`lms load`, GUI-adjacent catalog | Not covered in playbook | GUI-adjacent model catalog + LM Link zero-config remote access (Tailscale mesh) instead of manual SSH tunneling | | **NIM** (`nim-llm` playbook) | Docker (NGC-hosted per-model container, e.g. `llama-3.1-8b-instruct-dgx-spark`) | OpenAI-compatible | One container per model, NGC-managed | Not covered in playbook | NVIDIA-validated, pre-optimized per-model containers when reproducibility/support matters more than DIY flexibility | | **SGLang** (`sglang`/`station-sglang-inference`) | Install per playbook | OpenAI-compatible | Per-model | Not covered for Spark specifically | RadixAttention prefix caching for repeated-prefix workloads (agentic loops, few-shot prompting) | | **TensorRT-LLM** (`trt-llm`) | Docker, single or multi-Spark | OpenAI-compatible (`trtllm-serve`) | Per-model, often paired with NVFP4 quantization (see [[concepts/fine-tuning-and-quantization]]) | Yes, per playbook | Maximum throughput once a model is quantized/compiled for TensorRT; pairs with the `nvfp4-quantization` playbook's own serving step | ## Analysis **Simplicity vs. throughput is the main axis.** llama.cpp and Ollama sit at the "just run a model" end — llama.cpp for framework-free control, Ollama for zero-friction model switching and the widest downstream integration (it's the default backend nearly every agent/chatbot playbook falls back to). vLLM, TensorRT-LLM, and SGLang sit at the "serving infrastructure" end — Docker-first, tuned for concurrent requests, and the only stacks in this list with a documented multi-Spark path (vLLM's Ray cluster; TensorRT-LLM's own multi-Spark support). NIM is a distinct third axis: it trades DIY flexibility for NVIDIA-validated, per-model container reproducibility — useful when the question is "will this exact model/hardware combination behave the way NVIDIA tested it" rather than "give me maximum tuning control." **Unified memory changes the calculus versus a discrete-GPU box.** On a traditional multi-GPU server, the stack choice is often forced by which framework supports the available VRAM split. On Spark's single 128 GB unified pool (see [[concepts/unified-memory-and-arm64]]), a much wider range of model sizes fit on one unit regardless of stack, so the deciding factor shifts to serving characteristics (concurrency, tool-calling maturity, prefix caching) rather than "which stack can even fit this model." The exception is genuinely oversized models (200B+ single-unit ceiling, 405B needing two units — see [[concepts/clustering]]), where vLLM's Ray-based tensor parallelism or TensorRT-LLM's multi-Spark path become necessary, not just preferable. **ARM64/aarch64 affects install friction more than capability.** All five stacks run on Spark, but llama.cpp (build-from-source, exact `sm_121a` flag) and vLLM/TensorRT-LLM (NGC-published ARM64-native containers) sidestep the wheel-availability problem described in [[concepts/unified-memory-and-arm64]] by either compiling natively or shipping a pre-built ARM64 image; a bare `pip install vllm` outside the NGC container is more likely to hit the same ARM64/CUDA-13-wheel friction that fine-tuning playbooks report. **Agentic/tool-calling maturity favors vLLM currently.** NVIDIA's own Hermes Agent playbook (see [[concepts/agents-on-spark]]) standardizes on vLLM specifically for its `--tool-call-parser`/`--enable-auto-tool-choice` flags. The [[syntheses/troubleshooting-casebook]] casebook shows this isn't friction-free (Gemma 4 tool-calling reliability issues reported on vLLM, and separately on llama.cpp under OpenClaw-style grammar-constrained calling) — but vLLM's parser ecosystem is more actively maintained for structured tool-calling than llama.cpp's grammar-based approach or Ollama's simpler API. **NVFP4/quantization interacts with stack choice.** A model quantized to NVFP4 via the `nvfp4-quantization` playbook (see [[concepts/fine-tuning-and-quantization]]) is served via TensorRT-LLM (`trtllm-serve`) or vLLM in the official playbooks — llama.cpp and Ollama's ecosystems are GGUF/K-quant-centric and don't feature in the NVFP4 pipeline the same way. If NVFP4 specifically is the goal (for its ~3.5x memory reduction vs FP16), that pulls the stack choice toward vLLM or TensorRT-LLM regardless of the other tradeoffs above — with the important caveat from the troubleshooting casebook that NVFP4 accuracy should be independently validated, not assumed. ## Recommendations - **Pick llama.cpp** if the goal is a single dependency-free server, the widest same-day GGUF quant availability for brand-new model releases, and full manual control over context size/speculative decoding flags. - **Pick Ollama** if convenience and fast model switching matter most, or the target is a downstream tool (CLI coding agent, Open WebUI, Live VLM WebUI) that already expects an Ollama backend by default. - **Pick vLLM** if the workload is agentic (tool-calling), needs concurrent multi-request throughput, or the model is too large for one Spark and needs a multi-Spark Ray cluster. - **Pick LM Studio** if a GUI-adjacent model catalog and zero-config remote access (LM Link over Tailscale) matter more than scripted/production tuning — and if the downstream client already speaks LM Studio's local API (Claude Code, Codex, OpenCode all work unmodified through LM Link). - **Pick NIM** when reproducibility and NVIDIA-validated performance matter more than tuning flexibility — e.g. deploying a model NVIDIA has specifically packaged for DGX Spark (Llama 3.1 8B, Qwen3-32B). - **Pick TensorRT-LLM (or vLLM) specifically** when serving an NVFP4-quantized model, since that's the pipeline the official quantization playbook targets. - **Regardless of stack**, budget for the [[concepts/unified-memory-and-arm64]] UMA cache-flush workaround and validate any NVFP4 model's accuracy independently — see [[syntheses/troubleshooting-casebook]] for concrete, stack-specific failure cases before committing to a production deployment. ## Pages Compared - [[concepts/llama-cpp-on-spark]] - [[concepts/ollama-on-spark]] - [[concepts/vllm-on-spark]] - [[concepts/lm-studio-on-spark]] - [[concepts/unified-memory-and-arm64]] - [[concepts/fine-tuning-and-quantization]] - [[concepts/clustering]] - [[concepts/agents-on-spark]] - [[syntheses/troubleshooting-casebook]] - [[entities/playbooks-catalog]] — nim-llm, sglang, trt-llm, speculative-decoding, station-sglang-inference entries --- title: "Troubleshooting Casebook: llama.cpp, Ollama, and vLLM on Spark" type: synthesis tags: [troubleshooting, unified-memory, arm64, llama-cpp, ollama, vllm] created: 2026-07-18 updated: 2026-07-18 sources: ["raw/web_community-known-issues-dgx-spark-user-guide.md", "raw/github_issue-llamacpp-compile-bug-cuda-build-for-mmq-breaks-for-compute-capability.md", "raw/github_issue-llamacpp-eval-bug-data-corruption-on-cuda-experts-offload.md", "raw/github_issue-llamacpp-eval-bug-gemma-4-generates-unused24-tokens.md", "raw/github_issue-llamacpp-misc-bug-many-grammar-errors-when-called-by-openclaw.md", "raw/github_issue-llamacpp-misc-bug-vulkan-backend-llama-server-and-llama-bench-cannot-.md", "raw/github_issue-llamacpp-model-request-deepseek-v4-series.md", "raw/github_issue-ollama-can-t-pull-nvfp4.md", "raw/github_issue-ollama-i-can-t-run-gemma4-31b.md", "raw/github_issue-ollama-ollama-slow-performance.md", "raw/github_issue-ollama-overrun-of-a-stack-based-buffer-error-when-running-phi4.md", "raw/github_issue-ollama-qwen3-embedding-8b-model-does-not-support-embeddings.md", "raw/github_issue-ollama-version-11-bombing-out-and-responds-with-ggggggggggggggg.md", "raw/github_issue-vllm-bug-5090-cannot-run-qwen3-30b-a3b-nvfp4.md", "raw/github_issue-vllm-bug-qwen3-5-nvfp4-checkpoint-has-poor-accuracy.md", "raw/github_issue-vllm-bug-qwen3-vl-235b-a22b-instruct-grounding-accuracy-issue-in-.md", "raw/github_issue-vllm-bug-runtimeerror-cuda-error-cublas-status-invalid-value-when.md", "raw/github_issue-vllm-bug-streaming-true-causes-missing-or-scrambled-tokens-with-g.md", "raw/github_issue-vllm-bug-vllm-gemma-4-claude-code-tool-calling-problems.md"] confidence: medium dgx_os_version: 7.5.0 --- ## Comparison Eighteen GitHub issues (six each from llama.cpp, Ollama, and vLLM) were mirrored for this casebook, plus the official DGX Spark Known Issues page. **Only one of the 18 issues (the Gemma 4 `` token spam) was explicitly reproduced on DGX Spark hardware** (`Hardware: DGX Spark`, `NVIDIA GB10, compute capability 12.1`) — one more (Ollama slow performance) explicitly names "2x Nvidia DGX Spark." The rest come from a mix of consumer GPUs (RTX 5070 Ti, RTX 5090, GTX 1080), workstation Blackwell cards (RTX PRO 6000/4000), and datacenter GPUs (H20), or provide no hardware details at all. They are included here because they cover the exact software (llama.cpp/Ollama/vLLM), exact model families (Gemma 4, Qwen3/3.5/3-VL, GPT-OSS, DeepSeek), and exact mechanisms (NVFP4, CUDA/Blackwell compute-capability handling, UMA-adjacent memory behavior) that a Spark user will hit — but every case below is labeled with its actual hardware provenance so confidence isn't overstated. | Symptom Area | Cases | Spark-confirmed? | |---|---|---| | Memory & OOM reporting | Known Issues UMA/`cudaMemGetInfo` note; Ollama slow-performance (2x Spark) | Partially — the UMA note is Spark-official; the slow-performance report is on-device | | Builds, wheels & compute-capability | llama.cpp `sm_120` MMQ compile failure; llama.cpp Vulkan+mmap=0 bug | No (both off-Spark, but compute-capability mechanism is architecture-general) | | NVFP4 / Blackwell quantization | Ollama can't-pull-nvfp4; vLLM 5090 NVFP4 load failure; vLLM Qwen3.5 NVFP4 accuracy; vLLM CUBLAS error | No (Ollama case is a library gate, not hardware; three vLLM cases are off-Spark Blackwell) | | Model/sampling correctness | llama.cpp Gemma 4 ``; llama.cpp CUDA experts-offload corruption; Ollama Gemma4:31b won't run; Ollama embeddings regression; Ollama garbage-output/buffer-overrun; vLLM Qwen3-VL grounding; vLLM streaming scrambled tokens; vLLM Gemma 4 tool-calling | Yes for Gemma 4 `` (Spark-confirmed); no for the rest | | Agentic/tool-calling integration | llama.cpp grammar errors under OpenClaw; vLLM Gemma 4 + Claude Code tool calling | No (both off-Spark, but directly relevant to [[concepts/agents-on-spark]]) | | Feature/model-support gaps | llama.cpp DeepSeek-V4 model request | N/A (feature request, not a bug) | ## Analysis **Memory & OOM reporting.** The one fact every Spark user needs before touching any inference stack is that `cudaMemGetInfo` under-reports allocatable memory on DGX Spark's UMA design, because it does not account for DRAM that could be reclaimed from SWAP (see [[concepts/unified-memory-and-arm64]]). This means memory errors that look like genuine capacity problems are often reporting artifacts, fixable with `sudo sh -c 'sync; echo 3 > /proc/sys/vm/drop_caches'`. The on-device Ollama slow-performance report (two Sparks, Docker, `gpt-oss:20b`/`phi4:14b`/`gemma3:12b`/`qwen3:8b`/`granite3.3:8b`) showed response times degrading from fast to 10+ minutes after 20-30 minutes of uptime, with `nvidia-smi` pegged at 92% GPU utilization; the captured config showed `OLLAMA_FLASH_ATTENTION:false` and `OLLAMA_CONTEXT_LENGTH:4096` — both worth checking explicitly, since sustained near-100% utilization with ballooning wall-clock time looks like thermal throttling or a stuck generation loop rather than a genuine OOM. **Builds, wheels & compute-capability.** The most Spark-relevant lesson from an otherwise off-Spark issue: CUDA compute-capability `sm_120` (Blackwell consumer) failed to compile llama.cpp's MMQ kernels with `ptxas` errors (`Feature '.block_scale' not supported on .target 'sm_120'`) while DGX Spark's own compute capability is `sm_121`/`121a` — a different sub-variant. This is the concrete case behind the general rule in [[concepts/unified-memory-and-arm64]]: never assume a CUDA architecture flag is portable across Blackwell members, always match the exact string the official Spark playbook uses. A separate llama.cpp Vulkan-backend bug (llama-bench cannot load models with `--mmap 0`, introduced by a specific PR implementing `set_tensor_async`/event interfaces) shows that backend choice (CUDA vs. Vulkan) interacts with flags like `mmap` in backend-specific ways — relevant if a Spark deployment ever falls back to the Vulkan backend instead of CUDA. **NVFP4 / Blackwell quantization.** This is the deepest and most consequential cluster. Three independent, unrelated reports converge on the same warning: NVFP4 is not a drop-in accuracy-neutral win. (1) An Ollama user on Windows could not `ollama pull qwen3.5:27b-nvfp4` at all — the pull failed with `412: this model requires macOS`, a **library-side platform gate** unrelated to actual hardware/quantization compatibility, meaning an NVFP4 pull failure on Linux/DGX OS should be triaged as a tag restriction, not assumed to be a Spark limitation. (2) An RTX 5090 (Blackwell consumer, not Spark) failed to load `Qwen3-30B-A3B-NVFP4` in vLLM entirely, crashing during engine startup after model load. (3) A separate report found `nvidia/Qwen3.5-397B-A17B-NVFP4` scoring **~0.35 on GSM8K versus ~0.90 for the FP8 original** — a massive accuracy regression that persisted across `TRITON_ATTN`/`FLASHINFER` backends and both `auto` and `fp8_e4m3` KV-cache dtypes. (4) A `CUBLAS_STATUS_INVALID_VALUE` error during `cublasGemmEx` was reported on dual RTX PRO 6000 Blackwell GPUs. None of these four are confirmed on GB10 specifically, but they share the same underlying mechanism (Blackwell-native FP4 support still maturing across the software stack) that the [[concepts/fine-tuning-and-quantization]] NVFP4-quantization playbook itself warns about ("we recommend running evaluations to verify if the quantized model maintains acceptable performance"). Treat any NVFP4 deployment on Spark as requiring its own accuracy validation, not a trust-the-marketing-number decision. **Model/sampling correctness.** The one Spark-confirmed case in this casebook: Gemma 4 (`gemma-4-26B-A4B-it-f16.gguf`) on DGX Spark (GB10, compute capability 12.1) began spamming an unbounded stream of `` tokens partway through a general-knowledge response — a sampling/vocabulary bug (the model emitting a token that should probably be excluded from sampling), not a hardware or memory fault. This recurs in a different form with vLLM + Gemma 4 + Claude Code, where tool-calling was reported unreliable on Blackwell workstation GPUs — two independent reports of Gemma 4 specifically misbehaving in agentic/structured-output contexts across two different inference stacks is a pattern worth flagging, not two unrelated coincidences. Separately: a CUDA multi-GPU MoE expert-offload configuration (`-ot ...=CPU, ...=CUDA0`) produced data corruption on long prompts (2x RTX 5070 Ti, not Spark's single-GPU unified layout, so the exact CPU/GPU split mechanics differ, but the general caution against untested manual per-tensor offload configurations still applies); Ollama users hit an inability to run `Gemma4:31b` at all after a version update, a regression where `Qwen3-Embedding-8B` silently stopped supporting the embeddings endpoint after an Ollama upgrade, and (on non-Spark hardware — Jetson Orin Nano, AMD-CPU Windows box) garbage `GGGGG...` output and low-level buffer-overrun crashes tied to specific Ollama versions. vLLM separately showed missing/scrambled streamed tokens with GPT-OSS 120B (correct with `--enforce-eager` at a speed cost, or with streaming disabled) and a Qwen3-VL-235B grounding-accuracy regression specific to vLLM >= v0.11.1. **Agentic/tool-calling integration.** Two cases directly relevant to [[concepts/agents-on-spark]]: llama.cpp under Windows/CPU-only produced excessive grammar-constraint errors when driven by an OpenClaw-style tool-orchestrating agent (the grammar/tool-schema interaction, not the base model, is the likely culprit); vLLM + Gemma 4 showed inconsistent tool-calling behavior specifically when driven by Claude Code on Blackwell workstation GPUs. Neither is Spark-confirmed, but both point at the same class of failure: local models behave differently under agentic tool-calling harnesses than under plain chat, and this should be smoke-tested end-to-end with the actual agent client before depending on it unattended. **Feature/support gaps.** llama.cpp users requested support for the DeepSeek-V4 series (V4-Pro 1.6T/49B-active MoE, V4-Flash 284B/13B-active, 1M context) — a reminder that very new large MoE releases routinely outrun local-inference-engine support, and "unsupported model" should be checked against the engine's actual `general.architecture` support list before assuming a Spark-specific problem. ## Recommendations - **Before debugging any "out of memory" error on Spark**, run the manual cache flush (`sudo sh -c 'sync; echo 3 > /proc/sys/vm/drop_caches'`) and retry — see [[concepts/unified-memory-and-arm64]]. Do not trust `cudaMemGetInfo`-derived capacity checks as ground truth. - **Match compute-capability flags exactly to GB10 (`sm_121`/`121a`)** in any from-source CUDA build; do not reuse a flag from a generic or consumer-Blackwell guide (`sm_120` is a different, incompatible target). - **Validate NVFP4 accuracy yourself** on any model before deploying it, especially large MoE models — the one hard data point available (Qwen3.5-397B, GSM8K 0.90→0.35) is severe enough that "NVFP4 should be fine because it's Blackwell-native" is not a safe assumption. See [[concepts/fine-tuning-and-quantization]]. - **Watch Gemma 4 specifically** for both raw-sampling misbehavior (`` spam, llama.cpp, Spark-confirmed) and tool-calling unreliability (vLLM + Claude Code) — smoke-test any Gemma 4 deployment end-to-end before trusting it unattended in an agentic loop. - **If GPU utilization is pegged near 100% but wall-clock latency balloons over time** (not just a single slow request), suspect thermal throttling, a stuck/looping generation, or a flash-attention/context-length misconfiguration before assuming the hardware itself is underpowered — check `OLLAMA_FLASH_ATTENTION` and `OLLAMA_CONTEXT_LENGTH` explicitly rather than trusting Docker defaults. - **Pin inference-engine versions** once a working configuration is found — several of the worst-looking bugs here (Ollama garbage output, buffer overruns) were version-specific regressions that rolling back resolved, not fundamental incompatibilities. - **Smoke-test streaming and tool-calling separately from plain chat** before relying on either in production — both have independent failure modes (scrambled streamed tokens, unreliable tool-call parsing) that don't show up in a simple non-streaming chat test. ## Pages Compared - [[concepts/unified-memory-and-arm64]] — the UMA/`cudaMemGetInfo` and compute-capability mechanisms underlying most cases here - [[concepts/llama-cpp-on-spark]] — compile-flag and Gemma 4 sampling cases - [[concepts/ollama-on-spark]] — slow-performance, NVFP4-pull, and embeddings-regression cases - [[concepts/vllm-on-spark]] — NVFP4 accuracy/load-failure and streaming/tool-calling cases - [[concepts/fine-tuning-and-quantization]] — the NVFP4-quantization playbook's own accuracy caveat, corroborated here - [[concepts/agents-on-spark]] — the two agentic tool-calling/grammar cases - [[syntheses/stack-picker]] — how these failure patterns factor into choosing a stack ## Source Note 18 GitHub issue mirrors (6 llama.cpp, 6 Ollama, 6 vLLM) plus the official DGX Spark Known Issues page were read in full for this casebook. Hardware provenance is called out per case above; readers should weight Spark-confirmed cases (2 of 18) more heavily than off-Spark cases included for their software/mechanism relevance.