wikis / NVIDIA DGX Spark / wiki / syntheses / stack-picker.md view as markdown report a mistake

Comparison

Analysis

Simplicity vs. throughput is the main axis. llama.cpp and Ollama sit at the "just run a model" end — llama.cpp for framework-free control, Ollama for zero-friction model switching and the widest downstream integration (it's the default backend nearly every agent/chatbot playbook falls back to). vLLM, TensorRT-LLM, and SGLang sit at the "serving infrastructure" end — Docker-first, tuned for concurrent requests, and the only stacks in this list with a documented multi-Spark path (vLLM's Ray cluster; TensorRT-LLM's own multi-Spark support). NIM is a distinct third axis: it trades DIY flexibility for NVIDIA-validated, per-model container reproducibility — useful when the question is "will this exact model/hardware combination behave the way NVIDIA tested it" rather than "give me maximum tuning control."

Unified memory changes the calculus versus a discrete-GPU box. On a traditional multi-GPU server, the stack choice is often forced by which framework supports the available VRAM split. On Spark's single 128 GB unified pool (see unified memory and arm64), a much wider range of model sizes fit on one unit regardless of stack, so the deciding factor shifts to serving characteristics (concurrency, tool-calling maturity, prefix caching) rather than "which stack can even fit this model." The exception is genuinely oversized models (200B+ single-unit ceiling, 405B needing two units — see clustering), where vLLM's Ray-based tensor parallelism or TensorRT-LLM's multi-Spark path become necessary, not just preferable.

ARM64/aarch64 affects install friction more than capability. All five stacks run on Spark, but llama.cpp (build-from-source, exact sm_121a flag) and vLLM/TensorRT-LLM (NGC-published ARM64-native containers) sidestep the wheel-availability problem described in unified memory and arm64 by either compiling natively or shipping a pre-built ARM64 image; a bare pip install vllm outside the NGC container is more likely to hit the same ARM64/CUDA-13-wheel friction that fine-tuning playbooks report.

Agentic/tool-calling maturity favors vLLM currently. NVIDIA's own Hermes Agent playbook (see agents on spark) standardizes on vLLM specifically for its --tool-call-parser/--enable-auto-tool-choice flags. The troubleshooting casebook casebook shows this isn't friction-free (Gemma 4 tool-calling reliability issues reported on vLLM, and separately on llama.cpp under OpenClaw-style grammar-constrained calling) — but vLLM's parser ecosystem is more actively maintained for structured tool-calling than llama.cpp's grammar-based approach or Ollama's simpler API.

NVFP4/quantization interacts with stack choice. A model quantized to NVFP4 via the nvfp4-quantization playbook (see fine tuning and quantization) is served via TensorRT-LLM (trtllm-serve) or vLLM in the official playbooks — llama.cpp and Ollama's ecosystems are GGUF/K-quant-centric and don't feature in the NVFP4 pipeline the same way. If NVFP4 specifically is the goal (for its ~3.5x memory reduction vs FP16), that pulls the stack choice toward vLLM or TensorRT-LLM regardless of the other tradeoffs above — with the important caveat from the troubleshooting casebook that NVFP4 accuracy should be independently validated, not assumed.

Recommendations

• Pick llama.cpp if the goal is a single dependency-free server, the widest same-day GGUF quant availability for brand-new model releases, and full manual control over context size/speculative decoding flags.
• Pick Ollama if convenience and fast model switching matter most, or the target is a downstream tool (CLI coding agent, Open WebUI, Live VLM WebUI) that already expects an Ollama backend by default.
• Pick vLLM if the workload is agentic (tool-calling), needs concurrent multi-request throughput, or the model is too large for one Spark and needs a multi-Spark Ray cluster.
• Pick LM Studio if a GUI-adjacent model catalog and zero-config remote access (LM Link over Tailscale) matter more than scripted/production tuning — and if the downstream client already speaks LM Studio's local API (Claude Code, Codex, OpenCode all work unmodified through LM Link).
• Pick NIM when reproducibility and NVIDIA-validated performance matter more than tuning flexibility — e.g. deploying a model NVIDIA has specifically packaged for DGX Spark (Llama 3.1 8B, Qwen3-32B).
• Pick TensorRT-LLM (or vLLM) specifically when serving an NVFP4-quantized model, since that's the pipeline the official quantization playbook targets.
• Regardless of stack, budget for the unified memory and arm64 UMA cache-flush workaround and validate any NVFP4 model's accuracy independently — see troubleshooting casebook for concrete, stack-specific failure cases before committing to a production deployment.

Pages Compared

• llama cpp on spark
• ollama on spark
• vllm on spark
• lm studio on spark
• unified memory and arm64
• fine tuning and quantization
• clustering
• agents on spark
• troubleshooting casebook
• playbooks catalog — nim-llm, sglang, trt-llm, speculative-decoding, station-sglang-inference entries