---
title: "Stack Picker: Choosing an Inference Stack on Spark"
type: synthesis
tags: [inference, llama-cpp, ollama, vllm, lm-studio, nim, unified-memory]
created: 2026-07-18
updated: 2026-07-18
sources: ["raw/github_doc-nvidia-llama-cpp-readme-md.md", "raw/github_doc-nvidia-ollama-readme-md.md", "raw/github_doc-nvidia-vllm-readme-md.md", "raw/github_doc-nvidia-lm-studio-readme-md.md", "raw/github_doc-nvidia-nim-llm-readme-md.md", "raw/github_doc-nvidia-sglang-readme-md.md", "raw/github_doc-nvidia-trt-llm-readme-md.md", "raw/github_doc-nvidia-speculative-decoding-readme-md.md"]
confidence: medium
dgx_os_version: 7.5.0
---

## Comparison

| Stack | Install model | Serving API | Model management | Multi-Spark | Best-fit workload |
|---|---|---|---|---|---|
| **llama.cpp** ([[concepts/llama-cpp-on-spark]]) | Build from source, CMake pinned to `sm_121a` | `llama-server`, OpenAI-compatible | Manual GGUF (`-hf` auto-pull or `--model` local path) | Not built-in (single-process) | Simple, no-framework GGUF serving; widest day-one GGUF quant coverage for new model releases |
| **Ollama** ([[concepts/ollama-on-spark]]) | `curl install.sh` (prebuilt binary) | REST API on :11434 | `ollama pull`/library-managed | Not built-in per playbook (Docker-based multi-instance possible) | Convenience-first single-command model switching; backend for [[concepts/agents-on-spark]] CLI coding agents, Open WebUI |
| **vLLM** ([[concepts/vllm-on-spark]]) | Docker (NGC container, model-family-specific images for Gemma/DiffusionGemma) | OpenAI-compatible, PagedAttention + continuous batching | HF handle at container launch | Yes — Ray cluster over QSFP, `--tensor-parallel-size` | High-throughput/high-concurrency serving; models too large for one Spark; agentic tool-calling (mature `--tool-call-parser` ecosystem) |
| **LM Studio** ([[concepts/lm-studio-on-spark]]) | `llmster` headless daemon install script | REST API on :1234 (`lms` CLI/SDK) | `lms get`/`lms load`, GUI-adjacent catalog | Not covered in playbook | GUI-adjacent model catalog + LM Link zero-config remote access (Tailscale mesh) instead of manual SSH tunneling |
| **NIM** (`nim-llm` playbook) | Docker (NGC-hosted per-model container, e.g. `llama-3.1-8b-instruct-dgx-spark`) | OpenAI-compatible | One container per model, NGC-managed | Not covered in playbook | NVIDIA-validated, pre-optimized per-model containers when reproducibility/support matters more than DIY flexibility |
| **SGLang** (`sglang`/`station-sglang-inference`) | Install per playbook | OpenAI-compatible | Per-model | Not covered for Spark specifically | RadixAttention prefix caching for repeated-prefix workloads (agentic loops, few-shot prompting) |
| **TensorRT-LLM** (`trt-llm`) | Docker, single or multi-Spark | OpenAI-compatible (`trtllm-serve`) | Per-model, often paired with NVFP4 quantization (see [[concepts/fine-tuning-and-quantization]]) | Yes, per playbook | Maximum throughput once a model is quantized/compiled for TensorRT; pairs with the `nvfp4-quantization` playbook's own serving step |

## Analysis

**Simplicity vs. throughput is the main axis.** llama.cpp and Ollama sit at the "just run a model" end — llama.cpp for framework-free control, Ollama for zero-friction model switching and the widest downstream integration (it's the default backend nearly every agent/chatbot playbook falls back to). vLLM, TensorRT-LLM, and SGLang sit at the "serving infrastructure" end — Docker-first, tuned for concurrent requests, and the only stacks in this list with a documented multi-Spark path (vLLM's Ray cluster; TensorRT-LLM's own multi-Spark support). NIM is a distinct third axis: it trades DIY flexibility for NVIDIA-validated, per-model container reproducibility — useful when the question is "will this exact model/hardware combination behave the way NVIDIA tested it" rather than "give me maximum tuning control."

**Unified memory changes the calculus versus a discrete-GPU box.** On a traditional multi-GPU server, the stack choice is often forced by which framework supports the available VRAM split. On Spark's single 128 GB unified pool (see [[concepts/unified-memory-and-arm64]]), a much wider range of model sizes fit on one unit regardless of stack, so the deciding factor shifts to serving characteristics (concurrency, tool-calling maturity, prefix caching) rather than "which stack can even fit this model." The exception is genuinely oversized models (200B+ single-unit ceiling, 405B needing two units — see [[concepts/clustering]]), where vLLM's Ray-based tensor parallelism or TensorRT-LLM's multi-Spark path become necessary, not just preferable.

**ARM64/aarch64 affects install friction more than capability.** All five stacks run on Spark, but llama.cpp (build-from-source, exact `sm_121a` flag) and vLLM/TensorRT-LLM (NGC-published ARM64-native containers) sidestep the wheel-availability problem described in [[concepts/unified-memory-and-arm64]] by either compiling natively or shipping a pre-built ARM64 image; a bare `pip install vllm` outside the NGC container is more likely to hit the same ARM64/CUDA-13-wheel friction that fine-tuning playbooks report.

**Agentic/tool-calling maturity favors vLLM currently.** NVIDIA's own Hermes Agent playbook (see [[concepts/agents-on-spark]]) standardizes on vLLM specifically for its `--tool-call-parser`/`--enable-auto-tool-choice` flags. The [[syntheses/troubleshooting-casebook]] casebook shows this isn't friction-free (Gemma 4 tool-calling reliability issues reported on vLLM, and separately on llama.cpp under OpenClaw-style grammar-constrained calling) — but vLLM's parser ecosystem is more actively maintained for structured tool-calling than llama.cpp's grammar-based approach or Ollama's simpler API.

**NVFP4/quantization interacts with stack choice.** A model quantized to NVFP4 via the `nvfp4-quantization` playbook (see [[concepts/fine-tuning-and-quantization]]) is served via TensorRT-LLM (`trtllm-serve`) or vLLM in the official playbooks — llama.cpp and Ollama's ecosystems are GGUF/K-quant-centric and don't feature in the NVFP4 pipeline the same way. If NVFP4 specifically is the goal (for its ~3.5x memory reduction vs FP16), that pulls the stack choice toward vLLM or TensorRT-LLM regardless of the other tradeoffs above — with the important caveat from the troubleshooting casebook that NVFP4 accuracy should be independently validated, not assumed.

## Recommendations

- **Pick llama.cpp** if the goal is a single dependency-free server, the widest same-day GGUF quant availability for brand-new model releases, and full manual control over context size/speculative decoding flags.
- **Pick Ollama** if convenience and fast model switching matter most, or the target is a downstream tool (CLI coding agent, Open WebUI, Live VLM WebUI) that already expects an Ollama backend by default.
- **Pick vLLM** if the workload is agentic (tool-calling), needs concurrent multi-request throughput, or the model is too large for one Spark and needs a multi-Spark Ray cluster.
- **Pick LM Studio** if a GUI-adjacent model catalog and zero-config remote access (LM Link over Tailscale) matter more than scripted/production tuning — and if the downstream client already speaks LM Studio's local API (Claude Code, Codex, OpenCode all work unmodified through LM Link).
- **Pick NIM** when reproducibility and NVIDIA-validated performance matter more than tuning flexibility — e.g. deploying a model NVIDIA has specifically packaged for DGX Spark (Llama 3.1 8B, Qwen3-32B).
- **Pick TensorRT-LLM (or vLLM) specifically** when serving an NVFP4-quantized model, since that's the pipeline the official quantization playbook targets.
- **Regardless of stack**, budget for the [[concepts/unified-memory-and-arm64]] UMA cache-flush workaround and validate any NVFP4 model's accuracy independently — see [[syntheses/troubleshooting-casebook]] for concrete, stack-specific failure cases before committing to a production deployment.

## Pages Compared

- [[concepts/llama-cpp-on-spark]]
- [[concepts/ollama-on-spark]]
- [[concepts/vllm-on-spark]]
- [[concepts/lm-studio-on-spark]]
- [[concepts/unified-memory-and-arm64]]
- [[concepts/fine-tuning-and-quantization]]
- [[concepts/clustering]]
- [[concepts/agents-on-spark]]
- [[syntheses/troubleshooting-casebook]]
- [[entities/playbooks-catalog]] — nim-llm, sglang, trt-llm, speculative-decoding, station-sglang-inference entries