Tuning Gemma 4 on a DGX Spark with vLLM: NVFP4, MTP and a Reboot-Proof Endpoint

Gabe is my DGX Spark. It runs a small retrieval stack for my own projects: a vLLM server with Gemma 4 26B-A4B for generation, a second vLLM instance for embeddings, LightRAG, Neo4j and pgvector, plus the usual exporters. It worked, but slowly: under real load each stream got about 14 tokens a second, and the whole thing ran on a floating latest image with BF16 weights and settings nobody had revisited.

This post is the full record of tuning it. I start with the research, lay out the plan, then go step by step through every change, each with the benchmark that justified it (or killed it). I include the mistakes, because two of them cost more time than all the tuning combined. If you have a Spark, or any unified-memory box running vLLM next to other services, you should be able to follow along and get the same results.

Where it ended up, on the same RAG-shaped benchmark before and after:

"Before" here is already the NVFP4 checkpoint on the new vLLM; the original BF16 setup was slower still (numbers below). The final config passed a one-hour soak at 8 concurrent streams with zero failed requests, lives in git, and came back serving on its own after a reboot.

The machine, and why memory bandwidth is the whole story

The DGX Spark is a GB10 Grace Blackwell SoC: an ARM host and a Blackwell GPU sharing one pool of LPDDR5X. The spec sheet says 128 GB; the kernel sees about 122 GiB. The CPU, the OS page cache, Docker, the model weights and the KV cache all draw from that one pool.

The number that matters most isn't the 128 GB. It's the ~273 GB/s of memory bandwidth. An RTX 5090 has 1,792 GB/s. Token generation is a memory-read problem: every decoded token reads the active weights once. So a rough ceiling is:

decode tokens/s  ≈  memory bandwidth (GB/s)  ÷  GB of weights read per token

dense 31B at ~4 bits ≈ 15.5 GB read per token  → single digits
MoE 26B-A4B, 4B active at ~4–8 bits ≈ 2–4 GB   → tens of tok/s

That one rule drives almost every decision. The model nearly always fits; the question is how many bytes each token touches. Published numbers bear it out: people report 3–7 tok/s for the dense Gemma 4 31B in NVFP4 on a Spark, and 36–52 tok/s for the 26B-A4B mixture-of-experts.

The GPU matters too. GB10 is sm_121, a consumer-Blackwell compute capability, not the datacenter sm_100. Kernels built for other Blackwell parts can be missing, slow, or, worst of all, silently wrong: fluent output that's garbage, with no error. That's why correctness testing is part of this process and not an afterthought.

The research

Before touching anything I read what everyone else had learned running Gemma 4 on a Spark: the vLLM recipes and Spark blog post, Google's MTP drafter announcement, NVIDIA forum threads, and a few people who published careful benchmarks. The useful findings:

• Pick the 26B-A4B MoE. Only ~4B parameters are active per token, so it decodes roughly 7× fewer bytes than the dense 31B. The quality gap is small for this kind of work.
• Use vLLM ≥ 0.25 on a CUDA 13 aarch64 image, pinned by digest. The Gemma 4 recipe lists 0.25.0 as the minimum; v0.29.0 ships CUDA 13 by default. The spring gemma4 preview tags crash or lack GB10 MoE tuning.
• --attention-backend TRITON_ATTN is required for Gemma 4. FlashInfer and FlashAttention reject its 512-dim full-attention heads, and FlashInfer has had silent FP8 accuracy bugs on Blackwell.
• For NVFP4 MoE weights, use --moe-backend marlin. CUTLASS FP4 on sm_121 has produced wrong output or NaNs with no error. Pair it with VLLM_MARLIN_USE_ATOMIC_ADD=1.
• MTP speculative decoding is the biggest single win. Google ships small "assistant" drafters for every Gemma 4 size. One published Spark run went from 40.9 to 108.8 tok/s single-stream on FP8, and 674 tok/s aggregate at 8 streams. The drafter must be paired with the -it (instruction-tuned) target; on a base model it made decode 38% slower.
• The Spark can hard-freeze above ~0.8 GPU memory utilization when swap and page cache fight CUDA for the pool. Disabling swap and installing earlyoom fixed it in the forum thread.
• Stay on the 580.x driver. 590.x had a reported CUDA-graph deadlock on GB10.
• Never use --enforce-eager for speed; it disables CUDA graphs and costs 20–55%.

One conflict worth knowing: vLLM's own June 2026 Spark post says to leave MoE and linear backends on auto, because newer FlashInfer paths got faster. The Gemma-specific advice above predates that. I started with the explicit flags (safety first) and left auto as an A/B for later.

The research ended in a six-step rollout, each step with a gate:

1. Prep the host: OS update, driver on 580.x, swap off, earlyoom, stage weights. Gate: stable, no swap use.
2. Baseline without MTP: quantized weights, FP8 KV cache, TRITON_ATTN. Gate: faster than before, correctness set passes.
3. Add MTP: the assistant drafter, sweep the number of draft tokens. Gate: ≥ 90 tok/s at 1 stream.
4. Size for the workload, then soak: one hour at 8 streams. Gate: no freeze, KV cache under 80%.
5. A/B the variants: NVFP4 vs FP8, auto backends, async scheduling. Gate: keep only wins that pass correctness.
6. Pin and codify: image digest, flags and env in git. Gate: reboot, and it serves unattended.

Adjusting the plan for a shared box

The published recipes assume the Spark does nothing but serve one model at --gpu-memory-utilization 0.85. Gabe isn't that. It also runs an embeddings model, LightRAG, Neo4j and Postgres. So I made three changes to the plan before starting:

• Keep the memory budget at 0.55. In vLLM on unified memory, gpu-memory-utilization is a fraction of the whole shared pool. 0.55 is ~67 GiB, which leaves room for everything else.
• Benchmark my workload, not a demo. I pulled vLLM's lifetime metrics first. The server had processed 63.1M prompt tokens against 2.21M generated: a 28:1 prompt-to-output ratio. Mean time to first token was 3.38 s, mean inter-token latency 72.3 ms, and prefix-cache hit rate 86%. RAG is prefill-heavy, so the benchmark had to use long prompts and short answers.
• A variant only wins if it's faster and passes a correctness check against the original BF16 weights.

The starting config, for reference: vLLM 0.23 from the floating vllm/vllm-openai:latest tag, BF16 weights (49 GB), --max-model-len 262144, --max-num-seqs 8, FP8 KV cache, no MTP, no explicit attention backend. The host had 16 GB of swap with 5 GB in use by vLLM's engine processes, swappiness 60, and no earlyoom.

The measurement harness

Three small tools did all the work. Build these first; everything after depends on them.

1. A correctness set

Twenty prompts with known answers, run at temperature 0 with thinking off: arithmetic, word problems, unit conversion, facts, string manipulation, reading code, and JSON extraction (the thing my RAG pipeline actually does). Each run saves raw answers so a new config can be diffed against the BF16 reference, answer by answer.

CASES = [
  { id: "arith-1", prompt: "What is 347 * 29? Reply with only the answer.", expect: hasNumber(10063) },
  { id: "str-1",   prompt: 'Reverse the string "gabespark". Reply with only the answer.', expect: exact("krapsebag") },
  { id: "json-1",  prompt: 'Extract the person as JSON with keys "name" and "age" from: ...',
                   expect: json(v => v.name === "Priya Natarajan" && v.age === 41) },
  ... 17 more
]

run(label):
  for case in CASES:
    answer = POST /v1/chat/completions {
      model, messages: [user: case.prompt],
      temperature: 0, max_tokens: 96,
      chat_template_kwargs: { enable_thinking: false }    # thinking eats max_tokens
    }
    results.push({ id, pass: case.expect(answer), answer })
  save results-<label>.json

diff(a, b):
  for each case: report if pass/fail changed OR normalized answer text changed
  exit non-zero if b passes fewer than a

The diff matters more than the score. A config that scores 19/20 like the baseline but changes three answers is suspicious; one that changes nothing is safe.

2. A benchmark that reports draft acceptance

vllm bench serve does the heavy lifting. I ran it inside the serving container at 1, 4 and 8 concurrent streams, and read the speculative-decoding counters from /metrics before and after, so each run also reports the drafter's acceptance rate:

acceptance() = sum(vllm:spec_decode_num_accepted_tokens_total) / sum(vllm:spec_decode_num_draft_tokens_total)

for c in 1 4 8:
  before = counters()
  vllm bench serve --backend openai-chat --endpoint /v1/chat/completions \
    --model gemma-4-26b-a4b-it --tokenizer /models/Gemma-4-26B-A4B-it-bf16 \
    --dataset-name random --random-input-len 4096 --random-output-len 128 --ignore-eos \
    --max-concurrency $c --num-prompts $((c*6)) \
    --percentile-metrics ttft,tpot,itl,e2el
  after = counters()
  print Mean TTFT, Mean TPOT, Output token throughput, acceptance(after - before)

Random 4,096-token prompts with 128-token outputs mimic the RAG prompt shape. (Hold that thought; random tokens turned out to hide something important.)

3. A variant launcher with automatic rollback

Every experiment is a container relaunch. Doing that by hand is how you end up with a down server at midnight, so one script does it and rolls back on failure:

PINNED=vllm/vllm-openai@sha256:<v0.29.0 arm64 cu130 digest>

launch(label, vllm_args):
  docker rm -f rag-llm
  docker run -d --name rag-llm --restart unless-stopped --gpus all --ipc=host \
    -p 127.0.0.1:8001:8000 -v ~/models:/models \
    -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
    -e VLLM_MARLIN_USE_ATOMIC_ADD=1 \
    $PINNED $vllm_args --api-key $KEY
  poll every 15 s for up to 20 min:
    if GET /v1/models (with bearer key) == 200 → return OK
    if container exited or its restart count went up → break
  print the last error lines from docker logs
  return FAIL

if launch(new_label, new_args):
    send a 3-token warmup request
    save new_args as last-good
else:
    launch(last-good)          # never leave the endpoint down

I also kept the original BF16 container stopped under another name, so a full rollback was a rename away.

Step 1: Prep the host

OS update and driver

The Spark had 36 NVIDIA/DGX/kernel updates pending: the driver moved within the 580 branch (580.159.03 → 580.178.04) and the kernel from 6.17 to 7.0. Check the driver branch before rebooting into it:

sudo apt update && sudo apt dist-upgrade
dpkg -l | grep nvidia-driver        # refuse to reboot if this shows 59x
sudo reboot
nvidia-smi --query-gpu=driver_version --format=csv,noheader    # 580.178.04

Gotcha: dist-upgrade applies every pending Ubuntu update, not just the NVIDIA ones, and it restarted the network daemons and Docker mid-upgrade. My vLLM container publishes its port on the LAN address as well as loopback. When Docker came back, the DHCP address wasn't assigned yet, the bind failed with cannot assign requested address, and Docker doesn't retry that error. The server stayed down for about six minutes, until I started it by hand. More on the permanent fix in Step 6.

Swap off

sudo swapoff -a
sudo sed -i 's|^/swap.img|#/swap.img|' /etc/fstab
echo "vm.swappiness=1" | sudo tee /etc/sysctl.d/99-swappiness.conf
sudo sysctl -p /etc/sysctl.d/99-swappiness.conf

earlyoom, configured the right way

earlyoom kills a process before the kernel locks up. The wrong way to configure it on an inference box is the obvious one: "protect vLLM." I did exactly that at first, and it's the worst mistake in this post (full story below). The right config makes vLLM the first thing killed and protects the host's plumbing:

# /etc/default/earlyoom
EARLYOOM_ARGS="-m 4 -r 3600
  --prefer ^(VLLM::EngineCor|vllm)$
  --avoid ^(sshd|systemd.*|[(]systemd[)]|NetworkManager|dbus-daemon|dbus-broker|dockerd|containerd.*|runc.*|[(]udev-worker[)]|polkitd|nginx|node_exporter|dcgm-exporter|java|postgres|earlyoom)$"

• -m 4 sends SIGTERM when available memory drops to 4% (~5 GB here), SIGKILL at 2%.
• --prefer matters because vLLM's raw OOM score looks small: GPU allocations on unified memory don't show up in RSS, so without the bonus, earlyoom ranks desktop processes above it. With --prefer the top four candidates were all vLLM.
• Write literal parentheses as [(] and [)]. Backslash escapes get stripped when systemd reads this file.
• Add your own databases, exporters, and VPN or tunnel daemons to --avoid. Killing vLLM loses in-flight requests; Docker restarts it. Killing NetworkManager loses the box.

Make monitoring survive memory pressure

node_exporter ran under Restart=on-failure. A SIGTERM exit counts as success to systemd, so after a memory event it just stayed dead. A drop-in fixes it:

# /etc/systemd/system/node_exporter.service.d/10-resilience.conf
[Service]
Restart=always
RestartSec=5
OOMScoreAdjust=-900

sudo systemctl daemon-reload && sudo systemctl restart node_exporter
# prove it: kill -TERM $(pidof node_exporter); sleep 6; systemctl show node_exporter -p NRestarts

Stage the weights and the image

docker pull vllm/vllm-openai@sha256:<digest>      # v0.29.0, check it matches Docker Hub's index digest
hf download google/gemma-4-26B-A4B-it-assistant --local-dir ~/models/gemma-4-26B-A4B-it-assistant   # 832 MB drafter
hf download nvidia/Gemma-4-26B-A4B-NVFP4 --local-dir ~/models/Gemma-4-26B-A4B-NVFP4                  # 18 GB

Check that any quantized checkpoint's base_model is the -it model, because the MTP drafter only works with the instruction-tuned target. NVIDIA's NVFP4 checkpoint is google/gemma-4-26B-A4B-it with the nvfp4_experts_only recipe (attention, MLP and router stay in higher precision). Its model card reports GPQA Diamond 80.30% vs 79.90% for BF16, and MMLU Pro 85.00% vs 84.80%.

The baseline, captured before changing anything

BF16 on vLLM 0.23, random 4,096-in / 128-out:

Correctness: 19/20. The one miss, str-1, is BF16 reversing "gabespark" as krapsēbag. String reversal is tokenization-sensitive, and that case flips between checkpoints all the way through this post. One caveat on this baseline: it was captured before the kernel and driver update, so later comparisons against it include that change too.

Step 2: Quantized weights, no MTP yet

First attempt: the BF16 checkpoint with on-the-fly FP8 (--quantization fp8) on vLLM 0.29:

/models/Gemma-4-26B-A4B-it-bf16 --served-model-name gemma-4-26b-a4b-it
  --quantization fp8 --kv-cache-dtype fp8 --attention-backend TRITON_ATTN
  --gpu-memory-utilization 0.55 --max-num-seqs 8 --max-model-len 131072
  --reasoning-parser gemma4 --enable-auto-tool-choice --tool-call-parser gemma4

Weights dropped from 48.54 to 25.69 GiB, correctness stayed at 19/20 with zero changed answers, and every metric improved. But the load was ugly: 354 seconds to load and 825 seconds to serve. Quantizing on the fly reads full BF16 tensors into memory first. On a shared unified pool that spike got the first attempt killed by the kernel OOM killer (the engine process had 41 GB resident), and Docker had to restart it. It also triggered the earlyoom disaster described below.

Lesson: on a Spark, never quantize at load time. Use a pre-quantized checkpoint.

NVFP4 + Marlin vs FP8

So the next variant was NVIDIA's pre-quantized NVFP4 checkpoint, which needs no conversion:

/models/Gemma-4-26B-A4B-NVFP4 --quantization modelopt --moe-backend marlin  (+ same flags as above)

• Load: 270 s to serving (vs 825 s), weights 17.93 GiB, KV cache 42.63 GiB. No OOM kill, no restarts.
• Correctness: 20/20. The only change from BF16 was str-1, which NVFP4 got right.
• Speed: FP8 wins single-stream decode by ~1.35×. NVFP4 wins batched decode by 6–10%. Marlin logs a warning that FP4 is weight-only here: GB10 has no native FP4 compute, so the weights are dequantized on the fly.

I also downloaded a pre-quantized FP8 checkpoint (a third-party compressed-tensors build, 26.67 GiB, pinned to a specific revision) so FP8 got a fair re-test without the load spike. It shows up in the MTP sweep below.

Step 3: MTP speculative decoding

MTP (multi-token prediction) uses a small drafter to propose several tokens, which the big model verifies in one forward pass. When the guesses are right you get several tokens for the price of one read of the weights, which is exactly what a bandwidth-starved machine needs. One flag enables it:

--speculative-config '{"method":"mtp",
                       "model":"/models/gemma-4-26B-A4B-it-assistant",
                       "num_speculative_tokens":4}'

The first launch (NVFP4, 4 draft tokens) served in 315 s with weights at 18.71 GiB (the drafter adds 0.8). The engine log had two warnings worth reading closely:

1. "num_speculative_tokens > 1 will run multiple times of forward on same MTP layer, which may result in lower acceptance rate." So the sweep had to include 1 and 2 draft tokens, not just the recipe's 3–5.
2. "max_num_scheduled_tokens is set to 2496 based on the speculative decoding settings… Consider increasing max_num_batched_tokens." With MTP on, vLLM capped chunked prefill at 2,496 tokens per step. With ~4k-token RAG prompts, that throttles time to first token.

The numbers confirmed the second warning. Against NVFP4 without MTP, 4 draft tokens gave 1.61× faster decode at one stream (21.5 vs 34.6 ms per token), but TTFT got 38% worse at 4 streams and 60% worse at 8, and aggregate output at 8 streams fell 5%. Acceptance was only 41–43%. Every later variant adds --max-num-batched-tokens 16384.

The six-variant sweep

A script launched each variant through the rollback launcher, ran the correctness set, and benchmarked it. All six had --max-num-batched-tokens 16384. Values are TTFT ms / TPOT ms / output tok/s on random 4,096-in / 128-out:

What I read from this:

• The 16k batched-token setting fixed most of the TTFT damage (MTP 4 at 8 streams: 1,777 → 1,413 ms).
• On this benchmark, MTP roughly halved per-token time at 1 stream but was a wash at 8 streams. No-MTP had the best aggregate output (+6%, inside the noise of 48 requests).
• Pre-quantized FP8 had no speed edge over NVFP4 once MTP was on, plus bigger weights and a slower load. Dropped.
• Across all six variants, the only correctness difference from BF16 was the str-1 reversal flipping from one wrong answer to another (or to right). Nothing was eliminated on correctness.
• Stability: zero OOM kills and zero restarts across six relaunches. Pre-quantized checkpoints removed the load-spike problem entirely.

And the deck's Step 3 gate, ≥ 90 tok/s at one stream, was not met. The best was about 48.7 tok/s (1000 / 20.5 ms). Something was off, and I suspected the benchmark.

Random tokens lie about speculative decoding

A drafter predicts what the model will say next. Random-token prompts make that output hard to predict. My RAG pipeline sends real documents with extraction instructions, and its output is much easier to guess. So I built a natural-text benchmark from my own project documentation:

docs    = 14 markdown documents from my own projects
chunks  = split on paragraph boundaries, packed to roughly 1k tokens (measured with the model tokenizer)
pool    = 115 unique chunks

# disjoint sets per concurrency, so the prefix cache can't reuse an earlier round
c1 = 6 prompts  (mean 991 tokens)
c4 = 24 prompts (mean 946 tokens)
c8 = 48 prompts (mean 975 tokens)

each line of natural-cN.jsonl:
  {"prompt": "You are building a knowledge graph. From the text below, extract every named
              entity ... and every relationship between them. Return a bullet list ...\n\n" + chunk}

vllm bench serve --backend openai-chat --endpoint /v1/chat/completions \
  --dataset-name custom --dataset-path /models/bench/natural-c$c.jsonl \
  --custom-output-len 384 --temperature 0 \
  --max-concurrency $c --num-prompts $n          # no --ignore-eos: answers end naturally

Gotcha: the vLLM serving image doesn't include the benchmark extras, and the custom dataset loader imports pandas. My first natural round crashed on every call with ModuleNotFoundError: No module named 'pandas'. The random dataset doesn't need pandas, which is why the earlier sweep worked. Worse, my script logged "done" anyway. Two fixes: a tiny bench image, and a script that fails loudly when a run has zero successful requests.

# Dockerfile: same pinned vLLM, plus what the custom loader needs
FROM vllm/vllm-openai@sha256:<same digest as the server>
RUN pip install --no-cache-dir pandas
ENTRYPOINT []

docker build -t vllm-bench:0.29 .
docker run --rm --network host -v ~/models:/models -e OPENAI_API_KEY=$KEY vllm-bench:0.29 \
  vllm bench serve --base-url http://127.0.0.1:8001 ...

ok = parse "Successful requests" from the output
if ok is empty or ok == 0: log "BENCH FAILED" with the error line; don't record a result

A two-request smoke test already showed the effect: 64% draft acceptance on natural text vs 40–44% on random tokens with the same 4 draft tokens. The full natural round (TTFT ms / TPOT ms / output tok/s, 0 failed requests in every cell):

On real text the picture reverses. MTP 4 has the best per-token time and throughput at every concurrency: 2.3× faster decode at 1 stream and 1.7× the output throughput at 8. It even cuts TTFT at 8 streams, most likely because requests finish sooner and free their slots. The one cost is +115 ms TTFT at a single stream, trivial next to saving 19 ms on each of hundreds of output tokens.

Two draft tokens have higher acceptance (74–75%) but lose to four: more tokens per verified step outweigh the lower acceptance. I didn't measure 3 or 5. Acceptance at 4 already falls to 58% at 8 streams, so 5 is unlikely to win, but that's a guess.

If you take one thing from this post: benchmark speculative decoding on text that looks like your traffic. On random tokens I nearly shipped the no-MTP config.

Step 4: Size it for the workload, then soak

Context length and sequence count

vLLM's metrics include a histogram of prompt lengths. Over 7,138 lifetime requests: 59% were ≤ 5k tokens, 91% ≤ 20k, 98.6% ≤ 50k, 99.8% ≤ 100k, and all ≤ 200k. 87% of generations were ≤ 500 tokens.

• A 65k context limit would have rejected about 100 requests, so I kept --max-model-len 131072 (down from 262144, which nothing used).
• --max-num-seqs 8 matches the recipe's Spark setting and my client. LightRAG runs 16 async workers, so extra requests queue instead of thrashing the KV cache.
• With MTP 4 on NVFP4, vLLM sized the KV cache at 35–45 GiB, enough for 7.8–24.6 maximum-length requests at once. RAG prompts are ~1–5k tokens, so KV is nowhere near the limit.

The one-hour soak

The soak runs continuous load at the target concurrency and samples the box every 30 seconds. A gap between samples is the freeze detector: if the host locks up, the sampler stops too.

soak(label, c=8, duration=3600):
  record restart counts for rag-llm, embeddings, neo4j, postgres; preemptions counter
  in background, until deadline:
     vllm bench serve ... --random-input-len 4096 --random-output-len 256 --ignore-eos \
                          --max-concurrency $c --num-prompts $((c*6))
  every 30 s until deadline:
     track max(kv_cache_usage_perc), max(num_requests_waiting), min(MemAvailable), max(gap between samples)
  after:
     failed   = sum "Failed requests" across rounds
     oom      = journalctl -k since start | count "Killed process"
     earlyoom = journalctl -u earlyoom since start | count kill lines   # NOT "sending SIGTERM": that matches its startup banner
  FAIL if failed > 0, max_kv ≥ 0.80, max_gap > 90 s, oom > 0, or any restart count changed

Result: SOAK PASS.

One sample showed 14 requests waiting. That was real traffic from my pipeline queueing behind the soak's 8 slots, which is the intended behaviour. Nothing failed.

Step 5: The A/B tests, and what I didn't get to

Steps 2 and 3 already covered most of the planned A/Bs. Kept, all passing correctness: NVFP4 + Marlin over both FP8 variants, MTP with 4 draft tokens over 0, 1 and 2, and --max-num-batched-tokens 16384. Not run yet: auto attention and MoE backends instead of the explicit TRITON_ATTN/Marlin, and --async-scheduling. Those are the next experiments.

I also never hit the 90 tok/s single-stream gate, against a published 108.8. I haven't isolated why. Differences between the published run and mine include FP8 weights there vs NVFP4 through Marlin's weight-only dequant here, longer prompts, a 0.55 memory budget on a shared box, and a 131k context. Any of those could account for it; none is proven.

Step 6: Pin it, codify it, and prove it survives a reboot

The launch script

The config lives in my ops repo as an env-driven launcher. The repo's old launcher had drifted badly: wrong memory budget, 16k context, an alias served name, a floating image tag, no API key (it would have started an unauthenticated server), and no LAN binding. I rewrote it from the running container's arguments and tested it with stubbed docker, sudo and curl to prove its vLLM arguments matched the live container's exactly before committing.

# rag-llm.env (mode 600; the key lives only in the live copy)
RAG_LLM_MODEL_PATH=/models/Gemma-4-26B-A4B-NVFP4
RAG_LLM_MODEL_NAMES=gemma-4-26b-a4b-it
RAG_LLM_GPU_MEM_UTIL=0.55
RAG_LLM_LAN_IP=<your LAN address, or empty for loopback only>
RAG_LLM_API_KEY=<secret>
VLLM_IMAGE=vllm/vllm-openai@sha256:<v0.29.0 digest>
VLLM_MAX_MODEL_LEN=131072
VLLM_MAX_NUM_SEQS=8
VLLM_MAX_BATCHED_TOKENS=16384
VLLM_DRAFT_MODEL_PATH=/models/gemma-4-26B-A4B-it-assistant
VLLM_NUM_SPEC_TOKENS=4          # 0 disables MTP

# rag-llm-launch.sh (vLLM branch)
source rag-llm.env
touch $MAINT_FLAG; on exit: rm $MAINT_FLAG        # tells the watchdog this relaunch is planned
docker rm -f rag-llm
docker run -d --name rag-llm --restart unless-stopped --ipc=host --gpus all \
  -p 127.0.0.1:8001:8000 ${LAN_IP:+-p $LAN_IP:8001:8000} \
  -v ~/models:/models \
  -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True -e VLLM_MARLIN_USE_ATOMIC_ADD=1 \
  $VLLM_IMAGE $RAG_LLM_MODEL_PATH \
  --served-model-name gemma-4-26b-a4b-it \
  --quantization modelopt --moe-backend marlin --kv-cache-dtype fp8 --attention-backend TRITON_ATTN \
  --gpu-memory-utilization 0.55 --max-num-seqs 8 \
  --max-model-len 131072 --max-num-batched-tokens 16384 \
  --reasoning-parser gemma4 --enable-auto-tool-choice --tool-call-parser gemma4 --trust-remote-code \
  --speculative-config '{"method":"mtp","model":"'$VLLM_DRAFT_MODEL_PATH'","num_speculative_tokens":4}' \
  --api-key $RAG_LLM_API_KEY
health gate: poll GET /v1/models with "Authorization: Bearer $RAG_LLM_API_KEY" for up to 600 s

A watchdog for the failures Docker doesn't handle

Docker's restart policy covers crashes. It doesn't cover two failures I've actually hit: an engine that hangs silently (requests admitted, zero tokens generated, no error), and a container left exited because its start failed (the LAN-bind race). A small systemd service watches for both:

every 30 s:
  metrics = GET http://127.0.0.1:8001/metrics
  if metrics is empty:                       # loading, or down
    state = docker inspect -f {{.State.Status}} rag-llm
    if state in (exited, created, dead) and no maintenance flag newer than 2 h:
      exited_samples += 1
      if exited_samples == 3: docker start rag-llm; sleep 420     # model load takes minutes
    continue
  running = vllm:num_requests_running
  tokens  = vllm:generation_tokens_total
  if running > 0 and tokens hasn't moved for 3 samples:      # silent wedge
    if restarts this hour < 4: docker restart rag-llm; sleep 420
    else: log "backing off, needs a human"

The maintenance flag matters: without it, the watchdog "helpfully" restarts a container you're deliberately replacing. I tested the exited-container guard with a stubbed docker. It starts the container after three exited samples, stays quiet while the flag is fresh, and acts again once the flag is stale.

Host config in the repo too

Settings outside containers are the ones that disappear on a reinstall. The earlyoom file, the node_exporter drop-in, the wait-online setting and the swap config all went into the repo with an idempotent installer:

apply.sh:
  if repo/earlyoom.default differs from /etc/default/earlyoom: back up, install, restart earlyoom
  if repo/node_exporter drop-in differs: install, daemon-reload, restart node_exporter
  systemctl enable NetworkManager-wait-online.service
  swapoff -a if any swap is active; comment the swap line in fstab; swappiness=1
  print a verify line: earlyoom, node_exporter, restart policy, wait-online, swap, swappiness

The second run of apply.sh changed nothing, which is the idempotence proof.

The LAN-bind race, fixed at the root

The boot log explained Step 1's outage. docker.service is ordered after network-online.target, but both wait-online services were disabled, so that target was reached instantly. Docker started at 12.9 s into boot, the NIC got carrier at 13.7 s, and DHCP finished at 15.8 s. Any container publishing a port on the LAN address loses that race. One command fixes it:

sudo systemctl enable NetworkManager-wait-online.service
# network-online.target now actually waits for DHCP, so docker starts after the address exists

The reboot test

The last gate: reboot, touch nothing, and check everything.

reboot; wait for SSH with a new boot_id
poll GET /v1/models (with key) until 200, up to 20 min
check: container restart counts; LAN port answers; a warmup chat returns an answer;
       spec_decode counters are non-zero; KV cache size in the engine log;
       every exporter answers on the address it actually binds;
       node_exporter/earlyoom/watchdog active; no failed units; swap off;
       kernel OOM kills and earlyoom kill lines since boot

• The machine booted at 18:25:12. vLLM was serving at 18:29:29, about 4 minutes later, with zero restarts and no intervention.
• The LAN port answered, and a warmup "capital of France" came back "Paris" with the drafter's counters moving.
• node_exporter, earlyoom and the watchdog were all active, with no failed units, swap off, and zero OOM kills.

One thing did show up: a boot race between the two vLLM instances. Docker starts every container at once. The embeddings server profiled its memory while the big model was loading, computed a KV cache of −25.66 GiB, and failed. Docker's restart policy brought it up on the second try (KV 7.27 GiB), and the LLM's KV cache came out at 34.99 GiB instead of the 41.22 GiB it got when started after embeddings. It healed itself this time, but boot order on a shared pool shouldn't depend on luck. Enforcing it is on my list.

What went wrong, and what it taught me

1. The earlyoom rampage

In Step 1 I configured earlyoom to --avoid vLLM, reasoning that it was the most important process. Then the on-the-fly FP8 load spiked memory. earlyoom couldn't touch the only process that could actually free memory, so it killed everything else it could find: about 250 small host processes at 1–18 MiB each. That included NetworkManager 17 times, systemd-resolved 17 times, udev workers, nginx, my remote-access tunnel, dcgm-exporter, node_exporter, and runc in the middle of container restarts. The kernel OOM killer made exactly one kill, and it was the right one: vLLM.

The fallout: NetworkManager failed (no DHCP renewal), the tunnel went down, three containers were stuck exited because runc was killed mid-restart, and node_exporter stayed dead. I only found out because monitoring went quiet.

Never exempt the dominant memory consumer from your OOM killer. On an inference box that's the model server. Prefer it, protect the plumbing, and let Docker restart it.

2. Two vLLM instances starting at once

The embeddings server crash-looped for 25 minutes. It had --gpu-memory-utilization 0.08 (~10 GiB): 7.56 GiB of weights plus CUDA graphs left 0.21 GiB of KV cache against the 1.12 GiB its 8k context needed. Its first failure came while the LLM was loading. vLLM measures memory during startup profiling, and on a unified pool another process's allocations in that window get counted against you. The same effect runs in reverse: one earlier LLM start finished profiling while embeddings was down and grabbed an extra 5 GiB of KV cache.

Fixes: give small instances real headroom (0.12 here, which gives 5.06 GiB of KV), start instances one at a time with the small one first, and treat utilization fractions as shares of the whole machine.

3. The benchmark that silently didn't run

A sweep script that logged "done" without checking Successful requests burned a 16-minute round on pandas import errors. Every benchmark wrapper should parse the success count and refuse to record a result without one.

4. Trusting random tokens

Covered above, but it's the biggest analytical lesson. Random prompts cut draft acceptance from 64% to about 40%, which made MTP look pointless under load. On real traffic it's the single largest win.

5. Small ones

• pkill -f pattern over SSH matched its own shell and killed my session. Use the bracket trick: pkill -f '[s]oak.sh'.
• Grepping earlyoom's journal for "sending SIGTERM" matches its startup banner ("sending SIGTERM when mem <= 4%"). Count actual kill lines.
• Exporters bound to a VPN address answer "000" on 127.0.0.1. Check the port bindings before declaring monitoring dead.

The final state

On the random 4,096-in / 128-out benchmark I started with, the old BF16 setup vs the final config: time per token at 1 stream went from 42.9 to 21.6 ms, and output at 8 streams from 77.1 to 119.4 tok/s. (The kernel and driver update sits between those two measurements.) On natural text, the benchmark that reflects real traffic, the table at the top of this post is the honest comparison.

Still open: auto backends, async scheduling, 3 and 5 draft tokens, enforced boot order between the two vLLM instances, and working out why single-stream decode sits at ~66 tok/s rather than the published 108.8.

A checklist for your own Spark

1. Pull vLLM's lifetime metrics first: prompt-to-generation ratio, prompt-length histogram, TTFT and ITL. Tune for the workload you actually have.
2. Build the three tools: a correctness set diffed against the unquantized model, a benchmark that reports draft acceptance, and a launcher that rolls back.
3. Capture a baseline before changing anything.
4. Update the OS, stay on the 580.x driver, turn swap off, and install earlyoom that prefers vLLM and avoids everything else. Give exporters Restart=always.
5. Enable NetworkManager-wait-online if any container binds a DHCP address.
6. Use the 26B-A4B MoE with a pre-quantized checkpoint. Never quantize at load time on a shared unified pool.
7. Always set --attention-backend TRITON_ATTN, and --moe-backend marlin for NVFP4.
8. Add the MTP drafter, raise --max-num-batched-tokens (vLLM will warn you), and sweep the draft count on text that looks like your traffic.
9. Size context from the histogram, not the model maximum. Budget utilization as a share of the whole machine if anything else runs on it.
10. Soak for an hour at target concurrency with a freeze detector.
11. Put the launch config, env and host settings in git. Prove the launcher reproduces the running container, then reboot and touch nothing.

Sources

• vLLM on the DGX Spark: Architecture, Configuration, and Local Evaluation (vLLM blog)
• vLLM Recipes: gemma-4-26B-A4B-it
• vLLM v0.29.0 release
• Accelerating Gemma 4 with MTP drafters (Google)
• google/gemma-4-26B-A4B-it-assistant
• Gemma 4 hits 670 tok/s aggregate on DGX Spark (ai-muninn)
• Gemma 4 26B-A4B on DGX Spark: 52 tok/s with NVFP4 (ai-muninn)
• vLLM on DGX Spark: what SM121 actually requires (Conselara Labs)
• Gemma 4 on GB10: system freeze above 80% utilization (NVIDIA forums)
• eugr/spark-vllm-docker
• vLLM for Inference on DGX Spark playbook (NVIDIA)