docs(cookbook): Qwen3.8-27B deployment grid rework (#35065)

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Jimmy Shong
2026-08-17 00:56:08 -07:00
committed by GitHub
co-authored by Claude Fable 5
parent 4cad864361
commit e03c53fc13
3 changed files with 263 additions and 174 deletions
@@ -56,6 +56,14 @@ import { Qwen38MambaRatioCalculator } from "/src/snippets/_qwen38_mamba_ratio_ca
<Deployment config={config} />
<Note>
The RTX 5090 and RTX PRO 6000 cells above — including every Speculative
Decoding / Serving Strategy / SSM dtype combination — were validated at
ISL 8192 / OSL 1024, concurrency 1. The other platforms' recipes carry their
original validation, which covers the default overlay picks (plus MTP on
GB300); non-default overlay picks there are valid but unmeasured.
</Note>
### Mamba ratio calculator
<Qwen38MambaRatioCalculator />
@@ -188,7 +196,19 @@ checkpoint's calibration scales automatically.
- **DSpark**: the trained draft model is a separate checkpoint — add
`--speculative-algorithm DSPARK --speculative-draft-model-path
RadixArk/Qwen3.8-27B-DSpark` (the Playground's Speculative Decoding card
emits this pair).
emits this pair). DSpark does **not** take
`--speculative-num-draft-tokens`: its verify window is
`--speculative-dspark-block-size` (gamma) **+ 1**, and gamma is auto-inferred
from the draft checkpoint when the flag is omitted (7 for this checkpoint, so
D = 8). That `D` is a term in the balanced ratio —
`r = (S + D) x token_equiv / L`, where `token_equiv` is the state slot
expressed in KV tokens, `state_bytes / kv_bytes_per_token` (4698 at fp32
state / 2394 at bf16, over fp8 KV) — so DSpark needs a materially higher
`--mamba-full-memory-ratio` than no-spec at the same `S`, and pinning a
different gamma changes the ratio with it. MTP is the opposite case: with
`--enable-linear-replayssm-spec` its draft intermediates move onto a fixed
ring, so `D = 0` and the ratio returns to the no-spec value. The
[calculator](#mamba-ratio-calculator) applies both rules.
- **Hardware fit**: FP8 weights ~28.5GB (not serviceable beyond bs≤2 on
32GB cards); NVFP4 weights ~16.5GB (recommended for RTX 5090-class GPUs).
- `--mamba-radix-cache-strategy extra_buffer_lazy` lowers the state cost per
@@ -197,6 +217,24 @@ checkpoint's calibration scales automatically.
`S` (lazy strategy, or `--disable-radix-cache` for S=1); the
[calculator](#mamba-ratio-calculator) re-derives the ratio for the new `S`.
The balanced ratio itself is VRAM-independent.
- `--mamba-ssm-dtype`: the GDN state slot is **153.9 MB at `float32`** (the
checkpoint's declared precision) and **78.4 MB at `bfloat16`**, so bf16 roughly
halves the state pool and hands the difference to KV — measured on an RTX 5090
with no speculation, 97,280 KV tokens at bf16 against 68,588 at fp32. On 32GB
cards it also decides whether a config fits at all: EAGLE needs
`--mem-fraction-static 0.94` at fp32 but 0.92 at bf16. Speed is **not** a
one-way trade — with speculative decoding fp32 sometimes wins (NVFP4 + EAGLE:
152.9 vs 144.5 tok/s/user) and sometimes loses (FP8 + EAGLE: 106.3 vs 116.1);
measure both for your quantization. Treat
`bfloat16` as an accuracy gate and validate it for your workload. On SM120
both precisions run the Triton linear-attn prefill path — the FlashInfer GDN
prefill fast path gates on SM100, where its validated domain is in fact a
bf16 state pool — so no dtype forces an extra flag here. One interaction to
know: `--enable-linear-replayssm-spec` auto-selects fp32 state when
`--mamba-ssm-dtype` is unset, and an explicit non-fp32 value logs a
state-drift warning at boot. The SSM dtype row always emits the flag
explicitly, so the bf16 + EAGLE cells run with that warning — accounted for
in their validation.
- `--chunked-prefill-size 2048`: decode steps stall behind each prefill chunk
on hybrid GDN models, and 8192-token chunks stall them ~600ms at a time.
2048 keeps decode inter-token latency smooth under mixed load and also