docs(cookbook): Qwen3.8-27B deployment grid rework (#35065)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Fable 5
parent
4cad864361
commit
e03c53fc13
@@ -56,6 +56,14 @@ import { Qwen38MambaRatioCalculator } from "/src/snippets/_qwen38_mamba_ratio_ca
|
||||
|
||||
<Deployment config={config} />
|
||||
|
||||
<Note>
|
||||
The RTX 5090 and RTX PRO 6000 cells above — including every Speculative
|
||||
Decoding / Serving Strategy / SSM dtype combination — were validated at
|
||||
ISL 8192 / OSL 1024, concurrency 1. The other platforms' recipes carry their
|
||||
original validation, which covers the default overlay picks (plus MTP on
|
||||
GB300); non-default overlay picks there are valid but unmeasured.
|
||||
</Note>
|
||||
|
||||
### Mamba ratio calculator
|
||||
|
||||
<Qwen38MambaRatioCalculator />
|
||||
@@ -188,7 +196,19 @@ checkpoint's calibration scales automatically.
|
||||
- **DSpark**: the trained draft model is a separate checkpoint — add
|
||||
`--speculative-algorithm DSPARK --speculative-draft-model-path
|
||||
RadixArk/Qwen3.8-27B-DSpark` (the Playground's Speculative Decoding card
|
||||
emits this pair).
|
||||
emits this pair). DSpark does **not** take
|
||||
`--speculative-num-draft-tokens`: its verify window is
|
||||
`--speculative-dspark-block-size` (gamma) **+ 1**, and gamma is auto-inferred
|
||||
from the draft checkpoint when the flag is omitted (7 for this checkpoint, so
|
||||
D = 8). That `D` is a term in the balanced ratio —
|
||||
`r = (S + D) x token_equiv / L`, where `token_equiv` is the state slot
|
||||
expressed in KV tokens, `state_bytes / kv_bytes_per_token` (4698 at fp32
|
||||
state / 2394 at bf16, over fp8 KV) — so DSpark needs a materially higher
|
||||
`--mamba-full-memory-ratio` than no-spec at the same `S`, and pinning a
|
||||
different gamma changes the ratio with it. MTP is the opposite case: with
|
||||
`--enable-linear-replayssm-spec` its draft intermediates move onto a fixed
|
||||
ring, so `D = 0` and the ratio returns to the no-spec value. The
|
||||
[calculator](#mamba-ratio-calculator) applies both rules.
|
||||
- **Hardware fit**: FP8 weights ~28.5GB (not serviceable beyond bs≤2 on
|
||||
32GB cards); NVFP4 weights ~16.5GB (recommended for RTX 5090-class GPUs).
|
||||
- `--mamba-radix-cache-strategy extra_buffer_lazy` lowers the state cost per
|
||||
@@ -197,6 +217,24 @@ checkpoint's calibration scales automatically.
|
||||
`S` (lazy strategy, or `--disable-radix-cache` for S=1); the
|
||||
[calculator](#mamba-ratio-calculator) re-derives the ratio for the new `S`.
|
||||
The balanced ratio itself is VRAM-independent.
|
||||
- `--mamba-ssm-dtype`: the GDN state slot is **153.9 MB at `float32`** (the
|
||||
checkpoint's declared precision) and **78.4 MB at `bfloat16`**, so bf16 roughly
|
||||
halves the state pool and hands the difference to KV — measured on an RTX 5090
|
||||
with no speculation, 97,280 KV tokens at bf16 against 68,588 at fp32. On 32GB
|
||||
cards it also decides whether a config fits at all: EAGLE needs
|
||||
`--mem-fraction-static 0.94` at fp32 but 0.92 at bf16. Speed is **not** a
|
||||
one-way trade — with speculative decoding fp32 sometimes wins (NVFP4 + EAGLE:
|
||||
152.9 vs 144.5 tok/s/user) and sometimes loses (FP8 + EAGLE: 106.3 vs 116.1);
|
||||
measure both for your quantization. Treat
|
||||
`bfloat16` as an accuracy gate and validate it for your workload. On SM120
|
||||
both precisions run the Triton linear-attn prefill path — the FlashInfer GDN
|
||||
prefill fast path gates on SM100, where its validated domain is in fact a
|
||||
bf16 state pool — so no dtype forces an extra flag here. One interaction to
|
||||
know: `--enable-linear-replayssm-spec` auto-selects fp32 state when
|
||||
`--mamba-ssm-dtype` is unset, and an explicit non-fp32 value logs a
|
||||
state-drift warning at boot. The SSM dtype row always emits the flag
|
||||
explicitly, so the bf16 + EAGLE cells run with that warning — accounted for
|
||||
in their validation.
|
||||
- `--chunked-prefill-size 2048`: decode steps stall behind each prefill chunk
|
||||
on hybrid GDN models, and 8192-token chunks stall them ~600ms at a time.
|
||||
2048 keeps decode inter-token latency smooth under mixed load and also
|
||||
|
||||
Reference in New Issue
Block a user