[docs] Add DFlash2 speculative cells to the Qwen3.8-27B cookbook (#35663)

This commit is contained in:
Jimmy Shong
2026-08-20 13:26:55 -07:00
committed by GitHub
parent eac91ac362
commit d9f6861359
2 changed files with 79 additions and 3 deletions
@@ -59,8 +59,12 @@ import { Qwen38MambaRatioCalculator } from "/src/snippets/_qwen38_mamba_ratio_ca
<Note> <Note>
The RTX 5090 and RTX PRO 6000 cells above — including every Speculative The RTX 5090 and RTX PRO 6000 cells above — including every Speculative
Decoding / Serving Strategy / SSM dtype combination — were validated at Decoding / Serving Strategy / SSM dtype combination — were validated at
ISL 8192 / OSL 1024, concurrency 1. The DGX Spark cells cover that same full ISL 8192 / OSL 1024, concurrency 1 — for DFLASH2, to that full standard on
combination set, but to a weaker standard: each was confirmed to **boot and NVFP4, and to boot-and-serve on the RTX PRO 6000 BF16/FP8 cells. On the
remaining platforms the DFLASH2 pick is offered but not yet exercised, and
the composed command carries a `# DFLASH2 on this platform: final
verification in progress` hint line while it is selected. The DGX Spark cells cover that same combination set minus
DFLASH2, but to a weaker standard: each was confirmed to **boot and
serve** at ISL 8192 / OSL 1024, concurrency 1, with no throughput or serve** at ISL 8192 / OSL 1024, concurrency 1, with no throughput or
acceptance-length numbers taken. The remaining platforms' recipes carry their acceptance-length numbers taken. The remaining platforms' recipes carry their
original validation, which covers the default overlay picks (plus MTP on original validation, which covers the default overlay picks (plus MTP on
@@ -89,7 +93,8 @@ ratio = (S + D) x state_bytes / (L x kv_bytes_per_token)
slot, and `extra_buffer` frees one more with the overlap scheduler off; the slot, and `extra_buffer` frees one more with the overlap scheduler off; the
calculator reads both knobs. calculator reads both knobs.
- `D` — verify intermediate states under speculative decoding: - `D` — verify intermediate states under speculative decoding:
`--speculative-num-draft-tokens` for EAGLE/MTP (4 at the recommended 3/1/4); `--speculative-num-draft-tokens` for EAGLE/MTP (4 at the recommended 3/1/4)
and for DFLASH (8, DFlash2's block size);
`--speculative-dspark-block-size + 1` for DSPARK, where the block size falls `--speculative-dspark-block-size + 1` for DSPARK, where the block size falls
back to the draft checkpoint's `block_size` when the flag is omitted (7 for back to the draft checkpoint's `block_size` when the flag is omitted (7 for
`RadixArk/Qwen3.8-27B-DSpark`, so `D = 8`); 0 with speculation off or with `RadixArk/Qwen3.8-27B-DSpark`, so `D = 8`); 0 with speculation off or with
@@ -220,6 +225,29 @@ checkpoint's calibration scales automatically.
`--enable-linear-replayssm-spec` its draft intermediates move onto a fixed `--enable-linear-replayssm-spec` its draft intermediates move onto a fixed
ring, so `D = 0` and the ratio returns to the no-spec value. The ring, so `D = 0` and the ratio returns to the no-spec value. The
[calculator](#mamba-ratio-calculator) applies both rules. [calculator](#mamba-ratio-calculator) applies both rules.
- **DFlash2**: a trained block-diffusion draft in a separate checkpoint — add
`--speculative-algorithm DFLASH --speculative-draft-model-path
incoai/Qwen3.8-27B-DFlash2 --speculative-num-draft-tokens 8` (8 is the
draft's block size, and it is the `D` term in the ratio, same value as
DSpark's). The selector projects candidates through the target `lm_head`,
including quantized heads, so it runs on the NVFP4 checkpoint (whose head
is NVFP4-packed; the BF16 and FP8 checkpoints keep a dense head).
Validation: NVFP4 measured end-to-end on RTX PRO 6000 and RTX 5090; the
RTX PRO 6000 BF16/FP8 cells boot and serve; on H200, DGX Spark and GB300
the pick is offered with a final-verification-in-progress hint line
above the command. The
RTX PRO 6000 recipe needs no changes. On the 32GB RTX 5090 prefer
`--mamba-ssm-dtype bfloat16` at `--mem-fraction-static 0.90`: measured
strictly better than float32 for this draft (6.1 vs 8.3 ms TPOT, accept
3.30 vs 3.09) — the opposite of the EAGLE trade, so measure before assuming.
float32 still fits, but only at `--mem-fraction-static 0.945` with
`--mamba-full-memory-ratio 10` pinned in place of the balanced value: the
L = 9216 ratio leaves the fp32 state pool one slot short at every
serviceable mem-fraction (0.94 allocates four of Low-Latency's five slots;
0.95 OOMs at runtime), and the re-weighted split leaves the Low-Latency KV
pool a single-request envelope (~9.4k tokens) — no headroom for longer
requests or radix reuse. The panel's DFLASH2 option applies these re-pins
automatically.
- **Hardware fit**: FP8 weights ~28.5GB (not serviceable beyond bs≤2 on - **Hardware fit**: FP8 weights ~28.5GB (not serviceable beyond bs≤2 on
32GB cards); NVFP4 weights ~16.5GB (recommended for RTX 5090-class GPUs). 32GB cards); NVFP4 weights ~16.5GB (recommended for RTX 5090-class GPUs).
- `--mamba-radix-cache-strategy extra_buffer_lazy` lowers the state cost per - `--mamba-radix-cache-strategy extra_buffer_lazy` lowers the state cost per
@@ -116,6 +116,49 @@ export const config = {
: []), : []),
], ],
}, },
{
id: "dflash", label: "DFLASH2",
// Trained block-diffusion draft, a separate checkpoint. The
// selector projects through the target lm_head — including
// quantized heads — so it runs on the NVFP4 checkpoint too.
// Validated on the SM120 pair (NVFP4 measured end to end; the
// RTX PRO 6000 BF16/FP8 cells boot-and-serve). On the other
// platforms the pick is offered with the in-progress hint below
// — the recipe composes from validated cells but has not been
// exercised there yet.
disabled: (sel) => sel.hw === "rtx5090" && sel.quant !== "nvfp4",
disableReason:
"On the 32GB RTX 5090 the DFlash2 draft model only fits on top of the NVFP4 weights",
hints: (sel) =>
["rtx6000", "rtx5090"].includes(sel.hw)
? []
: ["DFLASH2 on this platform: final verification in progress"],
// 5090: mem-fraction re-pins like DSPARK's, and fp32 additionally
// re-pins the ratio — the balanced L=9216 value leaves the state
// pool one slot short at every serviceable mem-fraction (see the
// DFlash2 bullet in Configuration Tips).
stripPrefixes: (sel) =>
sel.hw === "rtx5090"
? sel.ssmDtype === "float32"
? ["--mem-fraction-static", "--mamba-full-memory-ratio"]
: ["--mem-fraction-static"]
: [],
flags: (sel) => [
"--speculative-algorithm DFLASH",
"--speculative-draft-model-path incoai/Qwen3.8-27B-DFlash2",
"--speculative-num-draft-tokens 8",
// Measured on the 5090: bf16 state serves at 0.90 (DSPARK's
// pin); fp32 fits only at 0.945 + ratio 10 (0.94 is one state
// slot short, 0.95 OOMs at runtime) and leaves the Low-Latency
// KV pool a single-request envelope.
...(sel.hw === "rtx5090"
? sel.ssmDtype === "float32"
? ["--mem-fraction-static 0.945",
"--mamba-full-memory-ratio 10"]
: ["--mem-fraction-static 0.90"]
: []),
],
},
], ],
}, },
{ {
@@ -260,6 +303,11 @@ export const config = {
flags: ["--speculative-algorithm DSPARK", flags: ["--speculative-algorithm DSPARK",
"--speculative-draft-model-path RadixArk/Qwen3.8-27B-DSpark", "--speculative-draft-model-path RadixArk/Qwen3.8-27B-DSpark",
"--speculative-draft-attention-backend flashinfer"] }, "--speculative-draft-attention-backend flashinfer"] },
{ id: "dflash", label: "DFlash2",
// Same trio as the Deploy panel's DFLASH2 option.
flags: ["--speculative-algorithm DFLASH",
"--speculative-draft-model-path incoai/Qwen3.8-27B-DFlash2",
"--speculative-num-draft-tokens 8"] },
], ],
}, },