[docs] Add DFlash2 speculative cells to the Qwen3.8-27B cookbook (#35663)

This commit is contained in:
Jimmy Shong
2026-08-20 13:26:55 -07:00
committed by GitHub
parent eac91ac362
commit d9f6861359
2 changed files with 79 additions and 3 deletions
@@ -59,8 +59,12 @@ import { Qwen38MambaRatioCalculator } from "/src/snippets/_qwen38_mamba_ratio_ca
<Note>
The RTX 5090 and RTX PRO 6000 cells above — including every Speculative
Decoding / Serving Strategy / SSM dtype combination — were validated at
ISL 8192 / OSL 1024, concurrency 1. The DGX Spark cells cover that same full
combination set, but to a weaker standard: each was confirmed to **boot and
ISL 8192 / OSL 1024, concurrency 1 — for DFLASH2, to that full standard on
NVFP4, and to boot-and-serve on the RTX PRO 6000 BF16/FP8 cells. On the
remaining platforms the DFLASH2 pick is offered but not yet exercised, and
the composed command carries a `# DFLASH2 on this platform: final
verification in progress` hint line while it is selected. The DGX Spark cells cover that same combination set minus
DFLASH2, but to a weaker standard: each was confirmed to **boot and
serve** at ISL 8192 / OSL 1024, concurrency 1, with no throughput or
acceptance-length numbers taken. The remaining platforms' recipes carry their
original validation, which covers the default overlay picks (plus MTP on
@@ -89,7 +93,8 @@ ratio = (S + D) x state_bytes / (L x kv_bytes_per_token)
slot, and `extra_buffer` frees one more with the overlap scheduler off; the
calculator reads both knobs.
- `D` — verify intermediate states under speculative decoding:
`--speculative-num-draft-tokens` for EAGLE/MTP (4 at the recommended 3/1/4);
`--speculative-num-draft-tokens` for EAGLE/MTP (4 at the recommended 3/1/4)
and for DFLASH (8, DFlash2's block size);
`--speculative-dspark-block-size + 1` for DSPARK, where the block size falls
back to the draft checkpoint's `block_size` when the flag is omitted (7 for
`RadixArk/Qwen3.8-27B-DSpark`, so `D = 8`); 0 with speculation off or with
@@ -220,6 +225,29 @@ checkpoint's calibration scales automatically.
`--enable-linear-replayssm-spec` its draft intermediates move onto a fixed
ring, so `D = 0` and the ratio returns to the no-spec value. The
[calculator](#mamba-ratio-calculator) applies both rules.
- **DFlash2**: a trained block-diffusion draft in a separate checkpoint — add
`--speculative-algorithm DFLASH --speculative-draft-model-path
incoai/Qwen3.8-27B-DFlash2 --speculative-num-draft-tokens 8` (8 is the
draft's block size, and it is the `D` term in the ratio, same value as
DSpark's). The selector projects candidates through the target `lm_head`,
including quantized heads, so it runs on the NVFP4 checkpoint (whose head
is NVFP4-packed; the BF16 and FP8 checkpoints keep a dense head).
Validation: NVFP4 measured end-to-end on RTX PRO 6000 and RTX 5090; the
RTX PRO 6000 BF16/FP8 cells boot and serve; on H200, DGX Spark and GB300
the pick is offered with a final-verification-in-progress hint line
above the command. The
RTX PRO 6000 recipe needs no changes. On the 32GB RTX 5090 prefer
`--mamba-ssm-dtype bfloat16` at `--mem-fraction-static 0.90`: measured
strictly better than float32 for this draft (6.1 vs 8.3 ms TPOT, accept
3.30 vs 3.09) — the opposite of the EAGLE trade, so measure before assuming.
float32 still fits, but only at `--mem-fraction-static 0.945` with
`--mamba-full-memory-ratio 10` pinned in place of the balanced value: the
L = 9216 ratio leaves the fp32 state pool one slot short at every
serviceable mem-fraction (0.94 allocates four of Low-Latency's five slots;
0.95 OOMs at runtime), and the re-weighted split leaves the Low-Latency KV
pool a single-request envelope (~9.4k tokens) — no headroom for longer
requests or radix reuse. The panel's DFLASH2 option applies these re-pins
automatically.
- **Hardware fit**: FP8 weights ~28.5GB (not serviceable beyond bs≤2 on
32GB cards); NVFP4 weights ~16.5GB (recommended for RTX 5090-class GPUs).
- `--mamba-radix-cache-strategy extra_buffer_lazy` lowers the state cost per