[docs] Add DFlash2 speculative cells to the Qwen3.8-27B cookbook (#35663)
This commit is contained in:
@@ -59,8 +59,12 @@ import { Qwen38MambaRatioCalculator } from "/src/snippets/_qwen38_mamba_ratio_ca
|
||||
<Note>
|
||||
The RTX 5090 and RTX PRO 6000 cells above — including every Speculative
|
||||
Decoding / Serving Strategy / SSM dtype combination — were validated at
|
||||
ISL 8192 / OSL 1024, concurrency 1. The DGX Spark cells cover that same full
|
||||
combination set, but to a weaker standard: each was confirmed to **boot and
|
||||
ISL 8192 / OSL 1024, concurrency 1 — for DFLASH2, to that full standard on
|
||||
NVFP4, and to boot-and-serve on the RTX PRO 6000 BF16/FP8 cells. On the
|
||||
remaining platforms the DFLASH2 pick is offered but not yet exercised, and
|
||||
the composed command carries a `# DFLASH2 on this platform: final
|
||||
verification in progress` hint line while it is selected. The DGX Spark cells cover that same combination set minus
|
||||
DFLASH2, but to a weaker standard: each was confirmed to **boot and
|
||||
serve** at ISL 8192 / OSL 1024, concurrency 1, with no throughput or
|
||||
acceptance-length numbers taken. The remaining platforms' recipes carry their
|
||||
original validation, which covers the default overlay picks (plus MTP on
|
||||
@@ -89,7 +93,8 @@ ratio = (S + D) x state_bytes / (L x kv_bytes_per_token)
|
||||
slot, and `extra_buffer` frees one more with the overlap scheduler off; the
|
||||
calculator reads both knobs.
|
||||
- `D` — verify intermediate states under speculative decoding:
|
||||
`--speculative-num-draft-tokens` for EAGLE/MTP (4 at the recommended 3/1/4);
|
||||
`--speculative-num-draft-tokens` for EAGLE/MTP (4 at the recommended 3/1/4)
|
||||
and for DFLASH (8, DFlash2's block size);
|
||||
`--speculative-dspark-block-size + 1` for DSPARK, where the block size falls
|
||||
back to the draft checkpoint's `block_size` when the flag is omitted (7 for
|
||||
`RadixArk/Qwen3.8-27B-DSpark`, so `D = 8`); 0 with speculation off or with
|
||||
@@ -220,6 +225,29 @@ checkpoint's calibration scales automatically.
|
||||
`--enable-linear-replayssm-spec` its draft intermediates move onto a fixed
|
||||
ring, so `D = 0` and the ratio returns to the no-spec value. The
|
||||
[calculator](#mamba-ratio-calculator) applies both rules.
|
||||
- **DFlash2**: a trained block-diffusion draft in a separate checkpoint — add
|
||||
`--speculative-algorithm DFLASH --speculative-draft-model-path
|
||||
incoai/Qwen3.8-27B-DFlash2 --speculative-num-draft-tokens 8` (8 is the
|
||||
draft's block size, and it is the `D` term in the ratio, same value as
|
||||
DSpark's). The selector projects candidates through the target `lm_head`,
|
||||
including quantized heads, so it runs on the NVFP4 checkpoint (whose head
|
||||
is NVFP4-packed; the BF16 and FP8 checkpoints keep a dense head).
|
||||
Validation: NVFP4 measured end-to-end on RTX PRO 6000 and RTX 5090; the
|
||||
RTX PRO 6000 BF16/FP8 cells boot and serve; on H200, DGX Spark and GB300
|
||||
the pick is offered with a final-verification-in-progress hint line
|
||||
above the command. The
|
||||
RTX PRO 6000 recipe needs no changes. On the 32GB RTX 5090 prefer
|
||||
`--mamba-ssm-dtype bfloat16` at `--mem-fraction-static 0.90`: measured
|
||||
strictly better than float32 for this draft (6.1 vs 8.3 ms TPOT, accept
|
||||
3.30 vs 3.09) — the opposite of the EAGLE trade, so measure before assuming.
|
||||
float32 still fits, but only at `--mem-fraction-static 0.945` with
|
||||
`--mamba-full-memory-ratio 10` pinned in place of the balanced value: the
|
||||
L = 9216 ratio leaves the fp32 state pool one slot short at every
|
||||
serviceable mem-fraction (0.94 allocates four of Low-Latency's five slots;
|
||||
0.95 OOMs at runtime), and the re-weighted split leaves the Low-Latency KV
|
||||
pool a single-request envelope (~9.4k tokens) — no headroom for longer
|
||||
requests or radix reuse. The panel's DFLASH2 option applies these re-pins
|
||||
automatically.
|
||||
- **Hardware fit**: FP8 weights ~28.5GB (not serviceable beyond bs≤2 on
|
||||
32GB cards); NVFP4 weights ~16.5GB (recommended for RTX 5090-class GPUs).
|
||||
- `--mamba-radix-cache-strategy extra_buffer_lazy` lowers the state cost per
|
||||
|
||||
Reference in New Issue
Block a user