[NPU] Adapt DFlash2 speculative decoding to Ascend NPUs (#35629)

Signed-off-by: syd520zy <529477025@qq.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
This commit is contained in:
faceless void
2026-09-07 09:08:39 +08:00
committed by GitHub
co-authored by github-actions[bot]
parent 6252993afe
commit 30d0eb2ca9
4 changed files with 202 additions and 24 deletions
@@ -247,9 +247,24 @@ checkpoint's calibration scales automatically.
`--speculative-algorithm DFLASH --speculative-draft-model-path
incoai/Qwen3.8-27B-DFlash2 --speculative-num-draft-tokens 8` (8 is the
draft's block size, and it is the `D` term in the ratio, same value as
DSpark's). The selector projects candidates through the target `lm_head`,
DSpark's). Runs on Ascend NPUs as well
([#35629](https://github.com/sgl-project/sglang/pull/35629)): the selector
verify falls back to argmax there, matching what EAGLE and first-generation
DFlash drafts (such as `z-lab/Qwen3-8B-DFlash-b16`) already do on NPU.
Therefore, NPU currently guarantees lossless verification only for greedy
requests; use `temperature=0` and `top_k=1`. Non-greedy requests log a warning;
both draft proposal and target verification fall back to greedy, so their
requested sampling distribution is not preserved. The selector projects
candidates through the target `lm_head`,
including quantized heads, so it runs on the NVFP4 checkpoint (whose head
is NVFP4-packed; the BF16 and FP8 checkpoints keep a dense head).
The Ascend comparison in #35629 used a 910C with BF16 target weights,
`--tp-size 2 --attention-backend ascend --mamba-ssm-dtype bfloat16
--mamba-scheduler-strategy extra_buffer`, and disabled RadixCache for both
baseline and DFlash2 to exclude cache warm-up and prefix reuse. The DFlash2
run added the three flags shown above.
Accuracy used zero-shot GSM8K with greedy sampling, `max_new_tokens=2048`,
128 examples, and concurrency levels 1, 2, 4, 8, and 16.
Validation: NVFP4 measured end-to-end on RTX PRO 6000 and RTX 5090; the
RTX PRO 6000 BF16/FP8 cells boot and serve; all 12 DGX Spark DFLASH2 cells
boot and serve on `1cf2b8c` with the selector folded into the draft CUDA