From f30a6f4d7e54f31761981b1f37b3b91768b7fbd0 Mon Sep 17 00:00:00 2001
From: shuwenn <47200617+alphabetc1@users.noreply.github.com>
Date: Sat, 25 Apr 2026 08:18:46 +0800
Subject: [PATCH] [DOC] Add DFLASH speculative decoding documentation (#23553)
---
.../speculative_decoding.mdx | 98 ++++++++++++++++++-
1 file changed, 95 insertions(+), 3 deletions(-)
diff --git a/docs_new/docs/advanced_features/speculative_decoding.mdx b/docs_new/docs/advanced_features/speculative_decoding.mdx
index 4946012f1..931e5c05a 100644
--- a/docs_new/docs/advanced_features/speculative_decoding.mdx
+++ b/docs_new/docs/advanced_features/speculative_decoding.mdx
@@ -1,9 +1,9 @@
---
title: "Speculative Decoding"
metatags:
- description: "SGLang EAGLE speculative decoding: EAGLE-2/EAGLE-3, up to 2.4x throughput improvement, draft model configuration, MTP for DeepSeek."
+ description: "SGLang speculative decoding: EAGLE-2/EAGLE-3, MTP, DFLASH, draft model configuration, and overlap-scheduler guidance."
---
-SGLang provides several speculative decoding options, including EAGLE-2/EAGLE-3, MTP, classic draft-model decoding, and an NGRAM-based variant. Our implementation aims to maximize speed and efficiency and is considered to be among the fastest in open-source LLM engines.
+SGLang provides several speculative decoding options, including EAGLE-2/EAGLE-3, MTP, DFLASH, classic draft-model decoding, and an NGRAM-based variant. Our implementation aims to maximize speed and efficiency and is considered to be among the fastest in open-source LLM engines.
## Summary
@@ -15,6 +15,7 @@ SGLang provides several speculative decoding options, including EAGLE-2/EAGLE-3,
- [EAGLE-2 Decoding via Frequency-Ranked Speculative Sampling](#eagle-2-decoding-via-frequency-ranked-speculative-sampling)
- [EAGLE-3 Decoding](#eagle-3-decoding)
- [Multi Token Prediction](#multi-token-prediction)
+- [DFlash Decoding](#dflash-decoding)
- [Standalone Speculative Decoding (Small Draft Model)](#standalone-speculative-decoding-small-draft-model)
- [Speculative Decoding V2 (Overlap Scheduler)](#speculative-decoding-v2-overlap-scheduler)
- [Ngram Speculative Decoding](#ngram-speculative-decoding)
@@ -29,6 +30,7 @@ SGLang provides several speculative decoding options, including EAGLE-2/EAGLE-3,
- **Workload acceptance changes over time**: Use [**Adaptive speculative decoding**](./adaptive_speculative_decoding) on top of **EAGLE** with `--speculative-eagle-topk 1`.
- **Lower `lm_head` overhead for EAGLE-2**: Enable **FR-Spec** with `--speculative-token-map`.
- **Model is MTP-enabled**: Use **MTP via speculative decoding** (often with small `speculative_num_steps/topk/num_draft_tokens`, see the example section).
+- **You have a DFlash draft checkpoint**: Use **DFLASH** with `--speculative-algorithm DFLASH` and `--speculative-draft-model-path ...`.
- **You have a smaller draft LLM**: Use **STANDALONE** (`--speculative-algorithm STANDALONE`).
- **No extra model available**: Use **NGRAM** (`--speculative-algorithm NGRAM`, CUDA-only).
- **Want overlap scheduler (experimental)**: Enable **SpecV2** with `SGLANG_ENABLE_SPEC_V2=True` (requires `--speculative-eagle-topk 1`).
@@ -85,6 +87,13 @@ SGLang provides several speculative decoding options, including EAGLE-2/EAGLE-3,
| See Multi Token Prediction section |
Uses speculative workflow; draft path may be auto-handled for some models |
+
+ | DFLASH |
+ DFlash draft model (linear block verification) |
+ Yes |
+ --speculative-algorithm DFLASH + --speculative-draft-model-path ... |
+ No --enable-dp-attention; pp_size == 1; disables overlap scheduler & mixed chunked prefill |
+
| STANDALONE |
Smaller draft LLM (token-level) |
@@ -426,6 +435,77 @@ print(response.json())
---
+## DFlash Decoding
+
+SGLang also supports **DFLASH** speculative decoding using a dedicated draft model checkpoint. Compared with EAGLE-style tree verification, DFLASH verifies a linear draft block and is configured around a block size / draft window. This path is useful when the target model has a matching DFlash draft checkpoint, such as `meta-llama/Llama-3.1-8B-Instruct` with `z-lab/LLaMA3.1-8B-Instruct-DFlash-UltraChat`.
+
+Relevant parameters:
+
+
+
+
+
+
+
+
+
+ | Parameter |
+ Description |
+ Default |
+
+
+
+
+ --speculative-draft-model-path |
+ Required DFlash draft model path/weights. |
+ None |
+
+
+ --speculative-num-draft-tokens |
+ DFlash verify block size. |
+ Inferred from draft config, otherwise 16 |
+
+
+ --speculative-dflash-block-size |
+ Alias of --speculative-num-draft-tokens for DFlash. |
+ None |
+
+
+ --speculative-dflash-draft-window-size |
+ Draft KV sliding-window size. Must be >= speculative-num-draft-tokens when set. |
+ None |
+
+
+
+
+```bash Command
+python3 -m sglang.launch_server \
+ --model meta-llama/Llama-3.1-8B-Instruct \
+ --speculative-algorithm DFLASH \
+ --speculative-draft-model-path z-lab/LLaMA3.1-8B-Instruct-DFlash-UltraChat
+```
+
+**Send a request:**
+
+```python Example
+import openai
+
+client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="None")
+
+response = client.chat.completions.create(
+ model="meta-llama/Llama-3.1-8B-Instruct",
+ messages=[
+ {"role": "user", "content": "Write a quicksort implementation in Python."},
+ ],
+ temperature=0,
+ max_tokens=128,
+)
+
+print(response.choices[0].message.content)
+```
+
+---
+
## Standalone Speculative Decoding (Small Draft Model)
Besides EAGLE/MTP, SGLang also supports **token-level speculative decoding** using a smaller **draft model**. Enable it with `--speculative-algorithm STANDALONE` and provide a draft model via `--speculative-draft-model-path`.
@@ -678,7 +758,7 @@ Below is a comprehensive list of all speculative decoding parameters available i
--speculative-algorithm |
str |
None |
- Algorithm to use: EAGLE, EAGLE3, STANDALONE, NGRAM, NEXTN (alias of EAGLE) |
+ Algorithm to use: DFLASH, EAGLE, EAGLE3, STANDALONE, NGRAM, NEXTN (alias of EAGLE) |
--speculative-draft-model-path |
@@ -716,6 +796,18 @@ Below is a comprehensive list of all speculative decoding parameters available i
None (auto-chosen when omitted) |
Maximum number of draft tokens for verification |
+
+ --speculative-dflash-block-size |
+ int |
+ None |
+ DFlash-only alias of --speculative-num-draft-tokens |
+
+
+ --speculative-dflash-draft-window-size |
+ int |
+ None |
+ DFlash-only draft KV sliding-window size |
+
--speculative-accept-threshold-single |
float |