[SPEC][1/N] feat: add adaptive speculative_num_steps for EAGLE topk=1 (#21599)

Co-authored-by: Qiaolin-Yu <liin1211@outlook.com>
2026-04-20 14:25:04 -07:00
parent dbcf7459b5
commit b65799cf83
13 changed files with 1296 additions and 33 deletions
@@ -0,0 +1,156 @@
+# Adaptive Speculative Decoding
+
+Adaptive speculative decoding lets SGLang adjust `speculative_num_steps/speculative_num_draft_tokens` at runtime instead of keeping a single fixed value for the whole server lifetime.
+It is designed for workloads whose accept length changes over time, where one static step count is rarely optimal.
+
+## Current support
+
+- Only `--speculative-algorithm EAGLE`
+- Only `--speculative-eagle-topk 1`
+- If either condition is not met, SGLang falls back to static speculative settings
+
+## Why adaptive steps help
+
+`speculative_num_steps` controls how many draft-model autoregressive steps run in each speculative round. In practice, the best value depends on the current workload.
+
+- If `num_steps` is too small, the draft model could have produced more accepted tokens, but the round stops too early.
+- If `num_steps` is too large, the draft model produces many candidate tokens that the target model rejects, so extra draft work is wasted.
+
+Real traffic often moves between high-acceptance and low-acceptance phases, so one fixed step count is usually a compromise. Adaptive mode tries to follow the workload instead of hard-coding a single global `num_steps`.
+
+## Design overview
+
+The adaptive mechanism has three pieces:
+
+- `AdaptiveSpeculativeParams`: the EMA-based policy
+- `SpecRuntimeState`: the per-tier runtime state bundle
+- `AdaptiveController`: the coordinator that chooses a tier and activates the matching runtime state
+
+At startup, SGLang pre-builds one runtime state per candidate tier. By default, the candidate tiers are `candidate_steps = [1, 3, 7]`.
+
+```text
+┌──────────────────────────────────────────────────────────┐
+│                      SpecRuntimeState                    │
+│                                                          │
+│  speculative_num_steps / speculative_num_draft_tokens   │
+│                                                          │
+│  ┌────────────────┐ ┌────────────────┐ ┌──────────────┐  │
+│  │  Draft stage   │ │  Verify stage  │ │ Extend stage │  │
+│  │                │ │                │ │              │  │
+│  │  attn_backend  │ │  attn_backend  │ │ attn_backend │  │
+│  │  cuda_graph    │ │  cuda_graph    │ │ cuda_graph   │  │
+│  └────────────────┘ └────────────────┘ └──────────────┘  │
+└──────────────────────────────────────────────────────────┘
+```
+
+This matters because `CudaGraphRunner` is shape-dependent. Each candidate tier owns its own graph and backend state, so runtime switching is a reference swap, not an online graph recapture.
+
+## Runtime flow
+
+The adaptive update happens after verify and affects the next round, not the current one:
+
+```text
+┌─────────────────────────────────────────────────────────────────────┐
+│           EAGLEWorker.forward_batch_generation() — decode path      │
+│                                                                     │
+│   ① draft(batch)                                                    │
+│       │  draft model multi-step generation with current tier        │
+│       v                                                             │
+│   ② verify(batch, spec_info)                                        │
+│       │  target model tree verification                             │
+│       │  → produces accept_length_per_req                           │
+│       v                                                             │
+│   ③ forward_draft_extend_after_decode(batch)                        │
+│       │  draft model KV-cache catch-up                              │
+│       v                                                             │
+│   ④ adaptive_controller.on_verify_complete(accept_lengths)          │
+│       │                                                             │
+│       │  update EMA, apply warmup / interval / hysteresis gates     │
+│       │  if tier changed, select a pre-built state from pool        │
+│       v                                                             │
+│     worker.apply_runtime_state(state)                               │
+│                                                                     │
+│   Tier switch happens after the current round completes.            │
+│   Backends and CUDA graphs are never swapped mid-round.             │
+└─────────────────────────────────────────────────────────────────────┘
+```
+
+## How the policy decides
+
+After each verify pass, SGLang reads the accepted draft length per request, computes the batch average, smooths it with an exponential moving average (EMA), and switches among the pre-built candidate tiers `[1, 3, 7]` by default.
+
+The decision logic is intentionally conservative:
+
+- `warmup_batches` skips the first few batches
+- `update_interval` avoids switching every batch
+- `down_hysteresis` and `up_hysteresis` reduce oscillation
+
+Conceptually, the policy probes one step beyond the observed acceptance:
+
+```text
+target_steps ≈ clamp(round(ema_accept_len) + 1, min(candidate_steps), max(candidate_steps))
+```
+
+So if recent requests consistently accept more drafted tokens, the policy tends to move up. If they start rejecting earlier, it tends to move down.
+
+## Usage
+
+`--speculative-adaptive-config` is optional, but the speculative setup still needs to be valid for adaptive mode.
+
+```bash
+python3 -m sglang.launch_server \
+    --model meta-llama/Llama-2-7b-chat-hf \
+    --speculative-algorithm EAGLE \
+    --speculative-draft-model-path lmsys/sglang-EAGLE-llama2-chat-7B \
+    --speculative-eagle-topk 1 \
+    --speculative-num-steps 3 \
+    --speculative-num-draft-tokens 4 \
+    --speculative-adaptive
+```
+
+If you want to override the defaults, add `--speculative-adaptive-config /path/to/adaptive_spec.json`.
+
+Example config:
+
+```json
+{
+  "candidate_steps": [1, 3, 7],
+  "ema_alpha": 0.2,
+  "warmup_batches": 10,
+  "update_interval": 5
+}
+```
+
+## Config file reference
+
+The config file is optional. Any omitted keys use defaults.
+
+| Key | Default | Meaning |
+|---|---|---|
+| `candidate_steps` | `[1, 3, 7]` | Discrete `speculative_num_steps` tiers that adaptive mode can switch between |
+| `ema_alpha` | `0.2` | EMA smoothing factor for accepted draft length |
+| `update_interval` | `5` | Recompute interval, in verify batches, after warmup |
+| `warmup_batches` | `10` | Number of verify batches to observe before switching |
+| `down_hysteresis` | `-0.25` | Extra margin before moving to a smaller step |
+| `up_hysteresis` | `0.0` | Extra margin before moving to a larger step |
+
+The initial `--speculative-num-steps` is snapped to the nearest value in `candidate_steps`.
+
+## Monitoring
+
+You can inspect the active tier and acceptance metric via `/server_info`:
+
+```bash
+curl -s http://127.0.0.1:30000/server_info | jq '.internal_states[0] | {speculative_num_steps, avg_spec_accept_length}'
+```
+
+- `speculative_num_steps` is the current active tier
+- `avg_spec_accept_length` helps explain whether the server is likely to move up or down
+
+## Tuning tips
+
+- Start with the default candidate tiers `[1, 3, 7]`
+- Use fewer tiers if you want lower startup and graph-memory overhead
+- Increase `ema_alpha` to react faster, or lower it for more stability
+- Increase `warmup_batches` or `update_interval` if tier switching is too noisy
+- If your workload is already stable and one static setting is well tuned, adaptive mode may not help much
@@ -23,6 +23,7 @@ SGLang provides several speculative decoding options, including EAGLE-2/EAGLE-3,

 - **Best speed/quality (recommended)**: Use **EAGLE-3** with `--speculative-algorithm EAGLE3`.
 - **Strong default / broad compatibility**: Use **EAGLE-2** with `--speculative-algorithm EAGLE`.
+- **Workload acceptance changes over time**: Use [**Adaptive speculative decoding**](adaptive_speculative_decoding.md) on top of **EAGLE** with `--speculative-eagle-topk 1`.
 - **Lower `lm_head` overhead for EAGLE-2**: Enable **FR-Spec** with `--speculative-token-map`.
 - **Model is MTP-enabled**: Use **MTP via speculative decoding** (often with small `speculative_num_steps/topk/num_draft_tokens`, see the example section).
 - **You have a smaller draft LLM**: Use **STANDALONE** (`--speculative-algorithm STANDALONE`).
@@ -75,6 +76,7 @@ To enable EAGLE speculative decoding the following parameters are relevant:

 These parameters are mostly the same for EAGLE-2 and EAGLE-3. `--speculative-token-map` is ignored for EAGLE-3 models.
 For `--speculative-num-steps`, `--speculative-eagle-topk`, and `--speculative-num-draft-tokens`: leave all three unset to use auto-tuning, or set all three explicitly when tuning.
+If you use EAGLE with `--speculative-eagle-topk 1` and your acceptance rate varies across requests, see [Adaptive Speculative Decoding](adaptive_speculative_decoding.md).

 You can find the best combinations of these parameters with [bench_speculative.py](https://github.com/sgl-project/sglang/blob/main/scripts/playground/bench_speculative.py).