[Docs] Sync docs_new with legacy docs and update migration redirects (#23337)

Co-authored-by: Mingyi <wisclmy0611@gmail.com>
2026-04-21 00:15:17 -07:00
parent f63def8510
commit 900aad5f72
179 changed files with 16014 additions and 8162 deletions
@@ -0,0 +1,197 @@
+---
+title: "Adaptive Speculative Decoding"
+metatags:
+    description: "Configure adaptive speculative decoding so SGLang can adjust speculative steps and draft tokens at runtime based on acceptance behavior."
+---
+
+Adaptive speculative decoding lets SGLang adjust `speculative_num_steps/speculative_num_draft_tokens` at runtime instead of keeping a single fixed value for the whole server lifetime.
+It is designed for workloads whose accept length changes over time, where one static step count is rarely optimal.
+
+## Current support
+
+- Only `--speculative-algorithm EAGLE`
+- Only `--speculative-eagle-topk 1`
+- If either condition is not met, SGLang falls back to static speculative settings
+
+## Why adaptive steps help
+
+`speculative_num_steps` controls how many draft-model autoregressive steps run in each speculative round. In practice, the best value depends on the current workload.
+
+- If `num_steps` is too small, the draft model could have produced more accepted tokens, but the round stops too early.
+- If `num_steps` is too large, the draft model produces many candidate tokens that the target model rejects, so extra draft work is wasted.
+
+Real traffic often moves between high-acceptance and low-acceptance phases, so one fixed step count is usually a compromise. Adaptive mode tries to follow the workload instead of hard-coding a single global `num_steps`.
+
+## Design overview
+
+The adaptive mechanism has three pieces:
+
+- `AdaptiveSpeculativeParams`: the EMA-based policy
+- `SpecRuntimeState`: the per-tier runtime state bundle
+- `AdaptiveController`: the coordinator that chooses a tier and activates the matching runtime state
+
+At startup, SGLang pre-builds one runtime state per candidate tier. By default, the candidate tiers are `candidate_steps = [1, 3, 7]`.
+
+```mermaid
+---
+title: "SpecRuntimeState — speculative_num_steps / speculative_num_draft_tokens"
+---
+graph LR
+  subgraph SR[" "]
+    direction LR
+    subgraph D["Draft stage"]
+      direction TB
+      d1[attn_backend]
+      d2[cuda_graph]
+    end
+    subgraph V["Verify stage"]
+      direction TB
+      v1[attn_backend]
+      v2[cuda_graph]
+    end
+    subgraph E["Extend stage"]
+      direction TB
+      e1[attn_backend]
+      e2[cuda_graph]
+    end
+  end
+```
+
+This matters because `CudaGraphRunner` is shape-dependent. Each candidate tier owns its own graph and backend state, so runtime switching is a reference swap, not an online graph recapture.
+
+## Runtime flow
+
+The adaptive update happens after verify and affects the next round, not the current one:
+
+```mermaid
+---
+title: "EAGLEWorker.forward_batch_generation() — decode path"
+---
+flowchart TD
+  A["① draft(batch)<br/>draft model multi-step generation with current tier"]
+  B["② verify(batch, spec_info)<br/>target model tree verification → produces accept_length_per_req"]
+  C["③ forward_draft_extend_after_decode(batch)<br/>draft model KV-cache catch-up"]
+  D["④ adaptive_controller.on_verify_complete(accept_lengths)<br/>update EMA, apply warmup / interval / hysteresis gates<br/>if tier changed, select a pre-built state from pool"]
+  E["worker.apply_runtime_state(state)"]
+  A --> B --> C --> D --> E
+```
+
+> Tier switch happens after the current round completes. Backends and CUDA graphs are never swapped mid-round.
+
+## How the policy decides
+
+After each verify pass, SGLang reads the accepted draft length per request, computes the batch average, smooths it with an exponential moving average (EMA), and switches among the pre-built candidate tiers `[1, 3, 7]` by default.
+
+The decision logic is intentionally conservative:
+
+- `warmup_batches` skips the first few batches
+- `update_interval` avoids switching every batch
+- `down_hysteresis` and `up_hysteresis` reduce oscillation
+
+Conceptually, the policy probes one step beyond the observed acceptance:
+
+```text
+target_steps ≈ clamp(round(ema_accept_len) + 1, min(candidate_steps), max(candidate_steps))
+```
+
+So if recent requests consistently accept more drafted tokens, the policy tends to move up. If they start rejecting earlier, it tends to move down.
+
+## Usage
+
+`--speculative-adaptive-config` is optional, but the speculative setup still needs to be valid for adaptive mode.
+
+```bash
+python3 -m sglang.launch_server \
+    --model meta-llama/Llama-2-7b-chat-hf \
+    --speculative-algorithm EAGLE \
+    --speculative-draft-model-path lmsys/sglang-EAGLE-llama2-chat-7B \
+    --speculative-eagle-topk 1 \
+    --speculative-num-steps 3 \
+    --speculative-num-draft-tokens 4 \
+    --speculative-adaptive
+```
+
+If you want to override the defaults, add `--speculative-adaptive-config /path/to/adaptive_spec.json`.
+
+Example config:
+
+```json
+{
+  "candidate_steps": [1, 3, 7],
+  "ema_alpha": 0.2,
+  "warmup_batches": 10,
+  "update_interval": 5
+}
+```
+
+## Config file reference
+
+The config file is optional. Any omitted keys use defaults.
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <colgroup>
+    <col style={{width: "33.33%"}} />
+    <col style={{width: "33.33%"}} />
+    <col style={{width: "33.33%"}} />
+  </colgroup>
+  <thead>
+    <tr>
+      <th>Key</th>
+      <th>Default</th>
+      <th>Meaning</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td><code>candidate_steps</code></td>
+      <td><code>[1, 3, 7]</code></td>
+      <td>Discrete <code>speculative_num_steps</code> tiers that adaptive mode can switch between</td>
+    </tr>
+    <tr>
+      <td><code>ema_alpha</code></td>
+      <td><code>0.2</code></td>
+      <td>EMA smoothing factor for accepted draft length</td>
+    </tr>
+    <tr>
+      <td><code>update_interval</code></td>
+      <td><code>5</code></td>
+      <td>Recompute interval, in verify batches, after warmup</td>
+    </tr>
+    <tr>
+      <td><code>warmup_batches</code></td>
+      <td><code>10</code></td>
+      <td>Number of verify batches to observe before switching</td>
+    </tr>
+    <tr>
+      <td><code>down_hysteresis</code></td>
+      <td><code>-0.25</code></td>
+      <td>Extra margin before moving to a smaller step</td>
+    </tr>
+    <tr>
+      <td><code>up_hysteresis</code></td>
+      <td><code>0.0</code></td>
+      <td>Extra margin before moving to a larger step</td>
+    </tr>
+  </tbody>
+</table>
+
+The initial `--speculative-num-steps` is snapped to the nearest value in `candidate_steps`.
+
+## Monitoring
+
+You can inspect the active tier and acceptance metric via `/server_info`:
+
+```bash
+curl -s http://127.0.0.1:30000/server_info | jq '.internal_states[0] | {speculative_num_steps, avg_spec_accept_length}'
+```
+
+- `speculative_num_steps` is the current active tier
+- `avg_spec_accept_length` helps explain whether the server is likely to move up or down
+
+## Tuning tips
+
+- Start with the default candidate tiers `[1, 3, 7]`
+- Use fewer tiers if you want lower startup and graph-memory overhead
+- Increase `ema_alpha` to react faster, or lower it for more stability
+- Increase `warmup_batches` or `update_interval` if tier switching is too noisy
+- If your workload is already stable and one static setting is well tuned, adaptive mode may not help much
@@ -14,7 +14,7 @@ If you don't specify `--attention-backend`, SGLang makes a best effort to automa

 ## Support Matrix

-The support matrix is split into two parts: MHA (standard attention) and MLA (multi-head latent attention). For an explanation of the key differences between MHA and MLA, please see the [SGLang documentation on DeepSeek MLA](../basic_usage/deepseek_v3.md#multi-head-latent-attention-mla-throughput-optimizations) and the original [DeepSeek MLA paper](https://arxiv.org/pdf/2405.04434).
+The support matrix is split into two parts: MHA (standard attention) and MLA (multi-head latent attention). For an explanation of the key differences between MHA and MLA, please see the [SGLang documentation on DeepSeek MLA](../basic_usage/deepseek_v3#multi-head-latent-attention-mla-throughput-optimizations) and the original [DeepSeek MLA paper](https://arxiv.org/pdf/2405.04434).

 ### MHA Backends

@@ -67,15 +67,15 @@ The support matrix is split into two parts: MHA (standard attention) and MLA (mu
      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>128</td>
      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
-      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
-      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
    </tr>
    <tr>
      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**Triton**</td>
      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
-      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
@@ -129,7 +129,7 @@ The support matrix is split into two parts: MHA (standard attention) and MLA (mu
      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
-      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
    </tr>
    <tr>
@@ -147,9 +147,9 @@ The support matrix is split into two parts: MHA (standard attention) and MLA (mu
      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
-      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
-      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
    </tr>
    <tr>
@@ -258,7 +258,7 @@ The support matrix is split into two parts: MHA (standard attention) and MLA (mu
      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1</td>
      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
-      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
    </tr>
@@ -279,10 +279,14 @@ Multimodal attention is selected by `--mm-attention-backend`. The "MultiModal" c
 </Note>

 <Note>
- FlashAttention 4 is prefill-only for now.
- NSA is specifically designed for [DeepSeek V3.2 DSA](https://lmsys.org/blog/2025-09-29-deepseek-V32/).
+- FlashAttention 4 supports both prefill and decode on SM90 (Hopper) and SM100 (Blackwell). FA4 MLA supports `page_size = 1`; FA4 MHA requires `page_size = 128`. On SM100, this is auto-enforced by the server; on SM90, users must set `--page-size 128` manually.
+- NSA is specifically designed for [DeepSeek V3.2 DSA](https://lmsys.org/blog/2025-09-29-deepseek-V32/). See the [DSA Attention Backend (NSA)](#dsa-attention-backend-nsa) section and [DeepSeek V3.2 deployment guide](../basic_usage/deepseek_v32) for details.
 </Note>

+<Warning>
+**FA4 on Hopper (SM90):** FA4 decode speed decreases as sequence length grows due to lack of SplitKV support. At batch=1 compared to FA3 on H100: ~-10% at 2K tokens, ~-18% at 4K, ~-31% at 8K, ~-49% at 16K. Larger batch sizes reduce the gap (e.g., batch=8: ~-2% at 2K, ~-8% at 4K). Blackwell (SM100) is not affected.
+</Warning>
+
 <Note>
 For the KV4 FA4 scenario, FA4 requires using a different --decode-attention-backend to run. Except for trtllm_mha being incompatible with FA4, all other decode backends behave as shown in the table.
 </Note>
@@ -291,8 +295,16 @@ For the KV4 FA4 scenario, FA4 requires using a different --decode-attention-back
 Speculative decoding topk: `topk` is the number of draft tokens sampled per step from the draft model. `topk = 1` follows classic EAGLE; `topk > 1` explores multiple branches and requires backend support in both draft and verification paths.
 </Tip>

+<Note>
+**Speculative Decoding V2 (Spec V2):** Spec V2 uses overlap scheduling (`SGLANG_ENABLE_SPEC_V2=True`) that benefits various attention backends. Requires `--speculative-eagle-topk 1` and currently applies to EAGLE and EAGLE3.
+
+**Verified backends:** TRTLLM MLA, TRTLLM MHA, FA3, Ascend (NPU), Triton.
+
+**Limited support:** FlashInfer can run under Spec V2, but its plan stream (used for split-KV optimization) introduces a synchronization point that limits overlap benefits.
+</Note>
+
 <Tip>
-Page size controls how many tokens are grouped into a KV cache block. For the prefix cache to take effect, the number of tokens must fill at least one complete page. For example, if your prompt is only 32 tokens and `page_size = 64`, it won't fill a complete page and cannot be matched in the prefix cache (pages cannot be padded). With 65 tokens and `page_size = 64`, only the first page of 64 tokens will be cached and matched; the remaining 1 token is discarded. Use `page_size = 1` for maximum prefix reuse (token-level matching).
+Page size controls how many tokens are grouped into a KV cache block. For the prefix cache to take effect, the number of tokens must fill at least one complete page. For example, if your prompt is only 32 tokens and `page_size = 64`, it won't fill a complete page and cannot be matched in the prefix cache (pages cannot be padded). With 65 tokens and `page_size = 64`, only the first page of 64 tokens will be cached and matched; the remaining 1 token is discarded. Use `page_size = 1` for maximum prefix reuse (token-level matching). Note that higher page sizes generally improve attention kernel performance, so prefer `page_size > 1` when prefix cache reuse is not critical.
 </Tip>

 Many backends that do not natively operate on pages can emulate `page_size > 1` at the wrapper layer by expanding page tables to per-token indices. The "Page Size > 1 (native)" column indicates true in-kernel paging. Some backends require fixed native page sizes and cannot be reduced/emulated differently: TRTLLM MHA (16/32/64), TRTLLM MLA (32/64), FlashMLA (64), Cutlass MLA (128), Ascend (128).
@@ -303,6 +315,138 @@ MLA page-size constraints:
 - Cutlass MLA: page_size = 128.
 - TRTLLM MLA: page_size ∈ &#123;32, 64&#125;.

+### GDN Attention Backends
+
+GDN (Gated Delta Network) is a linear attention mechanism with O(n) complexity, used in hybrid models that alternate GDN linear attention layers with standard full attention layers. GDN is **not** selected via `--attention-backend`; it is automatically activated when the model architecture requires it (e.g., Qwen 3.5, Qwen 3 Next, Jet Nemotron, Jet VLM).
+
+The GDN linear attention layers have their own kernel backends, selected via `--linear-attn-backend` (default: `triton`). You can override the kernel per phase with `--linear-attn-decode-backend` and `--linear-attn-prefill-backend`.
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <colgroup>
+    <col style={{width: "28%"}} />
+    <col style={{width: "16%"}} />
+    <col style={{width: "24%"}} />
+    <col style={{width: "32%"}} />
+  </colgroup>
+  <thead>
+    <tr style={{borderBottom: "2px solid #d55816"}}>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Backend</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Decode</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Prefill / Extend</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Spec Decoding (Target Verify)</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong>Triton (CUDA)</strong></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong>Triton (AMD/ROCm)</strong></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong>Triton (NPU)</strong></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong>Triton (CPU)</strong></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong>CuTe DSL (CUDA only)</strong></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
+    </tr>
+  </tbody>
+</table>
+
+<Warning>
+GDN models are hybrid: the full-attention layers still require a standard `--attention-backend`. Platform constraints for the full-attention backend on hybrid GDN models:
+- **Blackwell (e.g., B200)**: `triton`, `trtllm_mha`, or `fa4` only.
+- **NPU (Ascend)**: `ascend` only.
+- **AMD (ROCm)**: `triton` recommended.
+- **Other CUDA (Hopper, Ampere, etc.)**: auto-selection works; no special constraints.
+</Warning>
+
+### DSA Attention Backend (NSA)
+
+DSA (Deepseek Sparse Attention) is a native sparse attention mechanism used by [DeepSeek V3.2](https://lmsys.org/blog/2025-09-29-deepseek-V32/). It is activated automatically when the model architecture requires it and is selected via `--attention-backend nsa`.
+
+Internally, the NSA backend dispatches to different sub-backends for prefill and decode phases. You can override these with `--nsa-prefill-backend` and `--nsa-decode-backend`:
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <colgroup>
+    <col style={{width: "26%"}} />
+    <col style={{width: "16%"}} />
+    <col style={{width: "16%"}} />
+    <col style={{width: "42%"}} />
+  </colgroup>
+  <thead>
+    <tr style={{borderBottom: "2px solid #d55816"}}>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Sub-backend</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Prefill</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Decode</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Notes</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong>flashmla_sparse</strong></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Default prefill on Hopper and Blackwell (bf16)</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong>flashmla_kv</strong></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Default decode for FP8 on Blackwell with DP</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong>flashmla_auto</strong></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Auto-selects flashmla_sparse or flashmla_kv based on kv_cache_dtype</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong>fa3</strong></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Default decode on Hopper (bf16)</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong>trtllm</strong></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Default decode on Blackwell (bf16); default for both on Blackwell without DP</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong>tilelang</strong></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Default on AMD (ROCm)</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong>aiter</strong></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>AMD-specific kernel library (requires aiter package)</td>
+    </tr>
+  </tbody>
+</table>
+
+For deployment examples, see the [DeepSeek V3.2 deployment guide](../basic_usage/deepseek_v32).
+
 ### Hybrid attention (different backends for prefill vs decode) (Experimental)

 <Warning>
@@ -354,7 +498,7 @@ If the `--attention-backend` argument is not specified, SGLang automatically sel

 **2. MLA Models (e.g., DeepSeek V3)**
 - **Hopper**: Defaults to `fa3` (requires CUDA 12.3+).
- **Blackwell**: Defaults to `trtllm_mla`.
+- **Blackwell**: Defaults to `flashinfer`; `trtllm_mla` is auto-selected for DeepSeek V3 models specifically.
 - **Other Architectures**: Defaults to `triton`.


@@ -432,8 +576,34 @@ python3 -m sglang.launch_server \
  --trust-remote-code
 ```

+- TRTLLM MHA (Optimized for Blackwell Architecture, e.g., B200)
+```bash Command
+python3 -m sglang.launch_server \
+  --tp 4 \
+  --model Qwen/Qwen3.5-35B-A3B-FP8 \
+  --attention-backend trtllm_mha \
+  --trust-remote-code
+```
+
+- TRTLLM MHA (XQA backend) (Optimized for SM90 and SM120, e.g., H20, H200, 5090)
+  Note that TRTLLM XQA backend only works well for pagesize 64.
+```bash Command
+python3 -m sglang.launch_server \
+  --tp 4 \
+  --model Qwen/Qwen3.5-35B-A3B-FP8 \
+  --decode-attention-backend trtllm_mha \
+  --trust-remote-code
+```
+
 - FlashAttention 4 (MHA & MLA)
 ```bash Command
+# FA4 for both prefill and decode on SM90/SM100
+python3 -m sglang.launch_server \
+  --model-path Qwen/Qwen3-30B-A3B-Instruct-2507-FP8 \
+  --attention-backend fa4 \
+  --page-size 128 \
+  --trust-remote-code
+
 python3 -m sglang.launch_server \
  --tp 8 \
  --model deepseek-ai/DeepSeek-R1 \
@@ -497,6 +667,10 @@ To add a new attention backend, you can learn from the existing backends
 (`python/sglang/srt/layers/attention/triton_backend.py`, `python/sglang/srt/layers/attention/flashattention_backend.py`)
 and follow the steps below.

+<Note>
+Linear attention kernel backends (GDN, KDA) follow a different pattern. They implement `LinearAttnKernelBase` in `python/sglang/srt/layers/attention/linear/kernels/` and are dispatched by `GDNKernelDispatcher` / `KDAKernelDispatcher` rather than registered via `@register_attention_backend`.
+</Note>
+
 1. Run without cuda graph. Support the two forward functions
    - forward_extend
        - Will be used for prefill, prefill with KV cache, and target verification
@@ -19,6 +19,81 @@ When launching a language-only model, you must additionally specify the encoder

 We support multiple encoder transfer backends, including zmq_to_scheduler, zmq_to_tokenizer, and mooncake (the default is zmq_to_scheduler). The backend can be selected using `--encoder-transfer-backend`.

+### Encoder transfer with Mooncake
+
+`--encoder-transfer-backend mooncake` controls **how encoder outputs are transferred** between encoder and language/prefill services. It is an encoder transfer option and can be used independently of the global multimodal embedding cache.
+
+Example:
+
+```bash Command
+# encoder
+python -m sglang.launch_server \
+  --model-path Qwen/Qwen3-VL-8B-Instruct \
+  --encoder-only \
+  --encoder-transfer-backend mooncake \
+  --port 30000
+
+# language-only server
+python -m sglang.launch_server \
+  --model-path Qwen/Qwen3-VL-8B-Instruct \
+  --language-only \
+  --encoder-urls http://127.0.0.1:30000 \
+  --encoder-transfer-backend mooncake \
+  --port 30002
+```
+
+### Global multimodal embedding cache with Mooncake
+
+SGLang also supports a Mooncake-backed **global multimodal embedding cache** for EPD workloads. When enabled on encoder servers, repeated image inputs can reuse previously computed ViT embeddings across instances instead of running the vision encoder again.
+
+This feature is useful when:
+
+- the deployment serves repeated or overlapping image inputs,
+- encoder compute is the bottleneck, and
+- Mooncake is already available in the cluster.
+
+At a high level, the encoder checks whether the image embedding already exists in Mooncake. Cache hits are prefetched from the global store, while misses are encoded normally and inserted into the cache in the background.
+
+To enable it:
+
+- install and configure Mooncake in the same way as other SGLang Mooncake integrations,
+- add `--enable-mm-global-cache` on the encoder server.
+
+`--enable-mm-global-cache` controls **whether multimodal embeddings are looked up and stored in the global Mooncake cache**. It is separate from `--encoder-transfer-backend`, which only controls encoder output transport.
+
+For Mooncake deployment and configuration details, see [HiCache best practices](./hicache_best_practices#deployment-with-mooncake) and the [Mooncake backend README](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/mem_cache/storage/mooncake_store/README.md).
+
+Example:
+
+```bash Command
+# Shared Mooncake configuration
+export MOONCAKE_TE_META_DATA_SERVER="http://127.0.0.1:8080/metadata"
+export MOONCAKE_MASTER="127.0.0.1:50051"
+export MOONCAKE_PROTOCOL="rdma"
+export MOONCAKE_GLOBAL_SEGMENT_SIZE="4gb"
+
+# encoder with global multimodal cache enabled
+python -m sglang.launch_server \
+  --model-path Qwen/Qwen3-VL-8B-Instruct \
+  --encoder-only \
+  --enable-mm-global-cache \
+  --port 30000
+
+# language-only server
+python -m sglang.launch_server \
+  --model-path Qwen/Qwen3-VL-8B-Instruct \
+  --language-only \
+  --encoder-urls http://127.0.0.1:30000 \
+  --port 30002
+```
+
+Notes:
+
+- This cache is for **multimodal encoder embeddings**, not the language model KV cache.
+- The feature currently uses Mooncake as the shared backing store.
+- It can be enabled regardless of which `--encoder-transfer-backend` you use.
+- It is most relevant for EPD or encoder-disaggregated VLM deployments where the same images are likely to appear across requests or instances.
+
 #### Qwen VL

 - EP Disaggregation
@@ -81,3 +156,42 @@ python -m sglang_router.launch_router \
  --port 8000

 ```
+
+#### gRPC Encoder (EPD)
+
+You can run the encoder as a gRPC server while keeping prefill/decode as HTTP.
+When using gRPC encoders, set `SGLANG_ENCODER_MM_RECEIVER_MODE=grpc` for the
+prefill process so it uses the gRPC receiver.
+
+```bash Command
+# gRPC encoder
+python -m sglang.launch_server \
+  --model-path Qwen/Qwen3-VL-8B-Instruct \
+  --encoder-only \
+  --grpc-mode \
+  --encoder-transfer-backend zmq_to_scheduler \
+  --port 30000
+
+# prefill (HTTP) - tell it to use gRPC receiver
+SGLANG_ENCODER_MM_RECEIVER_MODE=grpc \
+python -m sglang.launch_server \
+  --model-path Qwen/Qwen3-VL-8B-Instruct \
+  --disaggregation-mode prefill \
+  --language-only \
+  --encoder-urls grpc://127.0.0.1:30000 \
+  --encoder-transfer-backend zmq_to_scheduler \
+  --port 30002
+
+# decode (HTTP)
+python -m sglang.launch_server \
+  --model-path Qwen/Qwen3-VL-8B-Instruct \
+  --disaggregation-mode decode \
+  --port 30003
+
+# router
+python -m sglang_router.launch_router \
+  --pd-disaggregation \
+  --prefill http://$PREFILL_HOST:30002 \
+  --decode http://$DECODE_HOST:30003 \
+  --port 8000
+```
@@ -42,6 +42,16 @@ SGLang's EP integrates diverse, highly efficient backends for different use case
      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>An extension of DeepEP for elastic inference, leveraging RDMA for high-performance data transfers.</td>
      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Elastic EP serving.</td>
    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>nixl</code></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><a href="https://github.com/ai-dynamo/nixl/tree/main/examples/device/ep">NIXL-EP</a>, an elastic EP communication library built on NVIDIA's <a href="https://github.com/ai-dynamo/nixl">NIXL</a> framework with native RDMA and NVLink support.</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Elastic EP serving with fault tolerance and dynamic scaling.</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>mori</code></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>MORI-EP, AMD's native all-to-all communication implementation optimized for ROCm.</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>AMD GPU deployments.</td>
+    </tr>
    <tr>
      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`flashinfer`</td>
      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Flashinfer implementation of all-to-all.</td>
@@ -55,9 +65,9 @@ SGLang's EP integrates diverse, highly efficient backends for different use case
  </tbody>
 </table>

-DeepEP and Mooncake backends support two modes for token dispatch: `normal` mode (optimized for prefill workloads with high throughput) and `low_latency` mode (optimized for decode workloads with low latency and CUDA Graph compatibility). Users are recommended to set `--deepep-mode auto` to enable automatic dispatch mode switching during runtime. Setting `--deepep-mode normal` or `--deepep-mode low_latency` is useful for debugging or development purposes.
+DeepEP and Mooncake backends support two modes for token dispatch: `normal` mode (optimized for prefill workloads with high throughput) and `low_latency` mode (optimized for decode workloads with low latency and CUDA Graph compatibility). MORI backend only supports `normal` mode now. NIXL-EP currently operates in low-latency mode with CUDA Graph support. Users are recommended to set `--deepep-mode auto` to enable automatic dispatch mode switching during runtime. Setting `--deepep-mode normal` or `--deepep-mode low_latency` is useful for debugging or development purposes.

-Currently, DeepEP and Mooncake only support cases where `ep_size = tp_size`. For hybrid EP and TP (i.e., `ep_size < tp_size`), only the `none` backend (All-Reduce or All-Gather-based dispatching) is supported.
+Currently, DeepEP, Mooncake, NIXL-EP, `ascend_fuseep` and MORI only support cases where `ep_size = tp_size`. For hybrid EP and TP (i.e., `ep_size < tp_size`), only the `none` backend (All-Reduce or All-Gather-based dispatching) is supported.

 ### Backends for MoE Computation

@@ -82,7 +92,7 @@ Currently, DeepEP and Mooncake only support cases where `ep_size = tp_size`. For
    </tr>
    <tr>
      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`triton`</td>
-      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Triton-based implementation for grouped GEMMs. To achieve higher performance, it's highly recommended to create [tuned configurations](https://github.com/sgl-project/sglang/blob/main/benchmark/kernels/fused_moe_triton/README).</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Triton-based implementation for grouped GEMMs. To achieve higher performance, it's highly recommended to create <a href="https://github.com/sgl-project/sglang/blob/main/benchmark/kernels/fused_moe_triton/README.md">tuned configurations</a>.</td>
      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Custom kernel development or scenarios requiring high extensibility with Torch compilation support.</td>
    </tr>
    <tr>
@@ -100,6 +110,11 @@ Currently, DeepEP and Mooncake only support cases where `ep_size = tp_size`. For
      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>FlashInfer integrated with TensorRT-LLM for accelerated MoE computations, supporting FP4 communication operators and high-performance GEMMs.</td>
      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Blackwell with TRT-LLM.</td>
    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>flashinfer_trtllm_routed</code></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>FlashInfer integrated with TensorRT-LLM for accelerated routed MoE computations, consuming SGLang-computed top-k expert assignments and weights.</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Blackwell with TRT-LLM.</td>
+    </tr>
    <tr>
      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`flashinfer_cutlass`</td>
      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>FlashInfer combined with CUTLASS for high-performance grouped GEMMs in MoE layers, handling FP4/FP8 quantization efficiently.</td>
@@ -242,7 +257,7 @@ For model like `nvidia/DeepSeek-R1-0528-NVFP4-v2`, the target model uses NVFP4 p

 ## Ascend NPU Guidance
 ### Guidance on SGLang configuration in Ascend NPU
- `--moe-a2a-backend` only supports deepep and ascend_fuseep backends,
+- `--moe-a2a-backend` only supports `deepep` and `ascend_fuseep` backends,

    - `deepep`: The mechanism is consistent with the above description.

@@ -252,12 +267,13 @@ For model like `nvidia/DeepSeek-R1-0528-NVFP4-v2`, the target model uses NVFP4 p

 - `--deepep-mode`:

-    - In PD mixed mode, please set `--deepep-mode` auto.
+    - In PD mixed mode, please set `--deepep-mode auto`.

-    - In PD Disaggregation Mode, prefill instance sets `--deepep-mode` normal, and decode instance sets `--deepep-mode` low_latency.
+    - In PD Disaggregation Mode, prefill instance sets `--deepep-mode normal`, and decode instance sets `--deepep-mode low_latency`.

 ### DeepEP Ascend Introduction
-DeepEP Ascend is the adapted version of the DeepEP communication library for Huawei Ascend NPUs, specifically designed for Mixture-of-Experts (MoE) model Expert Parallelism (EP). It supports the Ant-moving Function (Split the sequence length into rounds for streaming batch transmission) to optimize the buffer size occupied during collective communication in prefill stage, especially for long sequences.
+DeepEP Ascend is the adapted version of the DeepEP communication library for Huawei Ascend NPUs, specifically designed for Mixture-of-Experts (MoE) model Expert Parallelism (EP).
+It supports the Ant-moving Function (Split the sequence length into rounds for streaming batch transmission) to optimize the buffer size occupied during collective communication in prefill stage, especially for long sequences.

 Ant-moving Function can be enabled for both the dispatch and combine phases via the following environment variables:

@@ -42,6 +42,23 @@ Notes:
 - `page_first`: Only compatible with `kernel` I/O backend, automatically switches to `layer_first` with `direct` backend
 - `page_first_direct`: Specifically designed for `direct` I/O backend with optimized memory organization

+### Heterogeneous TP Support (GQA/MHA models)
+
+HiCache storage supports cross-cluster KV reuse when different deployments use different TP sizes (for example, `tp=4` and `tp=8`) and share the same storage backend namespace.
+
+Use `tp_lcm_size` in `--hicache-storage-backend-extra-config`:
+
+```bash Command
+# Example: heterogeneous TP = {4, 8}, so lcm = 8
+--hicache-storage-backend-extra-config '{"tp_lcm_size": 8}'
+```
+
+Guidelines:
+
+- Set `tp_lcm_size` to the least common multiple (LCM) of all TP sizes that will share the same HiCache storage.
+- For MHA models with Mooncake and `page_head` layout, HiCache will split head shards based on `tp_lcm_size` to make keys reusable across heterogeneous TP deployments.
+- If all clusters use the same TP size, this option is not needed.
+
 ### Prefetch Policies

 ```bash Command
@@ -108,7 +125,7 @@ python3 -m sglang.launch_server \

 ### Deployment with HF3FS

-Here is an example of deploying DeepSeek-R1 with HiCache-HF3FS. For more details, see the [HF3FS Documentation](https://github.com/sgl-project/sglang/tree/main/python/sglang/srt/mem_cache/storage/hf3fs/docs).
+Here is an example of deploying DeepSeek-R1 with HiCache-HF3FS. For more details, see the [HF3FS Documentation](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/mem_cache/storage/hf3fs/docs/README.md).

 ```bash Command
 python3 -m sglang.launch_server \
@@ -133,7 +150,7 @@ python3 -m sglang.launch_server \

 ### Deployment with Mooncake

-Here is an example of deploying Qwen3-235B-A22B-Instruct-2507 with Mooncake. For more details, see the [Mooncake Documentation](https://github.com/sgl-project/sglang/tree/main/python/sglang/srt/mem_cache/storage/mooncake_store).
+Here is an example of deploying Qwen3-235B-A22B-Instruct-2507 with Mooncake. For more details, see the [Mooncake Documentation](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/mem_cache/storage/mooncake_store/README.md).

 ```bash Command
 # Set Mooncake environment variables
@@ -21,8 +21,8 @@ The control path is:

 1. **HTTP Server** (`python/sglang/srt/entrypoints/http_server.py`)
   - Exposes `PUT /hicache/storage-backend`, `DELETE /hicache/storage-backend`, `GET /hicache/storage-backend`
-2. **TokenizerManager** (`python/sglang/srt/managers/tokenizer_communicator_mixin.py`)
-   - Sends the request to the Scheduler via `_Communicator`
+2. **TokenizerManager** (`python/sglang/srt/managers/tokenizer_control_mixin.py`)
+   - Sends the request to the Scheduler via `FanOutCommunicator`
 3. **Scheduler** (`python/sglang/srt/managers/scheduler.py`)
   - Performs a **strict idle check**
   - Calls `tree_cache.attach_storage_backend(...)` / `detach_storage_backend(...)`
@@ -36,11 +36,11 @@ The control path is:
 ***
 ## 2. Idle-state requirement (strict)

-The Scheduler uses a stricter `_is_idle_for_hicache_storage_op()`:
+The Scheduler uses `is_fully_idle()` which checks:

- `_is_no_request()` is true (covers running/overlap/pp/disagg and other active states)
- `waiting_queue` is empty
- `grammar_queue` is empty (if the grammar backend is enabled)
+- No running batches (including chunked prefill, overlap, pipeline-parallel, and disaggregation paths)
+- No waiting requests in any queue (waiting, grammar, disagg bootstrap/prealloc/transfer/inflight)
+- No DLLM staging requests

 If the condition is not met, attach/detach returns an error like:

@@ -0,0 +1,187 @@
+---
+title: "HiSparse: Hierarchical Sparse Attention"
+metatags:
+    description: "Use HiSparse hierarchical sparse attention to reduce decode GPU KV memory with CPU pinned host storage and PD disaggregation."
+---
+
+HiSparse reduces per-request GPU memory consumption during the decode phase by maintaining only a small "hot" KV buffer on GPU while keeping complete KV data in CPU pinned memory. Combined with PD disaggregation, it enables significantly higher decode concurrency.
+
+> **Prerequisites**: HiSparse only works with models that use **DeepSeek Sparse Attention (DSA)**  architectures (e.g., DeepSeek-V3.2, GLM-5). These models natively select a subset of tokens for attention, making it possible to keep only the top-k KV on GPU while storing the full KV in host memory — without accuracy loss.  Additionally, HiSparse currently requires **PD disaggregation mode** and is enabled on the **decode instance** only.
+
+## Why HiSparse?
+
+In long-context LLM inference, each decoding request holds a full-length KV cache on GPU, limiting the number of concurrent requests a decode instance can serve. HiSparse addresses this by:
+
+- **Reducing GPU memory per request**: Each request occupies only a fixed-size device buffer (e.g., 4KB tokens) instead of the full sequence length.
+- **On-demand swap-in**: A CUDA kernel dynamically loads the top-k most relevant KV entries from host memory based on attention scores.
+- **Transparent to prefill**: HiSparse is entirely a decode-side optimization; the prefill instance requires no changes.
+
+## Design Overview
+
+### Decode Workflow
+
+Each decode step follows this flow:
+
+1. **Forward decode** — generate the next token
+2. **Top-k selection** — select the most relevant token positions via attention scores
+3. **Swap-in** — the CUDA kernel loads top-k KV entries from host to device buffer:
+   - *Short sequences* (`seq_len ≤ device_buffer_size`): fast path, all KV already in buffer
+   - *Long sequences*: hit detection → LRU reordering → miss handling (host → device copy)
+4. **Decode attention** — compute attention using the top-k device locations
+5. **Eager backup** — asynchronously copy the previous token's KV from device to host
+
+### PD Disaggregation Integration (Direct-to-Host)
+
+In PD disaggregation mode, the prefill instance transfers KV cache directly into the decode instance's host pool via RDMA, bypassing the GPU entirely on the decode side. This eliminates the transient GPU memory spike during KV transfer and removes the staging DMA step.
+
+```
+Prefill GPU  ──RDMA──▶  Decode Host Pool (CPU pinned memory)
+                              │
+                              ▼
+                     alloc device buffer (4KB)
+                              │
+                              ▼
+                     swap-in kernel (on-demand top-k)
+```
+
+## Server Arguments
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <colgroup>
+    <col style={{width: "33.33%"}} />
+    <col style={{width: "33.33%"}} />
+    <col style={{width: "33.33%"}} />
+  </colgroup>
+  <thead>
+    <tr>
+      <th>Argument</th>
+      <th>Type / Default</th>
+      <th>Description</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td><code>--enable-hisparse</code></td>
+      <td>flag; default: disabled</td>
+      <td>Enable HiSparse on the decode instance</td>
+    </tr>
+    <tr>
+      <td><code>--hisparse-config</code></td>
+      <td>JSON string</td>
+      <td>Configuration for HiSparse (see below)</td>
+    </tr>
+  </tbody>
+</table>
+
+### HiSparse Config Parameters
+
+Pass as a JSON string via `--hisparse-config`:
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <colgroup>
+    <col style={{width: "33.33%"}} />
+    <col style={{width: "33.33%"}} />
+    <col style={{width: "33.33%"}} />
+  </colgroup>
+  <thead>
+    <tr>
+      <th>Parameter</th>
+      <th>Type / Default</th>
+      <th>Description</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td><code>top_k</code></td>
+      <td>int</td>
+      <td>Number of topk entries</td>
+    </tr>
+    <tr>
+      <td><code>device_buffer_size</code></td>
+      <td>int</td>
+      <td>Number of token slots in the per-request GPU device buffer</td>
+    </tr>
+    <tr>
+      <td><code>host_to_device_ratio</code></td>
+      <td>int</td>
+      <td>Ratio of logical pool size to device pool size, determining host memory capacity</td>
+    </tr>
+  </tbody>
+</table>
+
+Example: `--hisparse-config='{"top_k": 2048, "device_buffer_size": 6144, "host_to_device_ratio": 10}'`
+
+## Deployment
+
+HiSparse currently requires **PD disaggregation mode** and is enabled only on the **decode instance**.
+
+### Prefill Instance
+
+```bash Command
+python3 -m sglang.launch_server \
+    --model-path /path/to/model \
+    --trust-remote-code \
+    --port 8000 --host 0.0.0.0 \
+    --context-length 81920 \
+    --chunked-prefill-size 65536 \
+    --tp-size 8 --dp-size 8 --enable-dp-attention \
+    --mem-fraction-static 0.85 \
+    --disaggregation-mode prefill \
+    --disaggregation-ib-device mlx5_0,mlx5_1,mlx5_2,mlx5_3 \
+    --nnodes 1 --node-rank 0
+```
+
+### Decode Instance (with HiSparse)
+
+```bash Command
+python3 -m sglang.launch_server \
+    --model-path /path/to/model \
+    --trust-remote-code \
+    --port 8000 --host 0.0.0.0 \
+    --context-length 81920 \
+    --tp-size 8 --dp-size 8 --enable-dp-attention \
+    --mem-fraction-static 0.85 \
+    --kv-cache-dtype bfloat16 \
+    --nsa-decode-backend flashmla_sparse \
+    --disaggregation-mode decode \
+    --disaggregation-ib-device mlx5_0,mlx5_1,mlx5_2,mlx5_3 \
+    --dist-init-addr 127.0.0.1:5757 \
+    --nnodes 1 --node-rank 0 \
+    --enable-hisparse \
+    --hisparse-config='{"top_k": 2048, "device_buffer_size": 6144, "host_to_device_ratio": 10}'
+```
+
+### Benchmark
+
+```bash Command
+python3 -m sglang.bench_serving \
+    --backend sglang \
+    --dataset-path /path/to/ShareGPT_V3_unfiltered_cleaned_split.json \
+    --dataset-name random \
+    --random-input 40000 \
+    --random-output 20000 \
+    --num-prompts 200 \
+    --max-concurrency 200 \
+    --request-rate 40 \
+    --random-range-ratio 1.0 \
+    --host 127.0.0.1 \
+    --port 20000 \
+    --model /path/to/model \
+    --flush-cache \
+```
+
+### Key Notes
+
+- The prefill instance does not need `--enable-hisparse`; it is unaware of HiSparse.
+- On the decode instance, the following flags are **required** for HiSparse:
+  - `--kv-cache-dtype bfloat16` — currently only bfloat16 KV cache is supported (more dtypes planned).
+  - `--nsa-decode-backend flashmla_sparse` — currently only `flashmla_sparse` backend is supported.
+  - `--enable-hisparse` — enables HiSparse.
+  - `--hisparse-config` — HiSparse configuration (top_k, device_buffer_size, host_to_device_ratio).
+    - `host_to_device_ratio` should be configured based on the host machine's available memory. For example:
+      - **~1 TB** host memory → `host_to_device_ratio: 5`
+      - **~2 TB** host memory → `host_to_device_ratio: 10`
+
+## Acknowledgments
+
+We would like to thank the SGLang team and community for the implementation and generous support, especially Zhiqiang Xie, Zhangheng Huang, Tingwei Huang, Shangming Cai, Teng Ma, and many others. We also thank the Alibaba Cloud TairKVCache team and the AntGroup SCT Inference team for their valuable contributions.
@@ -102,7 +102,7 @@
    "\"\"\"\n",
    ")\n",
    "\n",
-    "wait_for_server(f\"http://localhost:{port}\")"
+    "wait_for_server(f\"http://localhost:{port}\", process=server_process)"
   ]
  },
  {
@@ -155,12 +155,12 @@
    "python3 -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \\\n",
    "    --enable-lora \\\n",
    "    --lora-paths lora0=algoprog/fact-generation-llama-3.1-8b-instruct-lora \\\n",
-    "    lora1=Nutanix/Meta-Llama-3.1-8B-Instruct_lora_4_alpha_16 \\\n",
+    "    lora1=Nutanix/Meta-Llama-3.1-8B-Instruct_SFT_lora_4_alpha_16_humaneval_raw_json \\\n",
    "    --max-loras-per-batch 2 \\\n",
    "    --log-level warning \\\n",
    "\"\"\")\n",
    "\n",
-    "wait_for_server(f\"http://localhost:{port}\")"
+    "wait_for_server(f\"http://localhost:{port}\", process=server_process)"
   ]
  },
  {
@@ -218,7 +218,7 @@
   "metadata": {},
   "outputs": [],
   "source": [
-    "lora0 = \"Nutanix/Meta-Llama-3.1-8B-Instruct_lora_4_alpha_16\"  # rank - 4, target modules - q_proj, k_proj, v_proj, o_proj, gate_proj\n",
+    "lora0 = \"Nutanix/Meta-Llama-3.1-8B-Instruct_SFT_lora_4_alpha_16_humaneval_raw_json\"  # rank - 4, target modules - q_proj, k_proj, v_proj, o_proj, gate_proj\n",
    "lora1 = \"algoprog/fact-generation-llama-3.1-8b-instruct-lora\"  # rank - 64, target modules - q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj\n",
    "lora0_new = \"philschmid/code-llama-3-1-8b-text-to-sql-lora\"  # rank - 256, target modules - q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj\n",
    "\n",
@@ -236,7 +236,7 @@
    "    \"\"\")\n",
    "\n",
    "url = f\"http://127.0.0.1:{port}\"\n",
-    "wait_for_server(url)"
+    "wait_for_server(url, process=server_process)"
   ]
  },
  {
@@ -404,7 +404,7 @@
   "source": [
    "### OpenAI-compatible API usage\n",
    "\n",
-    "You can use LoRA adapters via the OpenAI-compatible APIs by specifying the adapter in the `model` field using the `base-model:adapter-name` syntax (for example, `qwen/qwen2.5-0.5b-instruct:adapter_a`). For more details and examples, see the “Using LoRA Adapters” section in the OpenAI API documentation: [openai_api_completions](../basic_usage/openai_api_completions).\n"
+    "You can use LoRA adapters via the OpenAI-compatible APIs by specifying the adapter in the `model` field using the `base-model:adapter-name` syntax (for example, `qwen/qwen2.5-0.5b-instruct:adapter_a`). For more details and examples, see the “Using LoRA Adapters” section in the OpenAI API documentation: [openai_api_completions.ipynb](../basic_usage/openai_api_completions.ipynb).\n"
   ]
  },
  {
@@ -439,7 +439,7 @@
    "    --max-lora-rank 256 \\\n",
    "    --lora-target-modules all \\\n",
    "    --lora-paths \\\n",
-    "        {\"lora_name\":\"lora0\",\"lora_path\":\"Nutanix/Meta-Llama-3.1-8B-Instruct_lora_4_alpha_16\",\"pinned\":true} \\\n",
+    "        {\"lora_name\":\"lora0\",\"lora_path\":\"Nutanix/Meta-Llama-3.1-8B-Instruct_SFT_lora_4_alpha_16_humaneval_raw_json\",\"pinned\":true} \\\n",
    "        {\"lora_name\":\"lora1\",\"lora_path\":\"algoprog/fact-generation-llama-3.1-8b-instruct-lora\"} \\\n",
    "        lora2=philschmid/code-llama-3-1-8b-text-to-sql-lora\n",
    "    --log-level warning\n",
@@ -447,7 +447,7 @@
    "\n",
    "\n",
    "url = f\"http://127.0.0.1:{port}\"\n",
-    "wait_for_server(url)"
+    "wait_for_server(url, process=server_process)"
   ]
  },
  {
@@ -581,7 +581,7 @@
   "metadata": {},
   "outputs": [],
   "source": [
-    "lora0 = \"Nutanix/Meta-Llama-3.1-8B-Instruct_lora_4_alpha_16\"\n",
+    "lora0 = \"Nutanix/Meta-Llama-3.1-8B-Instruct_SFT_lora_4_alpha_16_humaneval_raw_json\"\n",
    "lora1 = \"algoprog/fact-generation-llama-3.1-8b-instruct-lora\"\n",
    "lora2 = \"philschmid/code-llama-3-1-8b-text-to-sql-lora\"\n",
    "\n",
@@ -591,7 +591,7 @@
    "    --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \\\n",
    "    --enable-lora \\\n",
    "    --enable-lora-overlap-loading \\\n",
-    "    --lora-paths lora0=Nutanix/Meta-Llama-3.1-8B-Instruct_lora_4_alpha_16 \\\n",
+    "    --lora-paths lora0=Nutanix/Meta-Llama-3.1-8B-Instruct_SFT_lora_4_alpha_16_humaneval_raw_json \\\n",
    "    lora1=algoprog/fact-generation-llama-3.1-8b-instruct-lora \\\n",
    "    lora2=philschmid/code-llama-3-1-8b-text-to-sql-lora \\\n",
    "    --max-lora-rank 256 \\\n",
@@ -600,7 +600,7 @@
    "    \"\"\")\n",
    "\n",
    "url = f\"http://127.0.0.1:{port}\"\n",
-    "wait_for_server(url)"
+    "wait_for_server(url, process=server_process)"
   ]
  },
  {
@@ -43,7 +43,7 @@ From client side, the user needs to provide a list of strings as input batch, an

 **Note:** SGLang supports LoRA adapters through two APIs:

-1. **OpenAI-Compatible API** (`/v1/chat/completions`, `/v1/completions`): Use the `model:adapter-name` syntax. See [OpenAI API with LoRA](../basic_usage/openai_api_completions.ipynb#Using-LoRA-Adapters) for examples.
+1. **OpenAI-Compatible API** (`/v1/chat/completions`, `/v1/completions`): Use the `model:adapter-name` syntax. See [OpenAI API with LoRA](../basic_usage/openai_api_completions#using-lora-adapters) for examples.

 2. **Native API** (`/generate`): Pass `lora_path` in the request body (shown below).

@@ -108,7 +108,7 @@ server_process, port = launch_server_cmd(
 python3 -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \
    --enable-lora \
    --lora-paths lora0=algoprog/fact-generation-llama-3.1-8b-instruct-lora \
-    lora1=Nutanix/Meta-Llama-3.1-8B-Instruct_lora_4_alpha_16 \
+    lora1=Nutanix/Meta-Llama-3.1-8B-Instruct_SFT_lora_4_alpha_16_humaneval_raw_json \
    --max-loras-per-batch 2 \
    --log-level warning \
 """
@@ -152,7 +152,7 @@ When using dynamic LoRA loading, it's recommended to explicitly specify both `--


 ```python Example
-lora0 = "Nutanix/Meta-Llama-3.1-8B-Instruct_lora_4_alpha_16"  # rank - 4, target modules - q_proj, k_proj, v_proj, o_proj, gate_proj
+lora0 = "Nutanix/Meta-Llama-3.1-8B-Instruct_SFT_lora_4_alpha_16_humaneval_raw_json"  # rank - 4, target modules - q_proj, k_proj, v_proj, o_proj, gate_proj
 lora1 = "algoprog/fact-generation-llama-3.1-8b-instruct-lora"  # rank - 64, target modules - q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
 lora0_new = "philschmid/code-llama-3-1-8b-text-to-sql-lora"  # rank - 256, target modules - q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj

@@ -317,7 +317,7 @@ server_process, port = launch_server_cmd(
    --max-lora-rank 256 \
    --lora-target-modules all \
    --lora-paths \
-        {"lora_name":"lora0","lora_path":"Nutanix/Meta-Llama-3.1-8B-Instruct_lora_4_alpha_16","pinned":true} \
+        {"lora_name":"lora0","lora_path":"Nutanix/Meta-Llama-3.1-8B-Instruct_SFT_lora_4_alpha_16_humaneval_raw_json","pinned":true} \
        {"lora_name":"lora1","lora_path":"algoprog/fact-generation-llama-3.1-8b-instruct-lora"} \
        lora2=philschmid/code-llama-3-1-8b-text-to-sql-lora
    --log-level warning
@@ -418,7 +418,7 @@ By using the `--enable-lora-overlap-loading` server argument, the SGLang engine


 ```python Example
-lora0 = "Nutanix/Meta-Llama-3.1-8B-Instruct_lora_4_alpha_16"
+lora0 = "Nutanix/Meta-Llama-3.1-8B-Instruct_SFT_lora_4_alpha_16_humaneval_raw_json"
 lora1 = "algoprog/fact-generation-llama-3.1-8b-instruct-lora"
 lora2 = "philschmid/code-llama-3-1-8b-text-to-sql-lora"

@@ -429,7 +429,7 @@ server_process, port = launch_server_cmd(
    --model-path meta-llama/Meta-Llama-3.1-8B-Instruct \
    --enable-lora \
    --enable-lora-overlap-loading \
-    --lora-paths lora0=Nutanix/Meta-Llama-3.1-8B-Instruct_lora_4_alpha_16 \
+    --lora-paths lora0=Nutanix/Meta-Llama-3.1-8B-Instruct_SFT_lora_4_alpha_16_humaneval_raw_json \
    lora1=algoprog/fact-generation-llama-3.1-8b-instruct-lora \
    lora2=philschmid/code-llama-3-1-8b-text-to-sql-lora \
    --max-lora-rank 256 \
@@ -26,7 +26,7 @@ When you need to profile prefill or decode workers in PD disaggregation mode, pl

 ## Router Integration

-For deploying PD disaggregation at scale with load balancing and fault tolerance, SGLang provides a router. The router can distribute requests between prefill and decode instances using various routing policies. For detailed information on setting up routing with PD disaggregation, including configuration options and deployment patterns, see the [SGLang Model Gateway (former Router)](../advanced_features/sgl_model_gateway.md#prefill-decode-disaggregation).
+For deploying PD disaggregation at scale with load balancing and fault tolerance, SGLang provides a router. The router can distribute requests between prefill and decode instances using various routing policies. For detailed information on setting up routing with PD disaggregation, including configuration options and deployment patterns, see the [SGLang Model Gateway (former Router)](./sgl_model_gateway#prefill-decode-disaggregation).


 ## Mooncake
@@ -132,11 +132,13 @@ PD Disaggregation with Mooncake supports the following environment variables for
 #### NVLink Transport Configuration
 To enable NVLink transport for KV cache transfers with the mooncake backend (recommended for NVL72 deployments), set the following environment variables. Note that auxiliary data transfer will still use TCP as a temporary workaround.

-```bash
-export SGLANG_MOONCAKE_CUSTOM_MEM_POOL=True
+```bash Command
+export SGLANG_MOONCAKE_CUSTOM_MEM_POOL=NVLINK
 export MC_FORCE_MNNVL=True
 ```

+The `SGLANG_MOONCAKE_CUSTOM_MEM_POOL` environment variable enables the custom memory pool. Supported values are `NVLINK` (or `True`), `BAREX`, and `INTRA_NODE_NVLINK`.
+
 #### Prefill Server Configuration
 <table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
  <colgroup>
@@ -155,11 +157,11 @@ export MC_FORCE_MNNVL=True
    <tr>
      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**`SGLANG_DISAGGREGATION_THREAD_POOL_SIZE`**</td>
      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Controls the total number of worker threads for KVCache transfer operations per TP rank</td>
-      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>A dynamic value calculated by `int(0.75 * os.cpu_count()) // 8)`, which is limited to be larger than 4 and less than 12 to ensure efficiency and prevent thread race conditions</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>A dynamic value calculated by <code>int(0.75 * os.cpu_count()) // 8)</code>, which is limited to be larger than 4 and less than 12 to ensure efficiency and prevent thread race conditions</td>
    </tr>
    <tr>
      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**`SGLANG_DISAGGREGATION_QUEUE_SIZE`**</td>
-      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Sets the number of parallel transfer queues. KVCache transfer requests from multiple decode instances will be sharded into these queues so that they can share the threads and the transfer bandwidth at the same time. If it is set to `1`, then we transfer requests one by one according to fcfs strategy</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Sets the number of parallel transfer queues. KVCache transfer requests from multiple decode instances will be sharded into these queues so that they can share the threads and the transfer bandwidth at the same time. If it is set to <code>1</code>, then we transfer requests one by one according to fcfs strategy</td>
      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`4`</td>
    </tr>
    <tr>
@@ -167,7 +169,12 @@ export MC_FORCE_MNNVL=True
      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Timeout (seconds) for receiving destination KV indices during request initialization</td>
      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`300`</td>
    </tr>
-  </tbody>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong><code>SGLANG_DISAGGREGATION_BOOTSTRAP_ENTRY_CLEANUP_INTERVAL</code></strong></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Interval (seconds) between cleanups of bootstrap entries</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><code>120</code></td>
+    </tr>
+</tbody>
 </table>

 If a greater mean TTFT is acceptable, you can `export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600` (10 minutes) to relax the timeout condition.
@@ -209,6 +216,84 @@ Please be aware that this setting will cause prefill instances to take a longer
 If a greater mean TTFT is acceptable, you can `export SGLANG_DISAGGREGATION_WAITING_TIMEOUT=600` (10 minutes) to relax the timeout condition.


+## Heterogeneous TP with GPU Staging Buffer
+
+When prefill and decode use different tensor parallelism (TP) sizes (e.g., prefill TP=4, decode DP attention with TP=1), the KV cache memory layout differs between the two sides. The **GPU staging buffer** solves this by gathering KV head slices into a contiguous buffer on the prefill side, performing bulk RDMA transfer, then scattering into the correct KV cache pages on the decode side. This provides **2–5x throughput improvement** over the default per-token slice approach at high concurrency and matches homogeneous TP baselines within ~5%.
+
+Enable the staging buffer when prefill and decode use **different TP sizes** with the **Mooncake** transfer backend. When both sides use the same TP size, staging is automatically bypassed even if enabled.
+
+> **Note:** The staging buffer is designed for non-MLA models (e.g. GQA, MHA). MLA models (e.g. DeepSeek-V2/V3) should not enable this flag.
+
+### Environment Variables
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <colgroup>
+    <col style={{width: "30%"}} />
+    <col style={{width: "50%"}} />
+    <col style={{width: "20%"}} />
+  </colgroup>
+  <thead>
+    <tr style={{borderBottom: "2px solid #d55816"}}>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Variable</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Description</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Default</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong><code>SGLANG_DISAGG_STAGING_BUFFER</code></strong></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Enable GPU staging buffer for heterogeneous TP KV transfer</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><code>False</code></td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong><code>SGLANG_DISAGG_STAGING_BUFFER_SIZE_MB</code></strong></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Prefill-side per-worker staging buffer size in MB</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><code>64</code></td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong><code>SGLANG_DISAGG_STAGING_POOL_SIZE_MB</code></strong></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Decode-side ring buffer pool total size in MB</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><code>4096</code></td>
+    </tr>
+  </tbody>
+</table>
+
+### Usage Example
+
+```bash Command
+# Set staging buffer environment variables on BOTH prefill and decode
+export SGLANG_DISAGG_STAGING_BUFFER=1
+export SGLANG_DISAGG_STAGING_BUFFER_SIZE_MB=64
+export SGLANG_DISAGG_STAGING_POOL_SIZE_MB=4096
+
+# Prefill with TP=4
+python -m sglang.launch_server \
+  --model-path $MODEL_PATH \
+  --disaggregation-mode prefill \
+  --port 30000 \
+  --tp 4 \
+  --trust-remote-code \
+  --disaggregation-ib-device mlx5_1,mlx5_2
+
+# Decode with TP=1 (or DP attention with effective attention TP=1)
+python -m sglang.launch_server \
+  --model-path $MODEL_PATH \
+  --disaggregation-mode decode \
+  --port 30001 \
+  --tp 4 \
+  --dp 4 \
+  --enable-dp-attention \
+  --trust-remote-code \
+  --disaggregation-ib-device mlx5_3,mlx5_4
+
+# Router
+python -m sglang_router.launch_router \
+  --pd-disaggregation \
+  --prefill http://127.0.0.1:30000 \
+  --decode http://127.0.0.1:30001 \
+  --host 0.0.0.0 --port 8000
+```
+
 ## NIXL
 ### Requirements

@@ -343,8 +428,8 @@ python -m sglang.launch_server \

 Use ascend backend with [memfabric_hybrid](https://gitcode.com/Ascend/memfabric_hybrid) and ASCEND_MF_STORE_URL being set

-```bash
-pip install memfabric-hybrid==1.0.5
+```bash Command
+pip install memfabric-hybrid==1.0.0
 export ASCEND_MF_STORE_URL="tcp://xxx.xx.xxx.xxx:xxxx"
 ```
 Use mooncake backend, more details can be found in mooncake section.
@@ -20,11 +20,276 @@ or [NeuralMagic](https://huggingface.co/collections/neuralmagic) collections on
 popular quality validated quantized models. Quantized models must be validated via benchmarks post-quantization
 to guard against abnormal quantization loss regressions.

+## Platform Compatibility
+
+The following table summarizes quantization method support across NVIDIA and AMD GPUs, Ascend NPUs.
+
+<table>
+  <thead>
+    <tr>
+      <th>Method</th>
+      <th>NVIDIA GPUs</th>
+      <th>AMD GPUs (MI300X/MI325X/MI350X)</th>
+      <th>Ascend NPUs (A2/A3)</th>
+      <th>Notes</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td><code>fp8</code></td>
+      <td>Yes</td>
+      <td>Yes</td>
+      <td>WIP</td>
+      <td>Aiter or Triton backend on AMD</td>
+    </tr>
+    <tr>
+      <td><code>mxfp4</code></td>
+      <td>Yes</td>
+      <td>Yes</td>
+      <td>WIP</td>
+      <td>Requires CDNA3/CDNA4 with MXFP support; uses Aiter</td>
+    </tr>
+    <tr>
+      <td><code>blockwise_int8</code></td>
+      <td>Yes</td>
+      <td>Yes</td>
+      <td>No</td>
+      <td>Triton-based, works on both platforms</td>
+    </tr>
+    <tr>
+      <td><code>w8a8_int8</code></td>
+      <td>Yes</td>
+      <td>Yes</td>
+      <td>No</td>
+      <td></td>
+    </tr>
+    <tr>
+      <td><code>w8a8_fp8</code></td>
+      <td>Yes</td>
+      <td>Yes</td>
+      <td>No</td>
+      <td>Aiter or Triton FP8 on AMD</td>
+    </tr>
+    <tr>
+      <td><code>awq</code></td>
+      <td>Yes</td>
+      <td>Yes</td>
+      <td>Yes</td>
+      <td>Uses Triton dequantize on AMD (vs. optimized CUDA kernels on NVIDIA). Uses CANN kernels on Ascend</td>
+    </tr>
+    <tr>
+      <td><code>gptq</code></td>
+      <td>Yes</td>
+      <td>Yes</td>
+      <td>Yes</td>
+      <td>Uses Triton or vLLM kernels on AMD. Uses CANN kernels on Ascend</td>
+    </tr>
+    <tr>
+      <td><code>compressed-tensors</code></td>
+      <td>Yes</td>
+      <td>Yes</td>
+      <td>Partial</td>
+      <td>Aiter paths for FP8/MoE on AMD. Uses CANN kernels on Ascend, <code>FP8</code> not supported yet</td>
+    </tr>
+    <tr>
+      <td><code>quark</code></td>
+      <td>Yes</td>
+      <td>Yes</td>
+      <td>No</td>
+      <td>AMD Quark quantization; Aiter GEMM paths on AMD</td>
+    </tr>
+    <tr>
+      <td><code>auto-round</code></td>
+      <td>Yes</td>
+      <td>Yes</td>
+      <td>Partial</td>
+      <td>Platform-agnostic (Intel auto-round). Uses CANN kernels on Ascend</td>
+    </tr>
+    <tr>
+      <td><code>quark_int4fp8_moe</code></td>
+      <td>No</td>
+      <td>Yes</td>
+      <td>No</td>
+      <td>AMD-only; online INT4-to-FP8 MoE quantization (CDNA3/CDNA4)</td>
+    </tr>
+    <tr>
+      <td><code>awq_marlin</code></td>
+      <td>Yes</td>
+      <td>No</td>
+      <td>No</td>
+      <td>Marlin kernels are CUDA-only</td>
+    </tr>
+    <tr>
+      <td><code>gptq_marlin</code></td>
+      <td>Yes</td>
+      <td>No</td>
+      <td>No</td>
+      <td>Marlin kernels are CUDA-only</td>
+    </tr>
+    <tr>
+      <td><code>gguf</code></td>
+      <td>Yes</td>
+      <td>No</td>
+      <td>WIP</td>
+      <td>CUDA-only kernels in sgl-kernel</td>
+    </tr>
+    <tr>
+      <td><code>modelopt</code> / <code>modelopt_fp8</code></td>
+      <td>Yes (Hopper/SM90+)</td>
+      <td>No</td>
+      <td>No</td>
+      <td><a href="https://github.com/NVIDIA/Model-Optimizer">NVIDIA ModelOpt</a>; requires NVIDIA hardware</td>
+    </tr>
+    <tr>
+      <td><code>modelopt_fp4</code></td>
+      <td>Yes (Blackwell/SM100+)</td>
+      <td>No</td>
+      <td>No</td>
+      <td><a href="https://github.com/NVIDIA/Model-Optimizer">NVIDIA ModelOpt</a>; native FP4 on Blackwell (B200, GB200)</td>
+    </tr>
+    <tr>
+      <td><code>petit_nvfp4</code></td>
+      <td>No</td>
+      <td>Yes (MI250/MI300X/MI325X)</td>
+      <td>No</td>
+      <td>Enables NVFP4 on ROCm via <a href="https://github.com/causalflow-ai/petit-kernel">Petit</a>; use <code>modelopt_fp4</code> on NVIDIA Blackwell. Auto-selected when loading NVFP4 models on AMD. See <a href="https://lmsys.org/blog/2025-09-21-petit-amdgpu/">LMSYS blog</a> and <a href="https://rocm.blogs.amd.com/artificial-intelligence/fp4-mixed-precision/README.html">AMD ROCm blog</a>.</td>
+    </tr>
+    <tr>
+      <td><code>bitsandbytes</code></td>
+      <td>Yes</td>
+      <td>Experimental</td>
+      <td>No</td>
+      <td>Depends on bitsandbytes ROCm support</td>
+    </tr>
+    <tr>
+      <td><code>torchao</code> (<code>int4wo</code>, etc.)</td>
+      <td>Yes</td>
+      <td>Partial</td>
+      <td>No</td>
+      <td><code>int4wo</code> not supported on AMD; other methods may work</td>
+    </tr>
+    <tr>
+      <td><code>modelslim</code></td>
+      <td>No</td>
+      <td>No</td>
+      <td>Yes</td>
+      <td>Ascend quantization; Uses CANN kernels</td>
+    </tr>
+  </tbody>
+</table>
+
+On AMD, several of these methods use [Aiter](https://github.com/ROCm/aiter) for acceleration -- set `SGLANG_USE_AITER=1` where noted. See [AMD GPU setup](../hardware-platforms/amd_gpu) for installation and configuration details.
+
+On Ascend, various layers quantization configurations are supported, see [Ascend NPU quantization](../hardware-platforms/ascend-npus/ascend_npu_quantization) for details.
+
+## GEMM Backends for FP4/FP8 Quantization
+
+<Note>
+Backend selection is supported only for **blockwise FP8** and **NVFP4** GEMM. When running FP8 or FP4 quantized models, you can select the GEMM backend via `--fp8-gemm-backend` and `--fp4-gemm-backend`.
+</Note>
+
+### `--fp8-gemm-backend` (Blockwise FP8 GEMM)
+
+<table>
+  <thead>
+    <tr>
+      <th>Backend</th>
+      <th>Hardware</th>
+      <th>Description</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td><code>auto</code></td>
+      <td>All</td>
+      <td>Auto-selects based on hardware</td>
+    </tr>
+    <tr>
+      <td><code>deep_gemm</code></td>
+      <td>SM90, SM100</td>
+      <td>JIT-compiled; enabled when DeepGEMM is installed</td>
+    </tr>
+    <tr>
+      <td><code>flashinfer_trtllm</code></td>
+      <td>SM100</td>
+      <td>FlashInfer TensorRT-LLM backend; optimal for low-latency</td>
+    </tr>
+    <tr>
+      <td><code>flashinfer_cutlass</code></td>
+      <td>SM100/120</td>
+      <td>FlashInfer CUTLASS groupwise FP8 GEMM</td>
+    </tr>
+    <tr>
+      <td><code>flashinfer_deepgemm</code></td>
+      <td>SM90</td>
+      <td>Uses swapAB optimization for small M dimensions in decoding</td>
+    </tr>
+    <tr>
+      <td><code>cutlass</code></td>
+      <td>SM90, SM100/120</td>
+      <td>sgl-kernel CUTLASS</td>
+    </tr>
+    <tr>
+      <td><code>triton</code></td>
+      <td>All</td>
+      <td>Fallback; widely compatible</td>
+    </tr>
+    <tr>
+      <td><code>aiter</code></td>
+      <td>ROCm</td>
+      <td>AMD AITER backend</td>
+    </tr>
+  </tbody>
+</table>
+
+**`auto` selection order:** 1) DeepGEMM (SM90/SM100, installed); 2) FlashInfer TRTLLM (SM100, FlashInfer available); 3) CUTLASS (SM90/SM100/120); 4) AITER (AMD); 5) Triton. **Exception:** SM120 always resolves to Triton.
+
+### `--fp4-gemm-backend` (NVFP4 GEMM)
+
+<table>
+  <thead>
+    <tr>
+      <th>Backend</th>
+      <th>Hardware</th>
+      <th>Description</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td><code>auto</code></td>
+      <td>SM100/120</td>
+      <td>Auto-selects: <code>flashinfer_cudnn</code> on SM120; <code>flashinfer_cutlass</code> on SM100</td>
+    </tr>
+    <tr>
+      <td><code>cutlass</code></td>
+      <td>SM100/120</td>
+      <td>SGLang CUTLASS kernel</td>
+    </tr>
+    <tr>
+      <td><code>flashinfer_cutlass</code></td>
+      <td>SM100/120</td>
+      <td>FlashInfer CUTLASS backend</td>
+    </tr>
+    <tr>
+      <td><code>flashinfer_cudnn</code></td>
+      <td>SM100/120 (CUDA 13+, cuDNN 9.15+)</td>
+      <td>FlashInfer cuDNN backend; used on SM120 for performance</td>
+    </tr>
+    <tr>
+      <td><code>flashinfer_trtllm</code></td>
+      <td>SM100</td>
+      <td>FlashInfer TensorRT-LLM backend</td>
+    </tr>
+  </tbody>
+</table>
+
+When FlashInfer is unavailable for NVFP4, the SGLang CUTLASS kernel is used as an automatic fallback.
+
 ## Offline Quantization

 To load already quantized models, simply load the model weights and config. **Again, if the model has been quantized offline,
 there's no need to add `--quantization` argument when starting the engine. The quantization method will be parsed from the
-downloaded Hugging Face config. For example, DeepSeek V3/R1 models are already in FP8, so do not add redundant parameters.**
+downloaded Hugging Face or msModelSlim config. For example, DeepSeek V3/R1 models are already in FP8, so do not add redundant parameters.**

 ```bash Command
 python3 -m sglang.launch_server \
@@ -194,23 +459,85 @@ python3 -m sglang.launch_server \

 #### Using [NVIDIA ModelOpt](https://github.com/NVIDIA/Model-Optimizer)

-NVIDIA Model Optimizer (ModelOpt) provides advanced quantization techniques optimized for NVIDIA hardware. SGLang includes a streamlined workflow for quantizing models with ModelOpt and automatically exporting them for deployment.
+NVIDIA Model Optimizer (ModelOpt) provides advanced quantization techniques optimized for NVIDIA hardware.
+
+**Offline vs. Online Quantization:**
+
+SGLang supports two modes for ModelOpt.
+
+* **Offline Quantization (pre-quantized):**
+    * **Usage:** Download a pre-quantized model from Hugging Face or run `hf_ptq.py` once to create a new quantized checkpoint. Then load this quantized checkpoint.
+    * **Pros:** Fast server startup, quantization can be validated before deployment, efficient resource usage.
+    * **Cons:** Requires an extra preparation step.
+
+* **Online Quantization (quant and serve):**
+    * **Usage:** Load a standard BF16/FP16 model and add a flag. The engine applies quantization *on startup*.
+    * **Pros:** Convenient (no new checkpoint needed).
+    * **Cons:** **High startup time**, increases VRAM usage during initialization (risk of OOM).
+
+The following sections guide you through using the Offline path: loading pre-quantized models or creating your own checkpoints.
+
+##### Using Pre-Quantized Checkpoints
+
+If a model is already quantized (e.g., from Hugging Face), you can load it directly.
+
+* **FP8 Models:**
+    Use `--quantization modelopt_fp8`.
+    ```bash Command
+    python3 -m sglang.launch_server \
+        --model-path nvidia/Llama-3.1-8B-Instruct-FP8 \
+        --quantization modelopt_fp8 \
+        --port 30000
+    ```
+
+* **FP4 Models:**
+    Use `--quantization modelopt_fp4`.
+    ```bash Command
+    python3 -m sglang.launch_server \
+        --model-path nvidia/Llama-3.3-70B-Instruct-NVFP4 \
+        --quantization modelopt_fp4 \
+        --port 30000
+    ```
+
+##### Creating Your Own Quantized Checkpoints
+
+If a pre-quantized checkpoint is not available for your model, you can create one using NVIDIA Model Optimizer's `hf_ptq.py` script.
+
+**Why quantize?**
+- Reduce VRAM usage
+- Higher throughput and lower latency
+- More flexible deployment (on smaller GPUs)
+
+**What can be quantized?**
+- The entire model
+- MLP layers only
+- KV cache
+
+**Key options in `hf_ptq.py`:**
+
+`--qformat`: Quantization formats `fp8`, `nvfp4`, `nvfp4_mlp_only`
+
+`--kv_cache_qformat`: KV cache quantization format (default: `fp8`)
+
+**Note:** The default `kv_cache_qformat` may not be optimal for all use cases. Consider setting this explicitly.
+
+**Hardware requirements:** Hopper and higher are recommended. Insufficient GPU memory may cause weight offloading, resulting in extremely long quantization time.
+
+For detailed usage and supported model architectures, see [NVIDIA Model Optimizer LLM PTQ](https://github.com/NVIDIA/Model-Optimizer/tree/main/examples/llm_ptq).
+
+SGLang includes a streamlined workflow for quantizing models with ModelOpt and automatically exporting them for deployment.

 ##### Installation

-First, install ModelOpt. You can either install it directly or as an optional SGLang dependency:
+First, install ModelOpt:

 ```bash Command
-# Option 1: Install ModelOpt directly
 pip install nvidia-modelopt
-
-# Option 2: Install SGLang with ModelOpt support (recommended)
-pip install sglang[modelopt]
 ```

 ##### Quantization and Export Workflow

-SGLang provides an example script that demonstrates the complete ModelOpt quantization and export workflow:
+SGLang provides an example script that demonstrates the complete ModelOpt quantization and export workflow. Run from the SGLang repository root (see [modelopt_quantize_and_export.py](https://github.com/sgl-project/sglang/blob/main/examples/usage/modelopt_quantize_and_export.py)):

 ```bash Command
 # Quantize and export a model using ModelOpt FP8 quantization
@@ -219,7 +546,7 @@ python examples/usage/modelopt_quantize_and_export.py quantize \
    --export-dir ./quantized_tinyllama_fp8 \
    --quantization-method modelopt_fp8

-# For FP4 quantization
+# For FP4 quantization (requires Blackwell GPU)
 python examples/usage/modelopt_quantize_and_export.py quantize \
    --model-path TinyLlama/TinyLlama-1.1B-Chat-v1.0 \
    --export-dir ./quantized_tinyllama_fp4 \
@@ -275,25 +602,39 @@ python -m sglang.launch_server \
    --port 30000 --host 0.0.0.0
 ```

-Or using the Python API:
+Or using the Python API (use the same path as `modelopt_export_path` from the quantize step):

 ```python Example
 import sglang as sgl

-# Deploy exported ModelOpt quantized model
-llm = sgl.Engine(
-    model_path="./quantized_tinyllama_fp8",
-    quantization="modelopt"
-)
+def main():
+    # Deploy exported ModelOpt quantized model
+    # Path must match modelopt_export_path from quantize step (e.g., ./exported_model)
+    llm = sgl.Engine(
+        model_path="./exported_model",
+        quantization="modelopt",
+    )

-# Run inference
-prompts = ["Hello, how are you?", "What is the capital of France?"]
-sampling_params = {"temperature": 0.8, "top_p": 0.95, "max_new_tokens": 100}
-outputs = llm.generate(prompts, sampling_params)
+    # Run inference
+    prompts = [
+        "Hello, how are you?",
+        "What is the capital of France?",
+    ]
+    sampling_params = {
+        "temperature": 0.8,
+        "top_p": 0.95,
+        "max_new_tokens": 100,
+    }
+
+    outputs = llm.generate(prompts, sampling_params)
+
+    for i, output in enumerate(outputs):
+        print(f"Prompt: {prompts[i]}")
+        print(f"Output: {output['text']}")
+
+if __name__ == "__main__":
+    main()

-for i, output in enumerate(outputs):
-    print(f"Prompt: {prompts[i]}")
-    print(f"Output: {output.outputs[0].text}")
 ```

 ##### Advanced Features
@@ -311,7 +652,7 @@ python examples/usage/modelopt_quantize_and_export.py quantize \
 # The checkpoint can be reused for future quantization runs and skip calibration
 ```

-**Export-only Workflow**: If you have a pre-existing fake quantized ModelOpt checkpoint, you can export it directly:
+**Export-only Workflow**: If you have a pre-existing fake quantized ModelOpt checkpoint, you can export it directly. See [LoadConfig](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/configs/load_config.py) for the full API:

 ```python Example
 from sglang.srt.configs.device_config import DeviceConfig
@@ -330,7 +671,7 @@ load_config = LoadConfig(
    modelopt_export_path="./exported_model",
 )

-# Load and export the model
+# Load and export the model (DeviceConfig defaults to device="cuda")
 model_loader = get_model_loader(load_config, model_config)
 model_loader.load_model(model_config=model_config, device_config=DeviceConfig())
 ```
@@ -343,6 +684,74 @@ model_loader.load_model(model_config=model_config, device_config=DeviceConfig())
 - **Calibration-based**: Uses calibration datasets for optimal quantization quality
 - **Production Ready**: Enterprise-grade quantization with NVIDIA support

+#### Using [ModelSlim](https://gitcode.com/Ascend/msmodelslim)
+MindStudio-ModelSlim (msModelSlim) is a model offline quantization compression tool launched by MindStudio and optimized for Ascend hardware.
+
+- **Installation**
+
+    ```bash Command
+    # Clone repo and install msmodelslim:
+    git clone https://gitcode.com/Ascend/msmodelslim.git
+    cd msmodelslim
+    bash install.sh
+    ```
+
+- **LLM quantization**
+
+    Download the original floating-point weights of the large model. Taking Qwen3-32B as an example, you can go to [Qwen3-32B](https://huggingface.co/Qwen/Qwen3-32B) to obtain the original model weights. Then install other dependencies (related to the model, refer to the huggingface model card).
+    > Note: You can find pre-quantized validated models on [modelscope/Eco-Tech](https://modelscope.cn/models/Eco-Tech).
+
+    _Traditional quantification methods require the preparation of calibration data files (```.jsonl``` formats) for calibration in the quantification process._
+    ```bash Command
+    Qwen3-32B/      # floating-point model downloaded from official HF (or modelscope) repo
+    msmodelslim/    # msmodelslim repo
+      |----- lab_calib # calibration date folder (put your dataset here in ```.jsonl``` format or use pre-prepared ones)
+          |----- some file (such as laos_calib.jsonl)
+      |----- lab_practice # best practice folder with configs for quantization
+          |----- model folder (such as qwen3_5_moe folder) # folder with quantization configs
+              |----- quant_config (such as qwen3_5_moe_w8a8.yaml) # quantization config
+      |----- another folders
+    output_folder/   # generated by below command
+      |----- quant_model_weights-00001-of-0001.safetensors # quantized weights
+      |----- quant_model_description.json # file with description of the quantization methods for each layer (```W4A4_DYNAMIC```, etc.)
+      |----- another files (such as config.json, tokenizer.json, etc.)
+    ```
+    Run quantization using one-click quantization (recommended):
+    ```bash Command
+    msmodelslim quant \
+    --model_path ${MODEL_PATH} \
+    --save_path ${SAVE_PATH} \
+    --device npu:0,1 \
+    --model_type Qwen3-32B \
+    --quant_type w8a8 \
+    --trust_remote_code True
+    ```
+
+- **Usage Example**
+    ```bash Command
+    python3 -m sglang.launch_server \
+    --model-path $PWD/Qwen3-32B-w8a8 \
+    --port 30000 --host 0.0.0.0
+    ```
+
+- **Available Quantization Methods**:
+    - [x]  ```W4A4_DYNAMIC``` linear with online quantization of activations
+    - [x]  ```W8A8``` linear with offline quantization of activations
+    - [x]  ```W8A8_DYNAMIC``` linear with online quantization of activations
+    - [x]  ```W4A4_DYNAMIC``` MOE with online quantization of activations
+    - [x]  ```W4A8_DYNAMIC``` MOE with online quantization of activations
+    - [x]  ```W8A8_DYNAMIC``` MOE with online quantization of activations
+    - [ ]  ```W4A8``` linear TBD
+    - [ ]  ```W4A16``` linear TBD
+    - [ ]  ```W48A16``` linear TBD
+    - [ ]  ```W4A16``` MoE in progress
+    - [ ]  ```W8A16``` MoE in progress
+    - [ ]  ```KV Cache``` in progress
+    - [ ]  ```Attention``` in progress
+
+
+For more detailed examples of quantization of models, as well as information about their support, see the [examples](https://gitcode.com/Ascend/msmodelslim/blob/master/example/README.md) section in ModelSLim repo.
+
 ## Online Quantization

 To enable online quantization, you can simply specify `--quantization` in the command line. For example, you can launch the server with the following command to enable `FP8` quantization for model `meta-llama/Meta-Llama-3.1-8B-Instruct`:
@@ -381,7 +790,7 @@ python3 -m sglang.launch_server \

 ### `quark_int4fp8_moe` online quantization method

-SGLang running on AMD GPUs (CDNA3 or CDNA4 architecture) supports the quantization method `--quantization quark_int4fp8_moe`, that will replace [MoE layers](https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/layers/moe/fused_moe_triton/layer.py) originally in high precision (bfloat16, float16 or float32) to use weights dynamically quantized to int4, that are upcasted to float8 during inference to run compute in float8 precision with activations dynamically quantized on the fly to float8.
+SGLang running on AMD GPUs (CDNA3 or CDNA4 architecture) supports the quantization method `--quantization quark_int4fp8_moe`, that will replace [MoE layers](https://github.com/sgl-project/sglang/blob/v0.4.8/python/sglang/srt/layers/moe/fused_moe_triton/layer.py#L271) originally in high precision (bfloat16, float16 or float32) to use weights dynamically quantized to int4, that are upcasted to float8 during inference to run compute in float8 precision with activations dynamically quantized on the fly to float8.

 Other layers (e.g. projections in the attention layers) have their weights quantized online to float8 directly.

@@ -390,6 +799,9 @@ Other layers (e.g. projections in the attention layers) have their weights quant
 - [GPTQModel](https://github.com/ModelCloud/GPTQModel)
 - [LLM Compressor](https://github.com/vllm-project/llm-compressor/)
 - [NVIDIA Model Optimizer (ModelOpt)](https://github.com/NVIDIA/Model-Optimizer)
+- [NVIDIA Model Optimizer LLM PTQ](https://github.com/NVIDIA/Model-Optimizer/tree/main/examples/llm_ptq)
+- [Petit: NVFP4 on ROCm](https://github.com/causalflow-ai/petit-kernel) — [LMSYS blog](https://lmsys.org/blog/2025-09-21-petit-amdgpu/), [AMD ROCm blog](https://rocm.blogs.amd.com/artificial-intelligence/fp4-mixed-precision/README.html)
 - [Torchao: PyTorch Architecture Optimization](https://github.com/pytorch/ao)
 - [vLLM Quantization](https://docs.vllm.ai/en/latest/quantization/)
 - [auto-round](https://github.com/intel/auto-round)
+- [ModelSlim](https://gitcode.com/Ascend/msmodelslim)
@@ -5,7 +5,7 @@ metatags:
 ---
 R-Fork (Tensor Remote Fork) is a novel weight loading methodology that leverages efficient inter-node GPU-to-GPU data transfer path to load tensors from a running SGLang instance to a new instance with zero-copy. It can significantly optimize the SGLang instance boot-up time by reducing model weights loading from several minutes to mere seconds.

-To learn more details about R-Fork, please check **[R-Fork blog](https://lmsys.org/blog/2025-12-10-rfork/)**
+To learn more details about R-Fork, please check **<a href="https://lmsys.org/blog/2025-12-10-rfork/"> R-Fork blog </a>**

 ## Usage

@@ -27,25 +27,29 @@ To learn more details about R-Fork, please check **[R-Fork blog](https://lmsys.o
    </tr>
    <tr>
      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>remote-instance-weight-loader-backend</td>
-      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`nccl` or `transfer_engine`, default value is `nccl`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><code>nccl</code>, <code>transfer_engine</code>, or <code>modelexpress</code>. Default is <code>nccl</code>.</td>
    </tr>
    <tr>
      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>remote-instance-weight-loader-seed-instance-ip</td>
-      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>IP address of the seed instance who will provide the model weight</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>IP address of the seed instance who will provide the model weight. Used by <code>nccl</code> and <code>transfer_engine</code> backends.</td>
    </tr>
    <tr>
      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>remote-instance-weight-loader-seed-instance-service-port</td>
-      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>the port that the seed instance's HTTP server is listening on</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>the port that the seed instance's HTTP server is listening on. Used by <code>nccl</code> and <code>transfer_engine</code> backends.</td>
    </tr>
    <tr>
      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>remote-instance-weight-loader-send-weights-group-ports</td>
-      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>the list of available ports on the seed instance that will be used to build NCCL communication groups between seed and client instance. This argument is only needed by `nccl` backend.</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>the list of available ports on the seed instance that will be used to build NCCL communication groups between seed and client instance. Only needed by <code>nccl</code> backend.</td>
    </tr>
    <tr>
      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>remote-instance-weight-loader-start-seed-via-transfer-engine</td>
-      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>set to start seed service that supports TransferEngine as backend. It is needed for seed instances when using `transfer_engine` as backend.</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>set to start seed service that supports TransferEngine as backend. Needed for seed instances when using <code>transfer_engine</code> as backend.</td>
    </tr>
-  </tbody>
+<tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>modelexpress-config</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>JSON config for <code>modelexpress</code> backend. Keys: <code>"url"</code> (required, gRPC host:port of ModelExpress server), <code>"model_name"</code> (optional, defaults to <code>--model-path</code>), <code>"source"</code> (optional bool, <code>true</code> for seed mode).</td>
+    </tr>
+</tbody>
 </table>

 ### NCCL as backend
@@ -80,3 +84,25 @@ python -m sglang.launch_server [args] \
  --remote-instance-weight-loader-seed-instance-service-port [seed_instance_service_port] \
  --remote-instance-weight-loader-backend transfer_engine
 ```
+
+### ModelExpress as backend
+
+[ModelExpress](https://github.com/ai-dynamo/modelexpress) is a coordination service that manages P2P weight transfer metadata. It removes the need for direct seed IP/port configuration by providing a centralized registry that seeds publish to and clients discover from. Under the hood it uses TransferEngine (Mooncake) for the actual RDMA data transfer.
+
+A running ModelExpress server is required. See the [ModelExpress documentation](https://github.com/ai-dynamo/modelexpress) for setup instructions.
+
+seed instance:
+```bash Command
+python -m sglang.launch_server [args] \
+  --modelexpress-config '{"url": "[modelexpress_grpc_host:port]", "model_name": "[model_name]", "source": true}'
+```
+
+client instance:
+```bash Command
+python -m sglang.launch_server [args] \
+  --load-format remote_instance \
+  --remote-instance-weight-loader-backend modelexpress \
+  --modelexpress-config '{"url": "[modelexpress_grpc_host:port]", "model_name": "[model_name]"}'
+```
+
+The seed publishes its TransferEngine session ID and tensor layout to ModelExpress. The client queries ModelExpress to discover the seed, then pulls weights directly via RDMA. This enables dynamic seed discovery without hardcoding IPs, and supports multiple models through a single ModelExpress instance.
@@ -70,7 +70,7 @@
    "    \"python3 -m sglang.launch_server --model-path deepseek-ai/DeepSeek-R1-Distill-Qwen-7B --host 0.0.0.0 --reasoning-parser deepseek-r1 --log-level warning\"\n",
    ")\n",
    "\n",
-    "wait_for_server(f\"http://localhost:{port}\")"
+    "wait_for_server(f\"http://localhost:{port}\", process=server_process)"
   ]
  },
  {
@@ -105,7 +105,7 @@ Enable memory saver support when launching the server:

 ## Open-To-Use Refit Functionality

-After training completes each step, rollout engines must be refit with new weights. SGLang supports three refit strategies so you can match your infrastructure style (co-located vs disaggregated) and scaling needs. Each strategy maps to a concrete API with clear request schemas. For a deeper dive into SGLang's weight update utilities, see [RL System Deep Thinking: Weight Update Mechanisms](https://github.com/zhaochenyang20/Awesome-ML-SYS-Tutorial/blob/main/rlhf/sys-design/readme-1-EN).
+After training completes each step, rollout engines must be refit with new weights. SGLang supports three refit strategies so you can match your infrastructure style (co-located vs disaggregated) and scaling needs. Each strategy maps to a concrete API with clear request schemas. For a deeper dive into SGLang's weight update utilities, see [RL System Deep Thinking: Weight Update Mechanisms](https://github.com/zhaochenyang20/Awesome-ML-SYS-Tutorial/blob/main/rlhf/sys-design/readme-1-EN.md).

 **How to choose:**

@@ -248,6 +248,86 @@ This path trades some I/O overhead for simplicity and flexibility. It integrates

 **Python Engine API:** `engine.update_weights_from_disk(model_path, load_format=None)`

+**Diffusion engine (SGLang-Diffusion):** The diffusion engine exposes the same `POST /update_weights_from_disk` endpoint with the following behavior:
+
+- **All-or-nothing with rollback:** if any module fails to load, all previously updated modules are rolled back to the original weights by reloading from the original model path. No partial updates are left behind. If rollback itself fails, the exception propagates so the caller knows the model is in an inconsistent state.
+- **Offload-aware:** when layerwise offload (`--dit-layerwise-offload`) is enabled, the diffusion offload manager replaces GPU parameters with small `torch.empty((1,))` placeholders while real weights live in consolidated pinned CPU buffers. A naive `param.data.copy_()` would fail with a shape mismatch. Instead, the updater dynamically detects active offload managers and writes new weights directly into their CPU buffers, bypassing the placeholders entirely. For any layer that happens to be prefetched on GPU at update time, the live GPU tensor is also updated so the change takes effect immediately. This requires no extra GPU memory and does not disturb the offload state.
+- **DTensor-aware:** parameters distributed via `torch.distributed.tensor` (tensor parallelism) are updated through `distribute_tensor` so that each shard is correctly placed on the right device mesh.
+
+**Request body:**
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <colgroup>
+    <col style={{width: "25%"}} />
+    <col style={{width: "25%"}} />
+    <col style={{width: "25%"}} />
+    <col style={{width: "25%"}} />
+  </colgroup>
+  <thead>
+    <tr style={{borderBottom: "2px solid #d55816"}}>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Field</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Description</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Defaults</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Options</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>model_path</code></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>The model path with the new weights.</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Required</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: str</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>flush_cache</code></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Flush TeaCache state after update.</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><code>True</code></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: bool</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>target_modules</code></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>List of module names to update (e.g. <code>["transformer"]</code>). If omitted, all <code>nn.Module</code> components are updated.</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><code>None</code></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: list[str]</td>
+    </tr>
+  </tbody>
+</table>
+
+**Response body:**
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <colgroup>
+    <col style={{width: "25%"}} />
+    <col style={{width: "25%"}} />
+    <col style={{width: "25%"}} />
+    <col style={{width: "25%"}} />
+  </colgroup>
+  <thead>
+    <tr style={{borderBottom: "2px solid #d55816"}}>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Field</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Description</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Defaults</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Options</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>success</code></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Whether the update succeeded.</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>-</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: bool</td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>message</code></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Status / error message.</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>-</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: str</td>
+    </tr>
+  </tbody>
+</table>
+
+> **Note:** The diffusion engine (SGLang-Diffusion) does not currently support hot refit (updating weights while inference is in progress). The diffusion scheduler processes one request at a time and completes the entire inference before handling the next request, so weight updates and inference never run concurrently.
+
 ### Update Weights from Tensor

 **When to use:**
@@ -280,33 +360,33 @@ This strategy requires the training process and rollout engine to share access t
  </thead>
  <tbody>
    <tr>
-      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`serialized_named_tensors`</td>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>serialized_named_tensors</code></td>
      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Per-TP serialized tensor payloads.</td>
      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Required</td>
-      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: list[str</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: list[str|bytes]</td>
    </tr>
    <tr>
-      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`load_format`</td>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>load_format</code></td>
      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Optional load format selector.</td>
-      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`None`</td>
-      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`None`, `direct`, `flattened_bucket`, or a custom loader path string</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><code>None</code></td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><code>None</code>, <code>direct</code>, <code>flattened_bucket</code>, or a custom loader path string</td>
    </tr>
    <tr>
-      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`flush_cache`</td>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>flush_cache</code></td>
      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Flush KV cache after update.</td>
-      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`True`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><code>True</code></td>
      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: bool</td>
    </tr>
    <tr>
-      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`abort_all_requests`</td>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>abort_all_requests</code></td>
      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Abort all running requests before update.</td>
-      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`False`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><code>False</code></td>
      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: bool</td>
    </tr>
    <tr>
-      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`weight_version`</td>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><code>weight_version</code></td>
      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Optional version label tracked by the server.</td>
-      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`None`</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><code>None</code></td>
      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: str</td>
    </tr>
  </tbody>
@@ -551,7 +631,7 @@ SGLang exposes explicit pause/resume APIs so you can pause slow requests and con

 In many RL stacks, rollout and training are implemented with different kernels or batching behavior. Even when weights are identical, token probabilities can drift, silently breaking the on-policy assumption. This is the training–inference mismatch problem.

-SGLang supports a deterministic inference mode that reduces non-determinism across batch shapes. This mitigates variance introduced by runtime batching and kernel selection. To further achieve true on-policy training, you need to modify the training engine to use the same deterministic kernels. For implementation details, see these miles examples: [True On-Policy](https://github.com/radixark/miles/tree/main/examples/true_on_policy) and [True On-Policy for VLM](https://github.com/radixark/miles/tree/main/examples/true_on_policy_vlm). For additional context, see the blog post [Let Speed Be With Stability: All-In-One Solution to Training-Inference Mismatch with Miles](https://github.com/zhaochenyang20/Awesome-ML-SYS-Tutorial/blob/main/rlhf/slime/mismatch/blog-en).
+SGLang supports a deterministic inference mode that reduces non-determinism across batch shapes. This mitigates variance introduced by runtime batching and kernel selection. To further achieve true on-policy training, you need to modify the training engine to use the same deterministic kernels. For implementation details, see these miles examples: [True On-Policy](https://github.com/radixark/miles/tree/main/examples/true_on_policy) and [True On-Policy for VLM](https://github.com/radixark/miles/tree/main/examples/true_on_policy_vlm). For additional context, see the blog post [Let Speed Be With Stability: All-In-One Solution to Training-Inference Mismatch with Miles](https://github.com/zhaochenyang20/Awesome-ML-SYS-Tutorial/blob/main/rlhf/slime/mismatch/blog-en.md).

 **Server flag:**

@@ -19,7 +19,7 @@
    "- [Outlines](https://github.com/dottxt-ai/outlines): Supports JSON schema and regular expression constraints.\n",
    "- [Llguidance](https://github.com/guidance-ai/llguidance): Supports JSON schema, regular expression, and EBNF constraints.\n",
    "\n",
-    "We suggest using XGrammar for its better performance and utility. XGrammar currently uses the [GGML BNF format](https://github.com/ggerganov/llama.cpp/blob/master/grammars/README). For more details, see [XGrammar technical overview](https://blog.mlc.ai/2024/11/22/achieving-efficient-flexible-portable-structured-generation-with-xgrammar).\n",
+    "We suggest using XGrammar for its better performance and utility. XGrammar currently uses the [GGML BNF format](https://github.com/ggerganov/llama.cpp/blob/master/grammars/README.md). For more details, see [XGrammar technical overview](https://blog.mlc.ai/2024/11/22/achieving-efficient-flexible-portable-structured-generation-with-xgrammar).\n",
    "\n",
    "To use Outlines, simply add `--grammar-backend outlines` when launching the server.\n",
    "To use llguidance, add `--grammar-backend llguidance`  when launching the server.\n",
@@ -54,7 +54,7 @@
    "    \"python -m sglang.launch_server --model-path meta-llama/Meta-Llama-3.1-8B-Instruct --host 0.0.0.0 --log-level warning\"\n",
    ")\n",
    "\n",
-    "wait_for_server(f\"http://localhost:{port}\")\n",
+    "wait_for_server(f\"http://localhost:{port}\", process=server_process)\n",
    "client = openai.Client(base_url=f\"http://127.0.0.1:{port}/v1\", api_key=\"None\")"
   ]
  },
@@ -356,8 +356,7 @@
   "outputs": [],
   "source": [
    "# Support for XGrammar latest structural tag format\n",
-    "# https://xgrammar.mlc.ai/docs/tutorials/structural_tag.html\n",
-    "\n",
+    "# <https://xgrammar.mlc.ai/docs/tutorials/structural_tag.html>\n",
    "response = client.chat.completions.create(\n",
    "    model=\"meta-llama/Meta-Llama-3.1-8B-Instruct\",\n",
    "    messages=messages,\n",
@@ -645,8 +644,7 @@
   "outputs": [],
   "source": [
    "# Support for XGrammar latest structural tag format\n",
-    "# https://xgrammar.mlc.ai/docs/tutorials/structural_tag.html\n",
-    "\n",
+    "# <https://xgrammar.mlc.ai/docs/tutorials/structural_tag.html>\n",
    "payload = {\n",
    "    \"text\": text,\n",
    "    \"sampling_params\": {\n",
@@ -925,8 +923,7 @@
   "outputs": [],
   "source": [
    "# Support for XGrammar latest structural tag format\n",
-    "# https://xgrammar.mlc.ai/docs/tutorials/structural_tag.html\n",
-    "\n",
+    "# <https://xgrammar.mlc.ai/docs/tutorials/structural_tag.html>\n",
    "sampling_params = {\n",
    "    \"temperature\": 0.8,\n",
    "    \"top_p\": 0.95,\n",
@@ -11,7 +11,7 @@ SGLang supports three grammar backends:
 - [Outlines](https://github.com/dottxt-ai/outlines): Supports JSON schema and regular expression constraints.
 - [Llguidance](https://github.com/guidance-ai/llguidance): Supports JSON schema, regular expression, and EBNF constraints.

-We suggest using XGrammar for its better performance and utility. XGrammar currently uses the [GGML BNF format](https://github.com/ggerganov/llama.cpp/blob/master/grammars/README). For more details, see [XGrammar technical overview](https://blog.mlc.ai/2024/11/22/achieving-efficient-flexible-portable-structured-generation-with-xgrammar).
+We suggest using XGrammar for its better performance and utility. XGrammar currently uses the [GGML BNF format](https://github.com/ggerganov/llama.cpp/blob/master/grammars/README.md). For more details, see [XGrammar technical overview](https://blog.mlc.ai/2024/11/22/achieving-efficient-flexible-portable-structured-generation-with-xgrammar).

 To use Outlines, simply add `--grammar-backend outlines` when launching the server.
 To use llguidance, add `--grammar-backend llguidance`  when launching the server.
@@ -247,9 +247,9 @@ If a you choose to call a function ONLY reply in the following format:
 where
 start_tag => `<function`
 parameters => a JSON dict with the function argument name as key and function argument value as value.
-end_tag => `&lt;/function&gt;`
+end_tag => `</function>`
 Here is an example,
-&lt;function=example_function_name>{{"example_name": "example_value"}}&lt;/function&gt;
+<function=example_function_name>{{"example_name": "example_value"}}</function>
 Reminder:
 - Function calls MUST follow the specified format
 - Required parameters MUST be specified
@@ -274,17 +274,17 @@ response = client.chat.completions.create(
        "type": "structural_tag",
        "structures": [
            {
-                "begin": "&lt;function=get_current_weather>",
+                "begin": "<function=get_current_weather>",
                "schema": schema_get_current_weather,
-                "end": "&lt;/function&gt;",
+                "end": "</function>",
            },
            {
-                "begin": "&lt;function=get_current_date>",
+                "begin": "<function=get_current_date>",
                "schema": schema_get_current_date,
-                "end": "&lt;/function&gt;",
+                "end": "</function>",
            },
        ],
-        "triggers": ["&lt;function="],
+        "triggers": ["<function="],
    },
 )

@@ -303,23 +303,23 @@ response = client.chat.completions.create(
        "type": "structural_tag",
        "format": {
            "type": "triggered_tags",
-            "triggers": ["&lt;function="],
+            "triggers": ["<function="],
            "tags": [
                {
-                    "begin": "&lt;function=get_current_weather>",
+                    "begin": "<function=get_current_weather>",
                    "content": {
                        "type": "json_schema",
                        "json_schema": schema_get_current_weather,
                    },
-                    "end": "&lt;/function&gt;",
+                    "end": "</function>",
                },
                {
-                    "begin": "&lt;function=get_current_date>",
+                    "begin": "<function=get_current_date>",
                    "content": {
                        "type": "json_schema",
                        "json_schema": schema_get_current_date,
                    },
-                    "end": "&lt;/function&gt;",
+                    "end": "</function>",
                },
            ],
            "at_least_one": False,
@@ -506,17 +506,17 @@ payload = {
                "type": "structural_tag",
                "structures": [
                    {
-                        "begin": "&lt;function=get_current_weather>",
+                        "begin": "<function=get_current_weather>",
                        "schema": schema_get_current_weather,
-                        "end": "&lt;/function&gt;",
+                        "end": "</function>",
                    },
                    {
-                        "begin": "&lt;function=get_current_date>",
+                        "begin": "<function=get_current_date>",
                        "schema": schema_get_current_date,
-                        "end": "&lt;/function&gt;",
+                        "end": "</function>",
                    },
                ],
-                "triggers": ["&lt;function="],
+                "triggers": ["<function="],
            }
        )
    },
@@ -541,23 +541,23 @@ payload = {
                "type": "structural_tag",
                "format": {
                    "type": "triggered_tags",
-                    "triggers": ["&lt;function="],
+                    "triggers": ["<function="],
                    "tags": [
                        {
-                            "begin": "&lt;function=get_current_weather>",
+                            "begin": "<function=get_current_weather>",
                            "content": {
                                "type": "json_schema",
                                "json_schema": schema_get_current_weather,
                            },
-                            "end": "&lt;/function&gt;",
+                            "end": "</function>",
                        },
                        {
-                            "begin": "&lt;function=get_current_date>",
+                            "begin": "<function=get_current_date>",
                            "content": {
                                "type": "json_schema",
                                "json_schema": schema_get_current_date,
                            },
-                            "end": "&lt;/function&gt;",
+                            "end": "</function>",
                        },
                    ],
                    "at_least_one": False,
@@ -727,17 +727,17 @@ sampling_params = {
            "type": "structural_tag",
            "structures": [
                {
-                    "begin": "&lt;function=get_current_weather>",
+                    "begin": "<function=get_current_weather>",
                    "schema": schema_get_current_weather,
-                    "end": "&lt;/function&gt;",
+                    "end": "</function>",
                },
                {
-                    "begin": "&lt;function=get_current_date>",
+                    "begin": "<function=get_current_date>",
                    "schema": schema_get_current_date,
-                    "end": "&lt;/function&gt;",
+                    "end": "</function>",
                },
            ],
-            "triggers": ["&lt;function="],
+            "triggers": ["<function="],
        }
    ),
 }
@@ -763,23 +763,23 @@ sampling_params = {
            "type": "structural_tag",
            "format": {
                "type": "triggered_tags",
-                "triggers": ["&lt;function="],
+                "triggers": ["<function="],
                "tags": [
                    {
-                        "begin": "&lt;function=get_current_weather>",
+                        "begin": "<function=get_current_weather>",
                        "content": {
                            "type": "json_schema",
                            "json_schema": schema_get_current_weather,
                        },
-                        "end": "&lt;/function&gt;",
+                        "end": "</function>",
                    },
                    {
-                        "begin": "&lt;function=get_current_date>",
+                        "begin": "<function=get_current_date>",
                        "content": {
                            "type": "json_schema",
                            "json_schema": schema_get_current_date,
                        },
-                        "end": "&lt;/function&gt;",
+                        "end": "</function>",
                    },
                ],
                "at_least_one": False,
@@ -50,7 +50,7 @@
    "    \"python -m sglang.launch_server --model-path deepseek-ai/DeepSeek-R1-Distill-Qwen-7B --host 0.0.0.0 --reasoning-parser deepseek-r1 --log-level warning\"\n",
    ")\n",
    "\n",
-    "wait_for_server(f\"http://localhost:{port}\")\n",
+    "wait_for_server(f\"http://localhost:{port}\", process=server_process)\n",
    "client = openai.Client(base_url=f\"http://127.0.0.1:{port}/v1\", api_key=\"None\")"
   ]
  },
@@ -3,17 +3,17 @@ title: "Structured Outputs For Reasoning Models"
 metatags:
    description: "SGLang structured outputs for reasoning models: free-form thinking with constrained final output for DeepSeek R1, QwQ models."
 ---
-When working with reasoning models that use special tokens like `&lt;think&gt;...&lt;/think&gt;` to denote reasoning sections, you might want to allow free-form text within these sections while still enforcing grammar constraints on the rest of the output.
+When working with reasoning models that use special tokens like `<think>...</think>` to denote reasoning sections, you might want to allow free-form text within these sections while still enforcing grammar constraints on the rest of the output.

 SGLang provides a feature to disable grammar restrictions within reasoning sections. This is particularly useful for models that need to perform complex reasoning steps before providing a structured output.

-To enable this feature, use the `--reasoning-parser` flag which decide the think_end_token, such as `&lt;/think&gt;`, when launching the server. You can also specify the reasoning parser using the `--reasoning-parser` flag.
+To enable this feature, use the `--reasoning-parser` flag which decide the think_end_token, such as `</think>`, when launching the server. You can also specify the reasoning parser using the `--reasoning-parser` flag.

 ## Supported Models

 Currently, SGLang supports the following reasoning models:
- [DeepSeek R1 series](https://huggingface.co/collections/deepseek-ai/deepseek-r1-678e1e131c0169c0bc89728d): The reasoning content is wrapped with `&lt;think&gt;` and `&lt;/think&gt;` tags.
- [QwQ](https://huggingface.co/Qwen/QwQ-32B): The reasoning content is wrapped with `&lt;think&gt;` and `&lt;/think&gt;` tags.
+- [DeepSeek R1 series](https://huggingface.co/collections/deepseek-ai/deepseek-r1-678e1e131c0169c0bc89728d): The reasoning content is wrapped with `<think>` and `</think>` tags.
+- [QwQ](https://huggingface.co/Qwen/QwQ-32B): The reasoning content is wrapped with `<think>` and `</think>` tags.


 ## Usage
@@ -252,9 +252,9 @@ If a you choose to call a function ONLY reply in the following format:
 where
 start_tag => `<function`
 parameters => a JSON dict with the function argument name as key and function argument value as value.
-end_tag => `&lt;/function&gt;`
+end_tag => `</function>`
 Here is an example,
-&lt;function=example_function_name>{{"example_name": "example_value"}}&lt;/function&gt;
+<function=example_function_name>{{"example_name": "example_value"}}</function>
 Reminder:
 - Function calls MUST follow the specified format
 - Required parameters MUST be specified
@@ -280,17 +280,17 @@ response = client.chat.completions.create(
        "max_new_tokens": 2048,
        "structures": [
            {
-                "begin": "&lt;function=get_current_weather>",
+                "begin": "<function=get_current_weather>",
                "schema": schema_get_current_weather,
-                "end": "&lt;/function&gt;",
+                "end": "</function>",
            },
            {
-                "begin": "&lt;function=get_current_date>",
+                "begin": "<function=get_current_date>",
                "schema": schema_get_current_date,
-                "end": "&lt;/function&gt;",
+                "end": "</function>",
            },
        ],
-        "triggers": ["&lt;function="],
+        "triggers": ["<function="],
    },
 )

@@ -351,8 +351,8 @@ response = requests.post(
 print(response.json())


-reasoing_content = response.json()["text"].split("&lt;/think&gt;")[0]
-content = response.json()["text"].split("&lt;/think&gt;")[1]
+reasoing_content = response.json()["text"].split("</think>")[0]
+content = response.json()["text"].split("</think>")[1]
 print_highlight(f"reasoing_content: {reasoing_content}\n\ncontent: {content}")
 ```

@@ -460,17 +460,17 @@ payload = {
                "type": "structural_tag",
                "structures": [
                    {
-                        "begin": "&lt;function=get_current_weather>",
+                        "begin": "<function=get_current_weather>",
                        "schema": schema_get_current_weather,
-                        "end": "&lt;/function&gt;",
+                        "end": "</function>",
                    },
                    {
-                        "begin": "&lt;function=get_current_date>",
+                        "begin": "<function=get_current_date>",
                        "schema": schema_get_current_date,
-                        "end": "&lt;/function&gt;",
+                        "end": "</function>",
                    },
                ],
-                "triggers": ["&lt;function="],
+                "triggers": ["<function="],
            }
        ),
    },
@@ -634,17 +634,17 @@ sampling_params = {
            "type": "structural_tag",
            "structures": [
                {
-                    "begin": "&lt;function=get_current_weather>",
+                    "begin": "<function=get_current_weather>",
                    "schema": schema_get_current_weather,
-                    "end": "&lt;/function&gt;",
+                    "end": "</function>",
                },
                {
-                    "begin": "&lt;function=get_current_date>",
+                    "begin": "<function=get_current_date>",
                    "schema": schema_get_current_date,
-                    "end": "&lt;/function&gt;",
+                    "end": "</function>",
                },
            ],
-            "triggers": ["&lt;function="],
+            "triggers": ["<function="],
        }
    ),
 }
@@ -60,7 +60,7 @@
    "server_process, port = launch_server_cmd(\n",
    "    \"python3 -m sglang.launch_server --model-path Qwen/Qwen2.5-7B-Instruct --tool-call-parser qwen25 --host 0.0.0.0 --log-level warning\"  # qwen25\n",
    ")\n",
-    "wait_for_server(f\"http://localhost:{port}\")"
+    "wait_for_server(f\"http://localhost:{port}\", process=server_process)"
   ]
  },
  {
@@ -550,7 +550,9 @@
    "server_process_tool_choice, port_tool_choice = launch_server_cmd(\n",
    "    \"python3 -m sglang.launch_server --model-path Qwen/Qwen2.5-7B-Instruct --tool-call-parser qwen25 --host 0.0.0.0  --log-level warning\"\n",
    ")\n",
-    "wait_for_server(f\"http://localhost:{port_tool_choice}\")\n",
+    "wait_for_server(\n",
+    "    f\"http://localhost:{port_tool_choice}\", process=server_process_tool_choice\n",
+    ")\n",
    "\n",
    "# Initialize client for tool choice examples\n",
    "client_tool_choice = OpenAI(\n",
@@ -695,7 +697,7 @@
    "server_process, port = launch_server_cmd(\n",
    "    \" python3 -m sglang.launch_server --model-path meta-llama/Llama-3.2-1B-Instruct --tool-call-parser pythonic --tp 1  --log-level warning\"  # llama-3.2-1b-instruct\n",
    ")\n",
-    "wait_for_server(f\"http://localhost:{port}\")\n",
+    "wait_for_server(f\"http://localhost:{port}\", process=server_process)\n",
    "\n",
    "tools = [\n",
    "    {\n",
@@ -64,8 +64,11 @@
    "\n",
    "nest_asyncio.apply()\n",
    "\n",
+    "import sglang.test.doc_patch  # noqa: F401\n",
+    "\n",
    "model_path = \"Qwen/Qwen2.5-VL-3B-Instruct\"\n",
-    "chat_template = \"qwen2-vl\""
+    "chat_template = \"qwen2-vl\"\n",
+    "example_image_url = \"https://raw.githubusercontent.com/sgl-project/sglang/main/examples/assets/example_image.png\""
   ]
  },
  {
@@ -81,13 +84,7 @@
    "\n",
    "from sglang.srt.parser.conversation import chat_templates\n",
    "\n",
-    "image = Image.open(\n",
-    "    BytesIO(\n",
-    "        requests.get(\n",
-    "            \"https://github.com/sgl-project/sglang/blob/main/examples/assets/example_image.png?raw=true\"\n",
-    "        ).content\n",
-    "    )\n",
-    ")\n",
+    "image = Image.open(BytesIO(requests.get(example_image_url).content))\n",
    "\n",
    "conv = chat_templates[chat_template].copy()\n",
    "conv.append_message(conv.roles[0], f\"What's shown here: {conv.image_token}?\")\n",
@@ -185,9 +182,8 @@
    "from transformers import Qwen2_5_VLForConditionalGeneration\n",
    "\n",
    "processor = AutoProcessor.from_pretrained(model_path, use_fast=True)\n",
-    "vision = (\n",
-    "    Qwen2_5_VLForConditionalGeneration.from_pretrained(model_path).eval().visual.cuda()\n",
-    ")"
+    "model = Qwen2_5_VLForConditionalGeneration.from_pretrained(model_path).eval()\n",
+    "vision = model.model.visual.cuda()"
   ]
  },
  {
@@ -206,6 +202,7 @@
    "precomputed_embeddings = vision(\n",
    "    processor_output[\"pixel_values\"].cuda(), processor_output[\"image_grid_thw\"].cuda()\n",
    ")\n",
+    "precomputed_embeddings = precomputed_embeddings.pooler_output\n",
    "\n",
    "multi_modal_item = dict(\n",
    "    processor_output,\n",
@@ -238,13 +235,7 @@
    "from sglang.srt.parser.conversation import chat_templates\n",
    "\n",
    "# Download the same example image\n",
-    "image = Image.open(\n",
-    "    BytesIO(\n",
-    "        requests.get(\n",
-    "            \"https://github.com/sgl-project/sglang/blob/main/examples/assets/example_image.png?raw=true\"\n",
-    "        ).content\n",
-    "    )\n",
-    ")\n",
+    "image = Image.open(BytesIO(requests.get(example_image_url).content))\n",
    "\n",
    "conv = chat_templates[chat_template].copy()\n",
    "conv.append_message(conv.roles[0], f\"What's shown here: {conv.image_token}?\")\n",
@@ -9,7 +9,6 @@ This tutorial demonstrates how to use SGLang's **offline Engine API** to query V
 2. **Processor Output**: Use HuggingFace processor for data preprocessing.
 3. **Precomputed Embeddings**: Pre-calculate image features to improve inference efficiency.

-
 ## Understanding the Three Input Formats

 SGLang supports three ways to pass visual data, each optimized for different scenarios:
@@ -35,21 +34,20 @@ SGLang supports three ways to pass visual data, each optimized for different sce

 The examples below demonstrate all three approaches with both Qwen2.5-VL and Llama 4 models.

-
 ## Querying Qwen2.5-VL Model

-
-
 ```python Example
 import nest_asyncio

 nest_asyncio.apply()

+import sglang.test.doc_patch  # noqa: F401
+
 model_path = "Qwen/Qwen2.5-VL-3B-Instruct"
 chat_template = "qwen2-vl"
+example_image_url = "https://raw.githubusercontent.com/sgl-project/sglang/main/examples/assets/example_image.png"
 ```

-
 ```python Example
 from io import BytesIO
 import requests
@@ -57,13 +55,7 @@ from PIL import Image

 from sglang.srt.parser.conversation import chat_templates

-image = Image.open(
-    BytesIO(
-        requests.get(
-            "https://github.com/sgl-project/sglang/blob/main/examples/assets/example_image.png?raw=true"
-        ).content
-    )
-)
+image = Image.open(BytesIO(requests.get(example_image_url).content))

 conv = chat_templates[chat_template].copy()
 conv.append_message(conv.roles[0], f"What's shown here: {conv.image_token}?")
@@ -78,16 +70,12 @@ image

 ### Basic Offline Engine API Call

-
-
 ```python Example
 from sglang import Engine

-
 llm = Engine(model_path=model_path, chat_template=chat_template, log_level="warning")
 ```

-
 ```python Example
 out = llm.generate(prompt=conv.get_prompt(), image_data=[image])
 print("Model response:")
@@ -98,8 +86,6 @@ print(out["text"])

 Using a HuggingFace processor to preprocess text and images, and passing the `processor_output` directly into `Engine.generate`.

-
-
 ```python Example
 from transformers import AutoProcessor

@@ -120,19 +106,15 @@ print(out["text"])

 You can pre-calculate image features to avoid repeated visual encoding processes.

-
-
 ```python Example
 from transformers import AutoProcessor
 from transformers import Qwen2_5_VLForConditionalGeneration

 processor = AutoProcessor.from_pretrained(model_path, use_fast=True)
-vision = (
-    Qwen2_5_VLForConditionalGeneration.from_pretrained(model_path).eval().visual.cuda()
-)
+model = Qwen2_5_VLForConditionalGeneration.from_pretrained(model_path).eval()
+vision = model.model.visual.cuda()
 ```

-
 ```python Example
 processor_output = processor(
    images=[image], text=conv.get_prompt(), return_tensors="pt"
@@ -143,6 +125,7 @@ input_ids = processor_output["input_ids"][0].detach().cpu().tolist()
 precomputed_embeddings = vision(
    processor_output["pixel_values"].cuda(), processor_output["image_grid_thw"].cuda()
 )
+precomputed_embeddings = precomputed_embeddings.pooler_output

 multi_modal_item = dict(
    processor_output,
@@ -170,13 +153,7 @@ from PIL import Image
 from sglang.srt.parser.conversation import chat_templates

 # Download the same example image
-image = Image.open(
-    BytesIO(
-        requests.get(
-            "https://github.com/sgl-project/sglang/blob/main/examples/assets/example_image.png?raw=true"
-        ).content
-    )
-)
+image = Image.open(BytesIO(requests.get(example_image_url).content))

 conv = chat_templates[chat_template].copy()
 conv.append_message(conv.roles[0], f"What's shown here: {conv.image_token}?")
@@ -190,7 +167,6 @@ print(f"Image size: {image.size}")
 image
 ```

-
 ### Llama 4 Basic Call

 Llama 4 requires more computational resources, so it's configured with multi-GPU parallelism (tp_size=4) and larger context length.
@@ -209,7 +185,6 @@ print("Llama 4 response:")
 print(out["text"])
 ```

-
 ### Call with Processor Output

 Using HuggingFace processor to preprocess data can reduce computational overhead during inference.
@@ -230,7 +205,6 @@ print("Response using processor output:")
 print(out)
 ```

-
 ### Call with Precomputed Embeddings

 ```python Example