[Feature] Add --attn-dp-size and deprecate --enable-dp-attention (#41818)

This commit is contained in:
Cheng Wan
2026-09-30 13:34:18 -07:00
committed by GitHub
parent 201c4b9b2a
commit a9c97c9f69
366 changed files with 1697 additions and 1900 deletions
@@ -134,7 +134,7 @@ Each cell describes one verified (or auto-estimated) launch recipe.
- DO NOT include `--host` / `--port` literally — use `{{HOST_IP}}` /
`{{PORT}}` placeholders so users can override through the Env modal.
- Order flags as: `--model-path` first (after any `--trust-remote-code`),
then parallelism (`--tp`, `--dp`, `--enable-dp-attention`), then MoE
then parallelism (`--tp`, `--attn-dp-size`), then MoE
flags, then tuning knobs, with `--host` / `--port` last. The playground
engine assumes this ordering when inserting overrides (its anchors target
`--model-path` / `--tp` / etc., and inserts before the `--host` tail).
@@ -171,7 +171,7 @@ schemas (full reference in the `_playground.jsx` header):
| Axis key | Widget | Use when |
|---|---|---|
| `attention` | TP / CP / DP-Attention sub-knobs (DP-Attention is a combined knob: its value is the DP degree AND toggles `--enable-dp-attention`) | Model exposes parallelism knobs in its cells (§2.2) and you want users to override them. |
| `attention` | TP / CP / DP-Attention sub-knobs (DP-Attention is a single knob: its value is the attention DP size, emitted as `--attn-dp-size N`) | Model exposes parallelism knobs in its cells (§2.2) and you want users to override them. |
| `moe` | Backend select (incl. MegaMoE) + EP knob; picking the MegaMoE backend reveals a Quantization sub-select (W4A8/W4A4) | Model is MoE and supports multiple `--moe-*-backend` choices. For Blackwell MoE kernel-fusion, give the `megamoe` backend option a `requiresHw` (and optional `excludesStrategy`) gate, then add a sibling `megamoeQuant` block (`{stripEnv, options}`): W4A8 = `NUM_MAX` only, W4A4 adds the FP4-activations env vars; both strip the DeepEP dispatch env. |
| `parsers` | Multi-toggle | Model has reasoning / tool-call parsers. |
| `speculative` | Single-select chip group | Model has spec-decoding presets you want to expose. |
@@ -208,7 +208,7 @@ schemas (full reference in the `_playground.jsx` header):
knobs/values that are disabled under the live facts, so stale picks never
emit a blocked combination. Interleave prefill-CP + DP-Attention is
deliberately NOT grayed (combined support is planned upstream even though
current releases assert `dp_size == 1` for interleave) — the engine shows
current releases assert `attn_dp_size == 1` for interleave) — the engine shows
a warning hint below the command box instead.
- The CP knob emits `--attn-cp-size N --enable-prefill-cp --cp-strategy S`,
where S is: an optional `{ id: "cpStrategy", values: [null, "interleave",
@@ -176,7 +176,7 @@ sgl-eval run gsm8k \\
playgroundFeatures: {
// ----- Card: "Attention Parallelism" ----- KEEP if the model exposes TP/CP/DP
// knobs. DP-Attention is a combined knob: value = DP degree AND toggles `--enable-dp-attention`.
// knobs. DP-Attention is a single knob: value = attention DP size, emitted as `--attn-dp-size N`.
attention: {
knobs: [
{ id: "tp", label: "TP", values: [
@@ -357,8 +357,7 @@ sgl-eval run gsm8k \\
"--trust-remote-code",
"--model-path {{MODEL_NAME}}",
"--tp 4",
"--dp 4",
"--enable-dp-attention",
"--attn-dp-size 4",
"--moe-a2a-backend deepep",
"--host {{HOST_IP}}",
"--port {{PORT}}",
@@ -403,8 +402,7 @@ sgl-eval run gsm8k \\
"--trust-remote-code",
"--model-path {{MODEL_NAME}}",
"--tp 16",
"--dp 16",
"--enable-dp-attention",
"--attn-dp-size 16",
"--moe-a2a-backend deepep",
"--host {{HOST_IP}}",
"--port {{PORT}}",
@@ -118,7 +118,7 @@ that **changes other parts of the command** (TP, mem) must (the Playground
can't do coupled changes), and so does a toggle the legacy page itself labels
with operating-point words — e.g. a `dpattention` radio whose options are
subtitled "Low Latency" / "High Throughput" (GLM-5.1 / Kimi-K2.6 pattern) —
even when its flags are uncoupled (`--dp N --enable-dp-attention` is a pure
even when its flags are uncoupled (`--attn-dp-size N` is a pure
flag add). Any other toggle that only adds/removes its own flags becomes a
Playground axis with the flags baked into cells when the legacy default was
ON — EXCEPT parsers:
@@ -12,7 +12,7 @@ this file is about the *mapping decisions*.
| hardware radio | `match.hw` | Catalog ids as-is. Off-catalog hardware → `config.hardware` entry — e.g. A100 `{id:"a100", label:"A100", vram:"80GB", vendor:"nvidia"}` (merges into the NVIDIA row), Xeon `{id:"xeon", label:"Xeon", vram:"host RAM", vendor:"intel"}` (engine renders a new INTEL row; any vendor key works). A merged chip like GLM-5's "MI300X/MI325X" splits into two ids with duplicated cells (cells are denormalized by design). |
| model-size / model-name radio | `variants` | One variant per deployable checkpoint family; single `{id:"default"}` when there's no variant axis (then `modelNames` keys drop the variant half). |
| quantization radio | `quantizations` | Real precision ids (`bf16`/`fp8`/`fp4`/`int4`/…). One `fp4` id even when checkpoints differ per vendor — route via `hw\|variant\|quant` triple keys in `modelNames` (NVFP4 on Blackwell vs AMD MXFP4 is the precedent); per-hw greying falls out of which cells exist. |
| toggle that **couples** with other parts of the command (changes TP/mem/EP), OR one the legacy page labels with **operating-point words** | `strategies` | The Playground applies pure flag diffs — it cannot do coupled changes. Example: Qwen3.5's MTP toggle bumps TP on three H100 combos → strategies `low-latency` (MTP on) / `high-throughput` (MTP off). **Naming counts like coupling**: GLM-5.1's / Kimi-K2.6's `dpattention` adds only `--dp N --enable-dp-attention` (uncoupled), but its options are subtitled "Low Latency" / "High Throughput" — the page's own named operating-point split → strategies; a flag-only spec toggle riding alongside it stays a Playground axis and bakes per its legacy default. GPU-count radios (GLM-4.7, MiniMax-M2.5/2.7) → budget-tier strategies with the legacy SUPPORT matrix preserved by which cells exist. Strategy count follows the page's operating points: 1 → `balanced`, 2 → `low-latency`+`high-throughput`, 3 → the full trio (§4). |
| toggle that **couples** with other parts of the command (changes TP/mem/EP), OR one the legacy page labels with **operating-point words** | `strategies` | The Playground applies pure flag diffs — it cannot do coupled changes. Example: Qwen3.5's MTP toggle bumps TP on three H100 combos → strategies `low-latency` (MTP on) / `high-throughput` (MTP off). **Naming counts like coupling**: GLM-5.1's / Kimi-K2.6's `dpattention` adds only `--attn-dp-size N` (uncoupled), but its options are subtitled "Low Latency" / "High Throughput" — the page's own named operating-point split → strategies; a flag-only spec toggle riding alongside it stays a Playground axis and bakes per its legacy default. GPU-count radios (GLM-4.7, MiniMax-M2.5/2.7) → budget-tier strategies with the legacy SUPPORT matrix preserved by which cells exist. Strategy count follows the page's operating points: 1 → `balanced`, 2 → `low-latency`+`high-throughput`, 3 → the full trio (§4). |
| toggle that only adds/removes its own flags | Playground axis (+ bake, EXCEPT parsers and accuracy-degrading flags) | **Parsers (`--reasoning-parser` / `--tool-call-parser`) are NEVER baked into cells** — Deployment commands ship without them regardless of the legacy default or the measured command; the `parsers` axis adds them on top (DSv4 convention; cells mirror the legacy generator's parsers-OFF output). Accuracy-degrading toggles are never baked either — §2 caveats (axis-only, accuracy-safe cells). Other flag-only toggles: legacy default ON → bake into cells AND declare the axis so users can strip (red strikethrough); default OFF → keep cells clean, axis preset only. MTP/EAGLE presets → `speculative` axis; dp-attention → a strategy when the legacy page labels it as the operating-point split or when coupled (see the row above), else `attention.dpAttn`. **EVERY legacy control survives as an interactive control** (a dimension or a Playground axis), never a tips-only mention — but a model-specific control is **config DATA, not engine code**: the axis handler reads options/flags/env/gating straight from `config.playgroundFeatures` (MegaMoE W4A4 is entirely DSv4 config data on the existing `moe` axis — no per-model engine edit). A control that fits an existing axis's data schema is therefore pure config, full stop. A **titled single-select that strips a flag family** (e.g. Nemotron3's "KV Cache DType" `--kv-cache-dtype`) is covered by the merged generic **`flagSelects`** axis → **config-only**: declare a `flagSelects` list of `{ id, title, stripPrefixes, options }` (see the Qwen3.5 mamba example), **no engine PR**. Only a control whose *shape* `flagSelects` still can't express would need a new ONE-TIME generic primitive (never a model-named handler) on a prior engine PR; the backward-compat reasoning (opt-in per key, not in the opt-out set) is in engine-axis.md. |
| per-combo hidden option (e.g. spec hidden on Xeon) | absent cells | Don't create cells for combos the legacy widget couldn't produce; the engine greys them automatically. `# Error:` pseudo-commands → no cell + explanation in §2 tips and/or a chip `disable`/`disableReason`. |
| coupled secondary knob (e.g. mamba cache V1/V2) | cells + Playground axis | Bake the correct value per cell following the legacy coupling (Qwen3.5: MTP ⇒ `--mamba-radix-cache-strategy extra_buffer` on NVIDIA; AMD/Xeon ⇒ V1/no flag) and document the coupling in §2 tips — AND surface the knob as a Playground axis like every other legacy feature (row above; add the axis when none fits). Baking alone is NOT enough — the every-feature rule supersedes the pilot's cells+prose-only treatment of Qwen3.5's mamba knob (retrofit pending). The mamba knob is the same single-select shape as KV Cache DType, so it rides the merged generic **`flagSelects`** axis — Qwen3.6 / Qwen3-Coder-Next declare it purely in config (a `flagSelects` block), **no engine PR**. |
@@ -30,7 +30,7 @@ this file is about the *mapping decisions*.
| (absent) | append `--host {{HOST_IP}}`, `--port {{PORT}}` to every cell |
| `--nnodes N --node-rank … --dist-init-addr …` literals | delete; `match.nodes: "multi-N"` + `nodesOptions` entry — the engine injects the trio after the last parallelism anchor plus the multi-node header comment |
| env-var command prefixes | verbatim into `cell.env[]` (never drop/normalize) |
| flag order as emitted | re-sort to canonical: `--trust-remote-code` → `--model-path` → parallelism (`--tp`/`--dp`/`--enable-dp-attention`/EP) → MoE → tuning → `--host`/`--port` (Playground insert anchors assume this). Keep the legacy relative order within the tuning span so commands stay eyeball-diffable. |
| flag order as emitted | re-sort to canonical: `--trust-remote-code` → `--model-path` → parallelism (`--tp`/`--attn-dp-size`/EP) → MoE → tuning → `--host`/`--port` (Playground insert anchors assume this). Keep the legacy relative order within the tuning span so commands stay eyeball-diffable. |
Caveats discovered in the pilot:
- The Playground `moe.ep` knob only understands `--ep` — normalize a legacy
@@ -227,7 +227,7 @@ slack (`>= len(...) - 1`) or compared key names instead of value sources.
read-through over the canonical getters, so they answer with the live process
groups. Everything else — `tp_size`, `pp_size`, `attn_cp_size`, `dcp_size`,
`moe_dp_size` included, alongside config-only leaves like `nccl_port`,
`enable_dp_attention`, `dp_size`, `ep_size`, `dwdp_size` — is answered from the
`dp_size`, `ep_size`, `dwdp_size` — is answered from the
published `parallel` bag. Reading a leaf before publish raises a `ValueError`
naming the namespace; an unknown name is an `AttributeError`.
+1 -1
View File
@@ -47,7 +47,7 @@ Add [performance optimization options](#performance-optimization-options) as nee
[MLA optimizations](https://lmsys.org/blog/2024-09-04-sglang-v0-3/#deepseek-multi-head-latent-attention-mla-throughput-optimizations) are enabled by default. Here are some optional optimizations can be enabled as needed.
- [Data Parallelism Attention](https://lmsys.org/blog/2024-12-04-sglang-v0-4/#data-parallelism-attention-for-deepseek-models): For high QPS scenarios, add the `--enable-dp-attention` argument to boost throughput.
- [Data Parallelism Attention](https://lmsys.org/blog/2024-12-04-sglang-v0-4/#data-parallelism-attention-for-deepseek-models): For high QPS scenarios, add the `--attn-dp-size <N>` argument (e.g. equal to `--tp`) to boost throughput.
- [Torch.compile Optimization](https://lmsys.org/blog/2024-09-04-sglang-v0-3/#torchcompile-latency-optimizations): Add `--enable-torch-compile` argument to enable it. This will take some time while server starts. The maximum batch size for torch.compile optimization can be controlled with `--torch-compile-max-bs`. It's recommended to set it between `1` and `8`. (e.g., `--torch-compile-max-bs 8`)
### Usage: Chat with DeepSeek
@@ -52,7 +52,7 @@ This section provides deployment configurations optimized for different hardware
**DP Attention:**
- Enable DP attention for high-throughput scenarios
- The `--dp` value commonly matches the `--tp` value
- The `--attn-dp-size` value commonly matches the `--tp` value
- Trade-off: Higher throughput at the cost of slightly increased latency
## 4. Model Invocation
@@ -192,8 +192,6 @@ sglang serve \
--enable-multimodal \
--tp 1 \
--ep 1 \
--dp 1 \
--enable-dp-attention \
--host 0.0.0.0 \
--port 30000
```
@@ -184,8 +184,6 @@ python3 -m sglang.launch_server \
--model-path deepseek-ai/DeepSeek-OCR \
--tp 1 \
--ep 1 \
--dp 1 \
--enable-dp-attention \
--dtype float16 \
--host 0.0.0.0 \
--port 8000
@@ -95,7 +95,7 @@ DeepSeek-R1 shares the same MoE architecture as DeepSeek-V3, so the same hardwar
python3 -m sglang.compile_deep_gemm --model deepseek-ai/DeepSeek-R1 --tp 8 --trust-remote-code
```
**Data Parallelism Attention (`--enable-dp-attention`):** Recommended for high-throughput scenarios. Use `--enable-dp-attention --tp 8 --dp 8` on a single 8-GPU node.
**Data Parallelism Attention (`--attn-dp-size`):** Recommended for high-throughput scenarios. Use `--tp 8 --attn-dp-size 8` on a single 8-GPU node.
**NCCL timeout:** If model loading is slow, increase: `--dist-timeout 3600`.
@@ -75,7 +75,7 @@ python3 -m sglang.compile_deep_gemm --model deepseek-ai/DeepSeek-V3 --tp 8 --tru
```
DeepGEMM is enabled by default on Hopper/Blackwell and can be disabled with `SGLANG_ENABLE_JIT_DEEPGEMM=0`.
**Data Parallelism Attention (`--enable-dp-attention`):** Recommended for high-throughput scenarios with large batch sizes. Reduces KV-cache duplication across TP ranks. Use `--enable-dp-attention --tp 8 --dp 8` on a single 8-GPU node. Not recommended for low-latency, small-batch workloads.
**Data Parallelism Attention (`--attn-dp-size`):** Recommended for high-throughput scenarios with large batch sizes. Reduces KV-cache duplication across TP ranks. Use `--tp 8 --attn-dp-size 8` on a single 8-GPU node. Not recommended for low-latency, small-batch workloads.
**NCCL timeout:** If model loading is slow and you hit an NCCL timeout, increase it: `--dist-timeout 3600`.
@@ -454,8 +454,7 @@ We use SGLang's built-in benchmarking tool to conduct performance evaluation on
python3 -m sglang.launch_server \
--model-path deepseek-ai/DeepSeek-V3 \
--tp 8 \
--dp 8 \
--enable-dp-attention \
--attn-dp-size 8 \
--speculative-algorithm EAGLE \
--speculative-num-steps 3 \
--speculative-eagle-topk 1 \
@@ -529,8 +528,7 @@ python3 -m sglang.launch_server \
--model-path deepseek-ai/DeepSeek-V3 \
--tp 8 \
--ep 8 \
--dp 8 \
--enable-dp-attention \
--attn-dp-size 8 \
--host 0.0.0.0 \
--port 8000
```
@@ -73,7 +73,7 @@ python3 -m sglang.compile_deep_gemm --model deepseek-ai/DeepSeek-V3.1 --tp 8 --t
```
DeepGEMM is enabled by default on Hopper/Blackwell and can be disabled with `SGLANG_ENABLE_JIT_DEEPGEMM=0`.
**Data Parallelism Attention (`--enable-dp-attention`):** Recommended for high-throughput scenarios with large batch sizes. Reduces KV-cache duplication across TP ranks. Use `--enable-dp-attention --tp 8 --dp 8` on a single 8-GPU node. Not recommended for low-latency, small-batch workloads.
**Data Parallelism Attention (`--attn-dp-size`):** Recommended for high-throughput scenarios with large batch sizes. Reduces KV-cache duplication across TP ranks. Use `--tp 8 --attn-dp-size 8` on a single 8-GPU node. Not recommended for low-latency, small-batch workloads.
**NCCL timeout:** If model loading is slow and you hit an NCCL timeout, increase it: `--dist-timeout 3600`.
@@ -173,7 +173,7 @@ DeepSeek-V3.2 and DeepSeek-V3.2-Exp support tool calling capabilities. But they
python3 -m sglang.launch_server \
--model-path deepseek-ai/DeepSeek-V3.2-Speciale \
--trust-remote-code \
--tp-size 8 --dp-size 8 --enable-dp-attention \
--tp-size 8 --attn-dp-size 8 \
--reasoning-parser deepseek-v3
```
@@ -357,8 +357,7 @@ SGLang implements Multi-Token Prediction (MTP) for DeepSeek V3.2 based on [EAGLE
**With DP Attention:**
```bash Command
python -m sglang.launch_server --model deepseek-ai/DeepSeek-V3.2-Exp --tp 8 --dp 8 \
--enable-dp-attention \
python -m sglang.launch_server --model deepseek-ai/DeepSeek-V3.2-Exp --tp 8 --attn-dp-size 8 \
--speculative-algorithm EAGLE \
--speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4
```
@@ -394,8 +393,7 @@ python -m sglang.launch_server \
--host $LOCAL_IP \
--port $PORT \
--tp 8 \
--dp 8 \
--enable-dp-attention \
--attn-dp-size 8 \
--dist-init-addr ${HOST}:${DIST_PORT} \
--trust-remote-code \
--disaggregation-bootstrap-port 8998 \
@@ -411,8 +409,7 @@ python -m sglang.launch_server \
--host $LOCAL_IP \
--port $PORT \
--tp 8 \
--dp 8 \
--enable-dp-attention \
--attn-dp-size 8 \
--dist-init-addr ${HOST}:${DIST_PORT} \
--trust-remote-code \
--mem-fraction-static 0.9
@@ -435,7 +432,7 @@ For production deployments (RBG / LWS-based, DeepEP EP parallelism), see [multi_
Enable prefill context parallelism (CP) on CUDA with `--enable-prefill-cp --cp-strategy interleave` for long-sequence workloads.
<Warning>
Zigzag prefill CP (`--cp-strategy zigzag`) is temporarily unavailable for DeepSeek V3.2, GLM-5, GLM-5.1, GLM-5.2, and GLM-5.3. Use `interleave` for these models and keep `--dp 1`; interleave DSA CP does not support `--dp` greater than 1.
Zigzag prefill CP (`--cp-strategy zigzag`) is temporarily unavailable for DeepSeek V3.2, GLM-5, GLM-5.1, GLM-5.2, and GLM-5.3. Use `interleave` for these models and keep `--attn-dp-size 1`; interleave DSA CP does not support `--attn-dp-size` greater than 1.
</Warning>
**Interleave** (`--cp-strategy interleave`): Distributes tokens by `token_idx % cp_size`. It supports fused MoE, FP8 KV cache, and multi-batch prefill.
@@ -462,7 +459,7 @@ export SGLANG_SET_CPU_AFFINITY=1
sglang_args=$(echo serve \
--model-path $MAPPED_MODEL_PATH \
--nccl-init $MASTER_ADDR:$MASTER_PORT --nnodes 2 --node-rank $RANK --tp 16 \
--dp 2 --enable-dp-attention --page-size 64 \
--attn-dp-size 2 --page-size 64 \
--trust-remote-code --host "0.0.0.0" --port 30000 \
--log-requests \
--context-length 65536 --max-running-requests 128 \
@@ -478,7 +475,7 @@ sglang_args=($sglang_args)
sglang "${sglang_args[@]}" 2>&1 | tee $LOG_DIR/$RANK.log
```
**CP + PP + EP + DP**
**CP + PP + EP**
`CP` is currently enabled with `PP=2` on Hopper platform and we can reduce TP=16 to TP=8 from standalone deployment:
@@ -486,7 +483,7 @@ sglang "${sglang_args[@]}" 2>&1 | tee $LOG_DIR/$RANK.log
# verified on Hopper platform
sglang_args=$(echo serve \
--model-path $MAPPED_MODEL_PATH \
--nccl-init $MASTER_ADDR:$MASTER_PORT --nnodes 2 --node-rank $RANK --tp 8 --pp-size 2 --dp 1 --enable-dp-attention \
--nccl-init $MASTER_ADDR:$MASTER_PORT --nnodes 2 --node-rank $RANK --tp 8 --pp-size 2 \
--moe-a2a-backend deepep --ep-size 16 \
--page-size 128 \
--chunked-prefill-size 16384 \
@@ -502,7 +499,7 @@ sglang_args=$(echo serve \
--context-length 65536 \
--allow-auto-truncate --enable-metrics \
--tool-call-parser deepseekv32 --reasoning-parser deepseek-v3 \
--served-model-name DeepSeek-V3.2-dsa-pp-cp-ep-dp
--served-model-name DeepSeek-V3.2-dsa-pp-cp-ep
)
sglang_args=($sglang_args)
@@ -512,14 +509,10 @@ sglang "${sglang_args[@]}" 2>&1 | tee $LOG_DIR/$RANK.log
**fp8 KV + CP + PP**
FP8 KV reduces the memory footprint. For DeepSeek V3.2, combine it with interleave CP and PP while keeping `--dp 1`:
FP8 KV reduces the memory footprint. For DeepSeek V3.2, combine it with interleave CP and PP, without attention DP:
```shell Command
# verified in Hopper platform
dp_config=" \
--dp 1 --enable-dp-attention \
"
cp_config=" \
--enable-prefill-cp \
--cp-strategy interleave \
@@ -529,7 +522,6 @@ cp_config=" \
sglang_args=$(echo serve \
--model-path $MAPPED_MODEL_PATH \
--nccl-init $MASTER_ADDR:$MASTER_PORT --nnodes 2 --node-rank $RANK --tp 8 --pp-size 2 --pp-async-batch-depth 1 \
$dp_config \
--trust-remote-code --host "0.0.0.0" --port 30000 \
--log-requests \
--context-length 65536 --max-running-requests 128 \
@@ -641,8 +633,7 @@ sglang serve \
--model-path deepseek-ai/DeepSeek-V3.2-Exp \
--tp 8 \
--ep 8 \
--dp 8 \
--enable-dp-attention \
--attn-dp-size 8 \
--host 0.0.0.0 \
--port 30000
```
@@ -875,7 +866,7 @@ export SGLANG_SET_CPU_AFFINITY=1
sglang_args=$(echo serve \
--model-path $MAPPED_MODEL_PATH \
--nccl-init $MASTER_ADDR:$MASTER_PORT --nnodes 2 --node-rank $RANK --tp 16 \
--dp 2 --enable-dp-attention --page-size 64 \
--attn-dp-size 2 --page-size 64 \
--trust-remote-code --host "0.0.0.0" --port 30000 \
--log-requests \
--context-length 65536 --max-running-requests 128 \
@@ -390,7 +390,7 @@ HiCache and MegaMoE are **not** supported on RTX PRO 6000.
- **Model checkpoints** — for correct accuracy, the FP4 model uses the stock `deepseek-ai/DeepSeek-V4-{Flash,Pro}`, and the FP8 model uses the repackaged `sgl-project/DeepSeek-V4-{Flash,Pro}-FP8`.
- **Supported models** — **MI300X** supports DeepSeek-V4-Flash in FP8; **MI355X** supports DeepSeek-V4-Flash / Pro in both FP4 and FP8. All recipes run single-node.
- **TP / DP setting (MI355X)** — both TP=4 and TP=8 are supported. At low concurrency we recommend **TP-only**; at high concurrency use **TP + DP** (balanced / high-throughput recipes). The verified MI355X DP recipes additionally set `--dp 8 --enable-dp-attention --enable-dp-attention-local-control-broadcast --tokenizer-worker-num 8 --stream-interval 20 --prefill-decode-interval 10`.
- **TP / DP setting (MI355X)** — both TP=4 and TP=8 are supported. At low concurrency we recommend **TP-only**; at high concurrency use **TP + DP** (balanced / high-throughput recipes). The verified MI355X DP recipes additionally set `--attn-dp-size 8 --enable-dp-attention-local-control-broadcast --tokenizer-worker-num 8 --stream-interval 20 --prefill-decode-interval 10`.
- **MTP** — speculative decoding on the original Flash / Pro checkpoints; add `--speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4`. The 0813 Pro Official checkpoint keeps its MTP head too, so the same flags work there as a fallback to DSpark.
- **DSpark (MI355X Pro Official 0813)** — the 0813 checkpoint bundles the DSpark draft head. Prefer it over EAGLE on 0813 for ordinary serving: enable `--speculative-algorithm DSPARK` (low-latency) and see the [MI355X agentic recipe](#3-7-agentic-long-context-with-hicache-dram-offload-mi355x-fp4-dspark) for the long-context path. DSpark also runs under **PD disaggregation** — see the Pro Official pairs in [§3.8](#3-8-pd-disaggregation-on-mi355x-mori-io).
- **Kernels** — uses the Unified KV attention and the flydsl MoE.
@@ -408,7 +408,7 @@ export SGLANG_MORI_RECV_BOUND=1
sglang serve \
--model-path {MODEL_PATH} \
--tp 8 --dp 8 --enable-dp-attention \
--tp 8 --attn-dp-size 8 \
--ep-size 8 --moe-a2a-backend mori --deepep-mode normal \
<other args from the Deploy panel>
```
@@ -769,8 +769,7 @@ python3 -m sglang.launch_server \
--model-path deepseek-ai/DeepSeek-V4-Pro-0813 \
--trust-remote-code \
--tp 8 \
--dp 8 \
--enable-dp-attention \
--attn-dp-size 8 \
--enable-dp-lm-head \
--ep-size 8 \
--moe-a2a-backend megamoe \
@@ -792,7 +791,7 @@ python3 -m sglang.launch_server \
--hicache-mem-layout page_first_direct
```
`--chunked-prefill-size` is a global budget divided by `--dp`, so this keeps 6144 tokens per rank.
`--chunked-prefill-size` is a global budget divided by `--attn-dp-size`, so this keeps 6144 tokens per rank.
**Disaggregated 1P1D (DEP8 prefill / DEP8 decode), concurrency 64–128.** Prefill, on one 8-GPU node:
```bash Command
@@ -804,8 +803,7 @@ python3 -m sglang.launch_server \
--model-path deepseek-ai/DeepSeek-V4-Pro-0813 \
--trust-remote-code \
--tp 8 \
--dp 8 \
--enable-dp-attention \
--attn-dp-size 8 \
--enable-dp-lm-head \
--ep-size 8 \
--moe-dense-tp-size 1 \
@@ -841,8 +839,7 @@ python3 -m sglang.launch_server \
--model-path deepseek-ai/DeepSeek-V4-Pro-0813 \
--trust-remote-code \
--tp 8 \
--dp 8 \
--enable-dp-attention \
--attn-dp-size 8 \
--enable-dp-lm-head \
--ep-size 8 \
--moe-dense-tp-size 1 \
@@ -941,8 +938,7 @@ python3 -m sglang.launch_server \
--model-path deepseek-ai/DeepSeek-V4-Pro-0813 \
--trust-remote-code \
--tp 8 \
--dp 8 \
--enable-dp-attention \
--attn-dp-size 8 \
--enable-dp-lm-head \
--enable-prefill-delayer \
--enable-dp-attention-local-control-broadcast \
@@ -971,7 +967,7 @@ python3 -m sglang.launch_server \
--watchdog-timeout 3600
```
`--chunked-prefill-size` is a global budget divided by `--dp`, so this keeps 8192 tokens per rank. `--enable-dp-lm-head` is required for DSpark under DP attention. Front the DP ranks with `sglang_router --policy cache_aware` so multi-turn sessions land on the rank holding the longest radix/HiCache prefix; above concurrency 160 add `--balance-abs-threshold 32`.
`--chunked-prefill-size` is a global budget divided by `--attn-dp-size`, so this keeps 8192 tokens per rank. `--enable-dp-lm-head` is required for DSpark under DP attention. Front the DP ranks with `sglang_router --policy cache_aware` so multi-turn sessions land on the rank holding the longest radix/HiCache prefix; above concurrency 160 add `--balance-abs-threshold 32`.
**DP8 EP8 MegaMoE, concurrency 128–384:**
```bash Command
@@ -996,9 +992,8 @@ python3 -m sglang.launch_server \
--model-path deepseek-ai/DeepSeek-V4-Pro-0813 \
--trust-remote-code \
--tp 8 \
--dp 8 \
--attn-dp-size 8 \
--ep 8 \
--enable-dp-attention \
--enable-dp-lm-head \
--enable-prefill-delayer \
--enable-dp-attention-local-control-broadcast \
+1 -1
View File
@@ -101,7 +101,7 @@ import { GLM51Deployment } from '/src/snippets/autoregressive/glm-51-deployment.
- **AMD GPUs**: BF16 and FP8 checkpoints run on MI300X/MI325X/MI355X at tp=8. On MI355X (gfx950), the MXFP4 checkpoint `amd/GLM-5.1-MXFP4` is also supported at tp=4 with `--kv-cache-dtype fp8_e4m3`. All AMD paths pass `--dsa-prefill-backend tilelang --dsa-decode-backend tilelang`, `--chunked-prefill-size 131072`, and `--watchdog-timeout 1200` (20 minutes for weight loading). FP8 uses approximately half the memory of BF16 (~89 GB/GPU vs ~175 GB/GPU). EAGLE speculative decoding is supported on AMD GPUs: MI300X/MI325X (gfx942) and MI355X (gfx950), but it **requires `--disable-custom-all-reduce`** — the aiter custom all-reduce kernel deadlocks during EAGLE verify at high concurrency, so without this flag the server will hang.
- For other configuration tips (MTP, DSA kernel, Context Parallel, HiSparse, NVFP4, Index Cache), see the [DeepSeek-V3.2 cookbook page](../DeepSeek/DeepSeek-V3_2). GLM-5.1 and DeepSeek-V3.2 share the same model structure, so the optimization techniques are common.
- **Prefill CP on CUDA**: Zigzag (`--cp-strategy zigzag`) is temporarily unavailable for GLM-5.1. Use `--enable-prefill-cp --cp-strategy interleave` with `--dp 1`.
- **Prefill CP on CUDA**: Zigzag (`--cp-strategy zigzag`) is temporarily unavailable for GLM-5.1. Use `--enable-prefill-cp --cp-strategy interleave` with `--attn-dp-size 1`.
- Use `--json-model-override-args '{"index_topk_pattern": "FFSFSSSFSSFFFSSSFFFSFSSSSSSFFSFFSFFSSFFFFFFSFFFFFSFFSSSSSSFSFFFSFSSSFSFFSFFSSS"}'` to enable the [IndexCache](https://github.com/THUDM/IndexCache) method for GLM-5.1. This can improve serving efficiency with only a small accuracy loss. If you are running rigorous accuracy evaluations, do not enable this feature.
+1 -1
View File
@@ -255,7 +255,7 @@ For the full setup (streaming, tool-use, count_tokens, persisting env in `~/.cla
### 3.5 Context Parallelism
<Warning>
Zigzag prefill CP (`--cp-strategy zigzag`) is temporarily unavailable for GLM-5.2. For prefill CP on CUDA, use `interleave` with `--dp 1` as shown below.
Zigzag prefill CP (`--cp-strategy zigzag`) is temporarily unavailable for GLM-5.2. For prefill CP on CUDA, use `interleave` with `--attn-dp-size 1` as shown below.
</Warning>
Prefill context parallelism can help with reduction of TTFT under long context. To enable prefill context parallelism for GLM 5.2, please append the following arguments:
+1 -1
View File
@@ -243,7 +243,7 @@ For the full setup (streaming, tool-use, count_tokens, persisting env in `~/.cla
### 3.5 Context Parallelism
<Warning>
Zigzag prefill CP (`--cp-strategy zigzag`) is temporarily unavailable for GLM-5.3. For prefill CP on CUDA, use `interleave` with `--dp 1` as shown below.
Zigzag prefill CP (`--cp-strategy zigzag`) is temporarily unavailable for GLM-5.3. For prefill CP on CUDA, use `interleave` with `--attn-dp-size 1` as shown below.
</Warning>
Prefill context parallelism can help with reduction of TTFT under long context. To enable prefill context parallelism for GLM 5.3, please append the following arguments:
+1 -1
View File
@@ -94,7 +94,7 @@ import { GLM5Deployment } from '/src/snippets/autoregressive/glm-5-deployment.js
- **AMD GPUs**: Use `--dsa-prefill-backend tilelang --dsa-decode-backend tilelang` for the DSA attention backend. Add `--chunked-prefill-size 131072` and `--watchdog-timeout 1200` (20 minutes for weight loading). EAGLE speculative decoding is not currently supported on AMD for GLM-5.
- For other configuration tips (MTP, DSA kernel, Context Parallel, HiSparse, NVFP4, Index Cache), see the [DeepSeek-V3.2 cookbook page](../DeepSeek/DeepSeek-V3_2). GLM-5 and DeepSeek-V3.2 share the same model structure, so the optimization techniques are common.
- **Prefill CP on CUDA**: Zigzag (`--cp-strategy zigzag`) is temporarily unavailable for GLM-5. Use `--enable-prefill-cp --cp-strategy interleave` with `--dp 1`.
- **Prefill CP on CUDA**: Zigzag (`--cp-strategy zigzag`) is temporarily unavailable for GLM-5. Use `--enable-prefill-cp --cp-strategy interleave` with `--attn-dp-size 1`.
- Use `--json-model-override-args '{"index_topk_pattern": "FFSFSSSFSSFFFSSSFFFSFSSSSSSFFSFFSFFSSFFFFFFSFFFFFSFFSSSSSSFSFFFSFSSSFSFFSFFSSS"}'` for GLM-5-FP8 if you want to enable the [IndexCache](https://github.com/THUDM/IndexCache) method. This feature is supported through [this PR](https://github.com/sgl-project/sglang/pull/21405) and introduces only a small accuracy loss. However, if you are running rigorous accuracy evaluations, it is not recommended to enable this feature.
@@ -40,7 +40,7 @@ import { KimiK25Deployment } from '/src/snippets/autoregressive/kimi-k25-deploym
- **Memory**: Requires GPUs with >=140GB each. Supported platforms: H200 (8x, TP=8), B300 (8x, TP=8), GB300 (4x, TP=4), MI300X/MI325X (4x, TP=4), MI350X/MI355X (4x, TP=4). Use `--context-length 128000` to conserve memory.
- **AMD GPU TP Constraint**: On AMD GPUs, TP must be &lt;= 4 (not 8). Kimi-K2.5 has 64 attention heads; the AITER MLA kernel requires `heads_per_gpu % 16 == 0`. With TP=4, each GPU gets 16 heads (valid). With TP=8, each GPU gets 8 heads (invalid).
- **AMD Docker Image**: Use `lmsysorg/sglang:v0.5.9-rocm700-mi35x` for MI350X/MI355X and `lmsysorg/sglang:v0.5.9-rocm700-mi30x` for MI300X/MI325X. The ROCm 7.2 images (`rocm720`) have an AITER compatibility issue.
- **DP Attention**: Enable with `--dp <N> --enable-dp-attention` for production throughput. A common choice is to set `--dp` equal to `--tp`, but this is not required.
- **DP Attention**: Enable with `--attn-dp-size <N>` for production throughput. A common choice is to set `--attn-dp-size` equal to `--tp`, but this is not required.
- **Reasoning Parser**: Add `--reasoning-parser kimi_k2` to separate thinking and content in model outputs.
- **Tool Call Parser**: Add `--tool-call-parser kimi_k2` for structured tool calls.
@@ -94,7 +94,7 @@ import { KimiK26Deployment } from '/src/snippets/autoregressive/kimi-k26-deploym
- **NVFP4**: Use `nvidia/Kimi-K2.6-NVFP4` with `--quantization modelopt_fp4` on NVIDIA Blackwell. Use `tp=8` on B300 and `tp=4` on GB300.
- **AMD GPU TP Constraint**: On AMD GPUs, TP must be ≤ 4 (not 8). Kimi-K2.6 has 64 attention heads; the AITER MLA kernel requires `heads_per_gpu % 16 == 0`. With TP=4, each GPU gets 16 heads (valid). With TP=8, each GPU gets 8 heads (invalid).
- **AMD Docker Image**: Use `lmsysorg/sglang:v0.5.9-rocm700-mi35x` for MI350X/MI355X and `lmsysorg/sglang:v0.5.9-rocm700-mi30x` for MI300X/MI325X.
- **DP Attention**: Enable with `--dp <N> --enable-dp-attention` for production throughput. A common choice is to set `--dp` equal to `--tp`, but this is not required.
- **DP Attention**: Enable with `--attn-dp-size <N>` for production throughput. A common choice is to set `--attn-dp-size` equal to `--tp`, but this is not required.
- **Reasoning Parser**: Add `--reasoning-parser kimi_k2` to separate thinking and content in model outputs.
- **Tool Call Parser**: Add `--tool-call-parser kimi_k2` for structured tool calls.
- **AMD FP8 KV Cache**: On AMD platforms the generator adds `--kv-cache-dtype fp8_e4m3` by default and sets `--mem-fraction-static 0.8` to fit the INT4 weights plus KV cache. FP8 KV cache trades a small amount of accuracy for memory; omit the flag if you observe accuracy regressions on your workload.
@@ -94,7 +94,7 @@ import { KimiK27CodeDeployment } from '/src/snippets/autoregressive/kimi-k27-cod
- **Transformers Version**: The model card requires `transformers>=4.57.1,<5.0.0`.
- **AMD GPU TP Constraint**: On AMD GPUs, TP must be ≤ 4 (not 8). Kimi-K2.7-Code has 64 attention heads; the AITER MLA kernel requires `heads_per_gpu % 16 == 0`. With TP=4, each GPU gets 16 heads (valid). With TP=8, each GPU gets 8 heads (invalid).
- **AMD Docker Image**: Use `lmsysorg/sglang:v0.5.9-rocm700-mi35x` for MI350X/MI355X and `lmsysorg/sglang:v0.5.9-rocm700-mi30x` for MI300X/MI325X.
- **DP Attention**: Enable with `--dp <N> --enable-dp-attention` for production throughput. A common choice is to set `--dp` equal to `--tp`, but this is not required.
- **DP Attention**: Enable with `--attn-dp-size <N>` for production throughput. A common choice is to set `--attn-dp-size` equal to `--tp`, but this is not required.
- **Reasoning Parser**: Add `--reasoning-parser kimi_k2` to separate thinking and content in model outputs.
- **Tool Call Parser**: Add `--tool-call-parser kimi_k2` for structured tool calls.
- **AMD FP8 KV Cache**: On AMD platforms the generator adds `--kv-cache-dtype fp8_e4m3` by default and sets `--mem-fraction-static 0.8` to fit the INT4 weights plus KV cache. FP8 KV cache trades a small amount of accuracy for memory; omit the flag if you observe accuracy regressions on your workload.
@@ -36,7 +36,7 @@ This section provides a progressive guide from quick deployment to performance o
- **Memory**: Requires 8 GPUs with ≥140GB each (H200/B200). Use `--context-length 128000` to conserve memory.
- **Expert Parallelism (EP)**: Use `--ep` for better MoE throughput. See [EP docs](../../../docs/advanced_features/expert_parallelism).
- **Data Parallel (DP)**: Enable with `--dp 4 --enable-dp-attention` for production throughput.
- **Data Parallel (DP)**: Enable with `--attn-dp-size 4` for production throughput.
- **KV Cache**: Use `--kv-cache-dtype fp8_e4m3` to reduce memory by 50% (CUDA 11.8+).
- **Reasoning Parser**: Add `--reasoning-parser kimi_k2` for Kimi-K2-Thinking to separate thinking and content.
- **Tool Call Parser**: Add `--tool-call-parser kimi_k2` for structured tool calls.
@@ -359,8 +359,7 @@ We use SGLang's built-in benchmarking tool to conduct performance evaluation on
python3 -m sglang.launch_server \
--model-path moonshotai/Kimi-K2-Instruct \
--tp 8 \
--dp 4 \
--enable-dp-attention \
--attn-dp-size 4 \
--trust-remote-code \
--host 0.0.0.0 \
--port 8000
@@ -427,9 +426,8 @@ Max ITL (ms): 19.94
python3 -m sglang.launch_server \
--model-path moonshotai/Kimi-K2-Instruct \
--tp 8 \
--dp 4 \
--attn-dp-size 4 \
--ep 4 \
--enable-dp-attention \
--trust-remote-code \
--host 0.0.0.0 \
--port 8000
@@ -437,7 +437,7 @@ Two presets come out of this, at `N = 8k` GPUs:
| **Peak Capacity (+DCP8)** — `dp = k` + `--dcp-size 8` | Deduplicates the attention-TP group's MLA KV: concurrency ceiling +72% at the same engine throughput, ~1.8× ITL. | Context ≥ ~16K, or per-replica concurrency past 128. |
- **Radix cache** is independent of the preset: for prefix-free traffic (offline batch, evals) switch it off (Playground's **Prefix Cache** card) — one state slot per request instead of 4–5.
- The fully data-parallel extreme (`--dp-size` = GPU count, attention-TP 1) — the shape behind the 64-GPU sweep's ~3K tok/s per GPU — is not a preset: 288 GB GPUs only, radix forced off, no head-to-head against the preset shape.
- The fully data-parallel extreme (`--attn-dp-size` = GPU count, attention-TP 1) — the shape behind the 64-GPU sweep's ~3K tok/s per GPU — is not a preset: 288 GB GPUs only, radix forced off, no head-to-head against the preset shape.
The Peak Throughput preset at 32 GPUs on B200/B300 (4 nodes × 8; every node runs the same command with its own `--node-rank`). On GB200/GB300 the same 32-GPU shape uses 8 nodes × 4, and the Playground emits `--nnodes 8`:
@@ -447,7 +447,7 @@ sglang serve \
--trust-remote-code \
--model-path moonshotai/Kimi-K3 \
--tp-size 32 --ep-size 32 \
--enable-dp-attention --dp-size 4 --enable-dp-lm-head \
--attn-dp-size 4 --enable-dp-lm-head \
--nnodes 4 --node-rank <rank> --dist-init-addr <node0-ip>:20000 \
--moe-a2a-backend megamoe --moe-runner-backend deep_gemm \
--kv-cache-dtype fp8_e4m3 \
@@ -460,7 +460,7 @@ sglang serve \
Scale by holding the per-replica shape fixed and moving only the replica count; pool sizing rides the calculator-driven `--mamba-full-memory-ratio`, which folds in DP, DCP, precision, and speculation:
| GPUs | B200/B300 nodes | GB200/GB300 nodes | `--tp-size` / `--ep-size` | `--dp-size` |
| GPUs | B200/B300 nodes | GB200/GB300 nodes | `--tp-size` / `--ep-size` | `--attn-dp-size` |
|---|---|---|---|---|
| 16 | 2×8 | 4×4 | 16 | 2 |
| 32 | 4×8 | 8×4 | 32 | 4 |
@@ -86,7 +86,7 @@ The generator only emits a runnable command for combinations that NVIDIA / SGLan
- **DP attention**:
By default the attention layers are tensor-parallel (sharded across all TP ranks). Enabling DP attention (the toggle above, or `--dp <N> --enable-dp-attention`) instead runs attention as `N` data-parallel groups: each DP rank serves its own slice of the requests with its own KV cache. `--dp` must divide `--tp`.
By default the attention layers are tensor-parallel (sharded across all TP ranks). Enabling DP attention (the toggle above, or `--attn-dp-size <N>`) instead runs attention as `N` data-parallel groups: each DP rank serves its own slice of the requests with its own KV cache. `--attn-dp-size` must divide `--tp`.
- **Expert parallel (EP)**:
@@ -84,7 +84,7 @@ import { LagunaXS2Deployment } from '/src/snippets/autoregressive/laguna-xs2-dep
- **Quantization**: NVFP4 requires Blackwell (B200 / B300); BF16 and FP8 run on either H200 or B200. FP8's first launch triggers a multi-session DeepGEMM JIT pre-compile (~10-20 min); pre-warm with `python3 -m sglang.compile_deep_gemm --model poolside/Laguna-XS.2-FP8` to avoid that cost on every restart.
- **Reasoning parser** (`--reasoning-parser poolside_v1`): Splits `<think>...</think>` segments into `reasoning_content` so `content` holds only the final answer. Disable only if you want the raw `<think>` tags in `content`.
- **Tool call parser** (`--tool-call-parser poolside_v1`): Required for OpenAI-compatible tool-call streaming. Disable only for chat-only deployments.
- **DP attention**: For higher-throughput deployments, enable the DP-Attention toggle — it emits `--dp <N> --enable-dp-attention` with `--dp` matching `--tp` (tune independently if needed).
- **DP attention**: For higher-throughput deployments, enable the DP-Attention toggle — it emits `--attn-dp-size <N>` with `--attn-dp-size` matching `--tp` (tune independently if needed).
- **Thinking default**: Thinking is **off by default** at the model level. Opt in per request with `extra_body={"chat_template_kwargs": {"enable_thinking": True}}`.
## 4. Model Invocation
@@ -51,7 +51,7 @@ This section provides deployment configurations verified on AMD MI300X, MI325X,
**NVIDIA (B200/GB200):**
* **GB200 Parallelism**: Use `--tp 4 --ep 4` on GB200. B200 uses the default NVIDIA settings generated above.
* **NVFP4 Quantization**: Requires `--quantization modelopt_fp4` and uses a different model path (`nvidia/Qwen3-Coder-...`).
* **DP Attention**: NVFP4 configuration supports `--enable-dp-attention` for improved throughput.
* **DP Attention**: NVFP4 configuration supports DP attention (`--attn-dp-size <N>`) for improved throughput.
**Intel Xeon CPU:**
* Please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings.
@@ -76,12 +76,12 @@ import { MiMoV25Deployment } from '/src/snippets/autoregressive/mimo-v25-deploym
- EAGLE speculative decoding (3 steps, topk=1) typically yields a 2–3× decode speedup. Requires `--enable-multi-layer-eagle` (both Hopper and Blackwell). See §5.4 for acceptance-rate behavior on natural text vs random prompts.
**MiMo-V2.5 (310B):**
- The checkpoint has a TP=4-interleaved fused `qkv_proj`; attention-TP per DP group **must** be 4. Use `--dp = TP / 4`; for TP > 4 this also requires DP-attention. Total GPUs must be a multiple of 4. A bare `--tp 8` without `--dp 2` will fail to load with `MiMoV2 fused qkv_proj checkpoint is TP=4-interleaved; got attention tp_size=8`.
- Single-node deployments: H100/H200 8× GPUs (`--tp 8 --dp 2`), B200 4× GPUs (`--tp 4`, dp=1, no DP-attn flag needed), GB300 4× GPUs (`--tp 4`, single NVL4 node). FP8 quantization.
- The checkpoint has a TP=4-interleaved fused `qkv_proj`; attention-TP per DP group **must** be 4. Use `--attn-dp-size = TP / 4`; for TP > 4 this means DP-attention is on. Total GPUs must be a multiple of 4. A bare `--tp 8` without `--attn-dp-size 2` will fail to load with `MiMoV2 fused qkv_proj checkpoint is TP=4-interleaved; got attention tp_size=8`.
- Single-node deployments: H100/H200 8× GPUs (`--tp 8 --attn-dp-size 2`), B200 4× GPUs (`--tp 4`, attention DP size 1, no DP-attn flag needed), GB300 4× GPUs (`--tp 4`, single NVL4 node). FP8 quantization.
- On Blackwell, pass `--attention-backend fa4`: MiMoV2's asymmetric KV (`head_dim` 192 / `v_head_dim` 128) fails on the SM100 default `trtllm_mha`, which requires equal K/V widths.
- On Blackwell, pass `--mm-attention-backend fa4` for the V2.5 vision encoder. The checkpoint config requests FlashAttention-3 internally, but SGLang rejects FA3 on Blackwell and expects FA4 for multimodal attention.
- On Blackwell, pass `--moe-runner-backend flashinfer_trtllm`; the default `auto` falls through to the triton fused-MoE runner, ~12% slower at bs=1 decode.
- `--enable-dp-lm-head` and `--mm-enable-dp-encoder` are required whenever `--enable-dp-attention` is on, to keep LM head and encoder sharding consistent.
- `--enable-dp-lm-head` and `--mm-enable-dp-encoder` are required whenever `--attn-dp-size` is greater than 1, to keep LM head and encoder sharding consistent.
- EAGLE MTP uses the checkpoint's MTP weights. Enable with `--speculative-algorithm EAGLE` and `--enable-multi-layer-eagle` (both Hopper and Blackwell).
- **Multimodal**: Supports image, video, and audio understanding; see Section 4.3 for invocation examples.
@@ -547,7 +547,7 @@ Max ITL (ms): 1600.00
- Hardware: NVIDIA H200 GPU (8×)
- Model: `XiaomiMiMo/MiMo-V2.5` (FP8)
- Tensor Parallelism: 8 (DP-attention with `--dp 2`)
- Tensor Parallelism: 8 (DP-attention with `--attn-dp-size 2`)
- Recipe: Balanced (DP-attn + EAGLE MTP)
- sglang version: `0.0.0.dev1+g7d99af439` (`lmsysorg/sglang:dev-mimo-v2.5`)
@@ -777,7 +777,7 @@ Reference numbers from the [day0 enablement PR](https://github.com/sgl-project/s
- Hardware: NVIDIA Hopper GPU (2 nodes × 8 GPUs, GPU SKU intentionally not disclosed)
- Model: `XiaomiMiMo/MiMo-V2.5-Pro` (FP8)
- Parallelism: `--tp 16 --dp 2 --ep 16 --moe-dense-tp-size 1 --enable-dp-attention`
- Parallelism: `--tp 16 --attn-dp-size 2 --ep 16 --moe-dense-tp-size 1`
- Recipe: Hopper EP16 (DeepEP + EAGLE multi-layer MTP)
#### 5.5.1 Long-Context Prefill Throughput
@@ -113,7 +113,7 @@ The shared prefill settings are `--chunked-prefill-size 49152 --max-prefill-toke
The base recipe uses `--speculative-algorithm DFLASH --speculative-num-draft-tokens 8`. Supply the DFlash checkpoint paired with the selected target model; the draft path is independently editable and does not have to be a subdirectory of the target checkpoint.
At the pinned commit, DFlash on NVIDIA GPUs requires PP=1 and rejects DP-attention. Do not add `--enable-dp-attention` or copy EAGLE-specific settings from the V2.5 cookbook.
At the pinned commit, DFlash on NVIDIA GPUs requires PP=1 and rejects DP-attention. Do not set `--attn-dp-size` above 1 or copy EAGLE-specific settings from the V2.5 cookbook.
## 3. Reasoning and tool calling
@@ -37,10 +37,15 @@ This guide explains the parallelism configuration fields used in SGLang model co
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`--ep-size`, `--expert-parallel-size`, `--ep`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Expert Parallelism - distributes MoE experts</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`attn_dp_size`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`--attn-dp-size`, `--attention-data-parallel-size`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Attention Data Parallelism - data-parallel attention groups inside the TP group, TP for FFN (hybrid)</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`enable_dp_attention`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`--enable-dp-attention`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>DP for attention, TP for FFN (hybrid)</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><strong>Deprecated.</strong> Use <code>--attn-dp-size &lt;dp-size&gt;</code> instead of <code>--dp-size &lt;dp-size&gt; --enable-dp-attention</code>.</td>
</tr>
</tbody>
</table>
+3 -4
View File
@@ -17,7 +17,7 @@ On Kimi K3, DCP applies only to MLA layers. Request-indexed KDA state is unchang
| `--dcp-size N` | Enables DCP and widens virtual capacity and page size by `N`. Alias: `--decode-context-parallel-size`. |
| `--dcp-comm-backend` | Selects the MLA partial-output merge: `ag_rs`, `a2a`, or `fi_a2a`. Unset resolves to `fi_a2a` where the DCP group shares an MNNVL domain on Blackwell, otherwise `a2a` on CUDA/ROCm. |
| `--dcp-replicate-q-proj` | Removes the query all-gather for supported MLA weights. Use `--no-dcp-replicate-q-proj` to disable a model-specific default. |
| `--enable-dp-attention` | Composes only when DCP groups nest inside attention TP. |
| `--attn-dp-size N` | Composes only when DCP groups nest inside attention TP. |
A DCP group must fit inside one attention-TP group and one attention-DP replica:
@@ -26,7 +26,7 @@ attn_tp_size = tp_size / attn_dp_size
attn_tp_size % dcp_size == 0
```
For example, `TP=64, DP=4, DCP=16` creates four valid 16-rank attention replicas. `DCP=32` would cross replica boundaries and is invalid.
For example, `TP=64, attention DP=4, DCP=16` creates four valid 16-rank attention replicas. `DCP=32` would cross replica boundaries and is invalid.
<Warning>
Startup currently checks only `tp_size % dcp_size == 0`, not the stronger condition above. Until that check is fixed, DPA+DCP deployments must enforce containment when choosing the topology.
@@ -52,8 +52,7 @@ sglang serve \
--model-path moonshotai/Kimi-K3 \
--tp-size 32 \
--ep-size 32 \
--enable-dp-attention \
--dp-size 4 \
--attn-dp-size 4 \
--dcp-size 8 \
--host 0.0.0.0 \
--port 30000
@@ -80,14 +80,13 @@ For MoE models like DeepSeek, DPA is **often** paired with Expert Parallelism (E
python -m sglang.launch_server \
--model-path deepseek-ai/DeepSeek-V3 \
--tp 8 \
--dp-size 8 \
--attn-dp-size 8 \
--ep 8 \
--enable-dp-attention \
--moe-a2a-backend deepep \
--moe-runner-backend deep_gemm
```
> **Note**: `--dp-size` must be explicitly set when using `--enable-dp-attention`. If `dp_size` is 1 (default), DPA will be disabled.
> **Note**: `--attn-dp-size` sets the number of attention data-parallel groups inside the TP group. If `attn_dp_size` is 1 (default), DPA is disabled.
For detailed EP configuration (DeepEP, Two-Batch Overlap, EPLB), see [Expert Parallelism](/docs/advanced_features/expert_parallelism).
@@ -106,21 +105,28 @@ DPA supports the following model architectures:
For models like Llama, with standard GQA, standard DP, or TP is typically recommended.
To enable DPA, add `--enable-dp-attention` to your server launch command.
To enable DPA, add `--attn-dp-size N` (N > 1) to your server launch command.
### Activation Logic
DPA is enabled explicitly via server arguments (CLI or config). You must set both `--dp-size` and `--enable-dp-attention`:
DPA is enabled explicitly via server arguments (CLI or config) with `--attn-dp-size` (alias `--attention-data-parallel-size`). It creates N attention data-parallel groups inside the TP group, one scheduler each, while the FFN stays tensor parallel:
```bash
python -m sglang.launch_server \
--model-path deepseek-ai/DeepSeek-V3 \
--tp 8 \
--dp-size 8 \
--enable-dp-attention
--attn-dp-size 8
```
**Important**: `--dp-size` must be greater than 1 for DPA to work. When `dp_size == 1` (default), `--enable-dp-attention` is automatically disabled. The constraint `tp_size % dp_size == 0` must also be satisfied.
**Important**: `--attn-dp-size` must be greater than 1 for DPA to work. When `attn_dp_size == 1` (default), DPA is disabled. The constraint `tp_size % attn_dp_size == 0` must also be satisfied. `--dp-size` only sets the number of data-parallel replicas of the model; combining replicas (`--dp-size` > 1) with DPA is not supported.
> **Deprecated**: `--enable-dp-attention` still works: `--dp-size N --enable-dp-attention` resolves to `--attn-dp-size N --dp-size 1` and prints a deprecation warning. Use `--attn-dp-size N` instead.
**Elastic EP scale joiners.** A rank that joins an existing DPA deployment through elastic EP scale-up runs DPA even when it is launched with `--attn-dp-size 1`: its own attention DP group is one rank wide.
**DP rank count.** The number of DP ranks (one scheduler each) is `dp_size × attn_dp_size`; one of the two is always 1.
- `/server_info` reports `dp_size` (replicas) and `attn_dp_size` separately, so under DPA its `dp_size` is 1. Its deprecated `enable_dp_attention` field is always `false`; read `attn_dp_size` instead.
- Wire formats that predate `--attn-dp-size` keep the key `dp_size` for the DP rank count: KV-event descriptors, the PD prefill server info, and load snapshots.
### TP LMHead with all-to-all
@@ -135,8 +141,7 @@ python -m sglang.launch_server \
--model-path MODEL_PATH \
--disaggregation-mode decode \
--tp 8 \
--dp-size 8 \
--enable-dp-attention
--attn-dp-size 8
```
Prefill / colocated workers will fallback to TP LMHead with allgather, as well as other parallel strategies. More details can be found in [PR #32313](https://github.com/sgl-project/sglang/pull/32313).
@@ -505,7 +510,7 @@ For detailed metrics and monitoring setup, see [SGLang Model Gateway Documentati
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Cache-aware routing, high availability</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong>DPA</strong> (<code>--dp-size N --enable-dp-attention</code>)</td>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}><strong>DPA</strong> (<code>--attn-dp-size N</code>)</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>DeepSeek/MLA models</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Eliminates KV cache duplication, improved throughput</td>
</tr>
@@ -518,7 +523,7 @@ For detailed metrics and monitoring setup, see [SGLang Model Gateway Documentati
</table>
**Recommended production setup for DeepSeek:**
1. Enable **DPA** for attention layers (`--dp-size 8 --enable-dp-attention`)
1. Enable **DPA** for attention layers (`--attn-dp-size 8`)
2. Enable **EP** for MoE layers (`--ep 8 --moe-a2a-backend deepep`)
3. Use **SMG** with **cache_aware** policy
@@ -72,7 +72,7 @@ SGLang's EP integrates diverse, highly efficient backends for different use case
DeepEP and Mooncake backends support two modes for token dispatch: `normal` mode (optimized for prefill workloads with high throughput) and `low_latency` mode (optimized for decode workloads with low latency and CUDA Graph compatibility). MORI backend only supports `normal` mode now. NIXL-EP and PPLX currently operate in low-latency mode with CUDA Graph support (PPLX reuses the DeepEP low-latency masked expert-compute path). Users are recommended to set `--deepep-mode auto` to enable automatic dispatch mode switching during runtime. Setting `--deepep-mode normal` or `--deepep-mode low_latency` is useful for debugging or development purposes.
Currently, DeepEP, Mooncake, NIXL-EP, `ascend_fuseep`, `pplx` and MORI only support cases where `ep_size = tp_size`. For hybrid EP and TP (i.e., `ep_size < tp_size`), only the `none` backend (All-Reduce or All-Gather-based dispatching) is supported. Note that `pplx` additionally requires `--enable-dp-attention` with at least 2 DP groups (i.e., `tp_size / attention_tp_size > 1`); otherwise pplx-kernels' AllToAll cannot be constructed.
Currently, DeepEP, Mooncake, NIXL-EP, `ascend_fuseep`, `pplx` and MORI only support cases where `ep_size = tp_size`. For hybrid EP and TP (i.e., `ep_size < tp_size`), only the `none` backend (All-Reduce or All-Gather-based dispatching) is supported. Note that `pplx` additionally requires DP attention with at least 2 attention DP groups (i.e., `--attn-dp-size` > 1, so `tp_size / attention_tp_size > 1`); otherwise pplx-kernels' AllToAll cannot be constructed.
### Backends for MoE Computation
@@ -137,7 +137,7 @@ python3 -m sglang.launch_server \
--port 8000 --host 0.0.0.0 \
--context-length 81920 \
--chunked-prefill-size 65536 \
--tp-size 8 --dp-size 8 --enable-dp-attention \
--tp-size 8 --attn-dp-size 8 \
--mem-fraction-static 0.85 \
--disaggregation-mode prefill \
--disaggregation-ib-device mlx5_0,mlx5_1,mlx5_2,mlx5_3 \
@@ -152,7 +152,7 @@ python3 -m sglang.launch_server \
--trust-remote-code \
--port 8000 --host 0.0.0.0 \
--context-length 81920 \
--tp-size 8 --dp-size 8 --enable-dp-attention \
--tp-size 8 --attn-dp-size 8 \
--mem-fraction-static 0.85 \
--disable-radix-cache \
--disaggregation-mode decode \
@@ -78,5 +78,5 @@ Data parallelism is better for throughput. When there is enough GPU memory, alwa
- `torch.compile` accelerates small models on small batch sizes. You can enable it with `--enable-torch-compile`.
- Try other quantization (e.g. FP8 quantization with `--quantization fp8`)
- Try other parallelism strategies (e.g. [expert parallelism](https://lmsys.org/blog/2025-05-05-large-scale-ep/)) or DP attention for deepseek models (with `--enable-dp-attention --dp-size 8`).
- Try other parallelism strategies (e.g. [expert parallelism](https://lmsys.org/blog/2025-05-05-large-scale-ep/)) or DP attention for deepseek models (with `--attn-dp-size 8`).
- If the workload has many shared prefixes, try `--schedule-policy lpm`. Here, `lpm` stands for longest prefix match. It reorders requests to encourage more cache hits but introduces more scheduling overhead.
+2 -3
View File
@@ -422,8 +422,7 @@ The following NVIDIA CUDA example uses four attention-DP ranks:
python3 -m sglang.launch_server \
--model-path MODEL_PATH \
--tp 4 \
--dp-size 4 \
--enable-dp-attention \
--attn-dp-size 4 \
--enable-lora \
--lora-backend triton \
--lora-paths '{"lora_name":"ADAPTER_NAME","lora_path":"ADAPTER_PATH","pinned":true}'
@@ -431,7 +430,7 @@ python3 -m sglang.launch_server \
Keep these constraints in mind:
- `--dp-size` must equal `--tp-size`, so every attention DP rank owns its whole token batch. The server rejects other layouts at startup.
- `--attn-dp-size` must equal `--tp-size`, so every attention DP rank owns its whole token batch. The server rejects other layouts at startup.
- All adapters must be GPU-pinned (`pinned: true`), including dynamically loaded adapters. You can pin at most `--max-loras-per-batch - 1` adapters; the remaining slot is reserved for the base model.
- Decode CUDA graphs are supported. LoRA prefill runs eagerly when DPA is enabled.
- `--max-loras-per-batch` limits the union of adapters active across all DP ranks in a synchronized forward, not the adapters on one rank alone.
@@ -132,8 +132,7 @@ python -m sglang.launch_server \
--nnodes 2 \
--node-rank 0 \
--tp-size 16 \
--dp-size 8 \
--enable-dp-attention \
--attn-dp-size 8 \
--moe-a2a-backend deepep \
--mem-fraction-static 0.8
# prefill 1
@@ -148,8 +147,7 @@ python -m sglang.launch_server \
--nnodes 2 \
--node-rank 1 \
--tp-size 16 \
--dp-size 8 \
--enable-dp-attention \
--attn-dp-size 8 \
--moe-a2a-backend deepep \
--mem-fraction-static 0.8
# decode 0
@@ -164,8 +162,7 @@ python -m sglang.launch_server \
--nnodes 2 \
--node-rank 0 \
--tp-size 16 \
--dp-size 8 \
--enable-dp-attention \
--attn-dp-size 8 \
--moe-a2a-backend deepep \
--mem-fraction-static 0.8 \
--max-running-requests 128
@@ -181,8 +178,7 @@ python -m sglang.launch_server \
--nnodes 2 \
--node-rank 1 \
--tp-size 16 \
--dp-size 8 \
--enable-dp-attention \
--attn-dp-size 8 \
--moe-a2a-backend deepep \
--mem-fraction-static 0.8 \
--max-running-requests 128
@@ -348,8 +344,7 @@ python -m sglang.launch_server \
--disaggregation-mode decode \
--port 30001 \
--tp 4 \
--dp 4 \
--enable-dp-attention \
--attn-dp-size 4 \
--trust-remote-code \
--disaggregation-ib-device mlx5_3,mlx5_4
@@ -413,8 +408,7 @@ python -m sglang.launch_server \
--nnodes 2 \
--node-rank 0 \
--tp-size 16 \
--dp-size 8 \
--enable-dp-attention \
--attn-dp-size 8 \
--moe-a2a-backend deepep \
--mem-fraction-static 0.8
# prefill 1
@@ -429,8 +423,7 @@ python -m sglang.launch_server \
--nnodes 2 \
--node-rank 1 \
--tp-size 16 \
--dp-size 8 \
--enable-dp-attention \
--attn-dp-size 8 \
--moe-a2a-backend deepep \
--mem-fraction-static 0.8
# decode 0
@@ -445,8 +438,7 @@ python -m sglang.launch_server \
--nnodes 2 \
--node-rank 0 \
--tp-size 16 \
--dp-size 8 \
--enable-dp-attention \
--attn-dp-size 8 \
--moe-a2a-backend deepep \
--mem-fraction-static 0.8 \
--max-running-requests 128
@@ -462,8 +454,7 @@ python -m sglang.launch_server \
--nnodes 2 \
--node-rank 1 \
--tp-size 16 \
--dp-size 8 \
--enable-dp-attention \
--attn-dp-size 8 \
--moe-a2a-backend deepep \
--mem-fraction-static 0.8 \
--max-running-requests 128
@@ -706,7 +697,7 @@ do
--disaggregation-transfer-backend ascend --max-running-requests 32 --context-length 8192 --disable-radix-cache \
--chunked-prefill-size -1 --max-prefill-tokens 10240 --moe-a2a-backend deepep --deepep-mode normal \
--speculative-algorithm NEXTN --speculative-num-steps 1 --speculative-eagle-topk 1 --speculative-num-draft-tokens 2 \
--dp-size 8 --enable-dp-attention --disable-shared-experts-fusion --dtype bfloat16
--attn-dp-size 8 --disable-shared-experts-fusion --dtype bfloat16
NODE_RANK=$i
break
fi
@@ -730,9 +721,9 @@ do
export HCCL_SOCKET_IFNAME=data0.3001
export GLOO_SOCKET_IFNAME=data0.3001
python -m sglang.launch_server --model-path ${MODEL_PATH} --disaggregation-mode decode --host ${D_IP[$i]} \
--port 8001 --trust-remote-code --nnodes 1 --node-rank 0 --tp-size 16 --dp-size 16 \
--port 8001 --trust-remote-code --nnodes 1 --node-rank 0 --tp-size 16 --attn-dp-size 16 \
--mem-fraction-static 0.8 --max-running-requests 448 --attention-backend ascend --device npu --quantization modelslim \
--moe-a2a-backend deepep --enable-dp-attention --deepep-mode low_latency --enable-dp-lm-head \
--moe-a2a-backend deepep --deepep-mode low_latency --enable-dp-lm-head \
--cuda-graph-bs-decode 2 4 6 8 10 12 14 16 18 20 22 24 26 28 --disaggregation-transfer-backend ascend --watchdog-timeout 9000 --context-length 8192 \
--speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 --disable-shared-experts-fusion --dtype bfloat16 --tokenizer-worker-num 4 \
--load-balance-method round_robin
@@ -1220,7 +1220,13 @@ Combining `--enable-response-store` with `--disaggregation-mode=prefill` or `dec
<tbody>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--data-parallel-size`<br/>`--dp-size`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>The data parallelism size.</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>The number of data-parallel replicas of the model.</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`1`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: int</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--attention-data-parallel-size`<br/>`--attn-dp-size`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>The attention data parallelism size: the number of data-parallel attention groups inside the TP group, one scheduler each, while the FFN stays tensor parallel.</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`1`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: int</td>
</tr>
@@ -2564,8 +2570,8 @@ Combining `--enable-response-store` with `--disaggregation-mode=prefill` or `dec
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--enable-dp-attention`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Enabling data parallelism for attention and tensor parallelism for FFN. The dp size should be equal to the tp size. Currently DeepSeek-V2 and Qwen 2/3 MoE models are supported.</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`False`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><strong>Deprecated.</strong> Use <code>--attn-dp-size &lt;dp-size&gt;</code> instead of <code>--dp-size &lt;dp-size&gt; --enable-dp-attention</code>.</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><code>False</code></td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>bool flag (set to enable)</td>
</tr>
<tr>
@@ -100,21 +100,21 @@ SGLang provides several speculative decoding options, including EAGLE-2/EAGLE-3,
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>DFlash draft model (linear block verification)</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Yes</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><code>--speculative-algorithm DFLASH</code> + <code>--speculative-draft-model-path ...</code></td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>No <code>--enable-dp-attention</code>; <code>pp_size == 1</code>; disables overlap scheduler &amp; mixed chunked prefill</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>No DP attention (<code>--attn-dp-size</code> &gt; 1); <code>pp_size == 1</code>; disables overlap scheduler &amp; mixed chunked prefill</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>STANDALONE</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Smaller draft LLM (token-level)</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Yes</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><code>--speculative-algorithm STANDALONE</code> + <code>--speculative-draft-model-path ...</code></td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Does <strong>not</strong> support <code>--enable-dp-attention</code></td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Does <strong>not</strong> support DP attention (<code>--attn-dp-size</code> &gt; 1)</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>NGRAM</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Ngram cache from previous tokens</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>No</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><code>--speculative-algorithm NGRAM</code></td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>CUDA-only; no <code>--enable-dp-attention</code>; disables overlap scheduler &amp; mixed chunked prefill</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>CUDA-only; no DP attention (<code>--attn-dp-size</code> &gt; 1); disables overlap scheduler &amp; mixed chunked prefill</td>
</tr>
</tbody>
</table>
@@ -678,7 +678,7 @@ Relevant parameters:
</tbody>
</table>
> **Note:** Standalone speculative decoding currently **does not support** `--enable-dp-attention`.
> **Note:** Standalone speculative decoding currently **does not support** DP attention (`--attn-dp-size` > 1).
```bash Command
python3 -m sglang.launch_server \
@@ -815,7 +815,7 @@ Enable it with:
Notes:
- Ngram speculative decoding **only supports CUDA**.
- It currently **does not support** `--enable-dp-attention`.
- It currently **does not support** DP attention (`--attn-dp-size` > 1).
- It disables the overlap scheduler and mixed chunked prefill.
- If `--speculative-ngram-max-bfs-breadth > 1` (thus `speculative_eagle_topk > 1`) and `page_size > 1`, use `--attention-backend flashinfer`; otherwise the server will error.
- Optional: set `SGLANG_NGRAM_FORCE_GREEDY_VERIFY=True` to force greedy verification.
@@ -125,8 +125,7 @@ do
--speculative-num-steps 1 \
--speculative-eagle-topk 1 \
--speculative-num-draft-tokens 2 \
--dp-size 8 \
--enable-dp-attention \
--attn-dp-size 8 \
--disable-shared-experts-fusion \
--dtype bfloat16 \
--reasoning-parser deepseek-r1 \
@@ -160,12 +159,11 @@ do
--port 8001 \
--nnodes 1 \
--tp-size 16 \
--dp-size 16 \
--attn-dp-size 16 \
--mem-fraction-static 0.9 \
--max-running-requests 416 \
--quantization modelslim \
--moe-a2a-backend deepep \
--enable-dp-attention \
--deepep-mode low_latency \
--enable-dp-lm-head \
--cuda-graph-bs-decode 2 4 6 8 10 12 14 16 18 20 22 24 26 \
@@ -302,8 +300,7 @@ python3 -m sglang.launch_server \
--max-prefill-tokens 3000 \
--moe-a2a-backend deepep \
--deepep-mode auto \
--enable-dp-attention \
--dp-size 16 \
--attn-dp-size 16 \
--enable-dp-lm-head \
--speculative-algorithm NEXTN \
--speculative-num-steps 3 \
@@ -430,8 +427,7 @@ do
--speculative-num-steps 1 \
--speculative-eagle-topk 1 \
--speculative-num-draft-tokens 2 \
--dp-size 2 \
--enable-dp-attention \
--attn-dp-size 2 \
--disable-shared-experts-fusion \
--dtype bfloat16 \
--enable-attn-tp-input-scattered \
@@ -468,12 +464,11 @@ do
--node-rank $i \
--nnodes 2 \
--tp-size 32 \
--dp-size 16 \
--attn-dp-size 16 \
--mem-fraction-static 0.75 \
--max-running-requests 32 \
--quantization modelslim \
--moe-a2a-backend deepep \
--enable-dp-attention \
--deepep-mode low_latency \
--enable-dp-lm-head \
--moe-dense-tp 1 \
@@ -636,8 +631,7 @@ do
--speculative-num-steps 1 \
--speculative-eagle-topk 1 \
--speculative-num-draft-tokens 2 \
--dp-size 4 \
--enable-dp-attention \
--attn-dp-size 4 \
--disable-shared-experts-fusion \
--dtype bfloat16 \
--enable-attn-tp-input-scattered \
@@ -675,12 +669,11 @@ do
--node-rank $i \
--nnodes 2 \
--tp-size 32 \
--dp-size 32 \
--attn-dp-size 32 \
--mem-fraction-static 0.82 \
--max-running-requests 1024 \
--quantization modelslim \
--moe-a2a-backend ascend_fuseep \
--enable-dp-attention \
--deepep-mode low_latency \
--moe-dense-tp 1 \
--cuda-graph-bs-decode 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 \
@@ -838,8 +831,7 @@ do
--speculative-num-steps 1 \
--speculative-eagle-topk 1 \
--speculative-num-draft-tokens 2 \
--dp-size 2 \
--enable-dp-attention \
--attn-dp-size 2 \
--disable-shared-experts-fusion \
--dtype bfloat16 \
--enable-attn-tp-input-scattered \
@@ -876,12 +868,11 @@ do
--node-rank $i \
--nnodes 2 \
--tp-size 32 \
--dp-size 16 \
--attn-dp-size 16 \
--mem-fraction-static 0.75 \
--max-running-requests 32 \
--quantization modelslim \
--moe-a2a-backend deepep \
--enable-dp-attention \
--deepep-mode low_latency \
--enable-dp-lm-head \
--moe-dense-tp 1 \
@@ -1040,8 +1031,7 @@ do
--speculative-num-steps 1 \
--speculative-eagle-topk 1 \
--speculative-num-draft-tokens 2 \
--dp-size 2 \
--enable-dp-attention \
--attn-dp-size 2 \
--disable-shared-experts-fusion \
--dtype bfloat16 \
--enable-attn-tp-input-scattered \
@@ -1078,12 +1068,11 @@ do
--node-rank $i \
--nnodes 2 \
--tp-size 32 \
--dp-size 16 \
--attn-dp-size 16 \
--mem-fraction-static 0.75 \
--max-running-requests 32 \
--quantization modelslim \
--moe-a2a-backend deepep \
--enable-dp-attention \
--deepep-mode low_latency \
--enable-dp-lm-head \
--moe-dense-tp 1 \
@@ -1241,8 +1230,7 @@ do
--speculative-num-steps 1 \
--speculative-eagle-topk 1 \
--speculative-num-draft-tokens 2 \
--dp-size 2 \
--enable-dp-attention \
--attn-dp-size 2 \
--disable-shared-experts-fusion \
--dtype bfloat16 \
--enable-attn-tp-input-scattered \
@@ -1279,12 +1267,11 @@ do
--node-rank $i \
--nnodes 2 \
--tp-size 32 \
--dp-size 8 \
--attn-dp-size 8 \
--mem-fraction-static 0.75 \
--max-running-requests 32 \
--quantization modelslim \
--moe-a2a-backend deepep \
--enable-dp-attention \
--deepep-mode low_latency \
--enable-dp-lm-head \
--moe-dense-tp 1 \
@@ -159,10 +159,9 @@ do
--node-rank $i \
--nnodes 2 \
--tp 32 \
--dp 8 \
--attn-dp-size 8 \
--ep 32 \
--moe-dense-tp-size 1 \
--enable-dp-attention \
--enable-dp-lm-head \
--watchdog-timeout 9000 \
--mem-fraction-static 0.79 \
@@ -356,10 +355,9 @@ do
--node-rank $i \
--nnodes 2 \
--tp 32 \
--dp 8 \
--attn-dp-size 8 \
--ep 32 \
--moe-dense-tp-size 1 \
--enable-dp-attention \
--enable-dp-lm-head \
--watchdog-timeout 9000 \
--mem-fraction-static 0.79 \
@@ -552,10 +550,9 @@ do
--node-rank $i \
--nnodes 2 \
--tp 32 \
--dp 8 \
--attn-dp-size 8 \
--ep 32 \
--moe-dense-tp-size 1 \
--enable-dp-attention \
--enable-dp-lm-head \
--watchdog-timeout 9000 \
--mem-fraction-static 0.79 \
@@ -131,8 +131,7 @@ do
--max-prefill-tokens 70000 \
--chunked-prefill-size -1 \
--max-running-requests 112 \
--dp-size 16 \
--enable-dp-attention \
--attn-dp-size 16 \
--moe-a2a-backend deepep \
--deepep-mode normal \
--quantization modelslim \
@@ -177,8 +176,7 @@ do
--chunked-prefill-size 32768 \
--disaggregation-transfer-backend ascend \
--max-running-requests 896 \
--dp-size 16 \
--enable-dp-attention \
--attn-dp-size 16 \
--moe-a2a-backend deepep \
--deepep-mode auto \
--quantization modelslim \
@@ -319,8 +317,7 @@ python3 -m sglang.launch_server \
--mem-fraction-static 0.7 \
--chunked-prefill-size 131072 \
--max-running-requests 64 \
--dp-size 16 \
--enable-dp-attention \
--attn-dp-size 16 \
--moe-a2a-backend deepep \
--deepep-mode auto \
--quantization modelslim \
@@ -444,8 +441,7 @@ python3 -m sglang.launch_server \
--mem-fraction-static 0.7 \
--chunked-prefill-size 131072 \
--max-running-requests 160 \
--dp-size 16 \
--enable-dp-attention \
--attn-dp-size 16 \
--moe-a2a-backend deepep \
--deepep-mode auto \
--quantization modelslim \
@@ -105,8 +105,7 @@ do
--attention-backend ascend \
--device npu \
--tp-size 32 \
--dp-size 16 \
--enable-dp-attention \
--attn-dp-size 16 \
--chunked-prefill-size 65536 \
--max-prefill-tokens 280000 \
--trust-remote-code \
@@ -294,9 +293,8 @@ do
--trust-remote-code \
--tp-size 32 \
--nnodes 2 \
--dp-size 32 \
--attn-dp-size 32 \
--ep-size 32 \
--enable-dp-attention \
--mem-fraction-static 0.85 \
--max-running-requests 32 \
--attention-backend ascend \
@@ -454,8 +452,7 @@ do
--disable-shared-experts-fusion \
--disable-cuda-graph \
--dtype bfloat16 \
--dp-size 4 \
--enable-dp-attention \
--attn-dp-size 4 \
--load-balance-method round_robin \
--enable-prefill-cp \
--cp-strategy zigzag \
@@ -492,9 +489,8 @@ do
--node-rank $i \
--tp-size 32 \
--nnodes 2 \
--dp-size 32 \
--attn-dp-size 32 \
--ep-size 32 \
--enable-dp-attention \
--mem-fraction-static 0.87 \
--max-running-requests 96 \
--attention-backend ascend \
@@ -702,8 +698,7 @@ do
--node-rank $i \
--tp-size 32 \
--nnodes 2 \
--dp-size 32 \
--enable-dp-attention \
--attn-dp-size 32 \
--ep-size 32 \
--mem-fraction-static 0.85 \
--max-running-requests 32 \
@@ -903,8 +898,7 @@ do
--node-rank $i \
--tp-size 32 \
--nnodes 2 \
--dp-size 32 \
--enable-dp-attention \
--attn-dp-size 32 \
--ep-size 32 \
--mem-fraction-static 0.85 \
--max-running-requests 32 \
@@ -1062,8 +1056,7 @@ do
--disable-shared-experts-fusion \
--disable-cuda-graph \
--dtype bfloat16 \
--dp-size 4 \
--enable-dp-attention \
--attn-dp-size 4 \
--load-balance-method round_robin \
--enable-prefill-cp \
--cp-strategy zigzag \
@@ -1100,9 +1093,8 @@ do
--node-rank $i \
--tp-size 32 \
--nnodes 2 \
--dp-size 32 \
--attn-dp-size 32 \
--ep-size 32 \
--enable-dp-attention \
--mem-fraction-static 0.87 \
--max-running-requests 96 \
--attention-backend ascend \
@@ -1308,9 +1300,8 @@ do
--node-rank $i \
--tp-size 32 \
--nnodes 2 \
--dp-size 32 \
--attn-dp-size 32 \
--ep-size 32 \
--enable-dp-attention \
--mem-fraction-static 0.865 \
--max-running-requests 96 \
--attention-backend ascend \
@@ -1516,9 +1507,8 @@ do
--node-rank $i \
--tp-size 32 \
--nnodes 2 \
--dp-size 32 \
--attn-dp-size 32 \
--ep-size 32 \
--enable-dp-attention \
--mem-fraction-static 0.85 \
--max-running-requests 320 \
--attention-backend ascend \
@@ -159,8 +159,7 @@ do
--port 8001 \
--trust-remote-code \
--tp-size 16 \
--dp-size 16 \
--enable-dp-attention \
--attn-dp-size 16 \
--ep-size 16 \
--mem-fraction-static 0.895 \
--max-running-requests 128 \
@@ -119,8 +119,7 @@ do
--enable-multimodal \
--mm-attention-backend ascend_attn \
--sampling-backend ascend \
--enable-dp-attention \
--dp-size 32 \
--attn-dp-size 32 \
--moe-a2a-backend deepep \
--deepep-mode auto \
--cuda-graph-bs-decode 1 \
@@ -295,8 +294,6 @@ do
--tp-size 16 \
--mem-fraction-static 0.82 \
--max-running-requests 2 \
--enable-dp-attention \
--dp-size 1 \
--enable-dp-lm-head \
--disable-radix-cache \
--enable-multimodal \
@@ -493,8 +490,6 @@ do
--tp-size 16 \
--mem-fraction-static 0.82 \
--max-running-requests 2 \
--enable-dp-attention \
--dp-size 1 \
--enable-dp-lm-head \
--disable-radix-cache \
--enable-multimodal \
@@ -690,8 +685,6 @@ do
--tp-size 16 \
--mem-fraction-static 0.82 \
--max-running-requests 16 \
--enable-dp-attention \
--dp-size 1 \
--enable-dp-lm-head \
--disable-radix-cache \
--enable-multimodal \
@@ -883,8 +876,7 @@ do
--tp-size 16 \
--mem-fraction-static 0.82 \
--max-running-requests 2 \
--enable-dp-attention \
--dp-size 2 \
--attn-dp-size 2 \
--enable-dp-lm-head \
--disable-radix-cache \
--enable-multimodal \
@@ -1019,8 +1011,7 @@ python3 -m sglang.launch_server \
--enable-multimodal \
--mm-attention-backend ascend_attn \
--sampling-backend ascend \
--enable-dp-attention \
--dp-size 16 \
--attn-dp-size 16 \
--moe-a2a-backend deepep \
--deepep-mode auto \
--cuda-graph-bs-decode 1 2 4 8 9 10 11 \
@@ -1129,8 +1120,7 @@ python3 -m sglang.launch_server \
--enable-multimodal \
--mm-attention-backend ascend_attn \
--sampling-backend ascend \
--enable-dp-attention \
--dp-size 16 \
--attn-dp-size 16 \
--moe-a2a-backend deepep \
--deepep-mode auto \
--cuda-graph-bs-decode 1 2 3 4 \
@@ -1244,8 +1234,7 @@ python3 -m sglang.launch_server \
--enable-multimodal \
--mm-attention-backend ascend_attn \
--sampling-backend ascend \
--enable-dp-attention \
--dp-size 16 \
--attn-dp-size 16 \
--moe-a2a-backend deepep \
--deepep-mode auto \
--cuda-graph-bs-decode 1 2 3 4 5 \
@@ -1357,8 +1346,7 @@ python3 -m sglang.launch_server \
--enable-multimodal \
--mm-attention-backend ascend_attn \
--sampling-backend ascend \
--enable-dp-attention \
--dp-size 16 \
--attn-dp-size 16 \
--moe-a2a-backend deepep \
--deepep-mode auto \
--cuda-graph-bs-decode 1 2 4 8 11 12 13 \
@@ -82,7 +82,7 @@ python3 -m sglang.launch_server \
--disaggregation-mode prefill --disaggregation-transfer-backend ascend \
--disaggregation-bootstrap-port 8996 \
--disable-piecewise-cuda-graph \
--dp-size 2 --enable-dp-attention --enable-dp-lm-head \
--attn-dp-size 2 --enable-dp-lm-head \
--moe-a2a-backend deepep --deepep-mode normal
```
@@ -110,7 +110,7 @@ python3 -m sglang.launch_server \
--disaggregation-mode decode --disaggregation-transfer-backend ascend \
--disaggregation-bootstrap-port 8996 \
--moe-a2a-backend deepep --deepep-mode low_latency \
--dp-size 2 --enable-dp-attention --enable-dp-lm-head \
--attn-dp-size 2 --enable-dp-lm-head \
--speculative-algorithm DFLASH \
--speculative-num-draft-tokens 8
```
@@ -166,8 +166,7 @@ do
--speculative-num-draft-tokens 4 \
--enable-multi-layer-eagle \
--disable-radix-cache \
--dp-size 2 \
--enable-dp-attention \
--attn-dp-size 2 \
--enable-dp-lm-head \
--moe-a2a-backend deepep \
--deepep-mode low_latency
@@ -350,8 +349,7 @@ do
--speculative-num-draft-tokens 4 \
--enable-multi-layer-eagle \
--disable-radix-cache \
--dp-size 2 \
--enable-dp-attention \
--attn-dp-size 2 \
--enable-dp-lm-head \
--moe-a2a-backend deepep \
--deepep-mode low_latency
@@ -534,8 +532,7 @@ do
--speculative-num-draft-tokens 4 \
--enable-multi-layer-eagle \
--disable-radix-cache \
--dp-size 2 \
--enable-dp-attention \
--attn-dp-size 2 \
--enable-dp-lm-head \
--moe-a2a-backend deepep \
--deepep-mode low_latency
@@ -718,8 +715,7 @@ do
--speculative-num-draft-tokens 4 \
--enable-multi-layer-eagle \
--disable-radix-cache \
--dp-size 2 \
--enable-dp-attention \
--attn-dp-size 2 \
--enable-dp-lm-head \
--moe-a2a-backend deepep \
--deepep-mode low_latency
@@ -312,8 +312,7 @@ python3 -m sglang.launch_server \
--model-path $MODEL_PATH \
--host 127.0.0.1 --port 6688 \
--tp-size 16 \
--dp-size 2 \
--enable-dp-attention \
--attn-dp-size 2 \
--mem-fraction-static 0.65 \
--max-running-requests 4 \
--reasoning-parser minimax-append-think \
@@ -422,8 +421,7 @@ python3 -m sglang.launch_server \
--model-path $MODEL_PATH \
--host 127.0.0.1 --port 6688 \
--tp-size 16 \
--enable-dp-attention \
--dp-size 16 \
--attn-dp-size 16 \
--mem-fraction-static 0.53 \
--max-running-requests 96 \
--disable-radix-cache \
@@ -527,8 +525,7 @@ python3 -m sglang.launch_server \
--model-path $MODEL_PATH \
--host 127.0.0.1 --port 6688 \
--tp-size 16 \
--enable-dp-attention \
--dp-size 16 \
--attn-dp-size 16 \
--mem-fraction-static 0.75 \
--max-running-requests 320 \
--disable-radix-cache \
@@ -204,8 +204,7 @@ python3 -m sglang.launch_server \
--speculative-num-draft-tokens 4 \
--speculative-draft-model-quantization unquant \
--tp 16 \
--dp-size 16 \
--enable-dp-attention \
--attn-dp-size 16 \
--enable-dp-lm-head \
--mem-fraction-static 0.8 \
--cuda-graph-bs-decode 1 2 4 8 16 20 24 26 27 \
@@ -105,8 +105,7 @@ python3 -m sglang.launch_server \
--speculative-eagle-topk 1 \
--speculative-num-draft-tokens 8 \
--tp-size 2 \
--enable-dp-attention \
--dp-size 2 \
--attn-dp-size 2 \
--mem-fraction-static 0.85 \
--cuda-graph-bs-decode 1 2 4 8 16 20 24 28 32 36 40 44 48 52 56 60 64 68 72 76 80 84 \
--dtype bfloat16 \
@@ -455,8 +455,7 @@ python3 -m sglang.launch_server \
--mm-attention-backend ascend_attn \
--dtype bfloat16 \
--mamba-ssm-dtype bfloat16 \
--dp-size 4 \
--enable-dp-attention \
--attn-dp-size 4 \
--enable-dp-lm-head \
--speculative-algorithm NEXTN \
--speculative-num-steps 3 \
@@ -572,8 +571,7 @@ python3 -m sglang.launch_server \
--mm-attention-backend ascend_attn \
--dtype bfloat16 \
--mamba-ssm-dtype bfloat16 \
--dp-size 4 \
--enable-dp-attention \
--attn-dp-size 4 \
--enable-dp-lm-head \
--speculative-algorithm NEXTN \
--speculative-num-steps 3 \
@@ -688,8 +686,7 @@ python3 -m sglang.launch_server \
--mm-attention-backend ascend_attn \
--dtype bfloat16 \
--mamba-ssm-dtype bfloat16 \
--dp-size 8 \
--enable-dp-attention \
--attn-dp-size 8 \
--enable-dp-lm-head \
--speculative-algorithm NEXTN \
--speculative-num-steps 3 \
@@ -804,8 +801,7 @@ python3 -m sglang.launch_server \
--mm-attention-backend ascend_attn \
--dtype bfloat16 \
--mamba-ssm-dtype bfloat16 \
--dp-size 8 \
--enable-dp-attention \
--attn-dp-size 8 \
--enable-dp-lm-head \
--speculative-algorithm NEXTN \
--speculative-num-steps 3 \
@@ -921,8 +917,7 @@ python3 -m sglang.launch_server \
--mm-attention-backend ascend_attn \
--dtype bfloat16 \
--mamba-ssm-dtype bfloat16 \
--dp-size 2 \
--enable-dp-attention \
--attn-dp-size 2 \
--enable-dp-lm-head \
--speculative-algorithm NEXTN \
--speculative-num-steps 3 \
@@ -1038,8 +1033,7 @@ python3 -m sglang.launch_server \
--mm-attention-backend ascend_attn \
--dtype bfloat16 \
--mamba-ssm-dtype bfloat16 \
--dp-size 2 \
--enable-dp-attention \
--attn-dp-size 2 \
--enable-dp-lm-head \
--speculative-algorithm NEXTN \
--speculative-num-steps 3 \
@@ -1149,8 +1143,7 @@ python3 -m sglang.launch_server \
--mm-attention-backend ascend_attn \
--dtype bfloat16 \
--mamba-ssm-dtype bfloat16 \
--dp-size 2 \
--enable-dp-attention \
--attn-dp-size 2 \
--enable-dp-lm-head \
--speculative-algorithm NEXTN \
--speculative-num-steps 3 \
@@ -228,8 +228,7 @@ python3 -m sglang.launch_server \
--speculative-num-draft-tokens 4 \
--speculative-draft-model-quantization unquant \
--speculative-draft-model-path $DRAFT_MODEL_PATH \
--dp-size 2 \
--enable-dp-attention \
--attn-dp-size 2 \
--enable-dp-lm-head \
--moe-a2a-backend deepep \
--deepep-mode auto \
@@ -337,8 +336,7 @@ python3 -m sglang.launch_server \
--max-prefill-tokens 28672 \
--context-length 81920 \
--max-total-tokens 122304 \
--dp-size 2 \
--enable-dp-attention \
--attn-dp-size 2 \
--enable-dp-lm-head \
--speculative-algorithm NEXTN \
--speculative-num-steps 3 \
@@ -31,7 +31,7 @@ recommended to use v0.5.16 or a later version.
| Speculative Decoding | `--speculative-algorithm EAGLE \`<br/>`--speculative-num-steps 2 \`<br/>`--speculative-eagle-topk 1 \`<br/>`--speculative-num-draft-tokens 3` |
| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1` |
| DP LM Head | `--enable-dp-lm-head` |
| DP Attention | `--enable-dp-attention` |
| DP Attention | `--attn-dp-size 16` |
<Note>
The values in the **Example usage** column are for illustration only. Adjust them according to your hardware, deployment
@@ -218,7 +218,7 @@ python3 -m sglang.launch_server \
--max-prefill-tokens 80000 \
--chunked-prefill-size 131072 \
--max-running-requests 160 \
--dp-size 8 --enable-dp-attention \
--attn-dp-size 8 \
--moe-a2a-backend deepep --deepep-mode auto \
--quantization modelslim --enable-dp-lm-head \
--kv-cache-dtype bfloat16
@@ -384,7 +384,7 @@ do
--disaggregation-transfer-backend ascend --max-running-requests 64 \
--served-model-name glm-5 --chunked-prefill-size 524288 --max-prefill-tokens 180000 --moe-a2a-backend deepep --deepep-mode normal \
--disable-shared-experts-fusion --disable-cuda-graph --dtype bfloat16 \
--dp-size 4 --enable-dp-attention \
--attn-dp-size 4 \
--load-balance-method round_robin \
--enable-dp-lm-head --moe-dense-tp 1 \
--speculative-draft-model-quantization unquant \
@@ -418,9 +418,9 @@ do
export SGLANG_NPU_USE_MULTI_STREAM=1
python -m sglang.launch_server --model-path ${MODEL_PATH} --disaggregation-mode decode --host ${D_IP[$i]} \
--port 8003 --trust-remote-code --nnodes 1 --node-rank $i --tp-size 16 --dp-size 16 --ep-size 16 \
--port 8003 --trust-remote-code --nnodes 1 --node-rank $i --tp-size 16 --attn-dp-size 16 --ep-size 16 \
--mem-fraction-static 0.8 --max-running-requests 128 --attention-backend ascend --device npu --quantization modelslim \
--served-model-name glm-5 --moe-a2a-backend deepep --enable-dp-attention --deepep-mode low_latency \
--served-model-name glm-5 --moe-a2a-backend deepep --deepep-mode low_latency \
--cuda-graph-max-bs-decode 4 --disaggregation-transfer-backend ascend --watchdog-timeout 9000 --context-length 180000 \
--tokenizer-worker-num 4 --disable-shared-experts-fusion --dtype bfloat16 --load-balance-method round_robin \
--speculative-draft-model-quantization unquant \
@@ -30,7 +30,7 @@ v0.5.16 or a later version.
| Speculative Decoding | `--speculative-algorithm EAGLE \`<br/>`--speculative-num-steps 3 \`<br/>`--speculative-eagle-topk 1 \`<br/>`--speculative-num-draft-tokens 4 \`<br/>`--enable-multi-layer-eagle` |
| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=0` |
| DP LM Head | `--enable-dp-lm-head` |
| DP Attention | `--enable-dp-attention` |
| DP Attention | `--attn-dp-size 2` |
<Note>
The values in the **Example usage** column are for illustration only. Adjust them according to your hardware, deployment
@@ -24,7 +24,7 @@ source.
| Feature | Example usage |
|-------------------------------|-----------------------------------------------------------------------------------------------|
| Tensor Parallelism | `--tp-size 64` |
| Data Parallelism | `--enable-dp-attention \`<br/>`--dp-size 4` |
| DP Attention | `--attn-dp-size 4` |
| Expert Parallelism | `--moe-a2a-backend deepep \`<br/>`--deepep-mode auto` |
| Quantization | `--quantization modelslim` |
| Chunked Prefill | auto based on device memory, or set explicit value;<br/>disable with `--chunked-prefill-size -1`; e.g., `--chunked-prefill-size 8192` |
@@ -136,7 +136,7 @@ quay.io/ascend/sglang:main-cann9.0.0-a3
Multi-node deployment distributes the model across multiple Ascend A3 Series nodes using tensor parallelism while keeping
prefill and decode on the same nodes (PD mixed mode), suitable for scenarios that need more device memory than a single
node can provide. The validated configuration deploys the W4A8 quantized checkpoint on 4 nodes with `--tp-size 64`
(16 dies per node), DP attention (`--dp-size 4`), and DeepEP in `auto` mode.
(16 dies per node), DP attention (`--attn-dp-size 4`), and DeepEP in `auto` mode.
Modify the IP addresses of the four nodes, then run the same script on all four nodes. Each node determines its own
rank by matching the local IP address against `IPS`.
@@ -211,7 +211,7 @@ do
--quantization modelslim \
--dtype bfloat16 \
--tp-size 64 \
--enable-dp-attention --dp-size 4 --enable-dp-lm-head \
--attn-dp-size 4 --enable-dp-lm-head \
--mem-fraction-static 0.8 \
--chunked-prefill-size 8192 \
--cuda-graph-bs-decode 16 \
@@ -218,8 +218,8 @@ These arguments and environment variables are critical for tuning decode perform
</thead>
<tbody>
<tr>
<td>`--dp-size` / `--data-parallel-size`</td>
<td>Sets the data parallelism degree for the decode server. With DP attention enabled, attention layers are sharded across DP ranks while FFN/MoE layers use tensor parallelism. Higher values create more independent decode instances, increasing throughput through parallel request processing. Choose a value that divides evenly into your total card count, with remaining cards used for TP/EP.</td>
<td>`--attn-dp-size` / `--attention-data-parallel-size`</td>
<td>Sets the attention DP size for the decode server: the number of data-parallel attention groups inside the TP group. This is a key optimization for decode throughput — attention is DP-sharded to reduce KV cache duplication, while FFN/MoE layers remain TP-sharded to leverage expert parallelism. Best suited for MoE models where attention is not the compute bottleneck. Higher values create more independent decode instances, increasing throughput through parallel request processing. Choose a value that divides the TP size evenly.</td>
<td>`8`</td>
</tr>
<tr>
@@ -227,11 +227,6 @@ These arguments and environment variables are critical for tuning decode perform
<td>Sets the expert parallelism degree. For MoE models, this distributes experts across cards, reducing per-card expert loading overhead and enabling all-to-all dispatch. The code default is `1`; set explicitly for MoE models. The optimal value depends on your model's expert count and architecture. DeepSeek V3.2 with 256 routed experts uses `ep=32`. For models with fewer experts, use a proportionally smaller value.</td>
<td>`32`</td>
</tr>
<tr>
<td>`--enable-dp-attention`</td>
<td>Enables data parallelism for attention layers while keeping tensor parallelism for FFN/MoE layers. This is a key optimization for decode throughput — attention is DP-sharded to reduce KV cache duplication, while MoE layers remain TP-sharded to leverage expert parallelism. Best suited for MoE models where attention is not the compute bottleneck.</td>
<td>Enabled</td>
</tr>
<tr>
<td>`--enable-dp-lm-head`</td>
<td>Enables vocabulary parallelism across the DP attention group, sharding the LM head weight across ranks. Each rank only computes logits for its vocabulary shard, avoiding a costly all-gather of logits across the DP group. This is essential when DP attention is enabled to maintain throughput.</td>
@@ -727,9 +722,9 @@ The symbols used are defined as follows:
Splits model weights across multiple NPU devices so that large models can be loaded and run cooperatively. Each device holds a shard of every weight tensor and communicates via HCCL all-reduce. This is the primary mechanism for deploying models that exceed single-device memory.
### Data Parallelism (`--dp-size`, `--enable-dp-attention`)
### Data Parallelism (`--dp-size`, `--attn-dp-size`)
Replicates the model across independent device groups to increase throughput by processing multiple requests in parallel. With --enable-dp-attention, the attention layers are replicated across DP ranks while the FFN/MoE layers remain tensor-parallel/expert-parallel. This reduces the communication overhead of attention layers in TP (which is significant during small-batch decode) while keeping the heavy compute and memory demands of MoE layers distributed.
`--dp-size` replicates the model across independent device groups to increase throughput by processing multiple requests in parallel. `--attn-dp-size` instead runs attention data parallel inside the TP group: the attention layers are replicated across attention DP ranks while the FFN/MoE layers remain tensor-parallel/expert-parallel. This reduces the communication overhead of attention layers in TP (which is significant during small-batch decode) while keeping the heavy compute and memory demands of MoE layers distributed. Model replicas (`--dp-size` > 1) cannot be combined with attention DP.
### Expert Parallelism (`--ep-size`)
@@ -779,7 +774,7 @@ On Ascend, this is primarily controlled via environment variable SGLANG_ENABLE_O
### DP LM Head (`--enable-dp-lm-head`)
Shards the language model head (the final vocabulary projection layer) across DP ranks to reduce memory consumption and improve decode throughput. Without this, each DP rank holds a full copy of the LM head, which is wasteful for large vocabularies. Typically used together with `--enable-dp-attention`.
Shards the language model head (the final vocabulary projection layer) across DP ranks to reduce memory consumption and improve decode throughput. Without this, each DP rank holds a full copy of the LM head, which is wasteful for large vocabularies. Typically used together with `--attn-dp-size`.
### MLAPO (`SGLANG_NPU_USE_MLAPO=1`)
@@ -1080,6 +1080,12 @@ click [Server Arguments](../../../advanced_features/server_arguments).
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Type: int</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>A2/A3 Series</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--attention-data-parallel-size`<br/>`--attn-dp-size`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`1`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Type: int</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>A2/A3 Series</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--load-balance-method`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`auto`</td>
@@ -2190,12 +2196,6 @@ If the value is int8, you must also set the environment variable:DEEP_NORMAL_MOD
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>bool flag<br/> (set to enable)</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>A2/A3 Series</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--enable-dp-attention`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`False`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>bool flag<br/> (set to enable)</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>A2/A3 Series</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--enable-dp-attention-local-control-broadcast`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`False`</td>
@@ -63,8 +63,7 @@ spec:
# - --deepep-config
# - /home/aiges/tuned/tuned_8sms.json
- --enable-dp-lm-head
- --enable-dp-attention
- --dp-size
- --attn-dp-size
- "16"
- --disable-radix-cache
- --moe-a2a-backend
@@ -210,8 +209,7 @@ spec:
# - --deepep-config
# - /home/aiges/tuned/tuned_8sms.json
- --enable-dp-lm-head
- --enable-dp-attention
- --dp-size
- --attn-dp-size
- "16"
- --disable-radix-cache
- --moe-a2a-backend
@@ -370,9 +368,8 @@ spec:
- "262144"
- --page-size
- "64"
- --enable-dp-attention
- --enable-dp-lm-head
- --dp-size
- --attn-dp-size
- "16"
- --moe-a2a-backend
- deepep
@@ -503,10 +500,9 @@ spec:
- "262144"
- --page-size
- "64"
- --enable-dp-attention
- --enable-dp-lm-head
#- --enable-two-batch-overlap
- --dp-size
- --attn-dp-size
- "16"
- --moe-a2a-backend
- deepep
@@ -69,8 +69,7 @@ spec:
- --eplb-algorithm
- deepseek
- --enable-dp-lm-head
- --enable-dp-attention
- --dp-size
- --attn-dp-size
- "8"
- --moe-a2a-backend
- deepep
@@ -267,9 +266,8 @@ spec:
- "29"
- --page-size
- "64"
- --enable-dp-attention
- --enable-dp-lm-head
- --dp-size
- --attn-dp-size
- "8"
- --moe-a2a-backend
- deepep
@@ -338,9 +336,8 @@ spec:
- "29"
- --page-size
- "64"
- --enable-dp-attention
- --enable-dp-lm-head
- --dp-size
- --attn-dp-size
- "32"
- --moe-a2a-backend
- "deepep"
+2 -1
View File
@@ -810,7 +810,8 @@ export const Deployment = ({ config, benchmarks }) => {
// Insert the multi-node trio after the last parallelism flag,
// falling back to right after --model-path.
const PARALLELISM_ANCHORS = new Set([
"--enable-dp-attention", "--dp-size", "--dp", "--tp-size", "--tp",
"--attn-dp-size", "--attention-data-parallel-size",
"--dp-size", "--dp", "--tp-size", "--tp",
"--sp-degree", "--ulysses-degree", "--ring-degree",
]);
let i = flags.reduce(
@@ -67,7 +67,7 @@ export const KimiK3MambaRatioCalculator = () => {
const hasFlag = (name) => flags.some((f) => f.split(/[\s=]/)[0] === name);
const tp = Number(flagArg("--tp-size")) || 8;
const dp = hasFlag("--enable-dp-attention") ? Number(flagArg("--dp-size")) || 1 : 1;
const dp = Number(flagArg("--attn-dp-size")) || 1;
// KDA state and the replicated MLA KV both live per attention-TP group.
// PP needs no term: it splits the layers of both pools equally.
const attnTp = Math.max(1, Math.round(tp / dp));
@@ -231,9 +231,9 @@ export const KimiK3MambaRatioCalculator = () => {
: `DSPARK (D = ${block + 1})`;
const derivedChips = [
// With DP attention on, show the whole topology so a large-scale preset is
// visibly understood: total GPUs = DP replicas x attention-TP group width.
// visibly understood: total GPUs = attention DP size x attention-TP group width.
dp > 1
? `${tp} GPUs = DP ${dp} × Attention TP ${attnTp}`
? `${tp} GPUs = Attention DP ${dp} × Attention TP ${attnTp}`
: `Attention TP ${attnTp}`,
`DCP ${dcp}`,
`KV ${kvDtype === "fp8_e4m3" ? "FP8" : "BF16"}`,
+35 -37
View File
@@ -360,6 +360,7 @@ export const Playground = ({ config }) => {
const TP_HEADS = ["--tp-size", "--tp", "--tensor-parallel-size"];
const EP_HEADS = ["--ep-size", "--ep", "--expert-parallel-size"];
const DP_HEADS = ["--dp-size", "--dp", "--data-parallel-size"];
const ATTN_DP_HEADS = ["--attn-dp-size", "--attention-data-parallel-size"];
const parseIntFlagAny = (flags, heads) => {
for (const head of heads) {
const n = parseIntFlag(flags, head);
@@ -370,22 +371,25 @@ export const Playground = ({ config }) => {
const flagSpelling = (flags, heads, fallback) =>
heads.find((head) =>
(flags || []).some((f) => f.split(/[\s=]/)[0] === head)) || fallback;
// DP-Attention is on when the attention DP size is above 1 (1 = off).
const dpAttnOnIn = (flags) => (parseIntFlagAny(flags, ATTN_DP_HEADS) ?? 1) > 1;
// Insertion-anchor sets (priority-ordered; each includes siblings so
// insertion still works in partial cells).
const ANCHOR_NEAR_MODEL_PATH = ["--model-path"];
const ANCHOR_NEAR_TP = ["--tp-size", "--tp", "--model-path"];
const ANCHOR_NEAR_DP = ["--dp-size", "--dp", "--tp-size", "--tp", "--model-path"];
const ANCHOR_NEAR_DPATTN = ["--enable-dp-attention", "--dp-size", "--dp", "--tp-size", "--tp", "--model-path"];
const ANCHOR_NEAR_DPATTN = [...ATTN_DP_HEADS, "--dp-size", "--dp", "--tp-size", "--tp", "--model-path"];
const ANCHOR_NEAR_MOE = ["--moe-a2a-backend", "--moe-runner-backend",
"--enable-dp-attention", "--dp-size", "--dp", "--tp-size", "--tp", "--model-path"];
...ATTN_DP_HEADS, "--dp-size", "--dp", "--tp-size", "--tp", "--model-path"];
// Helper bundle passed to every axis handler.
const helpers = {
matchConstraint, evaluateChip, findEntry, isHidden,
stripFlagsByFirstToken, stripEnvByPrefix, insertBeforeTail, insertAfter,
parseIntFlag, hasFlag, findFlagArg,
TP_HEADS, EP_HEADS, DP_HEADS, parseIntFlagAny, flagSpelling,
TP_HEADS, EP_HEADS, DP_HEADS, ATTN_DP_HEADS, parseIntFlagAny, flagSpelling,
dpAttnOnIn,
ANCHOR_NEAR_MODEL_PATH, ANCHOR_NEAR_TP, ANCHOR_NEAR_DP,
ANCHOR_NEAR_DPATTN, ANCHOR_NEAR_MOE,
};
@@ -452,27 +456,23 @@ export const Playground = ({ config }) => {
const AXIS_HANDLERS = {
// ---- Axis: Attention Parallelism ----------------------------------------
// TP / CP / DP-Attention sub-knobs; `null` = inherit. DP-Attention is
// combined: a numeric value emits `--dp N --enable-dp-attention`, `false`
// strips both. An optional `cpStrategy` knob (values from --cp-strategy:
// "zigzag" / "interleave") picks the CP layout; without it the strategy
// baked in the base is preserved, defaulting to "interleave" (the legacy
// knob's round-robin-split).
// TP / CP / DP-Attention sub-knobs; `null` = inherit. DP-Attention's
// value is the attention DP size: a numeric value emits
// `--attn-dp-size N`, `false` strips it. An optional `cpStrategy` knob
// (values from --cp-strategy: "zigzag" / "interleave") picks the CP
// layout; without it the strategy baked in the base is preserved,
// defaulting to "interleave" (the legacy knob's round-robin-split).
attention: {
initState: () => ({ tp: null, cp: null, cpStrategy: null, dpAttn: null }),
// DP-Attention: `--dp N --enable-dp-attention` → N; neither → false;
// bare `--enable-dp-attention` → 1. CP: any enable spelling →
// `--attn-cp-size N` (bare enable → 2, the legacy convention), plus the
// baked strategy (legacy mode flags mapped to zigzag/interleave).
// DP-Attention: `--attn-dp-size N` → N; absent → false. CP: any enable
// spelling → `--attn-cp-size N` (bare enable → 2, the legacy
// convention), plus the baked strategy (legacy mode flags mapped to
// zigzag/interleave).
deriveFromBase: (cell, fc, h) => {
const flags = (cell && cell.flags) || [];
const dpVal = h.parseIntFlagAny(flags, h.DP_HEADS);
const hasDpAttn = h.hasFlag(flags, "--enable-dp-attention");
let dpAttn;
if (dpVal !== null) dpAttn = dpVal;
else if (hasDpAttn) dpAttn = 1;
else dpAttn = false;
const dpVal = h.parseIntFlagAny(flags, h.ATTN_DP_HEADS);
const dpAttn = dpVal !== null ? dpVal : false;
const cpSize = h.parseIntFlag(flags, "--attn-cp-size");
return {
tp: h.parseIntFlagAny(flags, h.TP_HEADS),
@@ -488,7 +488,7 @@ export const Playground = ({ config }) => {
const knobEntry = (id) => (fc.knobs || []).find((k) => k.id === id) || {};
const factsNow = () => ({
...(sel || {}),
dpAttnOn: h.hasFlag(flags, "--enable-dp-attention"),
dpAttnOn: h.dpAttnOnIn(flags),
cpOn: cpEnabledIn(flags),
cpStrategy: bakedCpStrategy(flags) || "interleave",
effTp: h.parseIntFlagAny(flags, h.TP_HEADS),
@@ -503,8 +503,7 @@ export const Playground = ({ config }) => {
if (knobEntry("cp").freeSize) return null;
const dpIntent = (value.dpAttn !== null && value.dpAttn !== undefined)
? value.dpAttn
: (h.hasFlag(flags, "--enable-dp-attention")
? (h.parseIntFlagAny(flags, h.DP_HEADS) ?? 1) : false);
: (h.parseIntFlagAny(flags, h.ATTN_DP_HEADS) ?? false);
if (typeof dpIntent === "number" && dpIntent > 1) return null;
return h.parseIntFlagAny(flags, h.TP_HEADS);
};
@@ -524,7 +523,7 @@ export const Playground = ({ config }) => {
&& h.evaluateChip(e, facts).disabled);
};
// NOTE: interleave prefill-CP + DP-Attention currently fails the
// runtime's dp_size == 1 assert, but combined support is planned
// runtime's attn_dp_size == 1 assert, but combined support is planned
// upstream — the combination is allowed here (with a warning hint
// below the command box) rather than banned.
@@ -567,7 +566,7 @@ export const Playground = ({ config }) => {
(r) => r && h.matchConstraint(factsNow(), r.when));
if (dpForced) {
flags = h.stripFlagsByFirstToken(flags, [
...h.DP_HEADS, "--enable-dp-attention",
...h.ATTN_DP_HEADS,
"--enable-dp-attention-local-control-broadcast",
]);
// The cell's DP-only env would otherwise outlive the flags it tunes.
@@ -579,22 +578,21 @@ export const Playground = ({ config }) => {
&& !blocked("dpAttn", value.dpAttn)) {
// Capture the spelling before stripping — the TP/EP handlers do the
// same, and a lookup on the stripped array always hits the fallback.
const dpHead = h.flagSpelling(flags, h.DP_HEADS, "--dp-size");
const dpHead = h.flagSpelling(flags, h.ATTN_DP_HEADS, "--attn-dp-size");
// The local-control-broadcast companion only means anything with DP
// attention on, so it has to go down with it — stripping just
// `--enable-dp-attention` would leave it orphaned in the command.
// `--attn-dp-size` would leave it orphaned in the command.
// apply re-seeds from the base cell, so this restores it when the
// cell had it and the user is only re-sizing DP rather than disabling.
const hadLocalBroadcast =
h.hasFlag(flags, "--enable-dp-attention-local-control-broadcast");
flags = h.stripFlagsByFirstToken(flags, [
...h.DP_HEADS, "--enable-dp-attention",
...h.ATTN_DP_HEADS,
"--enable-dp-attention-local-control-broadcast",
]);
if (typeof value.dpAttn === "number" && value.dpAttn > 0) {
flags = h.insertAfter(flags, h.ANCHOR_NEAR_TP, [
`${dpHead} ${value.dpAttn}`,
"--enable-dp-attention",
...(hadLocalBroadcast
? ["--enable-dp-attention-local-control-broadcast"] : []),
]);
@@ -608,7 +606,7 @@ export const Playground = ({ config }) => {
if (!knobs.length) return null;
const setKnob = (k, v) => setValue({ ...value, [k]: v });
// Interleave prefill-CP + DP-Attention is deliberately NOT grayed:
// current releases assert dp_size == 1 for interleave, but combined
// current releases assert attn_dp_size == 1 for interleave, but combined
// support is planned upstream — a warning hint below the command box
// covers it instead.
const labelFor = (knob) => (c) => {
@@ -930,7 +928,7 @@ export const Playground = ({ config }) => {
const picked = (fc.options || []).find((p) => p.id === value);
if (picked && h.evaluateChip(picked, {
...sel,
dpAttnOn: h.hasFlag(flags, "--enable-dp-attention"),
dpAttnOn: h.dpAttnOnIn(flags),
}).disabled) {
return { flags, env };
}
@@ -1414,7 +1412,7 @@ export const Playground = ({ config }) => {
// flags rather than the Deploy dims — the attention axis runs first.
// A role override with `allowTp` is a TP-only shape validated as-is.
if (fc.requiresDpAttention && !(roleOverride && roleOverride.allowTp)
&& !flags.some((f) => f.split(/[\s=]/)[0] === "--enable-dp-attention")) {
&& !h.dpAttnOnIn(flags)) {
return { flags, env };
}
flags = h.stripFlagsByFirstToken(flags, HICACHE_HEADS);
@@ -1517,7 +1515,7 @@ export const Playground = ({ config }) => {
apply: ({ flags, env, value, fc, sel, h, derived }) => {
const evalBase = {
...(sel || {}),
dpAttnOn: h.hasFlag(flags, "--enable-dp-attention"),
dpAttnOn: h.dpAttnOnIn(flags),
pdMode: h.findFlagArg(flags, "--disaggregation-mode") || "off",
};
for (const spec of (fc || [])) {
@@ -1701,7 +1699,7 @@ export const Playground = ({ config }) => {
if (multinode && !f.some((x) => x.startsWith("--nnodes"))) {
// Insert the multi-node trio after the last parallelism flag (matches
// _deployment.jsx so untouched-base output is byte-identical).
const PARALLELISM_ANCHORS = ["--enable-dp-attention", "--dp-size", "--dp", "--tp-size", "--tp"];
const PARALLELISM_ANCHORS = [...ATTN_DP_HEADS, "--dp-size", "--dp", "--tp-size", "--tp"];
let at = -1;
for (const anchor of PARALLELISM_ANCHORS) {
at = f.findIndex((x) => x.split(/[\s=]/)[0] === anchor);
@@ -2418,7 +2416,7 @@ export const Playground = ({ config }) => {
? (staleExplicit("dpAttn", attnDelta.dpAttn) ? null : attnDelta.dpAttn)
: (attnDerived.dpAttn !== undefined ? attnDerived.dpAttn : null);
const dpAttnOn = (effDpAttn === true)
|| (typeof effDpAttn === "number" && effDpAttn > 0);
|| (typeof effDpAttn === "number" && effDpAttn > 1);
// Runtime derivation attn_cp_size = tp/dp: with DP-Attention off, the only
// enable-able CP size is TP. With DP-Attention on, sizes are NOT gated —
// CP + DP-Attention is an allowed experiment covered by a warning hint
@@ -2542,11 +2540,11 @@ export const Playground = ({ config }) => {
// Interleave prefill-CP + DP-Attention hint on the EFFECTIVE command:
// deliberately allowed (combined support is planned upstream), but current
// releases assert dp_size == 1 for the interleave layout at startup.
// releases assert attn_dp_size == 1 for the interleave layout at startup.
const pgCpDpHint =
cpEnabledIn(pgFlagsLatest)
&& (bakedCpStrategy(pgFlagsLatest) || "interleave") === "interleave"
&& pgFlagsLatest.some((f) => f.split(/[\s=]/)[0] === "--enable-dp-attention");
&& dpAttnOnIn(pgFlagsLatest);
// Submission snippets: proposed cell + existing cell at the same match.
const proposedCellSnippet = baseCell
@@ -2822,7 +2820,7 @@ export const Playground = ({ config }) => {
)}
{pgCpDpHint && (
<div style={s.mtpWarn}>
⚠️ Interleave prefill-CP together with DP-Attention: current SGLang releases assert <code>dp_size == 1</code> for the interleave layout, so this command fails at startup. Combined CP + DP-Attention support is planned upstream — keep one of the two off until it lands.
⚠️ Interleave prefill-CP together with DP-Attention: current SGLang releases assert <code>attn_dp_size == 1</code> for the interleave layout, so this command fails at startup. Combined CP + DP-Attention support is planned upstream — keep one of the two off until it lands.
</div>
)}
</div>
@@ -51,9 +51,9 @@ export const DeepSeekMathV2Deployment = () => {
// TP setting
cmd += ` \\\n --tp ${tpValue}`;
// DP Attention: --dp matches --tp
// DP Attention: --attn-dp-size matches --tp
if (values.dpattention === 'enabled') {
cmd += ` \\\n --dp ${tpValue} \\\n --enable-dp-attention`;
cmd += ` \\\n --attn-dp-size ${tpValue}`;
}
// EP setting (commonly matches tp for MoE models)
@@ -24,7 +24,6 @@ export const DeepSeekOCRDeployment = () => {
type: 'checkbox',
items: [
{ id: 'tp', label: 'TP', subtitle: 'Tensor Parallel', default: true, required: true },
{ id: 'dp', label: 'DP', subtitle: 'Data Parallel', default: false, disabledWhen: (v) => v.hardware === 'xeon', disabledReason: 'Intel Xeon CPUs only support Tensor Parallel (TP)' },
{ id: 'ep', label: 'EP', subtitle: 'Expert Parallel', default: false, disabledWhen: (v) => v.hardware === 'xeon', disabledReason: 'Intel Xeon CPUs only support Tensor Parallel (TP)' }
]
}
@@ -117,11 +116,6 @@ export const DeepSeekOCRDeployment = () => {
cmd += ` \\\n --tp 1`;
}
// DP strategy
if (strategyArray.includes('dp')) {
cmd += ` \\\n --dp 1 \\\n --enable-dp-attention`;
}
// EP strategy
if (strategyArray.includes('ep')) {
cmd += ` \\\n --ep 1`;
@@ -26,7 +26,6 @@ export const DeepSeekOCR2Deployment = () => {
type: 'checkbox',
items: [
{ id: 'tp', label: 'TP', subtitle: 'Tensor Parallel', default: true, required: true },
{ id: 'dp', label: 'DP', subtitle: 'Data Parallel', default: false, disabledWhen: (v) => v.hardware === 'xeon' || v.hardware === 'arc_b', disabledReason: 'Only Tensor Parallel (TP) is supported on this hardware' },
{ id: 'ep', label: 'EP', subtitle: 'Expert Parallel', default: false, disabledWhen: (v) => v.hardware === 'xeon' || v.hardware === 'arc_b', disabledReason: 'Only Tensor Parallel (TP) is supported on this hardware' }
]
},
@@ -52,10 +51,6 @@ export const DeepSeekOCR2Deployment = () => {
cmd += ` \\\n --tp 1`;
}
if (strategyArray.includes('dp')) {
cmd += ` \\\n --dp 1 \\\n --enable-dp-attention`;
}
if (strategyArray.includes('ep')) {
cmd += ` \\\n --ep 1`;
}
@@ -116,7 +116,7 @@ export const DeepSeekR1BasicDeployment = () => {
command += isXeon ? ' \\\n --tp 6' : ' \\\n --tp 8';
}
if (strategyValues.includes('dp')) {
command += ' \\\n --dp 8 \\\n --enable-dp-attention';
command += ' \\\n --attn-dp-size 8';
}
if (strategyValues.includes('ep')) {
command += ' \\\n --ep 8';
@@ -151,7 +151,7 @@ export const DeepSeekV3Deployment = () => {
cmd += ` --model-path ${modelPath}`;
if (strategyArray.includes('tp')) cmd += isXeon ? ' \\\n --tp 6' : ' \\\n --tp 8';
if (strategyArray.includes('dp')) cmd += ' \\\n --dp 8 \\\n --enable-dp-attention';
if (strategyArray.includes('dp')) cmd += ' \\\n --attn-dp-size 8';
if (strategyArray.includes('ep')) cmd += ' \\\n --ep 8';
if (strategyArray.includes('mtp')) {
cmd += ' \\\n --speculative-algorithm EAGLE \\\n --speculative-num-steps 3 \\\n --speculative-eagle-topk 1 \\\n --speculative-num-draft-tokens 4';
@@ -148,7 +148,7 @@ export const DeepSeekV31Deployment = () => {
// TP is mandatory
cmd += isXeon ? ` \\\n --tp 6` : ` \\\n --tp 8`;
if (strategyArray.includes('dp')) {
cmd += ` \\\n --dp 8 \\\n --enable-dp-attention`;
cmd += ` \\\n --attn-dp-size 8`;
}
if (strategyArray.includes('ep')) {
cmd += ` \\\n --ep 8`;
@@ -219,7 +219,7 @@ export const DeepSeekV32Deployment = () => {
const epSize = 8;
cmd += ` \\\n --tp ${tpSize}`;
if (strategyArray.includes('dp')) {
cmd += ` \\\n --dp ${dpSize} \\\n --enable-dp-attention`;
cmd += ` \\\n --attn-dp-size ${dpSize}`;
}
if (strategyArray.includes('ep')) {
cmd += ` \\\n --ep ${epSize}`;
@@ -55,7 +55,7 @@ export const Ernie45Deployment = () => {
}
if (strategyArray.includes('dp') && modelsize === '300b') {
cmd += ` \\\n --dp ${dpValue} \\\n --enable-dp-attention`;
cmd += ` \\\n --attn-dp-size ${dpValue}`;
}
if (strategyArray.includes('ep') && modelsize === '300b') {
@@ -121,7 +121,7 @@ export const GLM45Deployment = () => {
// Strategy-specific parameters
if (strategyArray.includes('dp')) {
cmd += ` \\\n --dp 8 \\\n --enable-dp-attention`;
cmd += ` \\\n --attn-dp-size 8`;
}
if (strategyArray.includes('ep')) {
cmd += ` \\\n --ep 8`;
@@ -132,7 +132,7 @@ export const GLM46Deployment = () => {
// Strategy-specific parameters
if (strategyArray.includes('dp')) {
cmd += ` \\\n --dp 8 \\\n --enable-dp-attention`;
cmd += ` \\\n --attn-dp-size 8`;
if (hardware === 'b300') {
cmd += ` \\\n --cuda-graph-max-bs-decode 256`;
}
@@ -176,7 +176,7 @@ export const GLM47Deployment = () => {
cmd += ` \\\n --max-context-length 8192 \\\n --mem-fraction-static 0.9`;
}
if (strategyArray.includes('dp')) {
cmd += ` \\\n --dp 8 \\\n --enable-dp-attention`;
cmd += ` \\\n --attn-dp-size 8`;
}
if (strategyArray.includes('ep')) {
cmd += ` \\\n --ep 8`;
@@ -23,7 +23,6 @@ export const GLM47FlashDeployment = () => {
type: 'checkbox',
items: [
{ id: 'tp', label: 'TP', subtitle: 'Tensor Parallel', default: true, required: true },
{ id: 'dp', label: 'DP', subtitle: 'Data Parallel', default: false },
{ id: 'mtp', label: 'MTP', subtitle: 'Multi-token Prediction', default: false }
]
},
@@ -113,9 +112,6 @@ export const GLM47FlashDeployment = () => {
}
// Strategy-specific parameters
if (strategyArray.includes('dp')) {
cmd += ` \\\n --dp 1 \\\n --enable-dp-attention`;
}
if (strategyArray.includes('mtp')) {
if (hardware === 'b200') {
@@ -199,7 +199,7 @@ export const GLM5Deployment = () => {
}
if (values.dpattention === 'enabled') {
cmd += ` \\\n --dp ${tpValue} \\\n --enable-dp-attention`;
cmd += ` \\\n --attn-dp-size ${tpValue}`;
if (hardware === 'b300') {
cmd += ' \\\n --cuda-graph-max-bs-decode 256';
}
@@ -166,7 +166,7 @@ export const GLM51Deployment = () => {
}
if (values.dpattention === 'enabled') {
cmd += ` \\\n --dp ${tpValue} \\\n --enable-dp-attention`;
cmd += ` \\\n --attn-dp-size ${tpValue}`;
}
if (values.reasoning === 'enabled') cmd += ' \\\n --reasoning-parser glm45';
if (values.toolcall === 'enabled') cmd += ' \\\n --tool-call-parser glm47';
@@ -75,7 +75,7 @@ export const KimiK2Deployment = () => {
const strategyArray = Array.isArray(strategy) ? strategy : [];
cmd += ` \\\n --tp 8`;
if (strategyArray.includes('dp')) {
cmd += ` \\\n --dp 4 \\\n --enable-dp-attention`;
cmd += ` \\\n --attn-dp-size 4`;
}
if (strategyArray.includes('ep')) {
cmd += ` \\\n --ep 4`;
@@ -176,9 +176,9 @@ export const KimiK25Deployment = () => {
cmd += ` \\\n --tp ${tpValue}`;
cmd += ' \\\n --trust-remote-code';
// DP Attention: --dp matches --tp
// DP Attention: --attn-dp-size matches --tp
if (values.dpattention === 'enabled') {
cmd += ` \\\n --dp ${tpValue} \\\n --enable-dp-attention`;
cmd += ` \\\n --attn-dp-size ${tpValue}`;
}
// Reasoning parser
@@ -175,7 +175,7 @@ export const KimiK26Deployment = () => {
cmd += ' \\\n --trust-remote-code';
if (dpattention === 'enabled') {
cmd += ` \\\n --dp ${tpValue} \\\n --enable-dp-attention`;
cmd += ` \\\n --attn-dp-size ${tpValue}`;
}
if (reasoning === 'enabled') {
@@ -179,7 +179,7 @@ export const KimiK27CodeDeployment = () => {
cmd += ' \\\n --trust-remote-code';
if (dpattention === 'enabled') {
cmd += ` \\\n --dp ${tpValue} \\\n --enable-dp-attention`;
cmd += ` \\\n --attn-dp-size ${tpValue}`;
}
if (reasoning === 'enabled') {
@@ -127,8 +127,7 @@ export const LagunaXS2Deployment = () => {
if (dpAttention === 'enabled') {
lines[lines.length - 1] += ' \\';
lines.push(` --dp ${tp} \\`);
lines.push(' --enable-dp-attention');
lines.push(` --attn-dp-size ${tp}`);
}
if (reasoning === 'enabled') {
@@ -235,7 +235,6 @@ export const Llama31Deployment = () => {
// NVIDIA-specific optimizations
if (!isAMD && !isXeon && hardware !== 'arc_b') {
if (optimization === 'throughput') {
args.push(`--enable-dp-attention`);
args.push(`--mem-fraction-static 0.85`);
} else if (optimization === 'latency') {
args.push(`--speculative-algorithm EAGLE3`);
@@ -112,7 +112,7 @@ export const MiMoV2FlashDeployment = () => {
// DP settings
if (!isMI355X && strategyArray.includes('dp')) {
cmd += ` \\\n --dp-size 2 \\\n --enable-dp-attention`;
cmd += ` \\\n --attn-dp-size 2`;
}
// Performance Optimizations
@@ -138,14 +138,14 @@ export const MiMoV25Deployment = () => {
const c = {};
// Both checkpoints are TP-interleaved (Pro: 8, base: 4), so attention-TP per
// DP group must equal that factor. When the spec carries dp>1 (Pro/Hopper
// tp=16, base tp=8) DP-attention with `--dp = tp/factor` is required; without
// tp=16, base tp=8) DP-attention with `--attn-dp-size = tp/factor` is required; without
// it a bare `--tp` gives attn_tp = tp != factor and the loader rejects the
// checkpoint. When no dp>1 (Pro/Blackwell tp=8, base tp=4) it's a single
// attention group and DP-attention must stay off.
if (spec && !jax) {
const factor = isPro ? 8 : 4;
if (spec.dp > 1) {
c.dpAttention = { force: "enabled", reason: `Checkpoint is TP=${factor}-interleaved; DP-attention is required (--dp = tp/${factor} = ${spec.dp}).` };
c.dpAttention = { force: "enabled", reason: `Checkpoint is TP=${factor}-interleaved; DP-attention is required (--attn-dp-size = tp/${factor} = ${spec.dp}).` };
} else {
c.dpAttention = { force: "disabled", reason: `Single attention group on this hardware (tp=${factor}, no dp-attention).` };
}
@@ -243,7 +243,8 @@ export const MiMoV25Deployment = () => {
// sgl-jax conventions:
// - `--tp-size` is always the total JAX device count; per-DP TP is
// derived automatically as tp/dp.
// - No `--enable-dp-attention` flag — DP attention is the default
// - No separate attention-DP flag (SGLang's `--attn-dp-size`) —
// sgl-jax's `--dp-size` is DP attention by default
// (FFN layers auto-pick EP-split for MoE, attn-TP-split for dense).
const isV7x = hardware === "tpu-v7x";
const useEp = expertParallelism === "enabled";
@@ -309,8 +310,7 @@ export const MiMoV25Deployment = () => {
flags.push(` --tp ${tp}`);
if (useDpAttn) {
flags.push(` --dp ${dpSize}`);
flags.push(" --enable-dp-attention");
flags.push(` --attn-dp-size ${dpSize}`);
if (!isPro) {
flags.push(" --enable-dp-lm-head");
flags.push(" --mm-enable-dp-encoder");
@@ -30,7 +30,7 @@ export const Nemotron3UltraDeployment = () => {
[...new Set(VERIFIED_CONFIGS.filter((c) => c.model === model && c.hardware === hardware).map((c) => c.tp))];
// DP attention is verified at dp=2 for BF16, and dp in {2,4,8} for NVFP4. SGLang
// requires tp_size % dp_size == 0, so dp is capped at both the selected TP and the
// requires tp_size % attn_dp_size == 0, so dp is capped at both the selected TP and the
// max verified TP for this model+hardware (whichever is smaller).
const dpCandidatesForModel = (model) => (model === 'bf16' ? ['2'] : ['2', '4', '8']);
@@ -128,12 +128,12 @@ export const Nemotron3UltraDeployment = () => {
};
});
},
// dp_size must divide tp_size; only emit when the selected DP is valid for the current TP.
// attn_dp_size must divide tp_size; only emit when the selected DP is valid for the current TP.
commandRule: (value, state) =>
value && value !== 'disabled' &&
dpCandidatesForModel(state.model).includes(value) &&
Number(value) <= Number(state.tp)
? `--dp ${value} \\\n --enable-dp-attention`
? `--attn-dp-size ${value}`
: null
},
mtp: {
@@ -119,11 +119,6 @@ export const Qwen3CoderDeployment = () => {
cmd += ` \\\n --ep 2`;
}
// DP attention setting
if (quantization === 'nvfp4') {
cmd += ` \\\n --enable-dp-attention`;
}
// MOE runner backend for NVIDIA
if (isNvidia) {
if (quantization === 'nvfp4') {
+3 -6
View File
@@ -367,9 +367,8 @@ export const config = {
"--trust-remote-code",
"--model-path {{MODEL_NAME}}",
"--tp-size 16",
"--dp-size 4",
"--attn-dp-size 4",
"--ep-size 16",
"--enable-dp-attention",
"--enable-dp-lm-head",
"--enable-dp-attention-local-control-broadcast",
"--moe-dense-tp-size 1",
@@ -452,9 +451,8 @@ export const config = {
"--trust-remote-code",
"--model-path {{MODEL_NAME}}",
"--tp-size 16",
"--dp-size 4",
"--attn-dp-size 4",
"--ep-size 16",
"--enable-dp-attention",
"--enable-dp-lm-head",
"--enable-dp-attention-local-control-broadcast",
"--moe-dense-tp-size 1",
@@ -540,8 +538,7 @@ export const config = {
"--trust-remote-code",
"--model-path {{MODEL_NAME}}",
"--tp-size 16",
"--dp-size 16",
"--enable-dp-attention",
"--attn-dp-size 16",
"--enable-dp-lm-head",
"--enable-dp-attention-local-control-broadcast",
"--moe-dense-tp-size 1",
@@ -235,11 +235,11 @@ sgl-eval run mmmu_pro \\
playgroundFeatures: {
// ----- Card 1: "Attention Parallelism" -----
// DP-Attention is a combined knob: value is the DP degree AND toggles `--enable-dp-attention`.
// DP-Attention is a single knob: value is the attention DP size, emitted as `--attn-dp-size N`.
// CP sizes auto-gate in the engine to the runtime derivation
// attn_cp_size = tp/dp (a user-passed --attn-cp-size is overridden).
// CP is single-machine only (tp_size <= 8). Interleave CP + DP-Attention
// currently fails the runtime's dp_size == 1 assert but is allowed here
// currently fails the runtime's attn_dp_size == 1 assert but is allowed here
// with a warning (combined support is planned upstream). No `cpStrategy`
// knob: DeepSeek-V4 supports only interleave (the runtime rejects zigzag).
attention: {
@@ -263,7 +263,7 @@ sgl-eval run mmmu_pro \\
// The low-latency and balanced PD roles run TP-only. That is what lets
// their decode ladders run to the full ceiling (8, 32 on Pro Official
// low-latency, and 96): the ceiling
// is server-wide and floor-divided by attn_dp_size, so only at dp_size
// is server-wide and floor-divided by attn_dp_size, so only at attn_dp_size
// 1 is it also the per-rank batch. Forced rather than left to the
// reader, because switching DP on would cut the slots per rank without
// changing either flag in the command — the ladder would still read 96
@@ -889,8 +889,7 @@ sgl-eval run mmmu_pro \\
"--trust-remote-code",
"--model-path {{MODEL_NAME}}",
"--tp 8",
"--dp 8",
"--enable-dp-attention",
"--attn-dp-size 8",
"--moe-a2a-backend deepep",
"--deepep-config '{\"normal_dispatch\":{\"num_sms\":96},\"normal_combine\":{\"num_sms\":96}}'",
"--host {{HOST_IP}}",
@@ -905,8 +904,7 @@ sgl-eval run mmmu_pro \\
"--trust-remote-code",
"--model-path {{MODEL_NAME}}",
"--tp 4",
"--dp 4",
"--enable-dp-attention",
"--attn-dp-size 4",
"--moe-a2a-backend deepep",
"--speculative-algorithm EAGLE",
"--speculative-num-steps 1",
@@ -927,8 +925,7 @@ sgl-eval run mmmu_pro \\
"--trust-remote-code",
"--model-path {{MODEL_NAME}}",
"--tp 8",
"--dp 8",
"--enable-dp-attention",
"--attn-dp-size 8",
"--moe-a2a-backend megamoe",
"--host {{HOST_IP}}",
"--port {{PORT}}",
@@ -944,8 +941,7 @@ sgl-eval run mmmu_pro \\
"--trust-remote-code",
"--model-path {{MODEL_NAME}}",
"--tp 4",
"--dp 4",
"--enable-dp-attention",
"--attn-dp-size 4",
"--moe-a2a-backend megamoe",
"--host {{HOST_IP}}",
"--port {{PORT}}",
@@ -982,8 +978,7 @@ sgl-eval run mmmu_pro \\
"--trust-remote-code",
"--model-path {{MODEL_NAME}}",
"--tp 8",
"--dp 8",
"--enable-dp-attention",
"--attn-dp-size 8",
"--moe-a2a-backend megamoe",
"--chunked-prefill-size 32768",
"--swa-full-tokens-ratio 0.1",
@@ -1007,8 +1002,7 @@ sgl-eval run mmmu_pro \\
"--trust-remote-code",
"--model-path {{MODEL_NAME}}",
"--tp 8",
"--dp 8",
"--enable-dp-attention",
"--attn-dp-size 8",
"--moe-a2a-backend megamoe",
"--mem-fraction-static 0.835",
"--cuda-graph-max-bs-decode 544",
@@ -1063,8 +1057,7 @@ sgl-eval run mmmu_pro \\
"--trust-remote-code",
"--model-path {{MODEL_NAME}}",
"--tp 4",
"--dp 4",
"--enable-dp-attention",
"--attn-dp-size 4",
"--moe-a2a-backend deepep",
"--deepep-config '{\"normal_dispatch\":{\"num_sms\":96},\"normal_combine\":{\"num_sms\":96}}'",
"--host {{HOST_IP}}",
@@ -1079,8 +1072,7 @@ sgl-eval run mmmu_pro \\
"--trust-remote-code",
"--model-path {{MODEL_NAME}}",
"--tp 4",
"--dp 4",
"--enable-dp-attention",
"--attn-dp-size 4",
"--moe-a2a-backend deepep",
"--speculative-algorithm EAGLE",
"--speculative-num-steps 1",
@@ -1101,8 +1093,7 @@ sgl-eval run mmmu_pro \\
"--trust-remote-code",
"--model-path {{MODEL_NAME}}",
"--tp 4",
"--dp 4",
"--enable-dp-attention",
"--attn-dp-size 4",
"--moe-a2a-backend megamoe",
"--host {{HOST_IP}}",
"--port {{PORT}}",
@@ -1118,8 +1109,7 @@ sgl-eval run mmmu_pro \\
"--trust-remote-code",
"--model-path {{MODEL_NAME}}",
"--tp 4",
"--dp 4",
"--enable-dp-attention",
"--attn-dp-size 4",
"--moe-a2a-backend megamoe",
"--host {{HOST_IP}}",
"--port {{PORT}}",
@@ -1154,8 +1144,7 @@ sgl-eval run mmmu_pro \\
"--trust-remote-code",
"--model-path {{MODEL_NAME}}",
"--tp 8",
"--dp 8",
"--enable-dp-attention",
"--attn-dp-size 8",
"--moe-runner-backend flashinfer_mxfp4",
"--disable-flashinfer-autotune",
"--chunked-prefill-size 32768",
@@ -1181,8 +1170,7 @@ sgl-eval run mmmu_pro \\
"--trust-remote-code",
"--model-path {{MODEL_NAME}}",
"--tp 8",
"--dp 8",
"--enable-dp-attention",
"--attn-dp-size 8",
"--moe-a2a-backend megamoe",
"--mem-fraction-static 0.835",
"--cuda-graph-max-bs-decode 544",
@@ -1408,8 +1396,7 @@ sgl-eval run mmmu_pro \\
"--trust-remote-code",
"--model-path {{MODEL_NAME}}",
"--tp 4",
"--dp 4",
"--enable-dp-attention",
"--attn-dp-size 4",
"--moe-a2a-backend deepep",
"--deepep-config '{\"normal_dispatch\":{\"num_sms\":96},\"normal_combine\":{\"num_sms\":96}}'",
"--host {{HOST_IP}}",
@@ -1424,8 +1411,7 @@ sgl-eval run mmmu_pro \\
"--trust-remote-code",
"--model-path {{MODEL_NAME}}",
"--tp 4",
"--dp 4",
"--enable-dp-attention",
"--attn-dp-size 4",
"--moe-a2a-backend deepep",
"--speculative-algorithm EAGLE",
"--speculative-num-steps 1",
@@ -1446,8 +1432,7 @@ sgl-eval run mmmu_pro \\
"--trust-remote-code",
"--model-path {{MODEL_NAME}}",
"--tp 4",
"--dp 4",
"--enable-dp-attention",
"--attn-dp-size 4",
"--moe-a2a-backend megamoe",
"--host {{HOST_IP}}",
"--port {{PORT}}",
@@ -1463,8 +1448,7 @@ sgl-eval run mmmu_pro \\
"--trust-remote-code",
"--model-path {{MODEL_NAME}}",
"--tp 4",
"--dp 4",
"--enable-dp-attention",
"--attn-dp-size 4",
"--moe-a2a-backend megamoe",
"--host {{HOST_IP}}",
"--port {{PORT}}",
@@ -1507,8 +1491,7 @@ sgl-eval run mmmu_pro \\
"--trust-remote-code",
"--model-path {{MODEL_NAME}}",
"--tp 8",
"--dp 8",
"--enable-dp-attention",
"--attn-dp-size 8",
"--moe-a2a-backend deepep",
"--speculative-algorithm EAGLE",
"--speculative-num-steps 1",
@@ -1533,8 +1516,7 @@ sgl-eval run mmmu_pro \\
"--trust-remote-code",
"--model-path {{MODEL_NAME}}",
"--tp 8",
"--dp 8",
"--enable-dp-attention",
"--attn-dp-size 8",
"--moe-a2a-backend megamoe",
"--mem-fraction-static 0.78",
"--cuda-graph-max-bs-decode 64",
@@ -1673,8 +1655,7 @@ sgl-eval run mmmu_pro \\
"--trust-remote-code",
"--model-path {{MODEL_NAME}}",
"--tp 4",
"--dp 4",
"--enable-dp-attention",
"--attn-dp-size 4",
"--moe-a2a-backend deepep",
"--deepep-config '{\"normal_dispatch\":{\"num_sms\":96},\"normal_combine\":{\"num_sms\":96}}'",
"--host {{HOST_IP}}",
@@ -1689,8 +1670,7 @@ sgl-eval run mmmu_pro \\
"--trust-remote-code",
"--model-path {{MODEL_NAME}}",
"--tp 4",
"--dp 4",
"--enable-dp-attention",
"--attn-dp-size 4",
"--moe-a2a-backend deepep",
"--speculative-algorithm EAGLE",
"--speculative-num-steps 1",
@@ -1711,8 +1691,7 @@ sgl-eval run mmmu_pro \\
"--trust-remote-code",
"--model-path {{MODEL_NAME}}",
"--tp 4",
"--dp 4",
"--enable-dp-attention",
"--attn-dp-size 4",
"--moe-a2a-backend megamoe",
"--host {{HOST_IP}}",
"--port {{PORT}}",
@@ -1728,8 +1707,7 @@ sgl-eval run mmmu_pro \\
"--trust-remote-code",
"--model-path {{MODEL_NAME}}",
"--tp 4",
"--dp 4",
"--enable-dp-attention",
"--attn-dp-size 4",
"--moe-a2a-backend megamoe",
"--host {{HOST_IP}}",
"--port {{PORT}}",
@@ -1764,8 +1742,7 @@ sgl-eval run mmmu_pro \\
"--trust-remote-code",
"--model-path {{MODEL_NAME}}",
"--tp 4",
"--dp 4",
"--enable-dp-attention",
"--attn-dp-size 4",
"--moe-a2a-backend deepep",
"--speculative-algorithm EAGLE",
"--speculative-num-steps 1",
@@ -1789,8 +1766,7 @@ sgl-eval run mmmu_pro \\
"--trust-remote-code",
"--model-path {{MODEL_NAME}}",
"--tp 4",
"--dp 4",
"--enable-dp-attention",
"--attn-dp-size 4",
"--moe-a2a-backend megamoe",
"--mem-fraction-static 0.9",
"--cuda-graph-max-bs-decode 128",
@@ -1833,8 +1809,7 @@ sgl-eval run mmmu_pro \\
"--trust-remote-code",
"--model-path {{MODEL_NAME}}",
"--tp 4",
"--dp 4",
"--enable-dp-attention",
"--attn-dp-size 4",
"--moe-a2a-backend deepep",
"--deepep-config '{\"normal_dispatch\":{\"num_sms\":96},\"normal_combine\":{\"num_sms\":96}}'",
"--host {{HOST_IP}}",
@@ -1855,8 +1830,7 @@ sgl-eval run mmmu_pro \\
"--trust-remote-code",
"--model-path {{MODEL_NAME}}",
"--tp 4",
"--dp 4",
"--enable-dp-attention",
"--attn-dp-size 4",
"--moe-a2a-backend megamoe",
"--mem-fraction-static 0.9",
"--cuda-graph-max-bs-decode 128",
@@ -1898,8 +1872,7 @@ sgl-eval run mmmu_pro \\
"--trust-remote-code",
"--model-path {{MODEL_NAME}}",
"--tp 8",
"--dp 8",
"--enable-dp-attention",
"--attn-dp-size 8",
"--moe-a2a-backend megamoe",
"--chunked-prefill-size 32768",
"--swa-full-tokens-ratio 0.1",
@@ -1917,8 +1890,7 @@ sgl-eval run mmmu_pro \\
"--trust-remote-code",
"--model-path {{MODEL_NAME}}",
"--tp 8",
"--dp 8",
"--enable-dp-attention",
"--attn-dp-size 8",
"--moe-a2a-backend megamoe",
"--mem-fraction-static 0.835",
"--cuda-graph-max-bs-decode 544",
@@ -1962,8 +1934,7 @@ sgl-eval run mmmu_pro \\
"--trust-remote-code",
"--model-path {{MODEL_NAME}}",
"--tp 8",
"--dp 8",
"--enable-dp-attention",
"--attn-dp-size 8",
"--moe-runner-backend flashinfer_mxfp4",
"--disable-flashinfer-autotune",
"--chunked-prefill-size 32768",
@@ -1983,8 +1954,7 @@ sgl-eval run mmmu_pro \\
"--trust-remote-code",
"--model-path {{MODEL_NAME}}",
"--tp 8",
"--dp 8",
"--enable-dp-attention",
"--attn-dp-size 8",
"--moe-a2a-backend megamoe",
"--mem-fraction-static 0.835",
"--cuda-graph-max-bs-decode 544",
@@ -2028,8 +1998,7 @@ sgl-eval run mmmu_pro \\
"--trust-remote-code",
"--model-path {{MODEL_NAME}}",
"--tp 8",
"--dp 8",
"--enable-dp-attention",
"--attn-dp-size 8",
"--moe-a2a-backend deepep",
"--mem-fraction-static 0.78",
"--cuda-graph-max-bs-decode 64",
@@ -2046,8 +2015,7 @@ sgl-eval run mmmu_pro \\
"--trust-remote-code",
"--model-path {{MODEL_NAME}}",
"--tp 8",
"--dp 8",
"--enable-dp-attention",
"--attn-dp-size 8",
"--moe-a2a-backend megamoe",
"--mem-fraction-static 0.78",
"--cuda-graph-max-bs-decode 64",
@@ -2195,8 +2163,7 @@ sgl-eval run mmmu_pro \\
"--trust-remote-code",
"--model-path {{MODEL_NAME}}",
"--tp 8",
"--dp 8",
"--enable-dp-attention",
"--attn-dp-size 8",
"--enable-dp-attention-local-control-broadcast",
"--tokenizer-worker-num 8",
"--stream-interval 20",
@@ -2223,8 +2190,7 @@ sgl-eval run mmmu_pro \\
"--trust-remote-code",
"--model-path {{MODEL_NAME}}",
"--tp 8",
"--dp 8",
"--enable-dp-attention",
"--attn-dp-size 8",
"--enable-dp-attention-local-control-broadcast",
"--tokenizer-worker-num 8",
"--stream-interval 20",
@@ -2356,8 +2322,7 @@ sgl-eval run mmmu_pro \\
"--trust-remote-code",
"--model-path {{MODEL_NAME}}",
"--tp 4",
"--dp 4",
"--enable-dp-attention",
"--attn-dp-size 4",
"--moe-a2a-backend deepep",
"--speculative-algorithm EAGLE",
"--speculative-num-steps 1",
@@ -2381,8 +2346,7 @@ sgl-eval run mmmu_pro \\
"--trust-remote-code",
"--model-path {{MODEL_NAME}}",
"--tp 4",
"--dp 4",
"--enable-dp-attention",
"--attn-dp-size 4",
"--moe-a2a-backend deepep",
"--cuda-graph-max-bs-decode 128",
"--max-running-requests 256",
@@ -2402,8 +2366,7 @@ sgl-eval run mmmu_pro \\
"--trust-remote-code",
"--model-path {{MODEL_NAME}}",
"--tp 16",
"--dp 16",
"--enable-dp-attention",
"--attn-dp-size 16",
"--moe-a2a-backend deepep",
"--cuda-graph-max-bs-decode 8",
"--max-running-requests 32",
@@ -2427,8 +2390,7 @@ sgl-eval run mmmu_pro \\
"--trust-remote-code",
"--model-path {{MODEL_NAME}}",
"--tp 16",
"--dp 16",
"--enable-dp-attention",
"--attn-dp-size 16",
"--moe-a2a-backend deepep",
"--speculative-algorithm EAGLE",
"--speculative-num-steps 1",
@@ -2452,8 +2414,7 @@ sgl-eval run mmmu_pro \\
"--trust-remote-code",
"--model-path {{MODEL_NAME}}",
"--tp 16",
"--dp 16",
"--enable-dp-attention",
"--attn-dp-size 16",
"--moe-a2a-backend deepep",
"--mem-fraction-static 0.88",
"--cuda-graph-max-bs-decode 128",
@@ -2857,8 +2818,7 @@ sgl-eval run mmmu_pro \\
"--trust-remote-code",
"--model-path {{MODEL_NAME}}",
"--tp 8",
"--dp 8",
"--enable-dp-attention",
"--attn-dp-size 8",
"--enable-prefill-delayer",
"--prefill-delayer-max-delay-ms 5000",
"--attention-backend dsv4",
@@ -2890,8 +2850,7 @@ sgl-eval run mmmu_pro \\
"--trust-remote-code",
"--model-path {{MODEL_NAME}}",
"--tp 8",
"--dp 8",
"--enable-dp-attention",
"--attn-dp-size 8",
"--enable-prefill-delayer",
"--prefill-delayer-max-delay-ms 5000",
"--attention-backend dsv4",
@@ -2983,8 +2942,7 @@ sgl-eval run mmmu_pro \\
"--trust-remote-code",
"--model-path {{MODEL_NAME}}",
"--tp 8",
"--dp 8",
"--enable-dp-attention",
"--attn-dp-size 8",
"--enable-dp-attention-local-control-broadcast",
"--tokenizer-worker-num 8",
"--stream-interval 20",
@@ -3019,8 +2977,7 @@ sgl-eval run mmmu_pro \\
"--trust-remote-code",
"--model-path {{MODEL_NAME}}",
"--tp 8",
"--dp 8",
"--enable-dp-attention",
"--attn-dp-size 8",
"--enable-dp-attention-local-control-broadcast",
"--tokenizer-worker-num 8",
"--stream-interval 20",
@@ -3059,8 +3016,7 @@ sgl-eval run mmmu_pro \\
"--trust-remote-code",
"--model-path {{MODEL_NAME}}",
"--tp 8",
"--dp 8",
"--enable-dp-attention",
"--attn-dp-size 8",
"--enable-dp-attention-local-control-broadcast",
"--tokenizer-worker-num 8",
"--stream-interval 20",
@@ -3095,8 +3051,7 @@ sgl-eval run mmmu_pro \\
"--trust-remote-code",
"--model-path {{MODEL_NAME}}",
"--tp 8",
"--dp 8",
"--enable-dp-attention",
"--attn-dp-size 8",
"--enable-dp-attention-local-control-broadcast",
"--tokenizer-worker-num 8",
"--stream-interval 20",
@@ -3166,8 +3121,7 @@ sgl-eval run mmmu_pro \\
"--trust-remote-code",
"--model-path {{MODEL_NAME}}",
"--tp 8",
"--dp 8",
"--enable-dp-attention",
"--attn-dp-size 8",
"--enable-dp-attention-local-control-broadcast",
"--tokenizer-worker-num 8",
"--stream-interval 20",
@@ -3206,8 +3160,7 @@ sgl-eval run mmmu_pro \\
"--trust-remote-code",
"--model-path {{MODEL_NAME}}",
"--tp 8",
"--dp 8",
"--enable-dp-attention",
"--attn-dp-size 8",
"--enable-dp-attention-local-control-broadcast",
"--tokenizer-worker-num 8",
"--stream-interval 20",
@@ -3278,8 +3231,7 @@ sgl-eval run mmmu_pro \\
"--trust-remote-code",
"--model-path {{MODEL_NAME}}",
"--tp 8",
"--dp 8",
"--enable-dp-attention",
"--attn-dp-size 8",
"--enable-dp-attention-local-control-broadcast",
"--tokenizer-worker-num 8",
"--stream-interval 20",
@@ -3318,8 +3270,7 @@ sgl-eval run mmmu_pro \\
"--trust-remote-code",
"--model-path {{MODEL_NAME}}",
"--tp 8",
"--dp 8",
"--enable-dp-attention",
"--attn-dp-size 8",
"--enable-dp-attention-local-control-broadcast",
"--tokenizer-worker-num 8",
"--stream-interval 20",
@@ -3389,8 +3340,7 @@ sgl-eval run mmmu_pro \\
"--trust-remote-code",
"--model-path {{MODEL_NAME}}",
"--tp 8",
"--dp 8",
"--enable-dp-attention",
"--attn-dp-size 8",
"--enable-dp-attention-local-control-broadcast",
"--tokenizer-worker-num 8",
"--stream-interval 20",
@@ -3429,8 +3379,7 @@ sgl-eval run mmmu_pro \\
"--trust-remote-code",
"--model-path {{MODEL_NAME}}",
"--tp 8",
"--dp 8",
"--enable-dp-attention",
"--attn-dp-size 8",
"--enable-dp-attention-local-control-broadcast",
"--tokenizer-worker-num 8",
"--stream-interval 20",
@@ -3606,8 +3555,7 @@ sgl-eval run mmmu_pro \\
flags: [
"--model-path {{MODEL_NAME}}",
"--tp 4",
"--dp 4",
"--enable-dp-attention",
"--attn-dp-size 4",
"--moe-a2a-backend deepep",
"--mem-fraction-static 0.85",
"--deepep-config '{\"normal_dispatch\":{\"num_sms\":96},\"normal_combine\":{\"num_sms\":96}}'",
@@ -3625,8 +3573,7 @@ sgl-eval run mmmu_pro \\
flags: [
"--model-path {{MODEL_NAME}}",
"--tp 4",
"--dp 4",
"--enable-dp-attention",
"--attn-dp-size 4",
"--moe-a2a-backend megamoe",
"--mem-fraction-static 0.85",
"--host {{HOST_IP}}",
@@ -3661,8 +3608,7 @@ sgl-eval run mmmu_pro \\
flags: [
"--model-path {{MODEL_NAME}}",
"--tp 4",
"--dp 4",
"--enable-dp-attention",
"--attn-dp-size 4",
"--moe-a2a-backend deepep",
"--mem-fraction-static 0.85",
"--deepep-config '{\"normal_dispatch\":{\"num_sms\":96},\"normal_combine\":{\"num_sms\":96}}'",
@@ -3681,8 +3627,7 @@ sgl-eval run mmmu_pro \\
flags: [
"--model-path {{MODEL_NAME}}",
"--tp 4",
"--dp 4",
"--enable-dp-attention",
"--attn-dp-size 4",
"--moe-a2a-backend megamoe",
"--mem-fraction-static 0.85",
"--host {{HOST_IP}}",
@@ -3713,8 +3658,7 @@ sgl-eval run mmmu_pro \\
flags: [
"--model-path {{MODEL_NAME}}",
"--tp 4",
"--dp 4",
"--enable-dp-attention",
"--attn-dp-size 4",
"--moe-a2a-backend deepep",
"--mem-fraction-static 0.85",
"--deepep-config '{\"normal_dispatch\":{\"num_sms\":96},\"normal_combine\":{\"num_sms\":96}}'",
@@ -3733,8 +3677,7 @@ sgl-eval run mmmu_pro \\
flags: [
"--model-path {{MODEL_NAME}}",
"--tp 4",
"--dp 4",
"--enable-dp-attention",
"--attn-dp-size 4",
"--moe-a2a-backend megamoe",
"--mem-fraction-static 0.85",
"--host {{HOST_IP}}",
@@ -3763,8 +3706,7 @@ sgl-eval run mmmu_pro \\
flags: [
"--model-path {{MODEL_NAME}}",
"--tp 4",
"--dp 4",
"--enable-dp-attention",
"--attn-dp-size 4",
"--moe-a2a-backend deepep",
"--mem-fraction-static 0.85",
"--deepep-config '{\"normal_dispatch\":{\"num_sms\":96},\"normal_combine\":{\"num_sms\":96}}'",
@@ -3782,8 +3724,7 @@ sgl-eval run mmmu_pro \\
flags: [
"--model-path {{MODEL_NAME}}",
"--tp 4",
"--dp 4",
"--enable-dp-attention",
"--attn-dp-size 4",
"--moe-a2a-backend megamoe",
"--mem-fraction-static 0.85",
"--host {{HOST_IP}}",
@@ -364,7 +364,7 @@ export const config = {
config.isNpuHw(s) ||
["h100", "h200", "mi350x", "mi355x"].includes(s.hw) ||
config.isPipelined(s) ||
!!config.flagOf(config.cellFor(s), "--enable-dp-attention"),
config.sizeOf(config.cellFor(s), "--attn-dp-size") > 1,
disableReason: (s) =>
config.isNpuHw(s)
? "Only DSPARK is supported on this recipe."
@@ -536,7 +536,7 @@ export const config = {
playgroundFeatures: {
// ----- Card: "Attention Parallelism" -----
// DP-Attention is a combined knob: value = DP degree AND toggles `--enable-dp-attention`.
// DP-Attention is a single knob: value = attention DP size, emitted as `--attn-dp-size N`.
// K3's MLA latent KV is TP-replicated, so DP-attention RAISES per-GPU KV pressure —
// dp=8/attn_tp=1 OOMs on a single node; use dp=2/attn_tp=4. No CP knob: K3 uses
// decode context parallel (`--dcp-size`), a different lever from prefill `--attn-cp-size`.
@@ -602,12 +602,6 @@ export const config = {
{ id: "dpAttn", label: "DP-Attention",
values: [
null,
{
// 950PR/DT Series only: the recipe enables DP-Attention at dp=1
// (attn-TP 32), unlike A3's dp=4.
value: 1,
hide: { hw: ["b300", "gb300", "b200", "gb200", "h200", "h100", "mi350x", "mi355x", "a3"] },
},
{
value: false,
get disable() { return [
@@ -615,10 +609,6 @@ export const config = {
when: { hw: ["a3"] },
reason: "Only DP-Attention=4 (2 on Prefill) is supported on this recipe.",
},
{
when: { hw: ["a5"] },
reason: "Only DP-Attention=1 is supported on this recipe.",
},
]; },
},
{
@@ -632,7 +622,7 @@ export const config = {
},
{
when: { hw: ["a5"] },
reason: "Only DP-Attention=1 is supported on this recipe.",
reason: "This recipe runs without DP-Attention.",
},
]; },
},
@@ -641,7 +631,7 @@ export const config = {
get disable() { return [
{
when: { hw: ["a5"] },
reason: "Only DP-Attention=1 is supported on this recipe.",
reason: "This recipe runs without DP-Attention.",
},
{
when: { hw: ["a3"], pdMode: ["prefill"] },
@@ -658,7 +648,7 @@ export const config = {
},
{
when: { hw: ["a5"] },
reason: "Only DP-Attention=1 is supported on this recipe.",
reason: "This recipe runs without DP-Attention.",
},
{
when: { hw: ["b300", "gb300"] },
@@ -680,7 +670,7 @@ export const config = {
},
{
when: { hw: ["a5"] },
reason: "Only DP-Attention=1 is supported on this recipe.",
reason: "This recipe runs without DP-Attention.",
},
{
when: { hw: ["b300", "gb300"] },
@@ -1058,7 +1048,7 @@ export const config = {
stripPrefixes: [
"--tp-size", "--tp", "--tensor-parallel-size",
"--ep-size", "--ep", "--expert-parallel-size",
"--enable-dp-attention", "--dp-size", "--enable-dp-lm-head",
"--attn-dp-size", "--enable-dp-lm-head",
"--dcp-size", "--dcp-comm-backend",
// Every B200 Unified cell carries --pp-size 2; left standing it
// multiplies against the preset's --tp-size for a world size the
@@ -1083,7 +1073,7 @@ export const config = {
const nnodes = n / gpusPerNode;
return [
`--tp-size ${n}`, `--ep-size ${n}`,
...(dp > 1 ? ["--enable-dp-attention", `--dp-size ${dp}`, "--enable-dp-lm-head"] : []),
...(dp > 1 ? [`--attn-dp-size ${dp}`, "--enable-dp-lm-head"] : []),
...(nnodes > 1 ? [`--nnodes ${nnodes}`, "--node-rank {{NODE_RANK}}", "--dist-init-addr {{NODE0_IP}}:20000"] : []),
"--moe-a2a-backend megamoe", "--moe-runner-backend deep_gemm",
"--kv-cache-dtype fp8_e4m3", "--mamba-ssm-dtype bfloat16",
@@ -1115,7 +1105,7 @@ export const config = {
const nnodes = n / gpusPerNode;
return [
`--tp-size ${n}`, `--ep-size ${n}`,
...(dp > 1 ? ["--enable-dp-attention", `--dp-size ${dp}`, "--enable-dp-lm-head"] : []),
...(dp > 1 ? [`--attn-dp-size ${dp}`, "--enable-dp-lm-head"] : []),
"--dcp-size 8",
...(nnodes > 1 ? [`--nnodes ${nnodes}`, "--node-rank {{NODE_RANK}}", "--dist-init-addr {{NODE0_IP}}:20000"] : []),
"--moe-a2a-backend megamoe", "--moe-runner-backend deep_gemm",
@@ -2452,8 +2442,7 @@ export const config = {
"--model-path {{MODEL_NAME}}",
"--tp-size 16",
"--dcp-size 8",
"--dp-size 2",
"--enable-dp-attention",
"--attn-dp-size 2",
"--ep-size 16",
"--mem-fraction-static 0.85",
"--disaggregation-decode-extra-slots 16",
@@ -2507,8 +2496,7 @@ export const config = {
"--quantization modelslim",
"--dtype bfloat16",
"--tp-size 64",
"--enable-dp-attention",
"--dp-size 4",
"--attn-dp-size 4",
"--enable-dp-lm-head",
"--enable-shared-experts-attn-tp",
"--enable-dense-mlp-attn-tp",
@@ -2578,8 +2566,7 @@ export const config = {
"--dtype bfloat16",
"--tp-size 16",
"--pp-size 2",
"--dp-size 2",
"--enable-dp-attention",
"--attn-dp-size 2",
"--enable-dp-lm-head",
"--enable-shared-experts-attn-tp",
"--enable-dense-mlp-attn-tp",
@@ -2646,8 +2633,7 @@ export const config = {
"--dtype bfloat16",
"--tp-size 64",
"--pp-size 1",
"--dp-size 4",
"--enable-dp-attention",
"--attn-dp-size 4",
"--enable-dp-lm-head",
"--enable-shared-experts-attn-tp",
"--enable-dense-mlp-attn-tp",
@@ -2672,8 +2658,8 @@ export const config = {
// Ascend 950PR/DT Series: 4 nodes × 8 cards, one rank per card (TP32).
// Unified PD, Balanced, DSPARK-only, with the product-line kernels armed
// per env: FIAS V2 BSND for the DSpark target-verify/draft attention
// paths and the fine-grained dual-stream MoE overlap. DP-attention runs
// at dp=1 (attn-TP 32), and the shared experts / dense MLP shard across
// paths and the fine-grained dual-stream MoE overlap. It runs without
// attention DP (attn-TP 32), and the shared experts / dense MLP shard across
// attention-TP through the server flags (--shared-experts-tp-size 4).
// Checkpoint: the official Moonshot MXFP4 build (moonshotai/Kimi-K3,
// fetched from ModelScope by SGLANG_USE_MODELSCOPE=1 above). Its routed
@@ -2729,7 +2715,6 @@ export const config = {
"--device npu",
"--dtype bfloat16",
"--tp-size 32",
"--enable-dp-attention",
"--enable-dp-lm-head",
"--mem-fraction-static 0.9",
"--chunked-prefill-size 8192",
@@ -2800,7 +2785,6 @@ export const config = {
"--device npu",
"--dtype bfloat16",
"--tp-size 32",
"--enable-dp-attention",
"--enable-dp-lm-head",
"--mem-fraction-static 0.85",
"--chunked-prefill-size 8192",
@@ -2872,7 +2856,6 @@ export const config = {
"--device npu",
"--dtype bfloat16",
"--tp-size 32",
"--enable-dp-attention",
"--enable-dp-lm-head",
"--mem-fraction-static 0.85",
"--chunked-prefill-size 8192",
@@ -135,9 +135,8 @@ export const config = {
flags: [
"--model-path {{MODEL_NAME}}",
"--trust-remote-code",
"--enable-dp-attention",
"--tp 8",
"--dp 8",
"--attn-dp-size 8",
"--ep 8",
"--moe-dense-tp-size 1",
"--moe-a2a-backend deepep",
@@ -184,9 +183,8 @@ export const config = {
flags: [
"--model-path {{MODEL_NAME}}",
"--trust-remote-code",
"--enable-dp-attention",
"--tp 8",
"--dp 8",
"--attn-dp-size 8",
"--ep 8",
"--moe-dense-tp-size 1",
"--moe-a2a-backend deepep",
@@ -233,9 +231,8 @@ export const config = {
flags: [
"--model-path {{MODEL_NAME}}",
"--trust-remote-code",
"--enable-dp-attention",
"--tp 8",
"--dp 8",
"--attn-dp-size 8",
"--ep 8",
"--moe-dense-tp-size 1",
"--moe-a2a-backend deepep",
@@ -282,9 +279,8 @@ export const config = {
flags: [
"--model-path {{MODEL_NAME}}",
"--trust-remote-code",
"--enable-dp-attention",
"--tp 8",
"--dp 8",
"--attn-dp-size 8",
"--ep 8",
"--moe-dense-tp-size 1",
"--moe-a2a-backend deepep",
+15 -29
View File
@@ -114,7 +114,7 @@ sgl-eval run aime25 \\
// CP sizes auto-gate in the engine to the runtime derivation
// attn_cp_size = tp/dp (a user-passed --attn-cp-size is overridden).
// CP is single-machine only (tp_size <= 8). Interleave CP + DP-Attention
// currently fails the runtime's dp_size == 1 assert but is allowed here
// currently fails the runtime's attn_dp_size == 1 assert but is allowed here
// with a warning (combined support is planned upstream).
// Strategy knob: interleave (ex round-robin-split) is the layout verified
// here and the default; zigzag (ex in-seq-split) is exposed as an
@@ -280,8 +280,7 @@ sgl-eval run aime25 \\
flags: [
"--model-path {{MODEL_NAME}}",
"--tp 8",
"--dp 8",
"--enable-dp-attention",
"--attn-dp-size 8",
"--moe-a2a-backend deepep",
"--speculative-algorithm EAGLE",
"--speculative-num-steps 1",
@@ -303,8 +302,7 @@ sgl-eval run aime25 \\
flags: [
"--model-path {{MODEL_NAME}}",
"--tp 8",
"--dp 8",
"--enable-dp-attention",
"--attn-dp-size 8",
"--moe-a2a-backend deepep",
"--mem-fraction-static 0.85",
"--max-running-requests 256",
@@ -339,8 +337,7 @@ sgl-eval run aime25 \\
flags: [
"--model-path {{MODEL_NAME}}",
"--tp 8",
"--dp 8",
"--enable-dp-attention",
"--attn-dp-size 8",
"--moe-a2a-backend deepep",
"--speculative-algorithm EAGLE",
"--speculative-num-steps 1",
@@ -362,8 +359,7 @@ sgl-eval run aime25 \\
flags: [
"--model-path {{MODEL_NAME}}",
"--tp 8",
"--dp 8",
"--enable-dp-attention",
"--attn-dp-size 8",
"--moe-a2a-backend deepep",
"--mem-fraction-static 0.85",
"--max-running-requests 256",
@@ -401,8 +397,7 @@ sgl-eval run aime25 \\
flags: [
"--model-path {{MODEL_NAME}}",
"--tp 4",
"--dp 4",
"--enable-dp-attention",
"--attn-dp-size 4",
"--moe-a2a-backend deepep",
"--speculative-algorithm EAGLE",
"--speculative-num-steps 1",
@@ -425,8 +420,7 @@ sgl-eval run aime25 \\
flags: [
"--model-path {{MODEL_NAME}}",
"--tp 4",
"--dp 4",
"--enable-dp-attention",
"--attn-dp-size 4",
"--moe-a2a-backend deepep",
"--mem-fraction-static 0.85",
"--host {{HOST_IP}}",
@@ -462,8 +456,7 @@ sgl-eval run aime25 \\
flags: [
"--model-path {{MODEL_NAME}}",
"--tp 8",
"--dp 8",
"--enable-dp-attention",
"--attn-dp-size 8",
"--moe-a2a-backend deepep",
"--speculative-algorithm EAGLE",
"--speculative-num-steps 1",
@@ -483,8 +476,7 @@ sgl-eval run aime25 \\
flags: [
"--model-path {{MODEL_NAME}}",
"--tp 8",
"--dp 8",
"--enable-dp-attention",
"--attn-dp-size 8",
"--moe-a2a-backend deepep",
"--mem-fraction-static 0.85",
"--max-running-requests 256",
@@ -728,8 +720,7 @@ sgl-eval run aime25 \\
"--model-path {{MODEL_NAME}}",
"--tp 8",
"--quantization modelopt_fp4",
"--dp 8",
"--enable-dp-attention",
"--attn-dp-size 8",
// Shorter draft (MTP 2-1-3) than low-latency's 5-1-6: at this concurrency the
// verify overhead of a long draft outweighs the accept-length gain.
"--speculative-algorithm EAGLE",
@@ -752,8 +743,7 @@ sgl-eval run aime25 \\
"--model-path {{MODEL_NAME}}",
"--tp 8",
"--quantization modelopt_fp4",
"--dp 8",
"--enable-dp-attention",
"--attn-dp-size 8",
"--chunked-prefill-size 32768",
"--mem-fraction-static 0.92",
"--max-running-requests 512",
@@ -789,8 +779,7 @@ sgl-eval run aime25 \\
flags: [
"--model-path {{MODEL_NAME}}",
"--tp 8",
"--dp 8",
"--enable-dp-attention",
"--attn-dp-size 8",
"--quantization modelopt_fp4",
// Shorter draft (MTP 2-1-3) than low-latency's 5-1-6: at this concurrency the
// verify overhead of a long draft outweighs the accept-length gain.
@@ -816,8 +805,7 @@ sgl-eval run aime25 \\
flags: [
"--model-path {{MODEL_NAME}}",
"--tp 8",
"--dp 8",
"--enable-dp-attention",
"--attn-dp-size 8",
"--quantization modelopt_fp4",
"--max-running-requests 1024",
"--chunked-prefill-size 8192",
@@ -855,8 +843,7 @@ sgl-eval run aime25 \\
"--model-path {{MODEL_NAME}}",
"--tp 4",
"--quantization modelopt_fp4",
"--dp 4",
"--enable-dp-attention",
"--attn-dp-size 4",
// Shorter draft (MTP 2-1-3) than low-latency's 5-1-6: at this concurrency the
// verify overhead of a long draft outweighs the accept-length gain.
"--speculative-algorithm EAGLE",
@@ -878,8 +865,7 @@ sgl-eval run aime25 \\
"--model-path {{MODEL_NAME}}",
"--tp 4",
"--quantization modelopt_fp4",
"--dp 4",
"--enable-dp-attention",
"--attn-dp-size 4",
"--chunked-prefill-size 8192",
"--mem-fraction-static 0.92",
"--max-running-requests 512",
+9 -17
View File
@@ -143,7 +143,7 @@ sgl-eval run aime25 \\
// CP sizes auto-gate in the engine to the runtime derivation
// attn_cp_size = tp/dp (a user-passed --attn-cp-size is overridden).
// CP is single-machine only (tp_size <= 8). Interleave CP + DP-Attention
// currently fails the runtime's dp_size == 1 assert but is allowed here
// currently fails the runtime's attn_dp_size == 1 assert but is allowed here
// with a warning (combined support is planned upstream).
// Strategy knob: interleave (ex round-robin-split) is the default;
// zigzag (ex in-seq-split) is exposed as an
@@ -317,8 +317,7 @@ sgl-eval run aime25 \\
flags: [
"--model-path {{MODEL_NAME}}",
"--tp 8",
"--dp 8",
"--enable-dp-attention",
"--attn-dp-size 8",
"--moe-a2a-backend deepep",
"--speculative-algorithm EAGLE",
"--speculative-num-steps 1",
@@ -360,8 +359,7 @@ sgl-eval run aime25 \\
flags: [
"--model-path {{MODEL_NAME}}",
"--tp 8",
"--dp 8",
"--enable-dp-attention",
"--attn-dp-size 8",
"--moe-a2a-backend deepep",
"--speculative-algorithm EAGLE",
"--speculative-num-steps 1",
@@ -405,8 +403,7 @@ sgl-eval run aime25 \\
flags: [
"--model-path {{MODEL_NAME}}",
"--tp 4",
"--dp 4",
"--enable-dp-attention",
"--attn-dp-size 4",
"--moe-a2a-backend deepep",
"--speculative-algorithm EAGLE",
"--speculative-num-steps 1",
@@ -448,8 +445,7 @@ sgl-eval run aime25 \\
flags: [
"--model-path {{MODEL_NAME}}",
"--tp 8",
"--dp 8",
"--enable-dp-attention",
"--attn-dp-size 8",
"--moe-a2a-backend deepep",
"--speculative-algorithm EAGLE",
"--speculative-num-steps 1",
@@ -698,8 +694,7 @@ sgl-eval run aime25 \\
"--model-path {{MODEL_NAME}}",
"--tp 8",
"--quantization modelopt_fp4",
"--dp 8",
"--enable-dp-attention",
"--attn-dp-size 8",
"--speculative-algorithm EAGLE",
"--speculative-num-steps 2",
"--speculative-eagle-topk 1",
@@ -719,8 +714,7 @@ sgl-eval run aime25 \\
"--model-path {{MODEL_NAME}}",
"--tp 8",
"--quantization modelopt_fp4",
"--dp 8",
"--enable-dp-attention",
"--attn-dp-size 8",
"--chunked-prefill-size 32768",
"--mem-fraction-static 0.92",
"--max-running-requests 512",
@@ -757,8 +751,7 @@ sgl-eval run aime25 \\
"--model-path {{MODEL_NAME}}",
"--tp 8",
"--quantization modelopt_fp4",
"--dp 8",
"--enable-dp-attention",
"--attn-dp-size 8",
"--speculative-algorithm EAGLE",
"--speculative-num-steps 2",
"--speculative-eagle-topk 1",
@@ -799,8 +792,7 @@ sgl-eval run aime25 \\
"--model-path {{MODEL_NAME}}",
"--tp 4",
"--quantization modelopt_fp4",
"--dp 4",
"--enable-dp-attention",
"--attn-dp-size 4",
"--speculative-algorithm EAGLE",
"--speculative-num-steps 2",
"--speculative-eagle-topk 1",
+1 -1
View File
@@ -65,7 +65,7 @@ async def warm_up_compile(
disaggregation_mode: str, tokenizer_manager: TokenizerManager
):
print("\nGenerate warm up request for compiling DeepGEMM...\n")
dp_size = get_parallel().dp_size
dp_size = get_parallel().num_dp_ranks
base_ids = [0, 1, 2, 3]
sampling_params = {
"temperature": 0.0,
@@ -42,11 +42,11 @@ def print_deprecated_warning(message: str):
# * an old valued flag replaced by a renamed valued flag
# -> `DeprecatedAliasStoreAction`.
#
# Only the second has a registration today (`--disable-cuda-graph`). The other
# three are kept because the shapes recur -- this package has retired flags of
# every one of them -- and the fiddly parts (`nargs=0` on a boolean, where the
# const goes, warn-and-continue versus `parser.error`) are what a
# reimplementation gets wrong. Pass `new_flag=` so the warning tells the
# Only the second has registrations today (`--disable-cuda-graph`,
# `--enable-dp-attention`). The other three are kept because the shapes recur
# -- this package has retired flags of every one of them -- and the fiddly
# parts (`nargs=0` on a boolean, where the const goes, warn-and-continue versus
# `parser.error`) are what a reimplementation gets wrong. Pass `new_flag=` so the warning tells the
# operator what to switch to; that pointer is the whole point.
@@ -23,7 +23,7 @@ from sglang.srt.model_executor.cuda_graph_config import (
with_phase,
)
from sglang.srt.platforms import current_platform
from sglang.srt.runtime_context import get_platform
from sglang.srt.runtime_context import attn_dp_enabled_of, get_platform
from sglang.srt.utils.common import (
is_cpu,
is_mps,
@@ -186,7 +186,7 @@ def disable_tc_piecewise_cudagraph_if_incompatible(server_args: Any):
"model-arch blacklist",
lambda: model_config_of(server_args).is_piecewise_cuda_graph_disabled_model,
),
("DP attention", lambda: resolved_view(server_args).enable_dp_attention),
("DP attention", lambda: attn_dp_enabled_of(resolved_view(server_args))),
("full torch.compile mode", lambda: cfg.enable_torch_compile),
("pipeline parallelism (pp_size > 1)", lambda: cfg.pp_size > 1),
(
@@ -11,7 +11,7 @@ from sglang.srt.arg_groups.overrides import (
run_post_process_pass,
)
from sglang.srt.environ import envs
from sglang.srt.runtime_context import get_platform
from sglang.srt.runtime_context import attn_dp_enabled_of, get_platform, num_dp_ranks_of
from sglang.srt.utils.common import is_gfx95_supported, is_npu
if TYPE_CHECKING:
@@ -123,10 +123,18 @@ def validate_deepseek_v4_cp(server_args: ServerArgs) -> None:
"reindexes with interleave order."
)
# DeepSeek-V4 CP runs data-parallel groups as attention DP.
assert not (cfg.attn_dp_size > 1 and cfg.dp_size > 1), (
f"--dp-size {cfg.dp_size} with --attn-dp-size {cfg.attn_dp_size}: "
"data-parallel replicas combined with attention data parallelism "
"are not supported."
)
attn_dp_size = cfg.attn_dp_size * cfg.dp_size
declare_resolution(
server_args,
"validate_deepseek_v4_cp",
enable_dp_attention=True,
attn_dp_size=attn_dp_size,
dp_size=1,
)
declare_resolution(
server_args,
@@ -136,10 +144,10 @@ def validate_deepseek_v4_cp(server_args: ServerArgs) -> None:
declare_resolution(
server_args,
"validate_deepseek_v4_cp",
attn_cp_size=cfg.tp_size // cfg.dp_size,
attn_cp_size=cfg.tp_size // attn_dp_size,
)
if not is_npu():
assert cfg.dp_size == 1, (
assert attn_dp_size == 1, (
"For round-robin split mode, dp attention is not supported."
)
assert cfg.tp_size <= 8, (
@@ -161,7 +169,7 @@ def validate_deepseek_v4_cp(server_args: ServerArgs) -> None:
logger.warning(
f"Enable Context Parallel for DeepSeekV4, "
f"strategy={cfg.cp_strategy}, "
f"dp_size={cfg.dp_size}, moe_dense_tp_size={cfg.moe_dense_tp_size}, "
f"attn_dp_size={attn_dp_size}, moe_dense_tp_size={cfg.moe_dense_tp_size}, "
f"attn_cp_size={cfg.attn_cp_size}, ep_size={cfg.ep_size}, tp_size={cfg.tp_size}"
)
@@ -193,7 +201,7 @@ def validate_deepseek_v41_features(server_args: ServerArgs) -> None:
"prefill CUDA graphs",
cfg.cuda_graph_config.prefill.backend != Backend.DISABLED,
),
("DP attention", cfg.enable_dp_attention),
("DP attention", attn_dp_enabled_of(cfg)),
("context parallelism", cfg.attn_cp_size > 1),
("external cache linker", cfg.enable_unified_cache_external_linker),
("unified memory", cfg.enable_unified_memory),
@@ -246,8 +254,8 @@ def validate_deepseek_v41_features(server_args: ServerArgs) -> None:
if (
read_ragged_verify_mode() is not RaggedVerifyMode.STATIC
or cfg.disaggregation_transfer_backend != "mooncake"
or cfg.dp_size != 1
or cfg.enable_dp_attention
or num_dp_ranks_of(cfg) != 1
or attn_dp_enabled_of(cfg)
or cfg.attn_cp_size != 1
or cfg.dcp_size != 1
):
@@ -285,7 +293,7 @@ def validate_deepseek_v41_features(server_args: ServerArgs) -> None:
cfg.cuda_graph_config.prefill.backend != Backend.DISABLED,
),
# input_ids_global is a DP-wide gather, so the tail slice cannot apply.
("DP attention", cfg.enable_dp_attention),
("DP attention", attn_dp_enabled_of(cfg)),
)
for feature, enabled in incompatible:
if enabled:
@@ -40,6 +40,7 @@ def handle_expert_pack(server_args: Any) -> None:
parallelism = (
("tensor", "--tp-size", cfg.tp_size),
("data", "--dp-size", cfg.dp_size),
("attention data", "--attn-dp-size", cfg.attn_dp_size),
("expert", "--ep-size", cfg.ep_size),
)
for label, option, size in parallelism:
@@ -79,6 +79,7 @@ POSITIONAL_FIELD_ORDER = (
"pp_max_micro_batch_size",
"pp_async_batch_depth",
"dp_size",
"attn_dp_size",
"load_balance_method",
"attn_cp_size",
"moe_dp_size",
+28 -14
View File
@@ -73,8 +73,19 @@ class Parallel(msgspec.Struct):
dp_size: A[
int,
Arg(
help="The data parallelism size.",
help="The number of data-parallel replicas of the model.",
aliases=["--data-parallel-size"],
resolvable=True,
),
] = 1
attn_dp_size: A[
int,
Arg(
help="The attention data parallelism size: the number of "
"data-parallel attention groups inside the TP group, one scheduler "
"each, while the FFN stays tensor parallel.",
aliases=["--attention-data-parallel-size"],
resolvable=True,
),
] = 1
load_balance_method: A[
@@ -159,14 +170,9 @@ class Parallel(msgspec.Struct):
bool,
"Enable attention tensor-parallel weight slicing during decode under context parallel (cp_size>1). Slices the replicated attention linears to the local CP partition, eliminating redundant decode GEMMs.",
] = False
# DP attention
enable_dp_attention: A[
bool,
Arg(
help="Enabling data parallelism for attention and tensor parallelism for FFN. The dp size should be equal to the tp size. Currently DeepSeek-V2 and Qwen 2/3 MoE models are supported.",
resolvable=True,
),
] = False
# Deprecated spelling of `attn_dp_size`: `--dp-size N --enable-dp-attention`
# resolves to `attn_dp_size = N`, `dp_size = 1`.
enable_dp_attention: A[bool, Arg(no_cli=True, resolvable=True)] = False
enable_dp_attention_local_control_broadcast: A[
bool,
"With DP-attention, send control messages to every DP group leader and broadcast within attn_tp_group instead of the full tp_group. Eliminates a costly all-ranks gloo sync on every scheduler iteration.",
@@ -184,7 +190,7 @@ class Parallel(msgspec.Struct):
help="Use all-to-all instead of TP all-gather followed by DP scatter "
"for the TP-sharded LM head under DP attention. By default this is "
"enabled only on decode-only PD nodes with pure DP attention "
"(tp_size == dp_size > 1 and attn_cp_size == 1), and disabled on "
"(tp_size == attn_dp_size > 1 and attn_cp_size == 1), and disabled on "
"prefill-only and colocated nodes. Pass "
"--no-enable-tp-lm-head-all-to-all to opt out. The path is "
"incompatible with --enable-dp-lm-head; batches without an equal "
@@ -282,10 +288,18 @@ class Parallel(msgspec.Struct):
doc="Attention tensor-parallel width: `tp_size` divided by the "
"attention-DP and attention-CP dimensions.",
)
attn_dp_size = Derived(
fn="sglang.srt.runtime_context.attn_dp_size_of",
doc="Attention data-parallel width: `dp_size` when DP attention is "
"on, otherwise one.",
attn_dp_enabled = Derived(
fn="sglang.srt.runtime_context.attn_dp_enabled_of",
doc="Whether attention runs data parallel: `attn_dp_size` is wider "
"than one, or this process is an elastic EP scale joiner, which joins "
"an attention-DP deployment with a group of its own that may be one "
"rank wide.",
)
num_dp_ranks = Derived(
fn="sglang.srt.runtime_context.num_dp_ranks_of",
doc="How many data-parallel ranks the deployment serves with, one "
"scheduler each: `dp_size * attn_dp_size`. Elastic EP scale-up widens "
"it as ranks join.",
)
attn_dcp_size = Derived(
fn="sglang.srt.runtime_context.attn_dcp_size_of",
@@ -18,7 +18,7 @@ from sglang.srt.arg_groups.overrides import (
from sglang.srt.configs.hybrid_arch import mambaish_config
from sglang.srt.environ import envs
from sglang.srt.model_executor.cuda_graph_config import Backend
from sglang.srt.runtime_context import get_platform
from sglang.srt.runtime_context import attn_dp_enabled_of, get_platform
logger = logging.getLogger(__name__)
@@ -450,7 +450,7 @@ def handle_cache_compatibility(server_args: Any) -> None:
raise NotImplementedError(
"LMCacheUnifiedRadixCache does not yet support speculative decoding"
)
if cfg.enable_dp_attention:
if attn_dp_enabled_of(cfg):
raise NotImplementedError(
"LMCacheUnifiedRadixCache does not yet support DP attention"
)

Some files were not shown because too many files have changed in this diff Show More