minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 13 Packages Projects Releases Wiki Activity

[CP V1 Deprecation 5/5] Update prefill CP documentation (#36230)

Browse Source
This commit is contained in:
Baizhou Zhang
2026-09-08 19:27:49 -07:00
committed by GitHub
parent 8ab9982851
commit e54ff1efb9
7 changed files with 30 additions and 73 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/cookbook/autoregressive/DeepSeek/DeepSeek-V3_2.mdx
+14 -35
View File
@@ -432,25 +432,19 @@ For production deployments (RBG / LWS-based, DeepEP EP parallelism), see [multi_
#### 4.2.5 DSA Long-Sequence Context Parallel and PP/CP
SGLang provides two context parallel (CP) modes for long-sequence workloads, controlled with `--dsa-prefill-cp-mode`.
Enable prefill context parallelism (CP) on CUDA with `--enable-prefill-cp --cp-strategy interleave` for long-sequence workloads.
**In-sequence splitting** (`--dsa-prefill-cp-mode in-seq-split`): Each CP rank handles a uniform shard of the sequence; KV cache is gathered via all-gather. Batch size is restricted to 1 during prefill. See [PR #12065](https://github.com/sgl-project/sglang/pull/12065).
<Warning>
Zigzag prefill CP (`--cp-strategy zigzag`) is temporarily unavailable for DeepSeek V3.2, GLM-5, GLM-5.1, GLM-5.2, and GLM-5.3. Use `interleave` for these models and keep `--dp 1`; interleave DSA CP does not support `--dp` greater than 1.
</Warning>
**Interleave** (`--cp-strategy interleave`): Distributes tokens by `token_idx % cp_size`. It supports fused MoE, FP8 KV cache, and multi-batch prefill.
```bash Command
# In-seq splitting mode — EP + DP, batch size 1
# Interleave — FusedMoE + CP8
python -m sglang.launch_server --model deepseek-ai/DeepSeek-V3.2-Exp \
--tp 8 --ep 8 --dp 2 --enable-dp-attention \
--enable-dsa-prefill-context-parallel --attn-cp-size 4 \
--dsa-prefill-cp-mode in-seq-split --max-running-requests 32
```
**Round-robin splitting** (`--dsa-prefill-cp-mode round-robin-split`, default): Distributes tokens by `token_idx % cp_size`. Supports fused MoE, FP8 KV cache, and multi-batch prefill. Cannot be combined with DP attention. See [PR #13959](https://github.com/sgl-project/sglang/pull/13959).
```bash Command
# Round-robin splitting — FusedMoE + CP8
python -m sglang.launch_server --model deepseek-ai/DeepSeek-V3.2-Exp \
--tp 8 --enable-dsa-prefill-context-parallel --attn-cp-size 8 \
--dsa-prefill-cp-mode round-robin-split --max-running-requests 32
--tp 8 --enable-prefill-cp --attn-cp-size 8 \
--cp-strategy interleave --max-running-requests 32
```
**PP + CP (multi-node):** Combines Pipeline Parallelism and Context Parallelism for cross-node scaling. The production-optimized configurations below have been verified on Hopper:
@@ -499,8 +493,8 @@ sglang_args=$(echo serve \
--attention-backend dsa \
--dsa-prefill-backend flashmla_sparse \
--dsa-decode-backend flashmla_sparse \
--enable-dsa-prefill-context-parallel \
--dsa-prefill-cp-mode round-robin-split \
--enable-prefill-cp \
--cp-strategy interleave \
--cuda-graph-max-bs-decode 128 \
--max-running-requests 128 \
--trust-remote-code --host "0.0.0.0" --port 30000 \
@@ -518,34 +512,19 @@ sglang "${sglang_args[@]}" 2>&1 | tee $LOG_DIR/$RANK.log
**fp8 KV + CP + PP**
With FP8 KV, we can have less memory footprint. This can be combined with various parallel schemes:
FP8 KV reduces the memory footprint. For DeepSeek V3.2, combine it with interleave CP and PP while keeping `--dp 1`:
```shell Command
# verified in Hopper platform
dp=1
dp_config=" \
--dp 1 --enable-dp-attention \
"
cp_config=" \
--enable-dsa-prefill-context-parallel \
--enable-prefill-cp \
--cp-strategy interleave \
"
if [ "$dp" -eq 1 ]; then
cp_config=" \
$cp_config \
--dsa-prefill-cp-mode round-robin-split \
"
else
cp_config=" \
$cp_config \
--dsa-prefill-cp-mode in-seq-split \
"
fi
# see discussion : https://github.com/sgl-project/sglang/pull/12065
sglang_args=$(echo serve \
--model-path $MAPPED_MODEL_PATH \
docs/cookbook/autoregressive/GLM/GLM-5.1.mdx
+3
View File
@@ -100,6 +100,9 @@ import { GLM51Deployment } from '/src/snippets/autoregressive/glm-51-deployment.
- **B300 and GB300**: NVFP4 is the recommended deployment path. Use `nvidia/GLM-5.1-NVFP4` with `--quantization modelopt_fp4`. Use `tp=8` on B300 and `tp=4` on GB300. The CUDA 13 image variant is required for B300 and GB300.
- **AMD GPUs**: BF16 and FP8 checkpoints run on MI300X/MI325X/MI355X at tp=8. On MI355X (gfx950), the MXFP4 checkpoint `amd/GLM-5.1-MXFP4` is also supported at tp=4 with `--kv-cache-dtype fp8_e4m3`. All AMD paths pass `--dsa-prefill-backend tilelang --dsa-decode-backend tilelang`, `--chunked-prefill-size 131072`, and `--watchdog-timeout 1200` (20 minutes for weight loading). FP8 uses approximately half the memory of BF16 (~89 GB/GPU vs ~175 GB/GPU). EAGLE speculative decoding is supported on AMD GPUs: MI300X/MI325X (gfx942) and MI355X (gfx950), but it **requires `--disable-custom-all-reduce`** — the aiter custom all-reduce kernel deadlocks during EAGLE verify at high concurrency, so without this flag the server will hang.
- For other configuration tips (MTP, DSA kernel, Context Parallel, HiSparse, NVFP4, Index Cache), see the [DeepSeek-V3.2 cookbook page](../DeepSeek/DeepSeek-V3_2). GLM-5.1 and DeepSeek-V3.2 share the same model structure, so the optimization techniques are common.
- **Prefill CP on CUDA**: Zigzag (`--cp-strategy zigzag`) is temporarily unavailable for GLM-5.1. Use `--enable-prefill-cp --cp-strategy interleave` with `--dp 1`.
- Use `--json-model-override-args '{"index_topk_pattern": "FFSFSSSFSSFFFSSSFFFSFSSSSSSFFSFFSFFSSFFFFFFSFFFFFSFFSSSSSSFSFFFSFSSSFSFFSFFSSS"}'` to enable the [IndexCache](https://github.com/THUDM/IndexCache) method for GLM-5.1. This can improve serving efficiency with only a small accuracy loss. If you are running rigorous accuracy evaluations, do not enable this feature.
## 4. Model Invocation
docs/cookbook/autoregressive/GLM/GLM-5.2.mdx
+4
View File
@@ -254,6 +254,10 @@ For the full setup (streaming, tool-use, count_tokens, persisting env in `~/.cla
### 3.5 Context Parallelism
<Warning>
Zigzag prefill CP (`--cp-strategy zigzag`) is temporarily unavailable for GLM-5.2. For prefill CP on CUDA, use `interleave` with `--dp 1` as shown below.
</Warning>
Prefill context parallelism can help with reduction of TTFT under long context. To enable prefill context parallelism for GLM 5.2, please append the following arguments:
```bash
--attn-cp-size 8 \
docs/cookbook/autoregressive/GLM/GLM-5.3.mdx
+4
View File
@@ -242,6 +242,10 @@ For the full setup (streaming, tool-use, count_tokens, persisting env in `~/.cla
### 3.5 Context Parallelism
<Warning>
Zigzag prefill CP (`--cp-strategy zigzag`) is temporarily unavailable for GLM-5.3. For prefill CP on CUDA, use `interleave` with `--dp 1` as shown below.
</Warning>
Prefill context parallelism can help with reduction of TTFT under long context. To enable prefill context parallelism for GLM 5.3, please append the following arguments:
```bash
--attn-cp-size 8 \
docs/cookbook/autoregressive/GLM/GLM-5.mdx
+3
View File
@@ -93,6 +93,9 @@ import { GLM5Deployment } from '/src/snippets/autoregressive/glm-5-deployment.js
- **AMD GPUs**: Use `--dsa-prefill-backend tilelang --dsa-decode-backend tilelang` for the DSA attention backend. Add `--chunked-prefill-size 131072` and `--watchdog-timeout 1200` (20 minutes for weight loading). EAGLE speculative decoding is not currently supported on AMD for GLM-5.
- For other configuration tips (MTP, DSA kernel, Context Parallel, HiSparse, NVFP4, Index Cache), see the [DeepSeek-V3.2 cookbook page](../DeepSeek/DeepSeek-V3_2). GLM-5 and DeepSeek-V3.2 share the same model structure, so the optimization techniques are common.
- **Prefill CP on CUDA**: Zigzag (`--cp-strategy zigzag`) is temporarily unavailable for GLM-5. Use `--enable-prefill-cp --cp-strategy interleave` with `--dp 1`.
- Use `--json-model-override-args '{"index_topk_pattern": "FFSFSSSFSSFFFSSSFFFSFSSSSSSFFSFFSFFSSFFFFFFSFFFFFSFFSSSSSSFSFFFSFSSSFSFFSFFSSS"}'` for GLM-5-FP8 if you want to enable the [IndexCache](https://github.com/THUDM/IndexCache) method. This feature is supported through [this PR](https://github.com/sgl-project/sglang/pull/21405) and introduces only a small accuracy loss. However, if you are running rigorous accuracy evaluations, it is not recommended to enable this feature.
## 4. Model Invocation
docs/docs/advanced_features/server_arguments.mdx
+1 -37
View File
@@ -2741,7 +2741,7 @@ Please consult the documentation below and [server_args.py](https://github.com/s
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--cp-strategy`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Sharding strategy for prefill CP. <code>zigzag</code> is the former <code>in-seq-split</code> mode; <code>interleave</code> is the former <code>round-robin-split</code> mode.</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Sharding strategy for prefill CP. <code>zigzag</code> assigns each rank one early and one late sequence block; <code>interleave</code> assigns token indices modulo the CP size.</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><code>None</code></td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><code>zigzag</code>, <code>interleave</code></td>
</tr>
@@ -2805,42 +2805,6 @@ Please consult the documentation below and [server_args.py](https://github.com/s
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`False`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>bool flag (set to enable)</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--enable-dsa-prefill-context-parallel`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>[Deprecated] Use --enable-prefill-cp instead.</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>—</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: str</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--enable-nsa-prefill-context-parallel`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>[Deprecated] Use --enable-prefill-cp instead.</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>—</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: str</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--enable-prefill-context-parallel`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>[Deprecated] Use --enable-prefill-cp instead.</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>—</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Type: str</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--dsa-prefill-cp-mode`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>[Deprecated] Use --cp-strategy &#123;zigzag,interleave&#125; instead. 'in-seq-split' maps to 'zigzag'; 'round-robin-split' maps to 'interleave'.</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`round-robin-split`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><code>in-seq-split</code>, <code>round-robin-split</code></td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--nsa-prefill-cp-mode`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>[Deprecated] Use --cp-strategy instead.</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Auto</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><code>in-seq-split</code>, <code>round-robin-split</code></td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--prefill-cp-mode`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>[Deprecated] Use --cp-strategy &#123;zigzag,interleave&#125; instead. 'in-seq-split' maps to 'zigzag'.</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`in-seq-split`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><code>in-seq-split</code></td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--enable-fused-moe-sum-all-reduce`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Enable fused moe triton and sum all reduce.</td>