diff --git a/docs/cookbook/autoregressive/DeepSeek/DeepSeek-V3_2.mdx b/docs/cookbook/autoregressive/DeepSeek/DeepSeek-V3_2.mdx index 9e85b4703..29d33e1d2 100644 --- a/docs/cookbook/autoregressive/DeepSeek/DeepSeek-V3_2.mdx +++ b/docs/cookbook/autoregressive/DeepSeek/DeepSeek-V3_2.mdx @@ -432,25 +432,19 @@ For production deployments (RBG / LWS-based, DeepEP EP parallelism), see [multi_ #### 4.2.5 DSA Long-Sequence Context Parallel and PP/CP -SGLang provides two context parallel (CP) modes for long-sequence workloads, controlled with `--dsa-prefill-cp-mode`. +Enable prefill context parallelism (CP) on CUDA with `--enable-prefill-cp --cp-strategy interleave` for long-sequence workloads. -**In-sequence splitting** (`--dsa-prefill-cp-mode in-seq-split`): Each CP rank handles a uniform shard of the sequence; KV cache is gathered via all-gather. Batch size is restricted to 1 during prefill. See [PR #12065](https://github.com/sgl-project/sglang/pull/12065). + +Zigzag prefill CP (`--cp-strategy zigzag`) is temporarily unavailable for DeepSeek V3.2, GLM-5, GLM-5.1, GLM-5.2, and GLM-5.3. Use `interleave` for these models and keep `--dp 1`; interleave DSA CP does not support `--dp` greater than 1. + + +**Interleave** (`--cp-strategy interleave`): Distributes tokens by `token_idx % cp_size`. It supports fused MoE, FP8 KV cache, and multi-batch prefill. ```bash Command -# In-seq splitting mode — EP + DP, batch size 1 +# Interleave — FusedMoE + CP8 python -m sglang.launch_server --model deepseek-ai/DeepSeek-V3.2-Exp \ - --tp 8 --ep 8 --dp 2 --enable-dp-attention \ - --enable-dsa-prefill-context-parallel --attn-cp-size 4 \ - --dsa-prefill-cp-mode in-seq-split --max-running-requests 32 -``` - -**Round-robin splitting** (`--dsa-prefill-cp-mode round-robin-split`, default): Distributes tokens by `token_idx % cp_size`. Supports fused MoE, FP8 KV cache, and multi-batch prefill. Cannot be combined with DP attention. See [PR #13959](https://github.com/sgl-project/sglang/pull/13959). - -```bash Command -# Round-robin splitting — FusedMoE + CP8 -python -m sglang.launch_server --model deepseek-ai/DeepSeek-V3.2-Exp \ - --tp 8 --enable-dsa-prefill-context-parallel --attn-cp-size 8 \ - --dsa-prefill-cp-mode round-robin-split --max-running-requests 32 + --tp 8 --enable-prefill-cp --attn-cp-size 8 \ + --cp-strategy interleave --max-running-requests 32 ``` **PP + CP (multi-node):** Combines Pipeline Parallelism and Context Parallelism for cross-node scaling. The production-optimized configurations below have been verified on Hopper: @@ -499,8 +493,8 @@ sglang_args=$(echo serve \ --attention-backend dsa \ --dsa-prefill-backend flashmla_sparse \ --dsa-decode-backend flashmla_sparse \ - --enable-dsa-prefill-context-parallel \ - --dsa-prefill-cp-mode round-robin-split \ + --enable-prefill-cp \ + --cp-strategy interleave \ --cuda-graph-max-bs-decode 128 \ --max-running-requests 128 \ --trust-remote-code --host "0.0.0.0" --port 30000 \ @@ -518,34 +512,19 @@ sglang "${sglang_args[@]}" 2>&1 | tee $LOG_DIR/$RANK.log **fp8 KV + CP + PP** -With FP8 KV, we can have less memory footprint. This can be combined with various parallel schemes: +FP8 KV reduces the memory footprint. For DeepSeek V3.2, combine it with interleave CP and PP while keeping `--dp 1`: ```shell Command # verified in Hopper platform -dp=1 - dp_config=" \ --dp 1 --enable-dp-attention \ " cp_config=" \ - --enable-dsa-prefill-context-parallel \ + --enable-prefill-cp \ + --cp-strategy interleave \ " -if [ "$dp" -eq 1 ]; then - -cp_config=" \ - $cp_config \ - --dsa-prefill-cp-mode round-robin-split \ -" - -else -cp_config=" \ - $cp_config \ - --dsa-prefill-cp-mode in-seq-split \ -" -fi - # see discussion : https://github.com/sgl-project/sglang/pull/12065 sglang_args=$(echo serve \ --model-path $MAPPED_MODEL_PATH \ diff --git a/docs/cookbook/autoregressive/GLM/GLM-5.1.mdx b/docs/cookbook/autoregressive/GLM/GLM-5.1.mdx index 39445e63e..bfa362e97 100644 --- a/docs/cookbook/autoregressive/GLM/GLM-5.1.mdx +++ b/docs/cookbook/autoregressive/GLM/GLM-5.1.mdx @@ -100,6 +100,9 @@ import { GLM51Deployment } from '/src/snippets/autoregressive/glm-51-deployment. - **B300 and GB300**: NVFP4 is the recommended deployment path. Use `nvidia/GLM-5.1-NVFP4` with `--quantization modelopt_fp4`. Use `tp=8` on B300 and `tp=4` on GB300. The CUDA 13 image variant is required for B300 and GB300. - **AMD GPUs**: BF16 and FP8 checkpoints run on MI300X/MI325X/MI355X at tp=8. On MI355X (gfx950), the MXFP4 checkpoint `amd/GLM-5.1-MXFP4` is also supported at tp=4 with `--kv-cache-dtype fp8_e4m3`. All AMD paths pass `--dsa-prefill-backend tilelang --dsa-decode-backend tilelang`, `--chunked-prefill-size 131072`, and `--watchdog-timeout 1200` (20 minutes for weight loading). FP8 uses approximately half the memory of BF16 (~89 GB/GPU vs ~175 GB/GPU). EAGLE speculative decoding is supported on AMD GPUs: MI300X/MI325X (gfx942) and MI355X (gfx950), but it **requires `--disable-custom-all-reduce`** — the aiter custom all-reduce kernel deadlocks during EAGLE verify at high concurrency, so without this flag the server will hang. - For other configuration tips (MTP, DSA kernel, Context Parallel, HiSparse, NVFP4, Index Cache), see the [DeepSeek-V3.2 cookbook page](../DeepSeek/DeepSeek-V3_2). GLM-5.1 and DeepSeek-V3.2 share the same model structure, so the optimization techniques are common. + +- **Prefill CP on CUDA**: Zigzag (`--cp-strategy zigzag`) is temporarily unavailable for GLM-5.1. Use `--enable-prefill-cp --cp-strategy interleave` with `--dp 1`. + - Use `--json-model-override-args '{"index_topk_pattern": "FFSFSSSFSSFFFSSSFFFSFSSSSSSFFSFFSFFSSFFFFFFSFFFFFSFFSSSSSSFSFFFSFSSSFSFFSFFSSS"}'` to enable the [IndexCache](https://github.com/THUDM/IndexCache) method for GLM-5.1. This can improve serving efficiency with only a small accuracy loss. If you are running rigorous accuracy evaluations, do not enable this feature. ## 4. Model Invocation diff --git a/docs/cookbook/autoregressive/GLM/GLM-5.2.mdx b/docs/cookbook/autoregressive/GLM/GLM-5.2.mdx index 3651e2e68..a62cb86ad 100644 --- a/docs/cookbook/autoregressive/GLM/GLM-5.2.mdx +++ b/docs/cookbook/autoregressive/GLM/GLM-5.2.mdx @@ -254,6 +254,10 @@ For the full setup (streaming, tool-use, count_tokens, persisting env in `~/.cla ### 3.5 Context Parallelism + +Zigzag prefill CP (`--cp-strategy zigzag`) is temporarily unavailable for GLM-5.2. For prefill CP on CUDA, use `interleave` with `--dp 1` as shown below. + + Prefill context parallelism can help with reduction of TTFT under long context. To enable prefill context parallelism for GLM 5.2, please append the following arguments: ```bash --attn-cp-size 8 \ diff --git a/docs/cookbook/autoregressive/GLM/GLM-5.3.mdx b/docs/cookbook/autoregressive/GLM/GLM-5.3.mdx index 778dbcd6f..8b0f63027 100644 --- a/docs/cookbook/autoregressive/GLM/GLM-5.3.mdx +++ b/docs/cookbook/autoregressive/GLM/GLM-5.3.mdx @@ -242,6 +242,10 @@ For the full setup (streaming, tool-use, count_tokens, persisting env in `~/.cla ### 3.5 Context Parallelism + +Zigzag prefill CP (`--cp-strategy zigzag`) is temporarily unavailable for GLM-5.3. For prefill CP on CUDA, use `interleave` with `--dp 1` as shown below. + + Prefill context parallelism can help with reduction of TTFT under long context. To enable prefill context parallelism for GLM 5.3, please append the following arguments: ```bash --attn-cp-size 8 \ diff --git a/docs/cookbook/autoregressive/GLM/GLM-5.mdx b/docs/cookbook/autoregressive/GLM/GLM-5.mdx index a5a781ca5..dfd9b6aa9 100644 --- a/docs/cookbook/autoregressive/GLM/GLM-5.mdx +++ b/docs/cookbook/autoregressive/GLM/GLM-5.mdx @@ -93,6 +93,9 @@ import { GLM5Deployment } from '/src/snippets/autoregressive/glm-5-deployment.js - **AMD GPUs**: Use `--dsa-prefill-backend tilelang --dsa-decode-backend tilelang` for the DSA attention backend. Add `--chunked-prefill-size 131072` and `--watchdog-timeout 1200` (20 minutes for weight loading). EAGLE speculative decoding is not currently supported on AMD for GLM-5. - For other configuration tips (MTP, DSA kernel, Context Parallel, HiSparse, NVFP4, Index Cache), see the [DeepSeek-V3.2 cookbook page](../DeepSeek/DeepSeek-V3_2). GLM-5 and DeepSeek-V3.2 share the same model structure, so the optimization techniques are common. + +- **Prefill CP on CUDA**: Zigzag (`--cp-strategy zigzag`) is temporarily unavailable for GLM-5. Use `--enable-prefill-cp --cp-strategy interleave` with `--dp 1`. + - Use `--json-model-override-args '{"index_topk_pattern": "FFSFSSSFSSFFFSSSFFFSFSSSSSSFFSFFSFFSSFFFFFFSFFFFFSFFSSSSSSFSFFFSFSSSFSFFSFFSSS"}'` for GLM-5-FP8 if you want to enable the [IndexCache](https://github.com/THUDM/IndexCache) method. This feature is supported through [this PR](https://github.com/sgl-project/sglang/pull/21405) and introduces only a small accuracy loss. However, if you are running rigorous accuracy evaluations, it is not recommended to enable this feature. ## 4. Model Invocation diff --git a/docs/docs/advanced_features/server_arguments.mdx b/docs/docs/advanced_features/server_arguments.mdx index b28326f9b..f6080eeee 100644 --- a/docs/docs/advanced_features/server_arguments.mdx +++ b/docs/docs/advanced_features/server_arguments.mdx @@ -2741,7 +2741,7 @@ Please consult the documentation below and [server_args.py](https://github.com/s `--cp-strategy` - Sharding strategy for prefill CP. zigzag is the former in-seq-split mode; interleave is the former round-robin-split mode. + Sharding strategy for prefill CP. zigzag assigns each rank one early and one late sequence block; interleave assigns token indices modulo the CP size. None zigzag, interleave @@ -2805,42 +2805,6 @@ Please consult the documentation below and [server_args.py](https://github.com/s `False` bool flag (set to enable) - - `--enable-dsa-prefill-context-parallel` - [Deprecated] Use --enable-prefill-cp instead. - — - Type: str - - - `--enable-nsa-prefill-context-parallel` - [Deprecated] Use --enable-prefill-cp instead. - — - Type: str - - - `--enable-prefill-context-parallel` - [Deprecated] Use --enable-prefill-cp instead. - — - Type: str - - - `--dsa-prefill-cp-mode` - [Deprecated] Use --cp-strategy {zigzag,interleave} instead. 'in-seq-split' maps to 'zigzag'; 'round-robin-split' maps to 'interleave'. - `round-robin-split` - in-seq-split, round-robin-split - - - `--nsa-prefill-cp-mode` - [Deprecated] Use --cp-strategy instead. - Auto - in-seq-split, round-robin-split - - - `--prefill-cp-mode` - [Deprecated] Use --cp-strategy {zigzag,interleave} instead. 'in-seq-split' maps to 'zigzag'. - `in-seq-split` - in-seq-split - `--enable-fused-moe-sum-all-reduce` Enable fused moe triton and sum all reduce. diff --git a/python/sglang/srt/arg_groups/fields/parallel.py b/python/sglang/srt/arg_groups/fields/parallel.py index a4bd69c9e..c323d65ec 100644 --- a/python/sglang/srt/arg_groups/fields/parallel.py +++ b/python/sglang/srt/arg_groups/fields/parallel.py @@ -152,7 +152,7 @@ class Parallel: cp_strategy: A[ Optional[str], Arg( - help="Sharding strategy for prefill CP. 'zigzag' is the former in-seq-split mode; 'interleave' is the former round-robin-split mode.", + help="Sharding strategy for prefill CP. 'zigzag' assigns each rank one early and one late sequence block; 'interleave' assigns token indices modulo the CP size.", choices=("zigzag", "interleave"), ), ] = None