minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity

Fix pipeline parallelism doc typos (#15688)

Browse Source 
Signed-off-by: Shangming Cai <csmthu@gmail.com>
This commit is contained in:
Shangming Cai
2025-12-23 22:19:15 +08:00
committed by GitHub
parent c15fa1c577
commit b3f83cc1c5
1 changed files with 9 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/advanced_features/pipeline_parallelism.md
+9 -7
View File
@@ -32,7 +32,7 @@ Optimizing the chunked prefill size is crucial for balancing pipeline efficiency
### Enable Dynamic Chunking and Adjust Smoothing Factor (Experimental feature) ### Enable Dynamic Chunking and Adjust Smoothing Factor (Experimental feature)
SGLang also offers a dynamic chunking solution that could further improve performance. This feature is currently an experimental feature that requires a certain amount of tuning experimentation and may not be suitable for all workloads. In addition, fine-tuning the smoothing factor can help optimize performance for specific workloads and model characteristics. SGLang also offers a dynamic chunking solution that could further improve performance. This feature is currently an experimental feature that requires a certain amount of tuning experimentation and may not be suitable for all workloads. In addition, fine-tuning the smoothing factor can help optimize performance for specific workloads and model characteristics.
### Case Study ### Case Study on NVIDIA H20
When evaluating pipeline parallelism with fixed chunked prefill sizes from 2K to 16K, experiment results show that a 4K chunk size delivered optimal prefill TTFT performance for the DeepSeek-V3.1, and a 6K chunk size delivered optimal prefill TTFT performance for the Qwen3-235B-A22B-Thinking-2507-FP8. When evaluating pipeline parallelism with fixed chunked prefill sizes from 2K to 16K, experiment results show that a 4K chunk size delivered optimal prefill TTFT performance for the DeepSeek-V3.1, and a 6K chunk size delivered optimal prefill TTFT performance for the Qwen3-235B-A22B-Thinking-2507-FP8.
@@ -46,8 +46,8 @@ python3 -m sglang.launch_server \
--nnodes 4 --node-rank 0 --tp 8 --pp-size 4 \ --nnodes 4 --node-rank 0 --tp 8 --pp-size 4 \
--port 30000 --dist-init-addr 192.168.0.137:62001 \ --port 30000 --dist-init-addr 192.168.0.137:62001 \
--disable-radix-cache --mem-fraction-static 0.8 \ --disable-radix-cache --mem-fraction-static 0.8 \
--attention-backend fa3 --watchdog-timeout 3600 --host 0.0.0.0 --watchdog-timeout 3600 \ --attention-backend fa3 --host 0.0.0.0 --watchdog-timeout 3600 \
--max-running-requests 128 --chunked-prefill-size 4096 --enable-dynamic-chunking --max-running-requests 128 --chunked-prefill-size 4096
``` ```
```bash ```bash
@@ -58,7 +58,7 @@ python3 -m sglang.launch_server \
--nnodes 4 --node-rank 0 --tp 8 --pp-size 4 \ --nnodes 4 --node-rank 0 --tp 8 --pp-size 4 \
--port 30000 --dist-init-addr 192.168.0.137:62001 \ --port 30000 --dist-init-addr 192.168.0.137:62001 \
--disable-radix-cache --mem-fraction-static 0.8 \ --disable-radix-cache --mem-fraction-static 0.8 \
--attention-backend fa3 --watchdog-timeout 3600 --host 0.0.0.0 --watchdog-timeout 3600 \ --attention-backend fa3 --host 0.0.0.0 --watchdog-timeout 3600 \
--max-running-requests 128 --chunked-prefill-size 12288 --enable-dynamic-chunking --max-running-requests 128 --chunked-prefill-size 12288 --enable-dynamic-chunking
``` ```
@@ -70,7 +70,7 @@ python3 -m sglang.launch_server \
--nnodes 2 --node-rank 0 --tp 4 --pp-size 2 \ --nnodes 2 --node-rank 0 --tp 4 --pp-size 2 \
--port 30000 --dist-init-addr 192.168.0.137:62001 \ --port 30000 --dist-init-addr 192.168.0.137:62001 \
--disable-radix-cache --mem-fraction-static 0.8 \ --disable-radix-cache --mem-fraction-static 0.8 \
--attention-backend fa3 --watchdog-timeout 3600 --host 0.0.0.0 --watchdog-timeout 3600 \ --attention-backend fa3 --host 0.0.0.0 --watchdog-timeout 3600 \
--max-running-requests 128 --chunked-prefill-size 6144 --max-running-requests 128 --chunked-prefill-size 6144
``` ```
@@ -78,13 +78,15 @@ python3 -m sglang.launch_server \
# prefill node 0 (with dynamic chunking) # prefill node 0 (with dynamic chunking)
export SGLANG_DYNAMIC_CHUNKING_SMOOTH_FACTOR=0.8 export SGLANG_DYNAMIC_CHUNKING_SMOOTH_FACTOR=0.8
python3 -m sglang.launch_server \ python3 -m sglang.launch_server \
--model-path deepseek-ai/DeepSeek-V3.1 --trust-remote-code \ --model-path Qwen/Qwen3-235B-A22B-Thinking-2507-FP8 --trust-remote-code \
--nnodes 2 --node-rank 0 --tp 4 --pp-size 2 \ --nnodes 2 --node-rank 0 --tp 4 --pp-size 2 \
--port 30000 --dist-init-addr 192.168.0.137:62001 \ --port 30000 --dist-init-addr 192.168.0.137:62001 \
--disable-radix-cache --mem-fraction-static 0.8 \ --disable-radix-cache --mem-fraction-static 0.8 \
--attention-backend fa3 --watchdog-timeout 3600 --host 0.0.0.0 --watchdog-timeout 3600 \ --attention-backend fa3 --host 0.0.0.0 --watchdog-timeout 3600 \
--max-running-requests 128 --chunked-prefill-size 18432 --enable-dynamic-chunking --max-running-requests 128 --chunked-prefill-size 18432 --enable-dynamic-chunking
``` ```
Note: `--disable-radix-cache` is enabled only for reproducible benchmarking purposes. It is not recommended to use it in production.
## Best Practice for Pipeline Parallelism with PD Disaggregation ## Best Practice for Pipeline Parallelism with PD Disaggregation
To be added. Stay tuned for the latest updates on Pipeline Parallelism with PD Disaggregation. To be added. Stay tuned for the latest updates on Pipeline Parallelism with PD Disaggregation.