minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity

[misc] Use --cuda-graph-max-bs-decode in tests, examples, and docs (#29591)

Browse Source
This commit is contained in:
Liangsheng Yin
2026-06-28 18:38:28 -07:00
committed by GitHub
parent 3217410cf6
commit 909123ddb8
140 changed files with 304 additions and 290 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs_new/cookbook/autoregressive/DeepSeek/DeepSeek-V3_2.mdx
+1 -1
View File
@@ -497,7 +497,7 @@ sglang_args=$(echo serve \
--dsa-decode-backend flashmla_sparse \
--enable-dsa-prefill-context-parallel \
--dsa-prefill-cp-mode round-robin-split \
--cuda-graph-max-bs 128 \
--cuda-graph-max-bs-decode 128 \
--max-running-requests 128 \
--trust-remote-code --host "0.0.0.0" --port 30000 \
--log-requests \
docs_new/cookbook/autoregressive/DeepSeek/DeepSeek-V4.mdx
+1 -1
View File
@@ -174,7 +174,7 @@ Both Instruct repos ship as **FP4 MoE experts + FP8 attention / dense** (one mix
**Concurrency & DeepEP dispatch buffer**
Must hold: `max-running-requests × MTP_draft_tokens ≤ SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK`. Violating it blows DeepEP's dispatch buffer at steady-state load (`deep_ep.cpp:1105`). When tuning, move `--cuda-graph-max-bs`, `--max-running-requests`, and the env together.
Must hold: `max-running-requests × MTP_draft_tokens ≤ SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK`. Violating it blows DeepEP's dispatch buffer at steady-state load (`deep_ep.cpp:1105`). When tuning, move `--cuda-graph-max-bs-decode`, `--max-running-requests`, and the env together.
The generator currently picks values on the **conservative** side (mirroring an internal stress-test matrix). They run safely out of the box but likely leave throughput on the table — please tune them up toward your actual workload's peak concurrency and report findings back so the defaults can be revised.