minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity

Add --enable-symm-mem for Qwen3.5 (#27296)

Browse Source 
Signed-off-by: Faradawn Yang <73060648+faradawn@users.noreply.github.com>
Co-authored-by: zijiexia <37504505+zijiexia@users.noreply.github.com>
This commit is contained in:
Faradawn Yang
2026-06-04 15:23:37 -07:00
committed by GitHub
co-authored by zijiexia
parent 4cfebbb95f
commit 0e4aa081ba
2 changed files with 6 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs_new/cookbook/autoregressive/Qwen/Qwen3.5.mdx
+1
View File
@@ -124,6 +124,7 @@ This section provides deployment configurations optimized for different hardware
### 3.2 Configuration Tips ### 3.2 Configuration Tips
- Speculative decoding (MTP) can significantly reduce latency for interactive use cases. - Speculative decoding (MTP) can significantly reduce latency for interactive use cases.
- **H100 FP8:** Add `--enable-symm-mem` to enable NCCL symmetric memory for faster collectives and better performance under multi-GPU settings.
- **AMD GPUs (MI300X / MI325X / MI355X):** Use `SGLANG_USE_AITER=1` and `--attention-backend triton`. Both the full attention layers and the Gated Delta Net (linear attention) layers use Triton-based kernels on ROCm. Example: `SGLANG_USE_AITER=1 python3 -m sglang.launch_server --model-path Qwen/Qwen3.5-397B-A17B --tp 8 --attention-backend triton --trust-remote-code`. See [AMD's Day-0 support article](https://www.amd.com/en/developer/resources/technical-articles/2026/day-0-support-for-qwen-3-5-on-amd-instinct-gpus.html) for details. - **AMD GPUs (MI300X / MI325X / MI355X):** Use `SGLANG_USE_AITER=1` and `--attention-backend triton`. Both the full attention layers and the Gated Delta Net (linear attention) layers use Triton-based kernels on ROCm. Example: `SGLANG_USE_AITER=1 python3 -m sglang.launch_server --model-path Qwen/Qwen3.5-397B-A17B --tp 8 --attention-backend triton --trust-remote-code`. See [AMD's Day-0 support article](https://www.amd.com/en/developer/resources/technical-articles/2026/day-0-support-for-qwen-3-5-on-amd-instinct-gpus.html) for details.
- **Watchdog timeout:** Increase `--watchdog-timeout` to `1200` or higher for this large model, as weight loading can take significant time. - **Watchdog timeout:** Increase `--watchdog-timeout` to `1200` or higher for this large model, as weight loading can take significant time.
- **Mamba Radix Cache**: Qwen3.5's hybrid Gated Delta Networks architecture supports two mamba scheduling strategies via `--mamba-scheduler-strategy`: - **Mamba Radix Cache**: Qwen3.5's hybrid Gated Delta Networks architecture supports two mamba scheduling strategies via `--mamba-scheduler-strategy`:
docs_new/src/snippets/autoregressive/qwen35-deployment.jsx
+5
View File
@@ -376,6 +376,11 @@ export const Qwen35Deployment = () => {
} }
} }
// Enable NCCL symmetric memory for H100 FP8 deployments.
if (hardware === 'h100' && quantization === 'fp8' && hwConfig.tp > 1) {
cmd += ` \\\n --enable-symm-mem`;
}
// Chunked prefill tuning for H200 FP8 + MTP (validated on H200 only) // Chunked prefill tuning for H200 FP8 + MTP (validated on H200 only)
if (hardware === 'h200' && quantization === 'fp8' && speculative === 'enabled') { if (hardware === 'h200' && quantization === 'fp8' && speculative === 'enabled') {
cmd += ` \\\n --max-running-requests 128`; cmd += ` \\\n --max-running-requests 128`;