minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity

[AMD] Update qwen3.5 cookbook (#31258)

Browse Source
This commit is contained in:
Thomas Wang
2026-07-15 10:59:12 +08:00
committed by GitHub
parent 43124cdd90
commit aafa706f8f
2 changed files with 8 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs_new/cookbook/autoregressive/Qwen/Qwen3.5.mdx
+3 -1
View File
@@ -127,7 +127,7 @@ This section provides deployment configurations optimized for different hardware
- Speculative decoding (MTP) can significantly reduce latency for interactive use cases. - Speculative decoding (MTP) can significantly reduce latency for interactive use cases.
- **H100 FP8:** Add `--enable-symm-mem` to enable NCCL symmetric memory for faster collectives and better performance under multi-GPU settings. - **H100 FP8:** Add `--enable-symm-mem` to enable NCCL symmetric memory for faster collectives and better performance under multi-GPU settings.
- **AMD GPUs (MI300X / MI325X / MI355X):** Use `SGLANG_USE_AITER=1` and `SGLANG_USE_AITER_UNIFIED_ATTN=1` with `--attention-backend aiter`, which requires `--page-size 16` and can also enable `--enable-aiter-allreduce-fusion`. - **AMD GPUs (MI300X / MI325X / MI355X):** Use `SGLANG_USE_AITER=1` and `SGLANG_USE_AITER_UNIFIED_ATTN=1` with `--attention-backend aiter`, which requires `--page-size 16` and can also enable `--enable-aiter-allreduce-fusion`. Additionally set `AITER_FLYDSL_FORCE=1` to force the AITER FlyDSL MoE kernels and `SGLANG_MAMBA_SSM_DTYPE=bfloat16` to store the Mamba SSM state in bfloat16 (instead of the default float32).
- **Watchdog timeout:** Increase `--watchdog-timeout` to `1200` or higher for this large model, as weight loading can take significant time. - **Watchdog timeout:** Increase `--watchdog-timeout` to `1200` or higher for this large model, as weight loading can take significant time.
- **Mamba Radix Cache**: Qwen3.5's hybrid Gated Delta Networks architecture supports two mamba scheduling strategies via `--mamba-scheduler-strategy`: - **Mamba Radix Cache**: Qwen3.5's hybrid Gated Delta Networks architecture supports two mamba scheduling strategies via `--mamba-scheduler-strategy`:
- **V1 (`no_buffer`)**: Default. No overlap scheduler, lower memory usage. Required for AMD MI GPUs. - **V1 (`no_buffer`)**: Default. No overlap scheduler, lower memory usage. Required for AMD MI GPUs.
@@ -258,6 +258,8 @@ Deploy Qwen3.5-397B-A17B with the following command (MI300X/MI325X/MI355X):
```shell Command ```shell Command
SGLANG_USE_AITER=1 \ SGLANG_USE_AITER=1 \
SGLANG_USE_AITER_UNIFIED_ATTN=1 \ SGLANG_USE_AITER_UNIFIED_ATTN=1 \
AITER_FLYDSL_FORCE=1 \
SGLANG_MAMBA_SSM_DTYPE=bfloat16 \
python3 -m sglang.launch_server \ python3 -m sglang.launch_server \
--model-path Qwen/Qwen3.5-397B-A17B \ --model-path Qwen/Qwen3.5-397B-A17B \
--tp 8 \ --tp 8 \
docs_new/src/snippets/autoregressive/qwen35-deployment.jsx
+5 -1
View File
@@ -435,7 +435,11 @@ export const Qwen35Deployment = () => {
// SGLANG_USE_AITER=1 and SGLANG_USE_AITER_UNIFIED_ATTN=1; see cookbook prose), // SGLANG_USE_AITER=1 and SGLANG_USE_AITER_UNIFIED_ATTN=1; see cookbook prose),
// which requires --page-size 16. Enable AITER allreduce fusion for multi-GPU. // which requires --page-size 16. Enable AITER allreduce fusion for multi-GPU.
if (amdGpu) { if (amdGpu) {
cmd = "SGLANG_USE_AITER=1 \\\nSGLANG_USE_AITER_UNIFIED_ATTN=1 \\\n" + cmd; let amdEnv = "SGLANG_USE_AITER=1 \\\nSGLANG_USE_AITER_UNIFIED_ATTN=1 \\\nAITER_FLYDSL_FORCE=1 \\\n";
if (MOE_MODELS.has(model)) {
amdEnv += "SGLANG_MAMBA_SSM_DTYPE=bfloat16 \\\n";
}
cmd = amdEnv + cmd;
cmd += " \\\n --attention-backend aiter"; cmd += " \\\n --attention-backend aiter";
cmd += " \\\n --page-size 16"; cmd += " \\\n --page-size 16";
if (hwConfig.tp > 1) { if (hwConfig.tp > 1) {