minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity

[Docs] Explain CUDA attention backend choices and aiter FP8 KV cache support (#17428)

Browse Source
This commit is contained in:
Ruihuan He
2026-01-20 11:10:36 -08:00
committed by GitHub
parent 6ea491e439
commit eb38d64413
1 changed files with 18 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/advanced_features/attention_backend.md
+18 -1
View File
@@ -25,7 +25,7 @@ The support matrix is split into two parts: MHA (standard attention) and MLA (mu
| **FlexAttention (PyTorch)** | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | | **FlexAttention (PyTorch)** | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ |
| **TRTLLM MHA** | 16, 32 or 64 | ✅ | ✅ | ✅ | ❌ | ✅ | ❌ | | **TRTLLM MHA** | 16, 32 or 64 | ✅ | ✅ | ✅ | ❌ | ✅ | ❌ |
| **Dual Chunk FlashAttention** | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | | **Dual Chunk FlashAttention** | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ |
| **AITER (ROCm)** | ✅ | ❌ | ❌ | ✅ | ✅ | ❌ | ✅ | | **AITER (ROCm)** | ✅ | ✅ | ❌ | ✅ | ✅ | ❌ | ✅ |
| **Wave (ROCm)** | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | | **Wave (ROCm)** | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ |
| **Ascend (NPU)** | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | | **Ascend (NPU)** | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| **Intel XPU** | ✅ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | | **Intel XPU** | ✅ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ |
@@ -111,6 +111,23 @@ If you set only one of `--prefill-attention-backend` or `--decode-attention-back
If both are specified and differ, SGLang automatically enables a hybrid wrapper to dispatch to the chosen backend per phase. If both are specified and differ, SGLang automatically enables a hybrid wrapper to dispatch to the chosen backend per phase.
``` ```
## Attention Backend Selection Guide (CUDA)
If the `--attention-backend` argument is not specified, SGLang automatically selects the best backend based on the hardware (CUDA) and model architecture.
### Automatic Selection Logic
**1. MHA Models (e.g., Llama, Qwen)**
- **Hopper (e.g., H100, H200)**: Defaults to `fa3` if using CUDA 12.3+ and the model configuration is supported.
- **Blackwell (e.g., B200)**: Defaults to `trtllm_mha`, unless using speculative decoding with `topk > 1`.
- **Other Architectures (Ampere, Ada, etc.)**: Defaults to `flashinfer` if available; otherwise falls back to `triton`.
**2. MLA Models (e.g., DeepSeek V3)**
- **Hopper**: Defaults to `fa3` (requires CUDA 12.3+).
- **Blackwell**: Defaults to `trtllm_mla`.
- **Other Architectures**: Defaults to `triton`.
## User Guide ## User Guide
### Launch Command for Different Attention Backends ### Launch Command for Different Attention Backends