minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity

Add KV4-capable backend flashmla and update server args (#14989)

Browse Source 
Signed-off-by: Ho-Ren (Jack) Chuang <horenchuang@bytedance.com>
This commit is contained in:
Ho-Ren (Jack) Chuang
2025-12-12 11:50:27 -08:00
committed by GitHub
parent 4b7b5af36a
commit 171b442ad3
2 changed files with 2 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/advanced_features/attention_backend.md
+1 -1
View File
@@ -36,7 +36,7 @@ The support matrix is split into two parts: MHA (standard attention) and MLA (mu
| **Backend** | **Native Page Sizes** | **FP8 KV Cache** | **FP4 KV Cache** | **Chunked Prefix Cache** | **Spec topk=1** | **Spec topk>1** | | **Backend** | **Native Page Sizes** | **FP8 KV Cache** | **FP4 KV Cache** | **Chunked Prefix Cache** | **Spec topk=1** | **Spec topk>1** |
|----------------------------|---------------------------|------------------|------------------|--------------------------|-----------------|-----------------| |----------------------------|---------------------------|------------------|------------------|--------------------------|-----------------|-----------------|
| **FlashInfer MLA** | 1 | ❌ | ✅ | ✅ | ✅ | ❌ | | **FlashInfer MLA** | 1 | ❌ | ✅ | ✅ | ✅ | ❌ |
| **FlashMLA** | 64 | ✅ | ❌ | ✅ | ✅ | ❌ | | **FlashMLA** | 64 | ✅ | ✅ | ✅ | ✅ | ❌ |
| **Cutlass MLA** | 128 | ✅ | ✅ | ✅ | ✅ | ❌ | | **Cutlass MLA** | 128 | ✅ | ✅ | ✅ | ✅ | ❌ |
| **TRTLLM MLA (Blackwell)** | 32 or 64 | ✅ | ✅ | ✅ | ✅ | ❌ | | **TRTLLM MLA (Blackwell)** | 32 or 64 | ✅ | ✅ | ✅ | ✅ | ❌ |
| **FA3 (FlashAttention 3)** | n/a | ❌ | ❌ | ✅ | ✅ | ⚠️ (page_size=1 only) | | **FA3 (FlashAttention 3)** | n/a | ❌ | ❌ | ✅ | ✅ | ⚠️ (page_size=1 only) |
python/sglang/srt/server_args.py
+1
View File
@@ -1607,6 +1607,7 @@ class ServerArgs:
"cutlass_mla", "cutlass_mla",
"flashinfer", "flashinfer",
"trtllm_mla", "trtllm_mla",
"flashmla",
] ]
assert ( assert (
self.attention_backend in KV4_ATTENTION_MLA_BACKEND_CHOICES self.attention_backend in KV4_ATTENTION_MLA_BACKEND_CHOICES