minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 13 Packages Projects Releases Wiki Activity

[Attention Backend] Add HPC-Ops attention backend (#30540)

Browse Source 
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Halcyon <56064364+VAthree@users.noreply.github.com>
This commit is contained in:
Xiaoyu Zhang
2026-07-22 22:06:22 +08:00
committed by GitHub
co-authored by Claude Fable 5 Halcyon
parent 004df6b520
commit 0a6d1930c3
6 changed files with 790 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs_new/docs/advanced_features/attention_backend.mdx
+32 -1
View File
@@ -122,6 +122,16 @@ The support matrix is split into two parts: MHA (standard attention) and MLA (mu
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**HPC-Ops**</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>64</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>**AITER (ROCm)**</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
@@ -312,7 +322,7 @@ Speculative decoding topk: `topk` is the number of draft tokens sampled per step
Page size controls how many tokens are grouped into a KV cache block. For the prefix cache to take effect, the number of tokens must fill at least one complete page. For example, if your prompt is only 32 tokens and `page_size = 64`, it won't fill a complete page and cannot be matched in the prefix cache (pages cannot be padded). With 65 tokens and `page_size = 64`, only the first page of 64 tokens will be cached and matched; the remaining 1 token is discarded. Use `page_size = 1` for maximum prefix reuse (token-level matching). Note that higher page sizes generally improve attention kernel performance, so prefer `page_size > 1` when prefix cache reuse is not critical.
</Tip>
Many backends that do not natively operate on pages can emulate `page_size > 1` at the wrapper layer by expanding page tables to per-token indices. The "Page Size > 1 (native)" column indicates true in-kernel paging. Some backends require fixed native page sizes and cannot be reduced/emulated differently: TRTLLM MHA (16/32/64), TRTLLM MLA (32/64), CuteDSL MLA (32/64), FlashMLA (64), Cutlass MLA (128), Ascend (128).
Many backends that do not natively operate on pages can emulate `page_size > 1` at the wrapper layer by expanding page tables to per-token indices. The "Page Size > 1 (native)" column indicates true in-kernel paging. Some backends require fixed native page sizes and cannot be reduced/emulated differently: TRTLLM MHA (16/32/64), TRTLLM MLA (32/64), CuteDSL MLA (32/64), FlashMLA (64), Cutlass MLA (128), Ascend (128), HPC-Ops (64).
MLA page-size constraints:
- FlashInfer MLA: page_size = 1.
@@ -617,6 +627,27 @@ python3 -m sglang.launch_server \
--trust-remote-code
```
- HPC-Ops (MHA kernels from [HPC-Ops](https://github.com/Tencent/hpc-ops) by the Tencent Hunyuan AI Infra team; Hopper+, requires installing the `hpc` package from source, page size 64, bf16 or fp8_e4m3 KV cache, head_dim 128, q/kv head group 4 or 8)
```bash Command
python3 -m sglang.launch_server \
--model Qwen/Qwen3-30B-A3B-Instruct-2507-FP8 \
--attention-backend hpc_ops \
--page-size 64 \
--trust-remote-code
# FP8 models should also set --kv-cache-dtype fp8_e4m3 to run the FP8 attention
# kernels. This enables the fused QKNorm+RoPE+FP8-quant+StoreKV op, which is
# currently wired for Hunyuan V3 and requires per-rank (q_heads, kv_heads) of
# (64, 8) or (8, 1), e.g. --tp 1 or --tp 8 for Hy3.
python3 -m sglang.launch_server \
--tp 8 \
--model tencent/Hy3-FP8 \
--attention-backend hpc_ops \
--kv-cache-dtype fp8_e4m3 \
--page-size 64 \
--trust-remote-code
```
- FlashAttention 4 (MHA & MLA)
```bash Command
# FA4 for both prefill and decode on SM90/SM100