minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity

CUTLASS NVFP4 GEMM improvement of SM120 (#21314)

Browse Source
This commit is contained in:
Brayden Zhong
2026-04-01 09:04:34 +08:00
committed by GitHub
parent 5bbf347bb3
commit 6a9b09847c
12 changed files with 845 additions and 901 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/advanced_features/quantization.md
+2 -1
View File
@@ -74,11 +74,12 @@ Backend selection is supported only for **blockwise FP8** and **NVFP4** GEMM. Wh
| Backend | Hardware | Description |
|---------|----------|-------------|
| `auto` | SM100/120 | Auto-selects: `flashinfer_cudnn` on SM120; `flashinfer_cutlass` on SM100 |
| `cutlass` | SM100/120 | SGLang CUTLASS kernel |
| `flashinfer_cutlass` | SM100/120 | FlashInfer CUTLASS backend |
| `flashinfer_cudnn` | SM100/120 (CUDA 13+, cuDNN 9.15+) | FlashInfer cuDNN backend; used on SM120 for performance |
| `flashinfer_trtllm` | SM100 | FlashInfer TensorRT-LLM backend |
When FlashInfer is unavailable for NVFP4, sgl-kernel CUTLASS is used as an automatic fallback.
When FlashInfer is unavailable for NVFP4, the SGLang CUTLASS kernel is used as an automatic fallback.
## Offline Quantization