[Refactor] kernels: Regroup operators by computation and dissolve ops/muse_spark (#45112)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5.5
parent
ab580b2e06
commit
693d333306
@@ -17,25 +17,47 @@ logical grouping and remove the old `_jit_` filename prefix.
|
||||
The `sgl_kernel` wheel retains its own public API and packaging tests.
|
||||
- Group by computation, not model or GPU: GEMM and GEMV in `gemm`, expert
|
||||
routing in `moe`, attention index selection in `attention`, sampling in
|
||||
`sampling`, normalization in `layernorm`. A quantized GEMM is still a GEMM;
|
||||
quantization alone belongs in `quantization`.
|
||||
`sampling`, normalization in `layernorm`.
|
||||
- Classify a fused operator by its complete contract; do not split a fused
|
||||
kernel into separate launches just to separate its parts. Split unrelated
|
||||
public entry points that happen to share a source file. Decided rules:
|
||||
- Any cross-rank collective or peer-memory access (all-reduce, all-gather,
|
||||
reduce-scatter, a2a, multicast/NVLS) puts the op in `communication`,
|
||||
whatever compute it fuses.
|
||||
- Producer + output quantization stays with the producer (norm + quant in
|
||||
`layernorm`, activation + quant in `activation`, GEMM with a quant epilogue
|
||||
in `gemm`, MoE dispatch/permute + quant in `moe`). `quantization` holds only
|
||||
pure format conversion: quant, dequant, scale-layout conversion, repack.
|
||||
- RoPE stays in `attention` (standalone RoPE, position ids, QK-norm + RoPE).
|
||||
- `kvcache` owns ops that read/write KV-cache contents; `attention` owns the
|
||||
index/shape/schedule tables (page tables, `cu_seqlens`, `kv_indices`, split
|
||||
metadata) and index selection. A fused compute + store stays with its
|
||||
compute group.
|
||||
- Model-specific files/subpackages and tuning data are allowed **inside** a
|
||||
logical group. Do not add a model bundle or an implementation file directly
|
||||
under `ops/`. Propose a new logical group only for a distinct responsibility,
|
||||
and update `ops._GROUPS`, documentation, and tests together.
|
||||
- Classify a fused operator by its complete contract. Do not split a fused
|
||||
kernel into separate launches just to separate norm, RoPE, or quantization.
|
||||
Split unrelated public entry points that happen to share a source file.
|
||||
- Keep shared CUDA build/runtime infrastructure in `kernels/jit`; operator
|
||||
wrappers call it from their logical group. Keep process groups, communicator
|
||||
state, model dispatch, and buffer ownership in `srt`. K3-specific adapters in
|
||||
`srt/layers/communication/` need not pretend to be generic interfaces.
|
||||
and update `ops._GROUPS`, the README group list, and tests together
|
||||
(`test_kernels_namespace.py` checks the README against `_GROUPS` and the
|
||||
README layout against the directories under `kernels/`).
|
||||
- Agent-generated implementations live in `kernels/kda_kernels/` (Kernel Design
|
||||
Agents, `KernelBackend.KDA`) and `kernels/cake_kernels/` (Cake adapters over
|
||||
FlashInfer). They are backends, not groups: register their public ops in the
|
||||
logical group with a `KernelSpec` whose target points into them, and do not
|
||||
reorganize those directories.
|
||||
- Keep shared build/runtime infrastructure (including the Helion runtime
|
||||
helper) in `kernels/jit`; operator wrappers call it from their logical group.
|
||||
Keep process groups, communicator state, workspace classes, model dispatch,
|
||||
and buffer ownership in `srt` (`srt/layers/communication/` for the K3 and Muse
|
||||
Spark adapters); kernels must not import srt runtime modules to inspect
|
||||
model objects, so pass flags in. Adapters there need not pretend to be generic
|
||||
interfaces.
|
||||
|
||||
## Register the public entry point
|
||||
|
||||
Add lazy `KernelSpec` metadata in the owning group's `__init__.py`, or use the
|
||||
existing `BaseFusedOp` registration when the operation has interchangeable
|
||||
backends. Group imports must not import GPU implementations or compile kernels.
|
||||
backends (a `BaseFusedOp` built per layer registers `module:Class.forward_<backend>`
|
||||
targets). Group imports must not import GPU implementations or compile kernels.
|
||||
|
||||
Use `<group>.<name>` for the op id and `module:callable` for the target. Preserve
|
||||
existing backend dispatch and describe actual device/architecture restrictions
|
||||
@@ -53,8 +75,9 @@ factories, reference helpers, and runtime classes are not separate kernel APIs.
|
||||
runtime module. A CUDA allocation does not make a runtime test a kernel test.
|
||||
- Non-CI smoke scripts and benchmarks: `test/manual/kernels/`; do not leave
|
||||
standalone test/benchmark entry points in production operator modules.
|
||||
- Shared test helpers: `sglang.test.kernels`. Preserve vendored upstream trees
|
||||
and AOT wheel packaging boundaries instead of reorganizing them incidentally.
|
||||
- Shared test helpers and reference oracles: `sglang.test.kernels`. Preserve
|
||||
vendored upstream trees and AOT wheel packaging boundaries instead of
|
||||
reorganizing them incidentally.
|
||||
|
||||
For a move/split, preserve assertions, parametrization, fixtures, platform skips,
|
||||
execution entry points, and CI stages/runners. Apportion existing time estimates
|
||||
|
||||
@@ -49,15 +49,15 @@ in-flight row as shipped.
|
||||
| FlashInfer unified `allreduce_fusion` | `cross_device_reduce_1stage*`<br>`all_reduce`<br>`FusedAddRMSNormKernel`<br>`rmsnorm*` | `python/sglang/srt/layers/flashinfer_comm_fusion.py`<br>`python/sglang/srt/layers/layernorm.py::forward_with_allreduce_fusion`<br>`python/sglang/srt/layers/layer_boundary/residual/add_norm.py::apply_flashinfer_allreduce_fusion` | FlashInfer workspace creation plus `allreduce_fusion(..., pattern=AllReduceFusionPattern.kARResidualRMSNorm, ...)` | First suspect missing / disabled / unsupported FlashInfer allreduce fusion, not a brand new TP fusion idea. |
|
||||
| AITER allreduce fusion | ROCm all-reduce plus RMSNorm still split | `python/sglang/srt/layers/layernorm.py::forward_with_allreduce_fusion`<br>`python/sglang/srt/distributed/communication_op.py::tensor_model_parallel_fused_allreduce_rmsnorm`<br>`python/sglang/srt/layers/layer_boundary/residual/add_norm.py::apply_aiter_all_reduce_fusion` | ROCm-side fused TP all-reduce + RMSNorm with fallback to plain all-reduce plus norm | On AMD, rule out existing AITER fusion before proposing a new communication fusion. |
|
||||
| Fused activation-and-mul (`SwiGLU` / `GeGLU`) | `silu_and_mul`<br>`gelu_and_mul`<br>`npu_swiglu` | `python/sglang/srt/layers/activation.py` | Single op covers activation plus elementwise multiply across CUDA / CPU / NPU / XPU backends | Treat separate activation + mul on packed MLP outputs as missing existing fusion. |
|
||||
| Fused dual residual RMSNorm | residual add plus two RMSNorm-like kernels around Grok blocks | `python/sglang/srt/layers/elementwise.py::fused_dual_residual_rmsnorm`<br>`python/sglang/srt/models/grok.py` | One Triton kernel computes intermediate residual update and next RMSNorm output together | On Grok-like residual layouts, treat split residual + norm as missing existing fusion. |
|
||||
| Fused dual residual RMSNorm | residual add plus two RMSNorm-like kernels around Grok blocks | `python/sglang/kernels/ops/layernorm/fused_rmsnorm.py::fused_dual_residual_rmsnorm`<br>`python/sglang/srt/models/grok.py` | One Triton kernel computes intermediate residual update and next RMSNorm output together | On Grok-like residual layouts, treat split residual + norm as missing existing fusion. |
|
||||
| In-place QK RMSNorm | split `q_norm` / `k_norm` kernels | `python/sglang/srt/models/utils.py::apply_qk_norm`<br>`python/sglang/kernels/ops/layernorm/norm.py::fused_inplace_qknorm` | In-place JIT QK norm plus optional `alt_stream` overlap for K | Check shape, dtype, deterministic mode, and in-place legality before proposing a new QK fuse. |
|
||||
| TorchInductor horizontal Q/K norm combo-kernels | `combo_kernels`<br>`benchmark_combo_kernel`<br>`q_norm`<br>`k_norm`<br>`split_with_sizes` | `torch._inductor.config.combo_kernels` | TorchInductor can horizontally fuse sibling Q-norm and K-norm kernels in compiled traces, often deleting `split_with_sizes` / `clone` ladders | Treat separate Q/K norm ladders in compile-heavy traces as an existing compiler-fusion family first. |
|
||||
| MiniMax TP fused QK RMSNorm | `MiniMaxM2RMSNormTP`<br>`rms_sumsq_serial`<br>`rms_apply_serial`<br>`forward_qk` | `python/sglang/srt/models/minimax_m2.py` | Triton kernels compute Q / K sumsq together, TP all-reduces shared stats, then apply both RMSNorms together | On MiniMax traces, separate Q norm and K norm are usually a missed model-specific Triton fusion. |
|
||||
| Fused QK RMSNorm + RoPE | `qknorm*` + `rope*` + `rotary*` as separate steps | `python/sglang/kernels/ops/attention/fused_qknorm_rope.py`<br>`python/sglang/srt/models/qwen3_moe.py` | One JIT kernel applies QK RMSNorm and RoPE in-place on packed QKV | For compatible LLMs, classify split QK norm + RoPE as a missing existing fusion. |
|
||||
| Fused QK RoPE reshape + KV cache write | `fused_qk_rope_reshape_and_cache*`<br>RoPE followed by reshape / cache DtoD | `python/sglang/srt/layers/attention/utils.py::fused_qk_rope_reshape_and_cache` | One Triton kernel applies RoPE to Q / K, reshapes cache layout, and writes K / V directly to paged cache | Treat separate RoPE + reshape + cache-write ladders as an existing attention-prep fusion family. |
|
||||
| Fused QK RoPE reshape + KV cache write | `fused_qk_rope_reshape_and_cache*`<br>RoPE followed by reshape / cache DtoD | `python/sglang/kernels/ops/attention/fused_qk_rope_reshape_and_cache.py::fused_qk_rope_reshape_and_cache` | One Triton kernel applies RoPE to Q / K, reshapes cache layout, and writes K / V directly to paged cache | Treat separate RoPE + reshape + cache-write ladders as an existing attention-prep fusion family. |
|
||||
| Fused RoPE + KV cache store | `fused_set_kv_buffer`<br>RoPE followed by KV-store, DtoD, or cache-write kernels | `python/sglang/kernels/ops/attention/rope.py`<br>`python/sglang/srt/models/utils.py::enable_fused_set_kv_buffer` | Shared entrypoints can route to fused RoPE + KV-store or model-side `fused_set_kv_buffer` fast paths | Compare against the fused cache-store path before proposing a new KV rewrite. |
|
||||
| Fused decode metadata setup | `normal_decode_set_metadata`<br>`cache_seqlens_int32`<br>`cu_seqlens_k`<br>`page_table`<br>`swa_page_table` | `python/sglang/srt/layers/attention/flashattention_backend.py::normal_decode_set_metadata` | Triton decode path fuses seq-len cast/add, prefix-sum, req-to-token gather, page-table divide, and optional SWA metadata build into 1-2 kernels | If decode exposes multiple tiny metadata kernels before attention, first compare against this existing fused metadata-prep path. |
|
||||
| NSA fused metadata copy for graph replay | `fused_metadata_copy`<br>`fused_metadata_copy_multi`<br>`fused_nsa_cache_seqlens`<br>`fused_flashmla_metadata` | `python/sglang/kernels/ops/attention/fused_metadata_copy.py` | CUDA graph replay path fuses multiple metadata copies into one kernel or one multi-destination kernel | Treat bursts of tiny metadata-copy kernels around NSA replay as a missed existing replay fusion. |
|
||||
| NSA fused metadata copy for graph replay | `fused_metadata_copy`<br>`fused_metadata_copy_multi`<br>`fused_nsa_cache_seqlens`<br>`fused_flashmla_metadata` | `python/sglang/kernels/ops/attention/dsa/fused_metadata_copy.py` | CUDA graph replay path fuses multiple metadata copies into one kernel or one multi-destination kernel | Treat bursts of tiny metadata-copy kernels around NSA replay as a missed existing replay fusion. |
|
||||
| DeepSeek MLA fused projection + norm + RoPE | `qkv_proj_with_rope_fused_weight`<br>`fused_qkv_a_proj_with_mqa`<br>`forward_absorb_fused_mla_rope*` | `python/sglang/srt/models/deepseek_common/attention_forward_methods/forward_mla_fused_rope_cpu.py`<br>`python/sglang/srt/models/deepseek_common/attention_forward_methods/forward_mla_fused_rope_rocm.py`<br>`python/sglang/srt/models/deepseek_v2.py` | CPU / ROCm paths fuse DeepSeek MLA projection packing with q / k norm, RoPE, and cache-oriented MLA prep | For DeepSeek MLA, split proj / norm / rope prep is usually an existing backend-specific fuse that did not fire. |
|
||||
| Fused QK RoPE concat + MLA cache write | `fused_qk_rope_cat_and_cache_mla`<br>`set_mla_kv_buffer` | `python/sglang/srt/layers/rocm_linear_utils.py`<br>`python/sglang/srt/models/deepseek_common/attention_forward_methods/forward_mla.py` | ROCm MLA path can fuse Q / K RoPE packing, concat, and MLA cache write in one backend-specific op | On DeepSeek / MLA traces, separate RoPE-cat-cache steps are not automatically novel. |
|
||||
| Qwen3 decode fused QK norm + 3D mRoPE + KV cache write | `fused_qk_norm_mrope_3d_cache_pts_quant_shuffle`<br>`mrope`<br>decode cache write | `python/sglang/srt/models/qwen3.py` | ROCm / AITER decode path fuses QK norm, 3D mRoPE, and paged KV cache write | On Qwen3-style decode, separate norm + mRoPE + cache-store kernels are not a novel opportunity. |
|
||||
@@ -69,18 +69,18 @@ in-flight row as shipped.
|
||||
| Qwen-style shared-expert append into routed top-k output | `_append_shared_to_topk_output`<br>`fused_append_shared_experts_with_weights`<br>`num_fused_shared_experts` | `python/sglang/srt/models/qwen2_moe.py`<br>`python/sglang/srt/layers/moe/moe_runner/triton_utils/fused_moe_triton_kernels.py` | Qwen-style MoE paths can append shared-expert ids and sigmoid gate weights to routed top-k output in one Triton kernel so the shared experts execute inside the fused MoE path | Treat routed top-k plus shared-expert pad / concat ladders as an existing MoE-prep fusion family first. |
|
||||
| Fused MoE dispatch / permute / combine | token permutation<br>dispatch / combine<br>grouped top-k<br>many small MoE support kernels | `python/sglang/srt/layers/moe/fused_moe_triton/layer.py`<br>`python/sglang/srt/layers/moe/fused_moe_triton/fused_moe.py` | `FusedMoE` plus DeepEP / FlashInfer / FuseEP / standard dispatch backends and `permute_fusion=True` | First ask whether the model is missing an existing `FusedMoE`-style path or backend-specific dispatcher path. |
|
||||
| Fused MoE sum + all-reduce | routed MoE followed by explicit sum-reduce kernels | `python/sglang/srt/layers/moe/fused_moe_triton/fused_moe.py`<br>`python/sglang/srt/layers/moe/fused_moe_triton/fused_moe_triton_kernels.py` | `fuse_sum_all_reduce=True` path in the second MoE GEMM | Before inventing a new MoE reduction fuse, check whether `enable_fused_moe_sum_all_reduce` is simply off or the quant path is incompatible. |
|
||||
| Fused MoE activation + quant / re-quant | `silu_and_mul_*quant*`<br>`npu_dequant_swiglu_quant`<br>`swiglu_quant` | `python/sglang/srt/layers/moe/ep_moe/kernels.py`<br>`python/sglang/kernels/ops/quantization/nvfp4_gemm_swiglu_nvfp4_quant.py`<br>`python/sglang/srt/layers/moe/cutlass_w4a8_moe.py`<br>`python/sglang/srt/hardware_backend/npu/quantization/moe_methods.py` | Quantized MoE backends fuse SwiGLU / SiLU-and-mul with FP8 / FP4 / NPU re-quant before the second expert GEMM | If MoE traces show standalone activation then quant kernels, first check whether the quantized fused path is missing. |
|
||||
| Fused MoE activation + quant / re-quant | `silu_and_mul_*quant*`<br>`npu_dequant_swiglu_quant`<br>`swiglu_quant` | `python/sglang/srt/layers/moe/ep_moe/kernels.py`<br>`python/sglang/kernels/ops/gemm/nvfp4_gemm_swiglu_nvfp4_quant.py`<br>`python/sglang/srt/layers/moe/cutlass_w4a8_moe.py`<br>`python/sglang/srt/hardware_backend/npu/quantization/moe_methods.py` | Quantized MoE backends fuse SwiGLU / SiLU-and-mul with FP8 / FP4 / NPU re-quant before the second expert GEMM | If MoE traces show standalone activation then quant kernels, first check whether the quantized fused path is missing. |
|
||||
| DeepSeek comm-prep fused RMSNorm + quant / flatten-quant | `fused_rms_fp8_group_quant`<br>`fused_rms_mxfp4_quant`<br>`fused_flatten_fp8_group_quant`<br>`fused_flatten_mxfp4_quant` | `python/sglang/srt/layers/layer_boundary/residual/add_norm.py`<br>`python/sglang/srt/models/deepseek_common/attention_forward_methods/forward_mla.py`<br>`python/sglang/srt/models/deepseek_common/attention_forward_methods/forward_mha.py` | DeepSeek MLA / MHA ROCm paths fuse RMSNorm or flatten with FP8 / MXFP4 quantization for comm / attention prep | On DeepSeek quant traces, split norm + quant or flatten + quant is an existing family, not a new idea. |
|
||||
| NSA fused top-k transform / page-table build | `fast_topk_transform_fused`<br>`fast_topk_transform_ragged_fused` | `python/sglang/srt/layers/attention/dsa_backend.py` | NSA can fuse top-k selection with paged / ragged index transform instead of separate top-k plus metadata scatter | If NSA top-k metadata work is split, check `SGLANG_DSA_FUSE_TOPK` and backend support first. |
|
||||
| NSA fused quantize + indexed K-cache store | `fused_store_index_k_cache`<br>`act_quant`<br>`index_k_with_scale_buffer` | `python/sglang/kernels/ops/attention/fused_store_index_cache.py`<br>`python/sglang/srt/layers/attention/nsa/nsa_indexer.py` | Single JIT kernel quantizes bf16 K to fp8 + scale and writes directly into NSA index cache | Treat split `act_quant` + buffer-store on CUDA as missing an existing fused store path. |
|
||||
| NSA fused quantize + indexed K-cache store | `fused_store_index_k_cache`<br>`act_quant`<br>`index_k_with_scale_buffer` | `python/sglang/kernels/ops/kvcache/dsa_fused_store_index_cache.py`<br>`python/sglang/srt/layers/attention/nsa/nsa_indexer.py` | Single JIT kernel quantizes bf16 K to fp8 + scale and writes directly into NSA index cache | Treat split `act_quant` + buffer-store on CUDA as missing an existing fused store path. |
|
||||
| Fused sampling temperature + softmax | `fused_temperature_softmax*` | `python/sglang/srt/layers/fused_sampling.py`<br>`python/sglang/srt/layers/sampler.py` | Triton single-pass / multi-pass kernels fuse temperature scaling and softmax during decode | Separate temp-divide + softmax at decode batch sizes is often a missed existing fusion. |
|
||||
| Fused logit softcap | `fused_softcap`<br>`final_logit_softcapping` | `python/sglang/srt/layers/elementwise.py`<br>`python/sglang/srt/layers/logits_processor.py` | Triton kernels fuse cast-to-float and softcap / tanh math for logits or generic elementwise softcapping | Treat exposed cast + softcap ladders as an existing Triton fuse family. |
|
||||
| Linear-attention packed projection reshuffle | `fused_qkvzba_split_reshape_cat*`<br>`qkvz_proj`<br>`ba_proj`<br>`qkvabz_proj`<br>`fused_qkvbfg_a_proj` | `python/sglang/kernels/ops/attention/triton_gdn_fused_proj.py`<br>`python/sglang/srt/models/qwen3_next.py`<br>`python/sglang/srt/models/qwen3_5.py`<br>`python/sglang/srt/models/kimi_linear.py`<br>`python/sglang/srt/models/jet_nemotron.py` | GDN / Kimi / Jet-style linear-attn models pack multiple projections, then fuse split / reshape / cat into one kernel | Treat split reshape / transpose / cat ladders as an existing linear-attention fusion family. |
|
||||
| Fused GDN gating prep | `fused_gdn_gating`<br>`softplus`<br>`beta_output` | `python/sglang/srt/layers/attention/fla/fused_gdn_gating.py` | Triton kernel computes GDN gate preparation such as `-exp(A_log) * softplus(...)` and `sigmoid(b)` together | On GDN traces, treat split gate-prep elementwise kernels as missing existing fusion first. |
|
||||
| Fused GDN gating prep | `fused_gdn_gating`<br>`softplus`<br>`beta_output` | `python/sglang/kernels/ops/attention/linear/fused_gdn_gating.py` | Triton kernel computes GDN gate preparation such as `-exp(A_log) * softplus(...)` and `sigmoid(b)` together | On GDN traces, treat split gate-prep elementwise kernels as missing existing fusion first. |
|
||||
| Fused RMSNorm-gated linear-attention output | `FusedRMSNormGated`<br>`layer_norm_gated_fwd` | `python/sglang/srt/layers/attention/fla/fused_norm_gate.py`<br>`python/sglang/srt/models/qwen3_next.py`<br>`python/sglang/srt/models/kimi_linear.py` | One Triton op covers residual-aware (RMS)Norm plus sigmoid / swish gating | If norm and output gate appear as separate kernels in GDN / Kimi-like blocks, first suspect a missing existing fusion. |
|
||||
| Fused gated RMSNorm / LayerNorm | `rms_norm_gated`<br>`layer_norm_gated` | `python/sglang/srt/layers/attention/mamba/ops/layernorm_gated.py` | Mamba-derived kernels can fuse normalization with the gating branch `z * sigmoid(z)` | Treat split norm and gate post-processing on Mamba-style blocks as an existing fusion family. |
|
||||
| Fused linear-attention chunk KKT + solve_tril | `chunk_gated_delta_rule_fwd_kkt_solve_kernel`<br>`scaled_dot_kkt`<br>`solve_tril`<br>`recompute_w_u` | `python/sglang/srt/layers/attention/fla/chunk_fwd.py`<br>`python/sglang/srt/layers/attention/fla/kda.py` | GDN / KDA chunk forward fuses `scaled_dot_kkt + solve_tril` in the prefill / intra-chunk path, then finishes `recompute_w_u` as the next step | Treat split KKT + triangular-solve ladders as an existing linear-attention fusion family first. |
|
||||
| Fused linear-attention recurrent / KDA update | `fused_sigmoid_gating_delta_rule_update`<br>`fused_recurrent_gated_delta_rule_update`<br>`fused_kda_gate` | `python/sglang/srt/layers/attention/fla/fused_sigmoid_gating_recurrent.py`<br>`python/sglang/srt/layers/attention/fla/fused_recurrent.py`<br>`python/sglang/srt/models/kimi_linear.py`<br>`python/sglang/srt/models/jet_nemotron.py` | Triton / CuTeDSL kernels fuse gating math, optional QK l2norm, recurrent state update, and output generation | Treat split gating + recurrent-update chains as existing linear-attention fusion, not a novel opportunity. |
|
||||
| Fused linear-attention recurrent / KDA update | `fused_sigmoid_gating_delta_rule_update`<br>`fused_recurrent_gated_delta_rule_update`<br>`fused_kda_gate` | `python/sglang/kernels/ops/attention/linear/fused_sigmoid_gating_recurrent.py`<br>`python/sglang/kernels/ops/attention/fla/fused_recurrent.py`<br>`python/sglang/srt/models/kimi_linear.py`<br>`python/sglang/srt/models/jet_nemotron.py` | Triton / CuTeDSL kernels fuse gating math, optional QK l2norm, recurrent state update, and output generation | Treat split gating + recurrent-update chains as existing linear-attention fusion, not a novel opportunity. |
|
||||
| Fused Mamba state gather/scatter with mask | `fused_mamba_state_scatter_with_mask`<br>`index_elementwise_kernel` | `python/sglang/srt/layers/attention/mamba/mamba_state_scatter_triton.py` | Triton kernel replaces multiple masked gather / scatter index kernels with one fused update | If Mamba verify/update shows many tiny index kernels, first compare against this existing fused path. |
|
||||
| Staging-buffer fused gather / scatter | `_fused_gather_to_staging_kernel`<br>`_fused_scatter_from_staging_kernel` | `python/sglang/srt/disaggregation/common/staging_buffer.py` | Triton kernels gather scattered KV slices into contiguous staging memory and scatter them back into KV cache on decode | Treat ladders of tiny gather/scatter/copy kernels in heterogeneous TP staging as missing an existing Triton fusion. |
|
||||
|
||||
@@ -146,7 +146,7 @@ Stable entries should be folded into the mainline family rows above.
|
||||
| PR `#22005` fused add + RMSNorm + per-token FP8 quant | `fused_add_rmsnorm_per_token_quant`<br>`per_token_quant_fp8` | `PR #22005`<br>`python/sglang/kernels/jit/csrc/elementwise/fused_add_rmsnorm_per_token_quant.cuh`<br>`python/sglang/kernels/jit/fused_add_rmsnorm_per_token_quant.py` | CUDA JIT kernel keeps normed values in registers and emits BF16 + FP8 outputs plus per-token scales | If FP8 online-quant traces show add+norm followed by per-token quant, treat this as an in-flight upstream CUDA fuse family. |
|
||||
| PR `#20667` Qwen3.5 fused QK norm + RoPE + KV cache write | `fused_qk_norm_rope_cache_pts_quant_shuffle`<br>`fused_qk_norm_mrope_3d_cache_pts_quant_shuffle`<br>`rotary_dim` | `PR #20667`<br>`python/sglang/srt/models/qwen3_5.py`<br>`python/sglang/srt/models/utils.py` | ROCm / AITER path fuses Q / K RMSNorm, partial or 3D RoPE, and direct KV cache write for Qwen3.5 attention | Treat split QK-norm + RoPE + cache-store on Qwen3.5 as a concrete in-flight upstream family, not a novel idea. |
|
||||
| PR `#22392` CUTLASS FP8 GEMM replacing nvjet | `cutlass_scaled_mm`<br>`fp8_scaled_mm`<br>`nvjet`<br>`cudaMemsetAsync` | `PR #22392`<br>`python/sglang/kernels/aot/python/sgl_kernel/gemm.py`<br>`python/sglang/srt/layers/quantization/fp8_utils.py` | Runtime replacement swaps nvjet FP8 GEMMs for CUTLASS kernels, removing per-launch memset bubbles and extra output-copy kernels | Treat nvjet GEMM + memset bubble ladders as an in-flight SGLang linear-kernel family before calling them novel. |
|
||||
| PR `#18612` NVFP4 CUTLASS MoE fused SiLU+Mul+quant | `silu_and_mul_scaled_nvfp4`<br>`nvfp4 expert quant`<br>`cutlass moe` | `PR #18612`<br>`python/sglang/srt/layers/moe/cutlass_w4a8_moe.py`<br>`python/sglang/kernels/ops/quantization/nvfp4_gemm_swiglu_nvfp4_quant.py` | Fuses MoE activation epilogue and NVFP4 expert quantization before the CUTLASS MoE second GEMM | Treat split SiLU+Mul then NVFP4 expert quant in CUTLASS MoE traces as an in-flight upstream SGLang family. |
|
||||
| PR `#18612` NVFP4 CUTLASS MoE fused SiLU+Mul+quant | `silu_and_mul_scaled_nvfp4`<br>`nvfp4 expert quant`<br>`cutlass moe` | `PR #18612`<br>`python/sglang/srt/layers/moe/cutlass_w4a8_moe.py`<br>`python/sglang/kernels/ops/gemm/nvfp4_gemm_swiglu_nvfp4_quant.py` | Fuses MoE activation epilogue and NVFP4 expert quantization before the CUTLASS MoE second GEMM | Treat split SiLU+Mul then NVFP4 expert quant in CUTLASS MoE traces as an in-flight upstream SGLang family. |
|
||||
| PR `#22918` FlashInfer per-token NVFP4 MoE | `per_token_nvfp4`<br>`trtllm_fp4_block_scale_moe`<br>`FlashInfer MoE` | `PR #22918`<br>`python/sglang/srt/layers/moe/fused_moe_triton/fused_moe.py` | Adds FlashInfer-backed per-token NVFP4 MoE execution so expert quant/dequant work can move into the fused MoE backend | Treat standalone per-token NVFP4 MoE support kernels as a candidate missing backend-selection path, not an automatically novel kernel idea. |
|
||||
| PR `#22851` NSA top-k backend and FlashInfer / PyTorch top-k split | `nsa topk`<br>`flashinfer_topk`<br>`pytorch_topk`<br>`fast_topk_transform` | `PR #22851`<br>`python/sglang/srt/layers/attention/dsa_backend.py` | Makes NSA top-k backend selection explicit and aligns fused top-k transform with FlashInfer / PyTorch fallbacks | When NSA top-k dominates decode, first classify it as backend selection or fused-transform eligibility work. |
|
||||
| PR `#24125` GLM5 NSA decode CatArrayBatchedCopy removal | `CatArrayBatchedCopy`<br>`GLM-5`<br>`NSA`<br>`TileLang decode` | `PR #24125`<br>`python/sglang/srt/layers/attention/dsa_backend.py` | Skips redundant cat/copy work in the GLM5 NSA TileLang decode path | Treat cat/copy bursts in GLM5 NSA decode as a concrete in-flight cleanup opportunity. |
|
||||
|
||||
@@ -482,7 +482,9 @@ FUSION_PATTERN_REGISTRY: Tuple[FusionPatternSpec, ...] = (
|
||||
),
|
||||
FusionPatternSpec(
|
||||
pattern="Fused QK RoPE reshape + KV cache write",
|
||||
candidate_path="python/sglang/srt/layers/attention/utils.py",
|
||||
candidate_path=(
|
||||
"python/sglang/kernels/ops/attention/fused_qk_rope_reshape_and_cache.py"
|
||||
),
|
||||
active_keywords=("fused_qk_rope_reshape_and_cache",),
|
||||
split_groups=(
|
||||
("rotary", "rope", "mrope"),
|
||||
@@ -533,7 +535,7 @@ FUSION_PATTERN_REGISTRY: Tuple[FusionPatternSpec, ...] = (
|
||||
),
|
||||
FusionPatternSpec(
|
||||
pattern="NSA fused metadata copy for graph replay",
|
||||
candidate_path="python/sglang/kernels/ops/attention/fused_metadata_copy.py",
|
||||
candidate_path="python/sglang/kernels/ops/attention/dsa/fused_metadata_copy.py",
|
||||
active_keywords=(
|
||||
"fused_metadata_copy",
|
||||
"fused_metadata_copy_multi",
|
||||
@@ -700,7 +702,7 @@ FUSION_PATTERN_REGISTRY: Tuple[FusionPatternSpec, ...] = (
|
||||
pattern="Fused MoE activation + quant / re-quant",
|
||||
candidate_path=(
|
||||
"python/sglang/srt/layers/moe/ep_moe/kernels.py"
|
||||
"<br>python/sglang/kernels/ops/quantization/nvfp4_gemm_swiglu_nvfp4_quant.py"
|
||||
"<br>python/sglang/kernels/ops/gemm/nvfp4_gemm_swiglu_nvfp4_quant.py"
|
||||
"<br>python/sglang/srt/layers/moe/cutlass_w4a8_moe.py"
|
||||
),
|
||||
active_keywords=(
|
||||
@@ -761,7 +763,7 @@ FUSION_PATTERN_REGISTRY: Tuple[FusionPatternSpec, ...] = (
|
||||
FusionPatternSpec(
|
||||
pattern="NSA fused quantize + indexed K-cache store",
|
||||
candidate_path=(
|
||||
"python/sglang/kernels/ops/attention/fused_store_index_cache.py"
|
||||
"python/sglang/kernels/ops/kvcache/dsa_fused_store_index_cache.py"
|
||||
"<br>python/sglang/srt/layers/attention/nsa/nsa_indexer.py"
|
||||
),
|
||||
active_keywords=("fused_store_index_k_cache",),
|
||||
|
||||
@@ -7,6 +7,7 @@
|
||||
/python/sglang/kernels @DarkSharpness @HaiShaw @BBuf @celve @HydraQYH @yuan-luo
|
||||
/python/sglang/kernels/ops/diffusion @yingluosanqian @BBuf @mickqian @triple-mu @niehen6174
|
||||
/python/sglang/kernels/ops/attention/fla @yizhang2077 @hebiao064 @yuan-luo
|
||||
/python/sglang/kernels/ops/attention/linear @yizhang2077 @hebiao064 @yuan-luo
|
||||
/python/sglang/kernels/ops/lora @Ying1123 @Fridge003 @lifuhuang @yushengsu-thu @jybsuper
|
||||
/python/sglang/kernels/jit/csrc/lora @Ying1123 @Fridge003 @lifuhuang @yushengsu-thu @jybsuper
|
||||
/python/sglang/multimodal_gen @mickqian @ping1jing2 @HaiShaw @yichiche @AgainstEntropy @BBuf @triple-mu @kevin-mii @niehen6174
|
||||
@@ -107,6 +108,7 @@
|
||||
/python/sglang/srt/speculative/cpp_ngram @hnyls2002 @Qiaolin-Yu @kpham-sgl
|
||||
/python/sglang/srt/speculative/frozen_kv_mtp_*.py @hnyls2002 @Qiaolin-Yu @kpham-sgl @pyc96
|
||||
/python/sglang/kernels/ops/speculative/ngram_*.py @hnyls2002 @Qiaolin-Yu @kpham-sgl
|
||||
/python/sglang/kernels/ops/embeddings/ngram_embedding.py @hnyls2002 @Qiaolin-Yu @kpham-sgl
|
||||
/python/sglang/kernels/jit/csrc/ngram_corpus @hnyls2002 @Qiaolin-Yu @kpham-sgl
|
||||
/python/sglang/srt/models/laguna.py @Jiminator
|
||||
/python/sglang/srt/configs/laguna.py @Jiminator
|
||||
|
||||
@@ -95,7 +95,7 @@ jobs:
|
||||
- "python/setup.py"
|
||||
- "python/sglang/!(multimodal_gen|kernels)/**/!(*.md)"
|
||||
# Diffusion-only kernels and tests are left to multimodal_gen and
|
||||
# jit_kernel; ops/diffusion/common stays since srt imports it via kda_kernels.
|
||||
# jit_kernel; ops/diffusion/common stays since kda_kernels (an srt path) imports it.
|
||||
- "python/sglang/kernels/{,ops/,jit/,jit/csrc/}!(*.md)"
|
||||
- "python/sglang/kernels/!(aot|ops|jit)/**/!(*.md)"
|
||||
- "python/sglang/kernels/jit/!(csrc)/**/!(*.md)"
|
||||
|
||||
@@ -20,10 +20,10 @@ import triton
|
||||
from sglang.kernels.ops.attention.cutedsl_kda import (
|
||||
cutedsl_fused_sigmoid_gating_kda_update,
|
||||
)
|
||||
from sglang.kernels.ops.attention.fla.fused_sigmoid_gating_recurrent import (
|
||||
from sglang.kernels.ops.attention.fla.kda import chunk_kda
|
||||
from sglang.kernels.ops.attention.linear.fused_sigmoid_gating_recurrent import (
|
||||
fused_sigmoid_gating_delta_rule_update,
|
||||
)
|
||||
from sglang.kernels.ops.attention.fla.kda import chunk_kda
|
||||
|
||||
|
||||
def make_inputs(
|
||||
|
||||
@@ -32,7 +32,7 @@ import triton
|
||||
from sglang.kernels.ops.attention.fla.fused_recurrent import (
|
||||
fused_recurrent_gated_delta_rule_packed_decode,
|
||||
)
|
||||
from sglang.kernels.ops.attention.fla.fused_sigmoid_gating_recurrent import (
|
||||
from sglang.kernels.ops.attention.linear.fused_sigmoid_gating_recurrent import (
|
||||
fused_sigmoid_gating_delta_rule_update,
|
||||
)
|
||||
|
||||
|
||||
@@ -26,12 +26,12 @@ import torch
|
||||
from sglang.kernels.ops.attention.fla.fused_recurrent import (
|
||||
fused_recurrent_kda_packed_decode,
|
||||
)
|
||||
from sglang.kernels.ops.attention.fla.fused_sigmoid_gating_recurrent import (
|
||||
fused_sigmoid_gating_delta_rule_update,
|
||||
)
|
||||
from sglang.kernels.ops.attention.helion.kda_decode import (
|
||||
helion_fused_recurrent_kda_packed_decode,
|
||||
)
|
||||
from sglang.kernels.ops.attention.linear.fused_sigmoid_gating_recurrent import (
|
||||
fused_sigmoid_gating_delta_rule_update,
|
||||
)
|
||||
|
||||
|
||||
def make_inputs(
|
||||
|
||||
@@ -37,7 +37,7 @@ import triton
|
||||
from sglang.kernels.ops.attention.fla.fused_recurrent import (
|
||||
fused_recurrent_gated_delta_rule_packed_decode,
|
||||
)
|
||||
from sglang.kernels.ops.attention.fla.fused_recurrent_linear_replayssm import (
|
||||
from sglang.kernels.ops.attention.linear.fused_recurrent_linear_replayssm import (
|
||||
fused_recurrent_gdn_replayssm_decode,
|
||||
)
|
||||
|
||||
|
||||
@@ -22,11 +22,11 @@ import argparse
|
||||
|
||||
import torch
|
||||
|
||||
import sglang.kernels.ops.attention.fla.fused_kda_conv_recurrent_verify as fused_mod
|
||||
from sglang.kernels.ops.attention.fla.fused_kda_conv_recurrent_verify import (
|
||||
import sglang.kernels.ops.attention.linear.fused_kda_conv_recurrent_verify as fused_mod
|
||||
from sglang.kernels.ops.attention.linear.fused_kda_conv_recurrent_verify import (
|
||||
fused_kda_conv_gating_verify,
|
||||
)
|
||||
from sglang.kernels.ops.attention.fla.fused_sigmoid_gating_recurrent import (
|
||||
from sglang.kernels.ops.attention.linear.fused_sigmoid_gating_recurrent import (
|
||||
fused_sigmoid_gating_delta_rule_update,
|
||||
)
|
||||
from sglang.kernels.ops.mamba.causal_conv1d_triton import causal_conv1d_update
|
||||
|
||||
@@ -3,7 +3,7 @@
|
||||
|
||||
Compares the legacy gather (``gather_dequant_requant_fp8_paged_legacy``:
|
||||
fresh ``torch.zeros`` destination + one program per (token, 128-elem
|
||||
slice)) against the new gather (``gather_dequant_requant_fp8_paged``:
|
||||
slice)) against the new gather (``gather_dequant_requant_fp8_paged_dsa``:
|
||||
no pre-zeroing needed, fused pad-row zero-fill, TOKENS_PER_PROG tokens
|
||||
per program with 16B-vectorized access), in three flavors:
|
||||
|
||||
@@ -43,6 +43,8 @@ import sys
|
||||
from pathlib import Path
|
||||
|
||||
import torch
|
||||
import triton
|
||||
import triton.language as tl
|
||||
|
||||
_REPO_ROOT = Path(__file__).resolve().parents[3]
|
||||
_MOD_PATH = _REPO_ROOT / "python/sglang/kernels/ops/attention/dsa/dequant_k_cache.py"
|
||||
@@ -54,8 +56,127 @@ _spec = importlib.util.spec_from_file_location("dequant_k_cache", _MOD_PATH)
|
||||
_mod = importlib.util.module_from_spec(_spec)
|
||||
_spec.loader.exec_module(_mod)
|
||||
|
||||
gather_new = _mod.gather_dequant_requant_fp8_paged
|
||||
gather_legacy = _mod.gather_dequant_requant_fp8_paged_legacy
|
||||
gather_new = _mod.gather_dequant_requant_fp8_paged_dsa
|
||||
_infer_dsa_dims = _mod._infer_dsa_dims
|
||||
|
||||
|
||||
def gather_dequant_requant_fp8_paged_legacy(
|
||||
quant_k_cache: torch.Tensor,
|
||||
page_table_1_flattened: torch.Tensor,
|
||||
group_size: int = 128,
|
||||
extra_rows: int = 0,
|
||||
) -> torch.Tensor:
|
||||
"""Legacy (pre-vectorization) gather + dequant + requant.
|
||||
|
||||
Kept as the bit-exactness / performance reference for
|
||||
``gather_dequant_requant_fp8_paged_dsa`` (see
|
||||
``benchmark/kernels/deepseek/benchmark_q8kv8_kv_gather.py``). Allocates and zero-fills the
|
||||
full destination each call, then launches one program per
|
||||
(token, 128-elem slice).
|
||||
"""
|
||||
dim_quant = quant_k_cache.shape[-1]
|
||||
dim_nope, dim_rope = _infer_dsa_dims(dim_quant)
|
||||
quant_k_cache = quant_k_cache.view((-1, dim_quant))
|
||||
|
||||
num_tokens = page_table_1_flattened.shape[0]
|
||||
assert quant_k_cache.dtype == torch.float8_e4m3fn
|
||||
num_tiles = dim_nope // group_size
|
||||
out_dim = dim_nope + dim_rope
|
||||
assert num_tiles * group_size == dim_nope
|
||||
|
||||
total_rows = num_tokens + extra_rows
|
||||
# Allocate a fresh zero-filled buffer. The extra landing-pad rows at
|
||||
# the tail must read as zeros (the kernel may over-read past
|
||||
# num_tokens for masked indices).
|
||||
output = torch.zeros(
|
||||
(total_rows, 1, out_dim),
|
||||
dtype=torch.float8_e4m3fn,
|
||||
device=quant_k_cache.device,
|
||||
)
|
||||
|
||||
num_blocks_per_token = num_tiles + triton.cdiv(dim_rope, group_size)
|
||||
|
||||
input_nope_q = quant_k_cache[:, :dim_nope]
|
||||
input_nope_s = quant_k_cache[:, dim_nope : dim_nope + num_tiles * 4].view(
|
||||
torch.float32
|
||||
)
|
||||
input_rope = quant_k_cache[:, dim_nope + num_tiles * 4 :].view(torch.bfloat16)
|
||||
|
||||
_gather_dequant_requant_fp8_paged_kernel[(num_tokens, num_blocks_per_token)](
|
||||
output,
|
||||
input_nope_q,
|
||||
input_nope_s,
|
||||
input_rope,
|
||||
page_table_1_flattened,
|
||||
output.stride(0),
|
||||
input_nope_q.stride(0),
|
||||
input_nope_s.stride(0),
|
||||
input_rope.stride(0),
|
||||
NUM_NOPE_BLOCKS=num_tiles,
|
||||
GROUP_SIZE=group_size,
|
||||
DIM_NOPE=dim_nope,
|
||||
DIM_ROPE=dim_rope,
|
||||
)
|
||||
|
||||
return output
|
||||
|
||||
|
||||
@triton.jit
|
||||
def _gather_dequant_requant_fp8_paged_kernel(
|
||||
output_ptr,
|
||||
input_nope_q_ptr,
|
||||
input_nope_s_ptr,
|
||||
input_rope_ptr,
|
||||
page_table_1_ptr,
|
||||
output_stride_0: int,
|
||||
input_nope_q_stride_0: int,
|
||||
input_nope_s_stride_0: int,
|
||||
input_rope_stride_0: int,
|
||||
NUM_NOPE_BLOCKS: tl.constexpr,
|
||||
GROUP_SIZE: tl.constexpr,
|
||||
DIM_NOPE: tl.constexpr,
|
||||
DIM_ROPE: tl.constexpr,
|
||||
):
|
||||
"""Fused gather + dequant(per-group) + requant(per-tensor) -> fp8."""
|
||||
token_id = tl.program_id(0)
|
||||
token_id_paged = tl.load(page_table_1_ptr + token_id).to(tl.int32)
|
||||
raw_block_id = tl.program_id(1)
|
||||
|
||||
if raw_block_id < NUM_NOPE_BLOCKS:
|
||||
# nope: read fp8, mul group scale -> f32, cast to fp8_e4m3fn
|
||||
effective_block_id = raw_block_id
|
||||
offs_q = effective_block_id * GROUP_SIZE + tl.arange(0, GROUP_SIZE)
|
||||
mask = offs_q < DIM_NOPE
|
||||
|
||||
ptr_q = input_nope_q_ptr + token_id_paged * input_nope_q_stride_0 + offs_q
|
||||
ptr_s = (
|
||||
input_nope_s_ptr
|
||||
+ token_id_paged * input_nope_s_stride_0
|
||||
+ effective_block_id
|
||||
)
|
||||
|
||||
y_q = tl.load(ptr_q, mask=mask, other=0.0).to(tl.float32)
|
||||
y_s = tl.load(ptr_s)
|
||||
|
||||
# dequant -> f32 -> requant to fp8
|
||||
y = (y_q * y_s).to(tl.float8e4nv)
|
||||
|
||||
dst_ptr = output_ptr + token_id * output_stride_0 + offs_q
|
||||
tl.store(dst_ptr, y, mask=mask)
|
||||
else:
|
||||
# rope: read bf16, cast to fp8
|
||||
effective_block_id = raw_block_id - NUM_NOPE_BLOCKS
|
||||
offs = effective_block_id * GROUP_SIZE + tl.arange(0, GROUP_SIZE)
|
||||
mask = offs < DIM_ROPE
|
||||
|
||||
src_ptr = input_rope_ptr + token_id_paged * input_rope_stride_0 + offs
|
||||
dst_ptr = output_ptr + token_id * output_stride_0 + DIM_NOPE + offs
|
||||
|
||||
data = tl.load(src_ptr, mask=mask).to(tl.float8e4nv)
|
||||
tl.store(dst_ptr, data, mask=mask)
|
||||
|
||||
|
||||
gather_legacy = gather_dequant_requant_fp8_paged_legacy
|
||||
|
||||
PAGE_SIZE = 64
|
||||
DIM_QUANT = 656 # 512 nope fp8 + 16 scale bytes + 128 rope bytes
|
||||
|
||||
@@ -54,7 +54,7 @@ import argparse
|
||||
|
||||
import torch
|
||||
|
||||
from sglang.kernels.ops.kvcache.cache_ops import (
|
||||
from sglang.kernels.ops.attention.mla_qk_prep import (
|
||||
absorbed_bmm_concat_cast_q_fp8,
|
||||
concat_and_cast_q_fp8_pad,
|
||||
)
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
"""Tune the gfx950 native MXFP8 dense route for one (N, K) and write its rows into
|
||||
python/sglang/kernels/ops/quantization/mxfp8_gemv_gfx95_configs.json.
|
||||
python/sglang/kernels/ops/gemm/mxfp8_gemv_gfx95_configs.json.
|
||||
|
||||
Two tables: the scaled-MFMA GEMV config per M bucket (1 ... 32) and the dot_scaled tile per
|
||||
M bucket (64 ... 16384). Each candidate is timed by HIP-graph replay rotating over weight
|
||||
@@ -18,7 +18,7 @@ import torch
|
||||
from triton.runtime.errors import OutOfResources
|
||||
|
||||
from sglang.kernels.jit.utils import empty_sentinel
|
||||
from sglang.kernels.ops.quantization import mxfp8_native_amd_gfx95 as native
|
||||
from sglang.kernels.ops.gemm import mxfp8_native_amd_gfx95 as native
|
||||
from sglang.kernels.ops.quantization.mxfp8_amd_gfx95 import fp8_grid_quantize
|
||||
|
||||
GRAPH_CALLS = 20
|
||||
|
||||
@@ -18,10 +18,16 @@ sglang/kernels/
|
||||
registry.py # process-wide KernelRegistry + register_kernel()
|
||||
selector.py # heuristic select_kernel() and cached get_kernel()
|
||||
fused_op.py # BaseFusedOp: per-operator multi-backend contract
|
||||
kernel_api_logging.py # kernel API crash-debugging decorator
|
||||
numerics.py # shared numerics helpers for bit-exact Triton kernels
|
||||
ops/
|
||||
<group>/ # one subpackage per operator group (see list below)
|
||||
jit/ # shared JIT CUDA build/runtime infra: utils/, csrc/,
|
||||
# include/, __main__ (KERNEL_PATH resolves here)
|
||||
jit/ # shared JIT build/runtime infra: utils/ (incl. the Helion
|
||||
# runtime helper), csrc/, include/, __main__ (KERNEL_PATH)
|
||||
aot/ # source tree of the AOT `sgl_kernel` wheel (own API and
|
||||
# packaging; not shipped in the sglang wheel)
|
||||
kda_kernels/ # agent-generated kernel backend (Kernel Design Agents)
|
||||
cake_kernels/ # agent-generated kernel backend (Cake, shipped by FlashInfer)
|
||||
```
|
||||
|
||||
Operator groups (all populated): `activation`, `attention`, `communication`,
|
||||
@@ -31,7 +37,7 @@ Operator groups (all populated): `activation`, `attention`, `communication`,
|
||||
|
||||
Place model-specific implementations and tuning data inside the corresponding
|
||||
logical operator group, rather than creating a model-specific top-level group
|
||||
under `ops/`.
|
||||
under `ops/` (a model-named file or subpackage inside a group is fine).
|
||||
|
||||
`lora` is its own group because it has a distinct responsibility: the adapter
|
||||
kernels that only the LoRA runtime in `srt/lora` calls (`dense/` SGMV GEMMs and
|
||||
@@ -51,18 +57,37 @@ that are not CI-registered belong in `test/manual/kernels/`, outside the shipped
|
||||
operator package. Keep vendored upstream trees under their existing maintenance
|
||||
policy.
|
||||
|
||||
Classify the public computation, not the model that first used it: expert
|
||||
routing belongs in `moe`, matrix multiplication (including quantized GEMM and
|
||||
its tuning data) in `gemm`, and attention index selection in `attention`.
|
||||
Keep a fused operation intact; an attention kernel that fuses RoPE with a
|
||||
projection may stay in `attention`. A model name inside a logical group is fine.
|
||||
Classify an operator by its complete public computation, not by the model
|
||||
that first used it, and keep a fused operation intact:
|
||||
|
||||
Runtime integration that owns process groups, symmetric buffers, or model
|
||||
dispatch stays in `srt`; the K3 communication adapters live in
|
||||
- **Collectives win.** Anything that performs a cross-rank collective or
|
||||
reads/writes peer memory (all-reduce, all-gather, reduce-scatter, all-to-all,
|
||||
multicast/NVLS) belongs in `communication`, whatever compute it fuses (GEMM,
|
||||
norm, MoE finalize).
|
||||
- **Producer + quantization stays with the producer.** Norm + quant is
|
||||
`layernorm`, activation + quant is `activation`, a GEMM with a quantizing
|
||||
epilogue is `gemm`, MoE dispatch/permute + quant is `moe`. `quantization`
|
||||
holds only pure format conversion: quant, dequant, scale-layout conversion,
|
||||
repack. A quantized GEMM is still a GEMM.
|
||||
- **RoPE stays in `attention`**: standalone RoPE, position ids, and QK-norm +
|
||||
RoPE fusions.
|
||||
- **`kvcache` owns KV-cache contents**: ops that read or write the cache
|
||||
(store, gather, page copy/move, host transfer, quantize-into-cache, layout
|
||||
conversion, clearing cache/state rows). **`attention` owns the metadata**:
|
||||
page tables, `cu_seqlens`, `kv_indices`, split/schedule tables, and attention
|
||||
index selection. A fused compute + store (e.g. norm + RoPE + quant + store)
|
||||
stays with its compute group.
|
||||
- Otherwise: expert routing is `moe`, matrix multiplication (and its tuning
|
||||
data) is `gemm`, normalization is `layernorm`.
|
||||
|
||||
Runtime integration that owns process groups, symmetric buffers, workspace
|
||||
classes, or model dispatch stays in `srt`: the K3, Muse Spark and SP
|
||||
collective-GEMM communication adapters and workspaces live in
|
||||
`srt/layers/communication/`, while their callable kernels live in
|
||||
`kernels/ops/communication/`. Tests of this runtime state belong with the
|
||||
runtime subsystem, even when they need a GPU. Kernel inventory/dispatch tests
|
||||
belong in `test/registered/unit/kernels/`.
|
||||
`kernels/ops/communication/` and take that state as arguments. Reference/test oracles belong in
|
||||
`sglang.test.kernels`, not in an operator module. Tests of runtime state belong
|
||||
with the runtime subsystem, even when they need a GPU. Kernel inventory/dispatch
|
||||
tests belong in `test/registered/unit/kernels/`.
|
||||
|
||||
When moving an operator, update its callers, lazy `KernelSpec` targets and op
|
||||
ids, tests, benchmark imports, and tuning-data paths together. Preserve CI
|
||||
@@ -70,6 +95,40 @@ stage/runner registrations and test cases when splitting files. See the
|
||||
[kernel organization skill](../../../.claude/skills/kernel-organization/SKILL.md)
|
||||
for the review checklist.
|
||||
|
||||
## Agent-generated kernel backends: `kda_kernels/` and `cake_kernels/`
|
||||
|
||||
These two directories hold implementations produced by kernel-generating agents
|
||||
and will keep growing; they are implementation homes, not operator groups.
|
||||
|
||||
- `kda_kernels/` — kernels produced or extended by the Kernel Design Agents
|
||||
(Humanize2) workflow, in CUDA, Triton, or CuTe DSL, with their JIT sources in
|
||||
`kda_kernels/csrc/`. `KernelBackend.KDA` records that provenance.
|
||||
- `cake_kernels/` — thin adapters to Cake-generated kernels that FlashInfer
|
||||
distributes (generated sources and caches stay in FlashInfer); they register
|
||||
with the provenance of that library (`KernelBackend.FLASHINFER`).
|
||||
|
||||
Their public ops are registered into the logical groups with
|
||||
`KernelSpec(op="<group>.<name>", backend=..., target="sglang.kernels.kda_kernels.<mod>:<fn>")`
|
||||
(or a `cake_kernels` target), and runtime code imports the `ops.<group>`
|
||||
facade, never these packages. Put a new agent-generated kernel in the matching
|
||||
directory with its provenance (see `kda_kernels/README.md`). List them from the
|
||||
registry by target (an agent kernel that is the op's only implementation may
|
||||
keep its language backend, e.g. `TRITON`):
|
||||
|
||||
```python
|
||||
import sglang.kernels.ops # populate the registry
|
||||
from sglang.kernels import KernelBackend, registry
|
||||
|
||||
agent_kernels = [
|
||||
spec
|
||||
for spec in registry.all_specs()
|
||||
if spec.backend is KernelBackend.KDA
|
||||
or spec.target.startswith(
|
||||
("sglang.kernels.kda_kernels.", "sglang.kernels.cake_kernels.")
|
||||
)
|
||||
]
|
||||
```
|
||||
|
||||
## How it works
|
||||
|
||||
Each `ops.<group>` function is a thin wrapper that forwards to a chosen
|
||||
@@ -146,6 +205,10 @@ module-level instances, so the import surface is unchanged; each instance also
|
||||
registers all of its backends as `KernelSpec`s so the registry inventory and
|
||||
`select_kernel(..., backend=...)` keep working.
|
||||
|
||||
An op that is built per layer around the model's own modules (e.g. the Muse
|
||||
Spark norms) has no module-level instance; its group registers each backend
|
||||
method as `module:Class.forward_<backend>` for the inventory.
|
||||
|
||||
What this buys (see the
|
||||
[RFC discussion](https://github.com/sgl-project/sglang/issues/29630#issuecomment-4920387930)):
|
||||
|
||||
|
||||
@@ -23,9 +23,6 @@ constexpr int kQuantBlock = 32;
|
||||
constexpr int kBlockBytes = 17;
|
||||
constexpr int kWarpsPerBlock = 4;
|
||||
constexpr int kRowsPerWarp = 4;
|
||||
constexpr int kMarlinTileK = 16;
|
||||
constexpr int kMarlinTileN = 64;
|
||||
constexpr int kMarlinTileWords = 128;
|
||||
|
||||
__device__ __forceinline__ float fp4_value(uint8_t value) {
|
||||
constexpr float table[16] = {
|
||||
@@ -230,127 +227,6 @@ __global__ void mxfp4_matvec_dual_kernel(
|
||||
}
|
||||
}
|
||||
|
||||
__device__ __forceinline__ uint32_t load_raw_word(
|
||||
const uint8_t* raw, int64_t cache_stride, int slot, int role_offset, int row, int blocks_per_row, int packed_word) {
|
||||
const int block = packed_word / 4;
|
||||
const int word_in_block = packed_word & 3;
|
||||
const int64_t row_bytes = static_cast<int64_t>(blocks_per_row) * kBlockBytes;
|
||||
const uint8_t* ptr = raw + static_cast<int64_t>(slot) * cache_stride + role_offset +
|
||||
static_cast<int64_t>(row) * row_bytes + static_cast<int64_t>(block) * kBlockBytes + 1 +
|
||||
word_in_block * 4;
|
||||
return static_cast<uint32_t>(ptr[0]) | (static_cast<uint32_t>(ptr[1]) << 8) | (static_cast<uint32_t>(ptr[2]) << 16) |
|
||||
(static_cast<uint32_t>(ptr[3]) << 24);
|
||||
}
|
||||
|
||||
__device__ __forceinline__ uint8_t load_raw_scale(
|
||||
const uint8_t* raw, int64_t cache_stride, int slot, int role_offset, int row, int blocks_per_row, int block) {
|
||||
const int64_t row_bytes = static_cast<int64_t>(blocks_per_row) * kBlockBytes;
|
||||
const uint8_t* ptr = raw + static_cast<int64_t>(slot) * cache_stride + role_offset +
|
||||
static_cast<int64_t>(row) * row_bytes + static_cast<int64_t>(block) * kBlockBytes;
|
||||
return *ptr;
|
||||
}
|
||||
|
||||
__device__ __forceinline__ uint8_t marlin_scale_perm(int index) {
|
||||
constexpr int local_perm[4] = {0, 2, 1, 3};
|
||||
const int interleaved = (index / 4) * 4 + local_perm[index & 3];
|
||||
return static_cast<uint8_t>(((interleaved & 7) * 8) + (interleaved >> 3));
|
||||
}
|
||||
|
||||
__device__ __forceinline__ uint8_t marlin_nibble(uint32_t word, int value_index) {
|
||||
return static_cast<uint8_t>((word >> ((value_index & 7) * 4)) & 0x0f);
|
||||
}
|
||||
|
||||
__global__ void mxfp4_marlin_repack_weight_kernel(
|
||||
const uint8_t* __restrict__ raw,
|
||||
int64_t raw_stride,
|
||||
const int32_t* __restrict__ source_slots,
|
||||
const int32_t* __restrict__ target_slots,
|
||||
int64_t role_bytes,
|
||||
int input_size,
|
||||
int output_size,
|
||||
bool gate_up,
|
||||
int32_t* __restrict__ output,
|
||||
int64_t output_stride) {
|
||||
const int batch = blockIdx.y;
|
||||
const int64_t total_words = static_cast<int64_t>(input_size / kMarlinTileK) * (output_size * 2);
|
||||
const int64_t index = static_cast<int64_t>(blockIdx.x) * blockDim.x + threadIdx.x;
|
||||
if (batch >= gridDim.y || index >= total_words) return;
|
||||
|
||||
const int64_t tile_span = static_cast<int64_t>(output_size / kMarlinTileN) * kMarlinTileWords;
|
||||
const int tile_k = static_cast<int>(index / tile_span);
|
||||
const int64_t tile_rem = index % tile_span;
|
||||
const int tile_n = static_cast<int>(tile_rem / kMarlinTileWords);
|
||||
const int local = static_cast<int>(tile_rem % kMarlinTileWords);
|
||||
const int warp = local & 3;
|
||||
const int thread = local >> 2;
|
||||
const int cur_n = warp * 16 + thread / 4;
|
||||
const int tc_row = (thread & 3) * 2;
|
||||
constexpr int offsets[4] = {0, 1, 8, 9};
|
||||
constexpr int pack_index[8] = {0, 2, 4, 6, 1, 3, 5, 7};
|
||||
|
||||
const int source_slot = source_slots[batch];
|
||||
const int target_slot = target_slots[batch];
|
||||
const int rows_per_role = gate_up ? output_size / 2 : output_size;
|
||||
const int blocks_per_row = input_size / kQuantBlock;
|
||||
const int role0_offset = 0;
|
||||
const int role1_offset = static_cast<int>(role_bytes);
|
||||
const int role2_offset = static_cast<int>(2 * role_bytes);
|
||||
uint8_t values[8];
|
||||
#pragma unroll
|
||||
for (int i = 0; i < 4; ++i) {
|
||||
const int value_index = tc_row + offsets[i];
|
||||
const int source_row = tile_n * kMarlinTileN + cur_n;
|
||||
const int role = gate_up && source_row >= rows_per_role ? 1 : (gate_up ? 0 : 2);
|
||||
const int row = gate_up ? source_row % rows_per_role : source_row;
|
||||
const int role_offset = role == 0 ? role0_offset : (role == 1 ? role1_offset : role2_offset);
|
||||
const uint32_t word =
|
||||
load_raw_word(raw, raw_stride, source_slot, role_offset, row, blocks_per_row, tile_k * 2 + value_index / 8);
|
||||
values[i] = marlin_nibble(word, value_index);
|
||||
const int high_source_row = tile_n * kMarlinTileN + cur_n + 8;
|
||||
const int high_role = gate_up && high_source_row >= rows_per_role ? 1 : (gate_up ? 0 : 2);
|
||||
const int high_role_offset = high_role == 0 ? role0_offset : (high_role == 1 ? role1_offset : role2_offset);
|
||||
const int high_row = gate_up ? high_source_row % rows_per_role : high_source_row;
|
||||
const uint32_t high_word = load_raw_word(
|
||||
raw, raw_stride, source_slot, high_role_offset, high_row, blocks_per_row, tile_k * 2 + value_index / 8);
|
||||
values[4 + i] = marlin_nibble(high_word, value_index);
|
||||
}
|
||||
|
||||
uint32_t packed = 0;
|
||||
#pragma unroll
|
||||
for (int i = 0; i < 8; ++i) {
|
||||
packed |= static_cast<uint32_t>(values[pack_index[i]]) << (i * 4);
|
||||
}
|
||||
output[static_cast<int64_t>(target_slot) * output_stride + index] = static_cast<int32_t>(packed);
|
||||
}
|
||||
|
||||
__global__ void mxfp4_marlin_repack_scale_kernel(
|
||||
const uint8_t* __restrict__ raw,
|
||||
int64_t raw_stride,
|
||||
const int32_t* __restrict__ source_slots,
|
||||
const int32_t* __restrict__ target_slots,
|
||||
int64_t role_bytes,
|
||||
int input_size,
|
||||
int output_size,
|
||||
bool gate_up,
|
||||
uint8_t* __restrict__ output,
|
||||
int64_t output_stride) {
|
||||
const int batch = blockIdx.y;
|
||||
const int groups = input_size / kQuantBlock;
|
||||
const int64_t total = static_cast<int64_t>(groups) * output_size;
|
||||
const int64_t index = static_cast<int64_t>(blockIdx.x) * blockDim.x + threadIdx.x;
|
||||
if (batch >= gridDim.y || index >= total) return;
|
||||
|
||||
const int group = static_cast<int>(index / output_size);
|
||||
const int column = static_cast<int>(index % output_size);
|
||||
const int source_column = (column / 64) * 64 + marlin_scale_perm(column & 63);
|
||||
const int rows_per_role = gate_up ? output_size / 2 : output_size;
|
||||
const int role = gate_up && source_column >= rows_per_role ? 1 : (gate_up ? 0 : 2);
|
||||
const int row = gate_up ? source_column % rows_per_role : source_column;
|
||||
const int role_offset = role == 0 ? 0 : (role == 1 ? static_cast<int>(role_bytes) : static_cast<int>(2 * role_bytes));
|
||||
const uint8_t value = load_raw_scale(raw, raw_stride, source_slots[batch], role_offset, row, groups, group);
|
||||
output[static_cast<int64_t>(target_slots[batch]) * output_stride + index] = value;
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
@@ -622,156 +498,4 @@ inline void mxfp4_matvec_dual(
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* \brief Repack raw GGUF MXFP4 objects into contiguous Marlin SoA cache tensors.
|
||||
*
|
||||
* One entry of `source_slots` / `target_slots` per object to move: the raw
|
||||
* matrices at `source_slots[i]` are read and the Marlin-layout weights and
|
||||
* scales for `target_slots[i]` are written. Slots absent from `target_slots`
|
||||
* are left untouched.
|
||||
*
|
||||
* \param raw Raw MXFP4 slot bank, `[source_slots, 3 * role_bytes]` uint8.
|
||||
* \param source_slots Row of `raw` to read per object, int32.
|
||||
* \param target_slots Row of the Marlin tensors to write per object, int32.
|
||||
* \param role_bytes Byte size of one role (gate, up, or down) per slot.
|
||||
* \param hidden_size Model hidden size; must be divisible by 32.
|
||||
* \param intermediate_size Expert intermediate size; must be divisible by 32.
|
||||
* \param w13 Marlin gate/up weights, int32.
|
||||
* \param w2 Marlin down weights, int32.
|
||||
* \param w13_scale Marlin gate/up scales, uint8.
|
||||
* \param w2_scale Marlin down scales, uint8.
|
||||
*/
|
||||
inline void mxfp4_marlin_repack(
|
||||
tvm::ffi::TensorView raw,
|
||||
tvm::ffi::TensorView source_slots,
|
||||
tvm::ffi::TensorView target_slots,
|
||||
int64_t role_bytes,
|
||||
int64_t hidden_size,
|
||||
int64_t intermediate_size,
|
||||
tvm::ffi::TensorView w13,
|
||||
tvm::ffi::TensorView w2,
|
||||
tvm::ffi::TensorView w13_scale,
|
||||
tvm::ffi::TensorView w2_scale) {
|
||||
using namespace host;
|
||||
|
||||
// The Marlin layout tiles the n dimension by `kMarlinTileN`, so a size that is
|
||||
// only a multiple of `kQuantBlock` leaves a partial tile: the scale
|
||||
// permutation then maps columns outside the role slice and the weight tile
|
||||
// span collapses to zero. `w2` takes its n from `hidden_size`, which is
|
||||
// therefore the stricter of the two; `w13` takes `2 * intermediate_size`, so
|
||||
// the quant-block check already makes it a whole number of tiles.
|
||||
// Not an OOB read -- memcheck is clean; the partial tile reads a neighbouring
|
||||
// slot from inside the tensor, and the collapsed span divides by zero on the
|
||||
// device, which is undefined rather than trapping. Both are silent.
|
||||
CHECK_HOST(hidden_size > 0 && hidden_size % kMarlinTileN == 0)
|
||||
<< "hidden_size must be divisible by " << kMarlinTileN << ", got " << hidden_size;
|
||||
CHECK_HOST(intermediate_size > 0 && intermediate_size % kQuantBlock == 0)
|
||||
<< "MXFP4 dimensions must be divisible by 32, got intermediate_size=" << intermediate_size;
|
||||
|
||||
// Every role holds one matrix of `intermediate_size x hidden_size` MXFP4
|
||||
// blocks -- transposed for the down projection, which is the same byte count
|
||||
// -- so the dimensions pin `role_bytes` exactly, the way they do for matvec.
|
||||
const int64_t expected_role_bytes = intermediate_size * (hidden_size / kQuantBlock) * kBlockBytes;
|
||||
CHECK_HOST(role_bytes == expected_role_bytes)
|
||||
<< "role byte count does not match matrix dimensions: " << role_bytes << " != " << expected_role_bytes;
|
||||
|
||||
const int64_t w13_n = 2 * intermediate_size;
|
||||
const int64_t w2_n = hidden_size;
|
||||
const int64_t w13_k = hidden_size;
|
||||
const int64_t w2_k = intermediate_size;
|
||||
const int64_t w13_words = (w13_k / kMarlinTileK) * w13_n * 2;
|
||||
const int64_t w2_words = (w2_k / kMarlinTileK) * w2_n * 2;
|
||||
const int64_t w13_scales = (w13_k / kQuantBlock) * w13_n;
|
||||
const int64_t w2_scales = (w2_k / kQuantBlock) * w2_n;
|
||||
|
||||
auto batch = SymbolicSize{"objects"};
|
||||
auto source_capacity = SymbolicSize{"raw_slots"};
|
||||
auto target_capacity = SymbolicSize{"marlin_slots"};
|
||||
auto device = SymbolicDevice{};
|
||||
|
||||
TensorMatcher({source_capacity, 3 * role_bytes}) //
|
||||
.with_dtype<uint8_t>()
|
||||
.with_device<kDLCUDA>(device)
|
||||
.verify(raw);
|
||||
TensorMatcher({batch}) //
|
||||
.with_dtype<int32_t>()
|
||||
.with_device<kDLCUDA>(device)
|
||||
.verify(source_slots)
|
||||
.verify(target_slots);
|
||||
TensorMatcher({target_capacity, w13_words}) //
|
||||
.with_dtype<int32_t>()
|
||||
.with_device<kDLCUDA>(device)
|
||||
.verify(w13);
|
||||
TensorMatcher({target_capacity, w2_words}) //
|
||||
.with_dtype<int32_t>()
|
||||
.with_device<kDLCUDA>(device)
|
||||
.verify(w2);
|
||||
TensorMatcher({target_capacity, w13_scales}) //
|
||||
.with_dtype<uint8_t>()
|
||||
.with_device<kDLCUDA>(device)
|
||||
.verify(w13_scale);
|
||||
TensorMatcher({target_capacity, w2_scales}) //
|
||||
.with_dtype<uint8_t>()
|
||||
.with_device<kDLCUDA>(device)
|
||||
.verify(w2_scale);
|
||||
|
||||
const uint32_t objects = static_cast<uint32_t>(batch.unwrap());
|
||||
if (objects == 0) return;
|
||||
|
||||
constexpr uint32_t kThreads = 256;
|
||||
const DLDevice dev = device.unwrap();
|
||||
const auto* raw_ptr = static_cast<const uint8_t*>(raw.data_ptr());
|
||||
const auto* source_ptr = static_cast<const int32_t*>(source_slots.data_ptr());
|
||||
const auto* target_ptr = static_cast<const int32_t*>(target_slots.data_ptr());
|
||||
|
||||
LaunchKernel(dim3(div_ceil(static_cast<uint32_t>(w13_words), kThreads), objects), kThreads, dev)(
|
||||
mxfp4_marlin_repack_weight_kernel,
|
||||
raw_ptr,
|
||||
raw.stride(0),
|
||||
source_ptr,
|
||||
target_ptr,
|
||||
role_bytes,
|
||||
static_cast<int>(w13_k),
|
||||
static_cast<int>(w13_n),
|
||||
true,
|
||||
static_cast<int32_t*>(w13.data_ptr()),
|
||||
w13.stride(0));
|
||||
LaunchKernel(dim3(div_ceil(static_cast<uint32_t>(w2_words), kThreads), objects), kThreads, dev)(
|
||||
mxfp4_marlin_repack_weight_kernel,
|
||||
raw_ptr,
|
||||
raw.stride(0),
|
||||
source_ptr,
|
||||
target_ptr,
|
||||
role_bytes,
|
||||
static_cast<int>(w2_k),
|
||||
static_cast<int>(w2_n),
|
||||
false,
|
||||
static_cast<int32_t*>(w2.data_ptr()),
|
||||
w2.stride(0));
|
||||
LaunchKernel(dim3(div_ceil(static_cast<uint32_t>(w13_scales), kThreads), objects), kThreads, dev)(
|
||||
mxfp4_marlin_repack_scale_kernel,
|
||||
raw_ptr,
|
||||
raw.stride(0),
|
||||
source_ptr,
|
||||
target_ptr,
|
||||
role_bytes,
|
||||
static_cast<int>(w13_k),
|
||||
static_cast<int>(w13_n),
|
||||
true,
|
||||
static_cast<uint8_t*>(w13_scale.data_ptr()),
|
||||
w13_scale.stride(0));
|
||||
LaunchKernel(dim3(div_ceil(static_cast<uint32_t>(w2_scales), kThreads), objects), kThreads, dev)(
|
||||
mxfp4_marlin_repack_scale_kernel,
|
||||
raw_ptr,
|
||||
raw.stride(0),
|
||||
source_ptr,
|
||||
target_ptr,
|
||||
role_bytes,
|
||||
static_cast<int>(w2_k),
|
||||
static_cast<int>(w2_n),
|
||||
false,
|
||||
static_cast<uint8_t*>(w2_scale.data_ptr()),
|
||||
w2_scale.stride(0));
|
||||
}
|
||||
|
||||
} // namespace sglang
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
// Fused MuseSpark MoE routing: normalize -> bias -> iterative top-k -> renormalize.
|
||||
//
|
||||
// Drop-in replacement for the Triton `fused_muse_spark_routing_kernel_topk8`
|
||||
// (python/sglang/kernels/ops/muse_spark/fused_muse_spark_routing.py). One warp owns one token
|
||||
// (python/sglang/kernels/ops/moe/muse_spark_routing.py). One warp owns one token
|
||||
// row; the whole row lives in registers (kExpertsPerLane per lane), each lane
|
||||
// sorts its slice once, and each of the kTopK rounds pops one head.
|
||||
//
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
// Row sum over [N, K] fp32, bitwise-identical to `sglang.kernels.ops.muse_spark.triton_sum`.
|
||||
// Row sum over [N, K] fp32, bitwise-identical to `sglang.test.kernels.triton_sum`.
|
||||
//
|
||||
// The point of that Triton kernel is a reduction order fixed by the compiler
|
||||
// rather than by the CUDA runtime version, so this kernel reproduces tl.sum's
|
||||
|
||||
@@ -20,7 +20,7 @@ limitations under the License.
|
||||
// out[:, h, 512:576] = fp8_e4m3(q_rope[:, h, :])
|
||||
//
|
||||
// This is the CUDA replacement for the Triton absorbed_bmm_concat_cast_q_fp8
|
||||
// kernel (triton_ops/cache_ops.py). The epilogue keeps the exact rounding
|
||||
// kernel (attention/mla_qk_prep.py). The epilogue keeps the exact rounding
|
||||
// chain of the Triton variants: fp32 WGMMA accumulate -> bf16 round-to-nearest
|
||||
// (cublas-equivalent output rounding) -> fp8_e4m3 rn/satfinite on store. The
|
||||
// K dimension is consumed as one in-order chain of k=16 WGMMA steps into a
|
||||
|
||||
@@ -17,7 +17,7 @@ namespace deepseek_v4::fp4 {
|
||||
|
||||
/// Largest finite e2m1 value.
|
||||
constexpr float kMax = 6.0f;
|
||||
/// `6 * 2^-126`, the amax floor `torch_quant.fake_quant_fp4` clamps to.
|
||||
/// `6 * 2^-126`, the amax floor `fp4_fake_quant.fake_quant_fp4` clamps to.
|
||||
constexpr float kAmaxFloor = 6.0f * 1.1754943508222875e-38f;
|
||||
/// Elements sharing one ue8m0 scale.
|
||||
constexpr uint32_t kBlockSize = 32;
|
||||
@@ -95,7 +95,7 @@ SGL_DEVICE fp32x2_t block_scale(float amax) {
|
||||
/// \brief Round a pair onto the e2m1 grid and back, through `scale`.
|
||||
///
|
||||
/// Every e2m1 value is exact in fp16, so the roundtrip is lossless. Adding `0.0f` during
|
||||
/// scaling clears negative zero to match `torch.sign(0) == 0` in `torch_quant.round_fp4`.
|
||||
/// scaling clears negative zero to match `torch.sign(0) == 0` in `fp4_fake_quant.round_fp4`.
|
||||
SGL_DEVICE fp32x2_t fake_quant_x2(fp32x2_t x, float scale, float inv_scale) {
|
||||
const fp32x2_t scaled{__fmaf_rn(x.x, inv_scale, 0.0f), __fmaf_rn(x.y, inv_scale, 0.0f)};
|
||||
#ifdef USE_ROCM
|
||||
|
||||
+8
-18
@@ -30,23 +30,20 @@ _compiled: dict[tuple[Any, str], Any] = {}
|
||||
|
||||
|
||||
@lru_cache(maxsize=None)
|
||||
def _warn_unsupported(reason: str) -> bool:
|
||||
def warn_helion_unsupported(reason: str) -> bool:
|
||||
logger.warning("Muse Spark Helion kernels are disabled: %s", reason)
|
||||
return False
|
||||
|
||||
|
||||
@lru_cache(maxsize=None)
|
||||
def _base_package_version(package: str) -> str | None:
|
||||
def base_package_version(package: str) -> str | None:
|
||||
try:
|
||||
return version(package).split("+", 1)[0]
|
||||
except PackageNotFoundError:
|
||||
return None
|
||||
|
||||
|
||||
def helion_supported(
|
||||
op_name: str,
|
||||
tensor: torch.Tensor,
|
||||
) -> bool:
|
||||
def helion_supported(tensor: torch.Tensor) -> bool:
|
||||
"""Return whether a selected kernel is validated for this runtime/device."""
|
||||
if torch.compiler.is_compiling():
|
||||
return False
|
||||
@@ -57,21 +54,12 @@ def helion_supported(
|
||||
forced_backend = get_fused_op_backend()
|
||||
if forced_backend is not None and forced_backend is not KernelBackend.HELION:
|
||||
return False
|
||||
helion_version = _base_package_version("helion")
|
||||
helion_version = base_package_version("helion")
|
||||
if helion_version != "1.4.0":
|
||||
return _warn_unsupported(
|
||||
return warn_helion_unsupported(
|
||||
f"Helion 1.4.0 is required, found {helion_version or 'not installed'}"
|
||||
)
|
||||
if not tensor.is_cuda:
|
||||
return False
|
||||
if op_name == "norm_fp4_quant":
|
||||
flashinfer_version = _base_package_version("flashinfer-python")
|
||||
if flashinfer_version != "0.6.17":
|
||||
return _warn_unsupported(
|
||||
"NVFP4 quantization requires FlashInfer 0.6.17, found "
|
||||
f"{flashinfer_version or 'not installed'}"
|
||||
)
|
||||
return True
|
||||
return tensor.is_cuda
|
||||
|
||||
|
||||
def token_count_specialization_key(first_tensor: torch.Tensor, *_: Any) -> int:
|
||||
@@ -167,9 +155,11 @@ def run_pinned_helion(
|
||||
|
||||
|
||||
__all__ = [
|
||||
"base_package_version",
|
||||
"current_helion_config_bucket",
|
||||
"helion_config_bucket",
|
||||
"helion_supported",
|
||||
"run_pinned_helion",
|
||||
"token_count_specialization_key",
|
||||
"warn_helion_unsupported",
|
||||
]
|
||||
@@ -0,0 +1,91 @@
|
||||
"""hipcc build and HIP module-launch helpers for the hand-written gfx950 kernels."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import ctypes
|
||||
import os
|
||||
|
||||
import torch
|
||||
|
||||
from sglang.srt.distributed.device_communicators.cuda_wrapper import (
|
||||
find_loaded_library,
|
||||
)
|
||||
|
||||
_hip = None
|
||||
|
||||
|
||||
def hip_lib():
|
||||
global _hip
|
||||
if _hip is None:
|
||||
# Bind to the HIP runtime torch already mapped. ROCm 10 ships a second
|
||||
# libamdhip64 in _rocm_sdk_devel; an unversioned CDLL("libamdhip64.so")
|
||||
# picks that one, and every launch on a torch stream then fails with
|
||||
# hipErrorContextIsDestroyed (709).
|
||||
torch.cuda.current_device()
|
||||
_hip = ctypes.CDLL(find_loaded_library("libamdhip64") or "libamdhip64.so")
|
||||
_hip.hipModuleLoad.restype = ctypes.c_int
|
||||
_hip.hipModuleLoad.argtypes = [ctypes.c_void_p, ctypes.c_char_p]
|
||||
_hip.hipModuleGetFunction.restype = ctypes.c_int
|
||||
_hip.hipModuleGetFunction.argtypes = [
|
||||
ctypes.c_void_p,
|
||||
ctypes.c_void_p,
|
||||
ctypes.c_char_p,
|
||||
]
|
||||
_hip.hipModuleLaunchKernel.restype = ctypes.c_int
|
||||
_hip.hipModuleLaunchKernel.argtypes = (
|
||||
[ctypes.c_void_p] + [ctypes.c_uint] * 7 + [ctypes.c_void_p] * 3
|
||||
)
|
||||
_hip.hipGetErrorString.restype = ctypes.c_char_p
|
||||
return _hip
|
||||
|
||||
|
||||
def check_hip(rc, what):
|
||||
if rc != 0:
|
||||
raise RuntimeError(
|
||||
f"smallm_moe: {what} failed: {rc} ({hip_lib().hipGetErrorString(rc).decode()})"
|
||||
)
|
||||
|
||||
|
||||
class HipKernel:
|
||||
def __init__(self, mod, name):
|
||||
self.fn = ctypes.c_void_p()
|
||||
check_hip(
|
||||
hip_lib().hipModuleGetFunction(ctypes.byref(self.fn), mod, name.encode()),
|
||||
f"hipModuleGetFunction {name}",
|
||||
)
|
||||
|
||||
def launch(self, grid, args, stream, block=256, shared=0):
|
||||
size = ctypes.c_size_t(ctypes.sizeof(args))
|
||||
extra = (ctypes.c_void_p * 5)(
|
||||
1,
|
||||
ctypes.cast(ctypes.byref(args), ctypes.c_void_p).value,
|
||||
2,
|
||||
ctypes.cast(ctypes.byref(size), ctypes.c_void_p).value,
|
||||
3,
|
||||
)
|
||||
check_hip(
|
||||
hip_lib().hipModuleLaunchKernel(
|
||||
self.fn,
|
||||
grid,
|
||||
1,
|
||||
1,
|
||||
block,
|
||||
1,
|
||||
1,
|
||||
shared,
|
||||
ctypes.c_void_p(stream),
|
||||
None,
|
||||
ctypes.cast(extra, ctypes.c_void_p),
|
||||
),
|
||||
"hipModuleLaunchKernel",
|
||||
)
|
||||
|
||||
|
||||
def hipcc_path():
|
||||
return os.path.join(os.environ.get("ROCM_PATH", "/opt/rocm"), "bin", "hipcc")
|
||||
|
||||
|
||||
def hip_version():
|
||||
if torch.version.hip:
|
||||
return tuple(int(x) for x in torch.version.hip.split("-")[0].split(".")[:3])
|
||||
return (0, 0, 0)
|
||||
@@ -30,7 +30,6 @@ _GROUPS = (
|
||||
"memory",
|
||||
"mm",
|
||||
"moe",
|
||||
"muse_spark",
|
||||
"quantization",
|
||||
"sampling",
|
||||
"speculative",
|
||||
|
||||
@@ -112,7 +112,7 @@ def situ_and_mul(
|
||||
beta: float,
|
||||
linear_beta: Optional[float],
|
||||
) -> torch.Tensor:
|
||||
from ._jit_situ_and_mul import situ_and_mul as impl
|
||||
from .kimi_k3_situ_and_mul import situ_and_mul as impl
|
||||
|
||||
return impl(input, out, beta, linear_beta)
|
||||
|
||||
@@ -150,7 +150,31 @@ register_kernel(
|
||||
KernelSpec(
|
||||
op="activation.situ_and_mul",
|
||||
backend=KernelBackend.JIT,
|
||||
target="sglang.kernels.ops.activation._jit_situ_and_mul:situ_and_mul",
|
||||
target="sglang.kernels.ops.activation.kimi_k3_situ_and_mul:situ_and_mul",
|
||||
capabilities=CUDA_ONLY,
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
# MXFP8-quantizing SwiGLU-OAI producers; registered for inventory.
|
||||
for _fn in ("swiglu_oai_mxfp8", "swiglu_oai_mxfp8_deepgemm"):
|
||||
register_kernel(
|
||||
KernelSpec(
|
||||
op=f"activation.{_fn}",
|
||||
backend=KernelBackend.TRITON,
|
||||
target=f"sglang.kernels.ops.activation.swiglu_mxfp8_swizzled:{_fn}",
|
||||
)
|
||||
)
|
||||
del _fn
|
||||
for _fn in ("silu_and_mul_clamp", "silu_and_mul_contig_post_quant"):
|
||||
register_kernel(
|
||||
KernelSpec(
|
||||
op=f"activation.{_fn}",
|
||||
backend=KernelBackend.JIT,
|
||||
target=f"sglang.kernels.ops.activation.silu_and_mul_clamp:{_fn}",
|
||||
capabilities=frozenset(
|
||||
{CapabilityRequirement.CUDA, CapabilityRequirement.HIP}
|
||||
),
|
||||
)
|
||||
)
|
||||
del _fn
|
||||
|
||||
@@ -0,0 +1,101 @@
|
||||
import torch
|
||||
import triton
|
||||
import triton.language as tl
|
||||
|
||||
from sglang.srt.utils import is_hip
|
||||
|
||||
_is_hip = is_hip()
|
||||
|
||||
|
||||
# gelu on first half of vector
|
||||
@triton.jit
|
||||
def gelu_and_mul_kernel(
|
||||
out_hidden_states_ptr, # (bs, hidden_dim)
|
||||
out_scales_ptr, # (bs,)
|
||||
hidden_states_ptr, # (bs, hidden_dim * 2)
|
||||
quant_max: tl.constexpr,
|
||||
static_scale: tl.constexpr,
|
||||
hidden_dim: tl.constexpr, # the output hidden_dim
|
||||
BLOCK_SIZE: tl.constexpr,
|
||||
):
|
||||
pid = tl.program_id(axis=0)
|
||||
|
||||
input_start = pid * hidden_dim * 2
|
||||
output_start = pid * hidden_dim
|
||||
|
||||
input1_offs = tl.arange(0, BLOCK_SIZE)
|
||||
mask = tl.arange(0, BLOCK_SIZE) < hidden_dim # shared for input1, input3, output
|
||||
input3_offs = hidden_dim + tl.arange(0, BLOCK_SIZE)
|
||||
output_offs = tl.arange(0, BLOCK_SIZE)
|
||||
|
||||
x1 = tl.load(
|
||||
hidden_states_ptr + input_start + input1_offs, mask=mask, other=0.0
|
||||
).to(tl.float32)
|
||||
x3 = tl.load(
|
||||
hidden_states_ptr + input_start + input3_offs, mask=mask, other=0.0
|
||||
).to(tl.float32)
|
||||
|
||||
# gelu
|
||||
# cast down before mul to better match training?
|
||||
gelu_x1 = 0.5 * (1.0 + tl.erf(x1 * 0.7071067811865475)) * x1
|
||||
out = x3 * gelu_x1.to(hidden_states_ptr.dtype.element_ty)
|
||||
|
||||
if quant_max is not None:
|
||||
raise NotImplementedError()
|
||||
|
||||
tl.store(out_hidden_states_ptr + output_start + output_offs, out, mask=mask)
|
||||
|
||||
|
||||
def gelu_and_mul_triton(
|
||||
hidden_states,
|
||||
scales=None,
|
||||
quantize=None, # dtype to quantize to
|
||||
out=None,
|
||||
):
|
||||
bs, in_hidden_dim = hidden_states.shape
|
||||
hidden_dim = in_hidden_dim // 2
|
||||
|
||||
if out is None:
|
||||
out_hidden_states = torch.empty(
|
||||
(bs, hidden_dim),
|
||||
dtype=quantize or hidden_states.dtype,
|
||||
device=hidden_states.device,
|
||||
)
|
||||
else:
|
||||
assert out.shape == (bs, hidden_dim)
|
||||
assert out.dtype == (quantize or hidden_states.dtype)
|
||||
out_hidden_states = out
|
||||
out_scales = None
|
||||
static_scale = False
|
||||
if quantize is not None:
|
||||
if scales is None:
|
||||
out_scales = torch.empty(
|
||||
(bs,), dtype=torch.float32, device=hidden_states.device
|
||||
)
|
||||
else:
|
||||
out_scales = scales
|
||||
static_scale = True
|
||||
|
||||
max_warps = 16 if _is_hip else 32
|
||||
config = {
|
||||
# 8 ele per thread (not tuned)
|
||||
"num_warps": max(
|
||||
min(triton.next_power_of_2(triton.cdiv(hidden_dim, 8 * 32)), max_warps), 4
|
||||
),
|
||||
}
|
||||
|
||||
gelu_and_mul_kernel[(bs,)](
|
||||
out_hidden_states,
|
||||
out_scales,
|
||||
hidden_states,
|
||||
quant_max=torch.finfo(quantize).max if quantize is not None else None,
|
||||
static_scale=static_scale,
|
||||
hidden_dim=hidden_dim,
|
||||
BLOCK_SIZE=triton.next_power_of_2(hidden_dim),
|
||||
**config,
|
||||
)
|
||||
|
||||
if quantize is not None:
|
||||
return out_hidden_states, out_scales
|
||||
else:
|
||||
return out_hidden_states, None
|
||||
+1
@@ -156,6 +156,7 @@ def _swiglu_oai_mxfp8_quant_kernel(
|
||||
)
|
||||
|
||||
|
||||
# Not swiglu_oai_mxfp8: no bf16 rounding before the quant, and plain [M, I // 32] scales.
|
||||
def swiglu_oai_mxfp8_quant(
|
||||
gate_up: torch.Tensor,
|
||||
alpha: float,
|
||||
@@ -0,0 +1,91 @@
|
||||
"""SwiGLU with clamped branches, optionally fused with per-token-group quantization."""
|
||||
|
||||
from typing import Optional
|
||||
|
||||
import torch
|
||||
|
||||
from sglang.kernels.jit.utils import (
|
||||
cache_once,
|
||||
is_arch_support_pdl,
|
||||
load_jit,
|
||||
make_cpp_args,
|
||||
)
|
||||
from sglang.srt.utils import is_xpu
|
||||
|
||||
_is_xpu = is_xpu()
|
||||
|
||||
|
||||
def make_name(name: str) -> str:
|
||||
return f"dpsk_v4_{name}"
|
||||
|
||||
|
||||
@cache_once
|
||||
def _jit_silu_mul_quant_contig_module(
|
||||
quant_group_size: int,
|
||||
scale_ue8m0: bool,
|
||||
swizzle: bool,
|
||||
apply_swiglu_limit: bool,
|
||||
):
|
||||
args = make_cpp_args(
|
||||
quant_group_size,
|
||||
scale_ue8m0,
|
||||
swizzle,
|
||||
is_arch_support_pdl(),
|
||||
apply_swiglu_limit,
|
||||
)
|
||||
return load_jit(
|
||||
make_name("silu_mul_quant_contig"),
|
||||
*args,
|
||||
cuda_files=["deepseek_v4/silu_and_mul_masked_post_quant.cuh"],
|
||||
cuda_wrappers=[("run", f"SiluAndMulContigPostQuantKernel<{args}>::run")],
|
||||
extra_cuda_cflags=["-use_fast_math"],
|
||||
)
|
||||
|
||||
|
||||
@cache_once
|
||||
def _jit_silu_and_mul_clamp_module(dtype: torch.dtype):
|
||||
args = make_cpp_args(dtype, is_arch_support_pdl())
|
||||
return load_jit(
|
||||
make_name("silu_and_mul_clamp"),
|
||||
*args,
|
||||
cuda_files=["deepseek_v4/silu_and_mul_masked_post_quant.cuh"],
|
||||
cuda_wrappers=[("run", f"SiluAndMulClampKernel<{args}>::run")],
|
||||
extra_cuda_cflags=["-use_fast_math"],
|
||||
)
|
||||
|
||||
|
||||
def silu_and_mul_clamp(
|
||||
input: torch.Tensor,
|
||||
output: torch.Tensor,
|
||||
swiglu_limit: float,
|
||||
) -> None:
|
||||
if _is_xpu:
|
||||
from sgl_kernel import silu_and_mul_clamp
|
||||
|
||||
silu_and_mul_clamp(input, output, float(swiglu_limit))
|
||||
else:
|
||||
module = _jit_silu_and_mul_clamp_module(input.dtype)
|
||||
module.run(input, output, float(swiglu_limit))
|
||||
|
||||
|
||||
def silu_and_mul_contig_post_quant(
|
||||
input: torch.Tensor,
|
||||
output: torch.Tensor,
|
||||
output_scale: torch.Tensor,
|
||||
quant_group_size: int,
|
||||
scale_ue8m0: bool = False,
|
||||
transposed: bool = False,
|
||||
swiglu_limit: Optional[float] = None,
|
||||
swizzle: bool = False,
|
||||
) -> None:
|
||||
apply_swiglu_limit = swiglu_limit is not None
|
||||
module = _jit_silu_mul_quant_contig_module(
|
||||
quant_group_size, scale_ue8m0, swizzle, apply_swiglu_limit
|
||||
)
|
||||
module.run(
|
||||
input,
|
||||
output,
|
||||
output_scale,
|
||||
transposed,
|
||||
float(swiglu_limit) if apply_swiglu_limit else 0.0,
|
||||
)
|
||||
@@ -131,53 +131,3 @@ def gathered_softcap_logits(gathered, final_logit_softcapping, output):
|
||||
BLOCK_SIZE=BLOCK_SIZE,
|
||||
)
|
||||
return output
|
||||
|
||||
|
||||
@triton.jit
|
||||
def _softcap_row_argmax_kernel(
|
||||
logits_ptr,
|
||||
out_ptr,
|
||||
row_stride,
|
||||
valid_width,
|
||||
col_offset,
|
||||
softcapping_value,
|
||||
BLOCK_SIZE: tl.constexpr,
|
||||
):
|
||||
row = tl.program_id(0).to(tl.int64)
|
||||
best_val = -float("inf")
|
||||
best_idx = 0
|
||||
for start in range(0, valid_width, BLOCK_SIZE):
|
||||
cols = start + tl.arange(0, BLOCK_SIZE)
|
||||
mask = cols < valid_width
|
||||
x = tl.load(logits_ptr + row * row_stride + cols, mask=mask, other=0.0)
|
||||
x = x / softcapping_value
|
||||
x = libdevice.tanh(x)
|
||||
x = x * softcapping_value
|
||||
x = tl.where(mask, x, -float("inf"))
|
||||
block_val, block_idx = tl.max(
|
||||
x, axis=0, return_indices=True, return_indices_tie_break_left=True
|
||||
)
|
||||
take = block_val > best_val
|
||||
best_idx = tl.where(take, start + block_idx, best_idx)
|
||||
best_val = tl.where(take, block_val, best_val)
|
||||
tl.store(out_ptr + row * 2, best_val)
|
||||
tl.store(out_ptr + row * 2 + 1, (best_idx + col_offset).to(tl.float32))
|
||||
|
||||
|
||||
def softcap_row_argmax(logits, final_logit_softcapping, valid_width, col_offset):
|
||||
"""Per row of a vocab shard: ``(max, first argmax + col_offset)`` of the
|
||||
softcapped logits over the first ``valid_width`` columns, as fp32 ``[rows, 2]``."""
|
||||
assert logits.dtype == torch.float32 and logits.stride(1) == 1
|
||||
assert col_offset + logits.shape[1] < 2**24
|
||||
out = torch.empty(logits.shape[0], 2, dtype=torch.float32, device=logits.device)
|
||||
_softcap_row_argmax_kernel[(logits.shape[0],)](
|
||||
logits,
|
||||
out,
|
||||
logits.stride(0),
|
||||
valid_width,
|
||||
col_offset,
|
||||
final_logit_softcapping,
|
||||
BLOCK_SIZE=4096,
|
||||
num_warps=8,
|
||||
)
|
||||
return out
|
||||
|
||||
@@ -0,0 +1,177 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
"""SwiGLU producers of MXFP8: ``silu(gate) * up`` or the clamped SwiGLU-OAI,
|
||||
quantized in the same pass with the scale layouts of
|
||||
``ops.quantization.mxfp8_swizzled_triton``. Each quantizes the bf16-rounded
|
||||
value the unfused activation stores, so it is byte-exact against that kernel
|
||||
followed by ``mxfp8_quantize_swizzled``.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import torch
|
||||
import triton
|
||||
import triton.language as tl
|
||||
|
||||
from sglang.kernels.ops.quantization.mxfp8_swizzled_triton import (
|
||||
_E4M3,
|
||||
_alloc,
|
||||
_mx_e8m0_from_amax,
|
||||
_mx_scale_offsets,
|
||||
can_use_mxfp8_swizzled,
|
||||
)
|
||||
|
||||
|
||||
@triton.jit
|
||||
def _silu_mul_mxfp8_kernel(
|
||||
x_ptr,
|
||||
q_ptr,
|
||||
s_ptr,
|
||||
rows,
|
||||
hidden,
|
||||
n_groups,
|
||||
n_col_blocks,
|
||||
stride_row,
|
||||
dg_rows,
|
||||
ALPHA: tl.constexpr,
|
||||
LIMIT: tl.constexpr,
|
||||
DEEPGEMM: tl.constexpr,
|
||||
BLOCK_R: tl.constexpr,
|
||||
G: tl.constexpr,
|
||||
):
|
||||
pid_r = tl.program_id(0)
|
||||
pid_c = tl.program_id(1)
|
||||
r = pid_r * BLOCK_R + tl.arange(0, BLOCK_R)
|
||||
g = pid_c * G + tl.arange(0, G)
|
||||
c = pid_c * (G * 32) + tl.arange(0, G * 32)
|
||||
rmask = r < rows
|
||||
mask = rmask[:, None] & (c < hidden)[None, :]
|
||||
base = x_ptr + r[:, None].to(tl.int64) * stride_row
|
||||
gate = tl.load(base + c[None, :], mask=mask, other=0.0).to(tl.float32)
|
||||
up = tl.load(base + hidden + c[None, :], mask=mask, other=0.0).to(tl.float32)
|
||||
if ALPHA is None:
|
||||
act = (gate * tl.sigmoid(gate)).to(tl.bfloat16).to(tl.float32)
|
||||
prod = (act * up).to(tl.bfloat16).to(tl.float32)
|
||||
else:
|
||||
# the op order Inductor emits for swiglu_no_interleaved_with_alpha_and_limit
|
||||
gate = tl.minimum(gate, LIMIT, propagate_nan=tl.PropagateNan.ALL)
|
||||
up = tl.maximum(up, -LIMIT, propagate_nan=tl.PropagateNan.ALL)
|
||||
up = tl.minimum(up, LIMIT, propagate_nan=tl.PropagateNan.ALL)
|
||||
prod = gate * tl.sigmoid(gate * ALPHA) * (up + 1.0)
|
||||
prod = prod.to(tl.bfloat16).to(tl.float32)
|
||||
p3 = tl.reshape(prod, [BLOCK_R, G, 32])
|
||||
amax = tl.max(tl.abs(p3), axis=2)
|
||||
if DEEPGEMM:
|
||||
amax = tl.maximum(amax, 1e-10)
|
||||
sbyte, inv = _mx_e8m0_from_amax(amax)
|
||||
q = tl.reshape(p3 * inv[:, :, None], [BLOCK_R, G * 32])
|
||||
tl.store(
|
||||
q_ptr + r[:, None].to(tl.int64) * hidden + c[None, :],
|
||||
q.to(tl.float8e4nv),
|
||||
mask=mask,
|
||||
)
|
||||
valid = rmask[:, None] & (g < n_groups)[None, :]
|
||||
if DEEPGEMM:
|
||||
# 4 bytes per int32 along K, MN-major, rows padded to 4
|
||||
soff = (
|
||||
(g[None, :] // 4).to(tl.int64) * dg_rows * 4
|
||||
+ r[:, None] * 4
|
||||
+ g[None, :] % 4
|
||||
)
|
||||
tl.store(s_ptr + soff, sbyte.to(tl.uint8), mask=valid)
|
||||
else:
|
||||
smask = (r < tl.cdiv(rows, 128) * 128)[:, None] & (g < n_col_blocks * 4)[
|
||||
None, :
|
||||
]
|
||||
tl.store(
|
||||
s_ptr + _mx_scale_offsets(r[:, None], g[None, :], n_col_blocks),
|
||||
tl.where(valid, sbyte, 0).to(tl.uint8),
|
||||
mask=smask,
|
||||
)
|
||||
|
||||
|
||||
def can_use_silu_mul_mxfp8(hidden: torch.Tensor) -> bool:
|
||||
return can_use_mxfp8_swizzled(hidden) and (hidden.shape[-1] // 2) % 32 == 0
|
||||
|
||||
|
||||
def _swiglu_mxfp8(
|
||||
hidden: torch.Tensor,
|
||||
alpha: float | None,
|
||||
limit: float | None,
|
||||
deepgemm: bool = False,
|
||||
) -> tuple[torch.Tensor, torch.Tensor]:
|
||||
if not can_use_silu_mul_mxfp8(hidden):
|
||||
raise ValueError(
|
||||
"expected a row-major bf16 CUDA [rows, 2 * n] tensor, n % 32 == 0"
|
||||
)
|
||||
rows, twice = hidden.shape
|
||||
n = twice // 2
|
||||
if deepgemm:
|
||||
if n % 128:
|
||||
raise ValueError(f"DeepGEMM MXFP8 scales need n % 128 == 0, got {n=}")
|
||||
q = torch.empty(rows, n, dtype=_E4M3, device=hidden.device)
|
||||
dg_rows = triton.cdiv(rows, 4) * 4
|
||||
base = torch.empty(n // 128, dg_rows, dtype=torch.int32, device=hidden.device)
|
||||
s = base.T[:rows]
|
||||
s_bytes = base.view(torch.uint8)
|
||||
padded_rows = rows
|
||||
else:
|
||||
q, s = _alloc(rows, n, hidden.device, zero_scales=False)
|
||||
s_bytes = s
|
||||
dg_rows = 0
|
||||
padded_rows = triton.cdiv(rows, 128) * 128
|
||||
if rows == 0:
|
||||
return q, s
|
||||
n_groups = n // 32
|
||||
n_col_blocks = triton.cdiv(n_groups, 4)
|
||||
block_r, g = (4 if rows <= 1024 else 16), 8
|
||||
grid = (triton.cdiv(padded_rows, block_r), triton.cdiv(n_groups, g))
|
||||
with torch.get_device_module().device(hidden.device):
|
||||
_silu_mul_mxfp8_kernel[grid](
|
||||
hidden,
|
||||
q,
|
||||
s_bytes,
|
||||
rows,
|
||||
n,
|
||||
n_groups,
|
||||
n_col_blocks,
|
||||
hidden.stride(0),
|
||||
dg_rows,
|
||||
ALPHA=alpha,
|
||||
LIMIT=limit,
|
||||
DEEPGEMM=deepgemm,
|
||||
BLOCK_R=block_r,
|
||||
G=g,
|
||||
num_warps=4,
|
||||
)
|
||||
return q, s
|
||||
|
||||
|
||||
def silu_mul_mxfp8(hidden: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor]:
|
||||
"""``hidden [rows, 2n]`` bf16 (gate | up) -> quantized ``silu(gate) * up``."""
|
||||
return _swiglu_mxfp8(hidden, None, None)
|
||||
|
||||
|
||||
def swiglu_oai_mxfp8(
|
||||
hidden: torch.Tensor, alpha: float, limit: float
|
||||
) -> tuple[torch.Tensor, torch.Tensor]:
|
||||
"""``hidden [rows, 2n]`` bf16 (gate | up) -> quantized SwiGLU-OAI,
|
||||
``clamp(gate, max=limit) * sigmoid(alpha * gate) * (clamp(up, +-limit) + 1)``."""
|
||||
return _swiglu_mxfp8(hidden, alpha, limit)
|
||||
|
||||
|
||||
def swiglu_oai_mxfp8_deepgemm(
|
||||
hidden: torch.Tensor, alpha: float, limit: float
|
||||
) -> tuple[torch.Tensor, torch.Tensor]:
|
||||
"""Same values as ``swiglu_oai_mxfp8``; scales in DeepGEMM's layout, the
|
||||
``sglang_per_token_group_quant_fp8(..., 32, column_major_scales=True,
|
||||
scale_tma_aligned=True, scale_ue8m0=True)`` output: int32 ``[rows, n / 128]``,
|
||||
four UE8M0 bytes each, MN-major with rows padded to 4."""
|
||||
return _swiglu_mxfp8(hidden, alpha, limit, deepgemm=True)
|
||||
|
||||
|
||||
__all__ = [
|
||||
"can_use_silu_mul_mxfp8",
|
||||
"silu_mul_mxfp8",
|
||||
"swiglu_oai_mxfp8",
|
||||
"swiglu_oai_mxfp8_deepgemm",
|
||||
]
|
||||
@@ -3,7 +3,7 @@
|
||||
The Triton kernels migrated here live in this package
|
||||
(``sglang.kernels.ops.attention.<module>``); import them from there. Their
|
||||
``KernelSpec`` metadata is registered below for inventory (backend = Triton).
|
||||
KV-cache index/write kernels went to the ``kvcache`` group instead.
|
||||
KV-cache write kernels went to the ``kvcache`` group instead.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -33,9 +33,12 @@ _TRITON_KERNELS = [
|
||||
("metadata", "get_num_kv_splits_triton"),
|
||||
("metadata", "prepare_swa_spec_page_table_triton"),
|
||||
("metadata", "normal_decode_set_metadata"),
|
||||
("dsa_metadata", "fused_dsa_decode_metadata"),
|
||||
("dsa_metadata", "fused_dsa_target_verify_metadata"),
|
||||
("dsa_metadata", "fused_dsa_draft_extend_metadata"),
|
||||
("dsa.metadata", "fused_dsa_decode_metadata"),
|
||||
("dsa.metadata", "fused_dsa_target_verify_metadata"),
|
||||
("dsa.metadata", "fused_dsa_draft_extend_metadata"),
|
||||
("dsa.kpool_metadata.decode", "fused_dsa_decode_metadata_kpool"),
|
||||
("dsa.kpool_metadata.verify", "fused_dsa_target_verify_metadata_kpool"),
|
||||
("dsa.kpool_metadata.draft_extend", "fused_dsa_draft_extend_metadata_kpool"),
|
||||
("rocm_mla_decode_rope", "decode_attention_fwd_grouped_rope"),
|
||||
("verify_splitkv", "verify_splitkv_fwd"),
|
||||
("unified_attention_3d_mtp", "unified_attention_3d_mtp_func"),
|
||||
@@ -44,9 +47,20 @@ _TRITON_KERNELS = [
|
||||
("pad", "unpad_draft_extend_output"),
|
||||
("pad", "seqlens_expand_triton"),
|
||||
("position", "compute_position_triton"),
|
||||
("dsv4_attn_metadata_kernels", "expand_prefill_causally"),
|
||||
("dsv4_attn_metadata_kernels", "build_page_table_positions"),
|
||||
("dsv4_attn_metadata_kernels", "build_causal_swa_page_indices"),
|
||||
("dsv4.attn_metadata_kernels", "expand_prefill_causally"),
|
||||
("dsv4.attn_metadata_kernels", "build_page_table_positions"),
|
||||
("dsv4.attn_metadata_kernels", "build_causal_swa_page_indices"),
|
||||
("kv_indices", "create_flashinfer_kv_indices_triton"),
|
||||
("kv_indices", "create_flashmla_kv_indices_triton"),
|
||||
("kv_indices", "create_chunked_prefix_cache_kv_indices"),
|
||||
("kv_indices", "get_num_kv_index_blocks_flashmla"),
|
||||
("kv_indices", "get_num_page_per_block_flashmla"),
|
||||
("trtllm_mha_page_table", "build_trtllm_mha_page_table"),
|
||||
("trtllm_mha_graph_metadata", "update_trtllm_mha_graph_metadata"),
|
||||
("aiter_unified_page_table", "scatter_ragged_to_page_table_kernel"),
|
||||
("aiter_unified_page_table", "scatter_req_to_token_to_page_table_kernel"),
|
||||
("fused_qk_rope_reshape_and_cache", "fused_qk_rope_reshape_and_cache"),
|
||||
("mla_qk_prep", "concat_and_cast_mha_k_triton"),
|
||||
]
|
||||
for _mod, _fn in _TRITON_KERNELS:
|
||||
register_kernel(
|
||||
@@ -173,9 +187,7 @@ for _mod, _fn in [
|
||||
("dsa.transform_index", "transform_index_page_table_decode"),
|
||||
("dsa.transform_index", "prepare_trtllm_nope_sparse_metadata"),
|
||||
("dsa.transform_index", "transform_index_page_table_dcp"),
|
||||
("dsa.cp_split", "dsa_cp_interleave_q_seqs_kernel"),
|
||||
("dsv4.fp4_indexer", "quantize_fp4_indexer_tensor"),
|
||||
("dsv4.fp4_indexer", "store_fp4_index_k_cache"),
|
||||
("dsa.cp_split", "dsa_cp_interleave_q_seqs"),
|
||||
]:
|
||||
register_kernel(
|
||||
KernelSpec(
|
||||
@@ -189,15 +201,13 @@ del _mod, _fn
|
||||
# Generic attention kernels migrated in Phase 2.5 (RFC #29630).
|
||||
for _mod, _fn in [
|
||||
("utils", "mla_quantize_and_rope_for_fp8"),
|
||||
("utils", "launch_reshape_and_cache_flash"),
|
||||
("utils", "launch_reshape_and_cache_shuffle_5d"),
|
||||
("flash_mla_sm120", "flash_mla_with_kvcache_sm120"),
|
||||
("dcp_kernels", "create_dcp_kv_indices"),
|
||||
("dcp_kernels", "correct_attn_out"),
|
||||
("dcp_kernels", "dcp_lse_combine_triton"),
|
||||
("dcp_kernels", "dcp_pack_a2a_send"),
|
||||
("pa_page_table", "_build_pa_page_table"),
|
||||
("nsa_triton_decode", "triton_sparse_attn_decode"),
|
||||
("dsv4.triton_sparse_decode", "triton_sparse_attn_decode"),
|
||||
]:
|
||||
register_kernel(
|
||||
KernelSpec(
|
||||
@@ -211,7 +221,7 @@ del _mod, _fn
|
||||
# RoPE / QK-norm fusion kernels migrated from srt/layers top-level strays
|
||||
# (RFC #29630, Phase 2.5); registered for inventory.
|
||||
for _mod, _fn in [
|
||||
("deepseek_v4_rope", "precompute_freqs_cis"),
|
||||
("dsv4.rope", "precompute_freqs_cis"),
|
||||
("fused_qk_norm_rope_store", "fused_qk_norm_rope_swa_store"),
|
||||
("fused_qk_rmsnorm_rope_gate", "fused_qk_gemma_rmsnorm_rope_gate"),
|
||||
("rotary_triton", "triton_mrope_fused"),
|
||||
@@ -231,25 +241,7 @@ del _mod, _fn
|
||||
# Kernels introduced with Kimi-K3, inventoried by logical operator group.
|
||||
for _mod, _fn, _backend, _device in [
|
||||
("attn_res", "attn_res_fused_tma", KernelBackend.JIT, CapabilityRequirement.CUDA),
|
||||
(
|
||||
"attn_res",
|
||||
"attn_res_fused_pull_rs",
|
||||
KernelBackend.JIT,
|
||||
CapabilityRequirement.CUDA,
|
||||
),
|
||||
(
|
||||
"attn_res",
|
||||
"attn_res_fused_direct_ag",
|
||||
KernelBackend.JIT,
|
||||
CapabilityRequirement.CUDA,
|
||||
),
|
||||
("attn_res_hip", "attn_res_hip", KernelBackend.TRITON, CapabilityRequirement.HIP),
|
||||
(
|
||||
"mla_output_gate",
|
||||
"kimi_k3_mla_output_gate",
|
||||
KernelBackend.JIT,
|
||||
CapabilityRequirement.CUDA,
|
||||
),
|
||||
(
|
||||
"kda_decode_mtp",
|
||||
"fused_kda_decode_mtp_dspark",
|
||||
@@ -412,3 +404,131 @@ for _fn in ("fp4_index_logits_paged", "finish_paged_indexer_topk"):
|
||||
)
|
||||
)
|
||||
del _fn
|
||||
|
||||
# Muse Spark QK norm (+ RoPE, + KV store), built per layer: inventoried through
|
||||
# its class methods.
|
||||
for _bk in ("torch", "triton", "helion"):
|
||||
register_kernel(
|
||||
KernelSpec(
|
||||
op="attention.muse_spark_qk_norm_rope_store",
|
||||
backend=KernelBackend(_bk),
|
||||
target="sglang.kernels.ops.attention.muse_spark_qk_norm_rope_store:FusedQKNorm."
|
||||
+ ("forward_native" if _bk == "torch" else f"forward_{_bk}"),
|
||||
capabilities=frozenset() if _bk == "torch" else CUDA_ONLY,
|
||||
)
|
||||
)
|
||||
del _bk
|
||||
for _mod, _fn in [
|
||||
("mla_qk_prep", "concat_and_cast_q_fp8_pad"),
|
||||
("mla_qk_prep", "absorbed_bmm_concat_cast_q_fp8"),
|
||||
("metadata", "q8kv8_topk_length_from_indices"),
|
||||
("minimax_sparse.decode.topk_sparse", "flash_decode_with_gqa_share_sparse"),
|
||||
("minimax_sparse.prefill.topk_sparse", "flash_prefill_with_gqa_share_sparse"),
|
||||
]:
|
||||
register_kernel(
|
||||
KernelSpec(
|
||||
op=f"attention.{_fn}",
|
||||
backend=KernelBackend.TRITON,
|
||||
target=f"sglang.kernels.ops.attention.{_mod}:{_fn}",
|
||||
)
|
||||
)
|
||||
del _mod, _fn
|
||||
register_kernel(
|
||||
KernelSpec(
|
||||
op="attention.absorbed_bmm_concat_cast_q_fp8",
|
||||
backend=KernelBackend.JIT,
|
||||
target="sglang.kernels.ops.attention.qprep_bf16_fp8_sm90:q8kv8_qprep_fwd",
|
||||
capabilities=frozenset(
|
||||
{CapabilityRequirement.cuda(min_sm=(9, 0), max_sm=(9, 9))}
|
||||
),
|
||||
)
|
||||
)
|
||||
register_kernel(
|
||||
KernelSpec(
|
||||
op="attention.topk_index_reduce",
|
||||
backend=KernelBackend.TORCH,
|
||||
target="sglang.kernels.ops.attention.minimax_sparse.common.index:topk_index_reduce",
|
||||
)
|
||||
)
|
||||
# The context-parallel indexer serves only the gfx950 ROCm top-k contract.
|
||||
for _fn in ("score_local_blocks", "select_local_candidates", "merge_candidates"):
|
||||
register_kernel(
|
||||
KernelSpec(
|
||||
op=f"attention.{_fn}",
|
||||
backend=KernelBackend.TRITON,
|
||||
target=f"sglang.kernels.ops.attention.minimax_sparse.decode.indexer_cp:{_fn}",
|
||||
capabilities=frozenset({CapabilityRequirement.HIP}),
|
||||
)
|
||||
)
|
||||
del _fn
|
||||
register_kernel(
|
||||
KernelSpec(
|
||||
op="attention.gluon_sparse_prefill",
|
||||
backend=KernelBackend.AITER,
|
||||
target="sglang.kernels.ops.attention.minimax_sparse.prefill.gluon_prefill:gluon_sparse_prefill",
|
||||
capabilities=frozenset({CapabilityRequirement.HIP}),
|
||||
)
|
||||
)
|
||||
register_kernel(
|
||||
KernelSpec(
|
||||
op="attention.mtp_verify_attn_fwd_asm",
|
||||
backend=KernelBackend.JIT,
|
||||
target="sglang.kernels.ops.attention.vattn_asm_gfx950:mtp_verify_attn_fwd_asm",
|
||||
capabilities=frozenset({CapabilityRequirement.HIP}),
|
||||
)
|
||||
)
|
||||
for _mod, _fn in [
|
||||
("helion.kda_decode", "helion_fused_recurrent_kda_packed_decode"),
|
||||
("helion.kda_replayssm", "helion_fused_recurrent_kda_replayssm_decode"),
|
||||
("helion.kda_prefill", "chunk_kda"),
|
||||
("helion.kda_verify", "helion_kda_verify"),
|
||||
]:
|
||||
register_kernel(
|
||||
KernelSpec(
|
||||
op=f"attention.{_fn}",
|
||||
backend=KernelBackend.HELION,
|
||||
target=f"sglang.kernels.ops.attention.{_mod}:{_fn}",
|
||||
capabilities=CUDA_ONLY,
|
||||
)
|
||||
)
|
||||
del _mod, _fn
|
||||
# tcgen05 + TMEM: datacenter Blackwell only (SM100 / SM103), not SM120.
|
||||
for _mod, _fn in [
|
||||
("linear.gdn_blackwell", "chunk_gated_delta_rule_cutedsl"),
|
||||
("linear.kda_blackwell", "chunk_kda_cutedsl"),
|
||||
("linear.kda_nvidia_prefill", "chunk_kda_fwd"),
|
||||
]:
|
||||
register_kernel(
|
||||
KernelSpec(
|
||||
op=f"attention.{_fn}",
|
||||
backend=KernelBackend.CUTE_DSL,
|
||||
target=f"sglang.kernels.ops.attention.{_mod}:{_fn}",
|
||||
capabilities=frozenset(
|
||||
{CapabilityRequirement.cuda(min_sm=(10, 0), max_sm=(10, 9))}
|
||||
),
|
||||
)
|
||||
)
|
||||
del _mod, _fn
|
||||
register_kernel(
|
||||
KernelSpec(
|
||||
op="attention.chunk_kda_fwd",
|
||||
backend=KernelBackend.JIT,
|
||||
target="sglang.kernels.ops.attention.linear.kda_ptx_prefill:chunk_kda_fwd",
|
||||
# Exactly the kda_ptx_prefill.SM_ARCHS build targets (sm_100a, sm_103a).
|
||||
capabilities=frozenset(
|
||||
CapabilityRequirement.cuda(min_sm=sm, max_sm=sm)
|
||||
for sm in ((10, 0), (10, 3))
|
||||
),
|
||||
)
|
||||
)
|
||||
register_kernel(
|
||||
KernelSpec(
|
||||
op="attention.fp8_attention",
|
||||
backend=KernelBackend.CUTE_DSL,
|
||||
target="sglang.kernels.ops.attention.fp8_fa_sm120:fp8_attention",
|
||||
# MmaMXF8Op assembles only for SM120 / SM121.
|
||||
capabilities=frozenset(
|
||||
{CapabilityRequirement.cuda(min_sm=(12, 0), max_sm=(12, 1))}
|
||||
),
|
||||
)
|
||||
)
|
||||
|
||||
@@ -12,13 +12,10 @@ from sglang.kernels.jit.utils import (
|
||||
make_cpp_args,
|
||||
override_jit_cuda_arch,
|
||||
)
|
||||
from sglang.srt.utils.custom_op import register_custom_op
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from tvm_ffi.module import Module
|
||||
|
||||
from sglang.kernels.ops.communication.all_reduce import Communicator
|
||||
|
||||
_DIM: int = 7168 # K3 hidden size, template parameter of the TMA kernel
|
||||
_MAX_BANK_ROWS: int = 8 # K3 has <= 8 snapshots, upper bound of the nvb dispatch tables
|
||||
|
||||
@@ -86,8 +83,9 @@ _TMA_BEST_CONFIG: dict[int, tuple[int, int, int]] = {
|
||||
def _tuning(nvb: int, num_tokens: int) -> tuple[int, int, int]:
|
||||
"""(chunk_rows, occupancy, consumer_regs) for this aggregation point.
|
||||
|
||||
Shared by all three entry points below: they run the same kernel template
|
||||
and differ only in the collective fused onto it."""
|
||||
Shared by attn_res_fused_tma and its collective variants in
|
||||
communication.attn_res_collective: they run the same kernel template and
|
||||
differ only in the collective fused onto it."""
|
||||
if not 1 <= nvb <= _MAX_BANK_ROWS:
|
||||
raise ValueError(f"attn_res: nvb must be in [1, {_MAX_BANK_ROWS}], got {nvb}")
|
||||
best = _TMA_BEST_CONFIG[nvb]
|
||||
@@ -98,56 +96,6 @@ def _tuning(nvb: int, num_tokens: int) -> tuple[int, int, int]:
|
||||
return best
|
||||
|
||||
|
||||
_COMM_MAP: dict[int, Communicator] = {}
|
||||
|
||||
|
||||
def register_comm(comm: Communicator) -> None:
|
||||
# One communicator per world_size per process -- see the note in
|
||||
# communication/all_reduce_residual.py::register_comm. The ops key only on world_size, so an
|
||||
# overwrite here would hand the old group's callers the new group's peer
|
||||
# pointers.
|
||||
prev = _COMM_MAP.get(comm.world_size)
|
||||
assert prev is None or prev is comm, (
|
||||
f"a different communicator is already registered for world_size="
|
||||
f"{comm.world_size}"
|
||||
)
|
||||
_COMM_MAP[comm.world_size] = comm
|
||||
|
||||
|
||||
@register_custom_op(mutates_args=["out", "prefix_out"])
|
||||
def _attn_res_fused_pull_rs_op(
|
||||
world_size: int,
|
||||
input: torch.Tensor,
|
||||
residual: torch.Tensor | None,
|
||||
bank: torch.Tensor,
|
||||
cw: torch.Tensor,
|
||||
ow: torch.Tensor,
|
||||
out: torch.Tensor,
|
||||
prefix_out: torch.Tensor,
|
||||
nvb: int,
|
||||
eps: float,
|
||||
input_mc_ptr: int,
|
||||
max_blocks: int,
|
||||
chunk_rows: int,
|
||||
occupancy: int,
|
||||
consumer_regs: int,
|
||||
) -> None:
|
||||
_jit_fused_tma_module(chunk_rows, occupancy, consumer_regs).run_pull_rs(
|
||||
_COMM_MAP[world_size],
|
||||
input,
|
||||
residual,
|
||||
bank,
|
||||
cw,
|
||||
ow,
|
||||
out,
|
||||
prefix_out,
|
||||
nvb,
|
||||
eps,
|
||||
input_mc_ptr,
|
||||
max_blocks,
|
||||
)
|
||||
|
||||
|
||||
def attn_res_fused_tma(
|
||||
prefix_sum: torch.Tensor,
|
||||
bank: torch.Tensor,
|
||||
@@ -186,113 +134,3 @@ def attn_res_fused_tma(
|
||||
_jit_fused_tma_module(*_tuning(nvb, prefix_sum.shape[0])).run(
|
||||
prefix_sum, bank, cw, ow, out, nvb, eps, write_prefix
|
||||
)
|
||||
|
||||
|
||||
@register_custom_op(mutates_args=["bank", "out"])
|
||||
def _attn_res_fused_direct_ag_op(
|
||||
world_size: int,
|
||||
prefix_sum: torch.Tensor,
|
||||
bank: torch.Tensor,
|
||||
cw: torch.Tensor,
|
||||
ow: torch.Tensor,
|
||||
out: torch.Tensor,
|
||||
nvb: int,
|
||||
eps: float,
|
||||
output_mc_ptr: int,
|
||||
max_blocks: int,
|
||||
write_prefix: bool,
|
||||
chunk_rows: int,
|
||||
occupancy: int,
|
||||
consumer_regs: int,
|
||||
) -> None:
|
||||
_jit_fused_tma_module(chunk_rows, occupancy, consumer_regs).run_direct_ag(
|
||||
_COMM_MAP[world_size],
|
||||
prefix_sum,
|
||||
bank,
|
||||
cw,
|
||||
ow,
|
||||
out,
|
||||
nvb,
|
||||
eps,
|
||||
output_mc_ptr,
|
||||
max_blocks,
|
||||
write_prefix,
|
||||
)
|
||||
|
||||
|
||||
def attn_res_fused_direct_ag(
|
||||
world_size: int,
|
||||
prefix_sum: torch.Tensor,
|
||||
bank: torch.Tensor,
|
||||
cw: torch.Tensor,
|
||||
ow: torch.Tensor,
|
||||
out: torch.Tensor,
|
||||
nvb: int,
|
||||
eps: float,
|
||||
*,
|
||||
output_mc_ptr: int,
|
||||
max_blocks: int = 128,
|
||||
write_prefix: bool = False,
|
||||
) -> torch.Tensor:
|
||||
"""Fuse local attention-residual aggregation with direct multicast AG.
|
||||
|
||||
`out` is the full symmetric [world * local_tokens, H] output. Each rank
|
||||
aggregates its local token shard and multicast-stores normalized vectors
|
||||
directly from consumer registers into that rank's slice on every peer.
|
||||
"""
|
||||
_attn_res_fused_direct_ag_op(
|
||||
world_size,
|
||||
prefix_sum,
|
||||
bank,
|
||||
cw,
|
||||
ow,
|
||||
out,
|
||||
nvb,
|
||||
eps,
|
||||
output_mc_ptr,
|
||||
max_blocks,
|
||||
write_prefix,
|
||||
*_tuning(nvb, prefix_sum.shape[0]),
|
||||
)
|
||||
return out
|
||||
|
||||
|
||||
def attn_res_fused_pull_rs(
|
||||
world_size: int,
|
||||
input: torch.Tensor,
|
||||
residual: torch.Tensor | None,
|
||||
bank: torch.Tensor,
|
||||
cw: torch.Tensor,
|
||||
ow: torch.Tensor,
|
||||
out: torch.Tensor,
|
||||
prefix_out: torch.Tensor,
|
||||
nvb: int,
|
||||
eps: float,
|
||||
*,
|
||||
input_mc_ptr: int,
|
||||
max_blocks: int = 128,
|
||||
) -> tuple[torch.Tensor, torch.Tensor]:
|
||||
"""Fused NVLS pull RS + local residual + TMA attention aggregation.
|
||||
|
||||
`input` is the full TP-partial o_proj tensor in multicast symmetric
|
||||
memory. Every rank reduces only its contiguous token shard, optionally
|
||||
adds its already-local residual, materializes that prefix in `prefix_out`,
|
||||
and feeds it directly to the fused attention-residual aggregation/norm.
|
||||
"""
|
||||
_attn_res_fused_pull_rs_op(
|
||||
world_size,
|
||||
input,
|
||||
residual,
|
||||
bank,
|
||||
cw,
|
||||
ow,
|
||||
out,
|
||||
prefix_out,
|
||||
nvb,
|
||||
eps,
|
||||
input_mc_ptr,
|
||||
max_blocks,
|
||||
# prefix_out, not input: the shard this rank actually aggregates.
|
||||
*_tuning(nvb, prefix_out.shape[0]),
|
||||
)
|
||||
return out, prefix_out
|
||||
|
||||
@@ -3,12 +3,13 @@
|
||||
Migrated from ``sglang.srt.layers.attention.dsa.utils`` (RFC #29630, Phase 2.5).
|
||||
"""
|
||||
|
||||
import torch
|
||||
import triton
|
||||
import triton.language as tl
|
||||
|
||||
|
||||
@triton.jit
|
||||
def dsa_cp_interleave_q_seqs_kernel(
|
||||
def _dsa_cp_interleave_q_seqs_kernel(
|
||||
in_seqs_ptr,
|
||||
out_seqs_ptr,
|
||||
bs_idx_ptr,
|
||||
@@ -27,3 +28,18 @@ def dsa_cp_interleave_q_seqs_kernel(
|
||||
tl.store(out_seqs_ptr + bs_idx, cur_seq)
|
||||
bs_idx += 1
|
||||
extra_seq = cur_len - cur_seq * cp_size
|
||||
|
||||
|
||||
def dsa_cp_interleave_q_seqs(
|
||||
extend_seqs: torch.Tensor, num_rank_seqs: int, cp_size: int, cp_rank: int
|
||||
) -> tuple[torch.Tensor, torch.Tensor]:
|
||||
"""This rank's interleaved share of each request's extend length, as
|
||||
``(q_lens, bs_idx)`` over the ``num_rank_seqs`` requests left non-empty."""
|
||||
q_lens = torch.empty(
|
||||
(num_rank_seqs,), device=extend_seqs.device, dtype=extend_seqs.dtype
|
||||
)
|
||||
bs_idx = torch.empty((num_rank_seqs,), device=extend_seqs.device, dtype=torch.int32)
|
||||
_dsa_cp_interleave_q_seqs_kernel[(1,)](
|
||||
extend_seqs, q_lens, bs_idx, len(extend_seqs), cp_size, cp_rank
|
||||
)
|
||||
return q_lens, bs_idx
|
||||
|
||||
@@ -17,7 +17,9 @@ import cutlass.cute as cute
|
||||
import torch
|
||||
from cutlass.utils import HardwareInfo
|
||||
|
||||
from sglang.kernels.ops.attention.cutedsl_fp8_paged_mqa_logits import FP8MQALogitsKernel
|
||||
from sglang.kernels.ops.attention.dsa.cutedsl_fp8_paged_mqa_logits import (
|
||||
FP8MQALogitsKernel,
|
||||
)
|
||||
from sglang.srt.utils import is_sm100_supported
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@@ -141,7 +141,7 @@ def _dequantize_k_cache_fast_kernel(
|
||||
tl.store(dst_ptr, data, mask=mask)
|
||||
|
||||
|
||||
def dequantize_k_cache_paged(
|
||||
def dequantize_k_cache_paged_dsa(
|
||||
quant_k_cache: torch.Tensor,
|
||||
page_table_1_flattened: torch.Tensor,
|
||||
group_size: int = 128,
|
||||
@@ -262,7 +262,7 @@ def _dequantize_k_cache_paged_kernel(
|
||||
_GATHER_TOKENS_PER_PROG = 4
|
||||
|
||||
|
||||
def gather_dequant_requant_fp8_paged(
|
||||
def gather_dequant_requant_fp8_paged_dsa(
|
||||
quant_k_cache: torch.Tensor,
|
||||
page_table_1_flattened: torch.Tensor,
|
||||
group_size: int = 128,
|
||||
@@ -441,122 +441,6 @@ def _gather_dequant_requant_fp8_paged_vec_kernel(
|
||||
tl.store(dst_r, data, mask=row_in_range[:, None])
|
||||
|
||||
|
||||
def gather_dequant_requant_fp8_paged_legacy(
|
||||
quant_k_cache: torch.Tensor,
|
||||
page_table_1_flattened: torch.Tensor,
|
||||
group_size: int = 128,
|
||||
extra_rows: int = 0,
|
||||
) -> torch.Tensor:
|
||||
"""Legacy (pre-vectorization) gather + dequant + requant.
|
||||
|
||||
Kept as the bit-exactness / performance reference for
|
||||
``gather_dequant_requant_fp8_paged`` (see
|
||||
``benchmark/kernels/deepseek/benchmark_q8kv8_kv_gather.py``). Allocates and zero-fills the
|
||||
full destination each call, then launches one program per
|
||||
(token, 128-elem slice).
|
||||
"""
|
||||
dim_quant = quant_k_cache.shape[-1]
|
||||
dim_nope, dim_rope = _infer_dsa_dims(dim_quant)
|
||||
quant_k_cache = quant_k_cache.view((-1, dim_quant))
|
||||
|
||||
num_tokens = page_table_1_flattened.shape[0]
|
||||
assert quant_k_cache.dtype == torch.float8_e4m3fn
|
||||
num_tiles = dim_nope // group_size
|
||||
out_dim = dim_nope + dim_rope
|
||||
assert num_tiles * group_size == dim_nope
|
||||
|
||||
total_rows = num_tokens + extra_rows
|
||||
# Allocate a fresh zero-filled buffer. The extra landing-pad rows at
|
||||
# the tail must read as zeros (the kernel may over-read past
|
||||
# num_tokens for masked indices).
|
||||
output = torch.zeros(
|
||||
(total_rows, 1, out_dim),
|
||||
dtype=torch.float8_e4m3fn,
|
||||
device=quant_k_cache.device,
|
||||
)
|
||||
|
||||
num_blocks_per_token = num_tiles + triton.cdiv(dim_rope, group_size)
|
||||
|
||||
input_nope_q = quant_k_cache[:, :dim_nope]
|
||||
input_nope_s = quant_k_cache[:, dim_nope : dim_nope + num_tiles * 4].view(
|
||||
torch.float32
|
||||
)
|
||||
input_rope = quant_k_cache[:, dim_nope + num_tiles * 4 :].view(torch.bfloat16)
|
||||
|
||||
_gather_dequant_requant_fp8_paged_kernel[(num_tokens, num_blocks_per_token)](
|
||||
output,
|
||||
input_nope_q,
|
||||
input_nope_s,
|
||||
input_rope,
|
||||
page_table_1_flattened,
|
||||
output.stride(0),
|
||||
input_nope_q.stride(0),
|
||||
input_nope_s.stride(0),
|
||||
input_rope.stride(0),
|
||||
NUM_NOPE_BLOCKS=num_tiles,
|
||||
GROUP_SIZE=group_size,
|
||||
DIM_NOPE=dim_nope,
|
||||
DIM_ROPE=dim_rope,
|
||||
)
|
||||
|
||||
return output
|
||||
|
||||
|
||||
@triton.jit
|
||||
def _gather_dequant_requant_fp8_paged_kernel(
|
||||
output_ptr,
|
||||
input_nope_q_ptr,
|
||||
input_nope_s_ptr,
|
||||
input_rope_ptr,
|
||||
page_table_1_ptr,
|
||||
output_stride_0: int,
|
||||
input_nope_q_stride_0: int,
|
||||
input_nope_s_stride_0: int,
|
||||
input_rope_stride_0: int,
|
||||
NUM_NOPE_BLOCKS: tl.constexpr,
|
||||
GROUP_SIZE: tl.constexpr,
|
||||
DIM_NOPE: tl.constexpr,
|
||||
DIM_ROPE: tl.constexpr,
|
||||
):
|
||||
"""Fused gather + dequant(per-group) + requant(per-tensor) -> fp8."""
|
||||
token_id = tl.program_id(0)
|
||||
token_id_paged = tl.load(page_table_1_ptr + token_id).to(tl.int32)
|
||||
raw_block_id = tl.program_id(1)
|
||||
|
||||
if raw_block_id < NUM_NOPE_BLOCKS:
|
||||
# nope: read fp8, mul group scale -> f32, cast to fp8_e4m3fn
|
||||
effective_block_id = raw_block_id
|
||||
offs_q = effective_block_id * GROUP_SIZE + tl.arange(0, GROUP_SIZE)
|
||||
mask = offs_q < DIM_NOPE
|
||||
|
||||
ptr_q = input_nope_q_ptr + token_id_paged * input_nope_q_stride_0 + offs_q
|
||||
ptr_s = (
|
||||
input_nope_s_ptr
|
||||
+ token_id_paged * input_nope_s_stride_0
|
||||
+ effective_block_id
|
||||
)
|
||||
|
||||
y_q = tl.load(ptr_q, mask=mask, other=0.0).to(tl.float32)
|
||||
y_s = tl.load(ptr_s)
|
||||
|
||||
# dequant -> f32 -> requant to fp8
|
||||
y = (y_q * y_s).to(tl.float8e4nv)
|
||||
|
||||
dst_ptr = output_ptr + token_id * output_stride_0 + offs_q
|
||||
tl.store(dst_ptr, y, mask=mask)
|
||||
else:
|
||||
# rope: read bf16, cast to fp8
|
||||
effective_block_id = raw_block_id - NUM_NOPE_BLOCKS
|
||||
offs = effective_block_id * GROUP_SIZE + tl.arange(0, GROUP_SIZE)
|
||||
mask = offs < DIM_ROPE
|
||||
|
||||
src_ptr = input_rope_ptr + token_id_paged * input_rope_stride_0 + offs
|
||||
dst_ptr = output_ptr + token_id * output_stride_0 + DIM_NOPE + offs
|
||||
|
||||
data = tl.load(src_ptr, mask=mask).to(tl.float8e4nv)
|
||||
tl.store(dst_ptr, data, mask=mask)
|
||||
|
||||
|
||||
@triton.jit
|
||||
def _concat_cast_kv_fp8_pad_kernel(
|
||||
out_ptr,
|
||||
|
||||
@@ -1,13 +1,4 @@
|
||||
"""gfx950 fused DSA indexer decode path: four raw-HIP kernels replacing the
|
||||
12-launch ROCm chain. Enabled wherever supported; the gate is
|
||||
``utils.gfx950_fused_indexer_runtime_ok``."""
|
||||
|
||||
from sglang.kernels.ops.attention.dsa.hip_gfx950.fused_decode import ( # noqa: F401
|
||||
CACHE_TOK_STRIDE,
|
||||
MAX_ROWS,
|
||||
PAGE_SIZE,
|
||||
Gfx950FusedIndexer,
|
||||
model_shape_supported,
|
||||
prealloc_workspace,
|
||||
supported_hardware,
|
||||
)
|
||||
``utils.gfx950_fused_indexer_runtime_ok``, the runtime wrapper
|
||||
``srt.layers.attention.dsa.gfx950_fused_indexer``."""
|
||||
|
||||
@@ -0,0 +1,53 @@
|
||||
"""DSA only. Indexer Q kernels (JIT)."""
|
||||
|
||||
from typing import Tuple
|
||||
|
||||
import torch
|
||||
|
||||
from sglang.kernels.jit.utils import (
|
||||
cache_once,
|
||||
is_arch_support_pdl,
|
||||
load_jit,
|
||||
make_cpp_args,
|
||||
)
|
||||
from sglang.kernels.ops.attention.dsv4.utils import make_name
|
||||
|
||||
|
||||
# V3.2 lays q out as [rope | nope] (V4 is [nope | rope]) -> kRopeFirst=true, and
|
||||
# drops the Hadamard rotation (kHadamard=false).
|
||||
@cache_once
|
||||
def _jit_main_q_indexer_rope_first_quant_module(dtype: torch.dtype):
|
||||
args = make_cpp_args(dtype, is_arch_support_pdl(), True, False)
|
||||
return load_jit(
|
||||
make_name("main_q_indexer_rope_first_quant"),
|
||||
*args,
|
||||
cuda_files=["deepseek_v4/main_norm_rope.cuh"],
|
||||
cuda_wrappers=[
|
||||
("forward", f"FusedQIndexerRopeHadamardQuantKernel<{args}>::forward"),
|
||||
],
|
||||
)
|
||||
|
||||
|
||||
def fused_q_indexer_rope_first_quant(
|
||||
q_input: torch.Tensor,
|
||||
weight: torch.Tensor,
|
||||
weight_scale: float,
|
||||
cos_sin_cache: torch.Tensor,
|
||||
positions: torch.Tensor,
|
||||
) -> Tuple[torch.Tensor, torch.Tensor]:
|
||||
"""DeepSeek-V3.2 only. Indexer Q: RoPE on the leading dims + fp8 act-quant. CUDA only."""
|
||||
q_fp8 = torch.empty(q_input.shape, dtype=torch.float8_e4m3fn, device=q_input.device)
|
||||
weights_out = torch.empty(
|
||||
(*q_input.shape[:-1], 1), dtype=torch.float32, device=q_input.device
|
||||
)
|
||||
module = _jit_main_q_indexer_rope_first_quant_module(q_input.dtype)
|
||||
module.forward(
|
||||
q_input,
|
||||
q_fp8,
|
||||
weight,
|
||||
weights_out,
|
||||
float(weight_scale),
|
||||
cos_sin_cache,
|
||||
positions,
|
||||
)
|
||||
return q_fp8, weights_out
|
||||
+4
-4
@@ -6,7 +6,7 @@ import torch
|
||||
import triton
|
||||
import triton.language as tl
|
||||
|
||||
from sglang.kernels.ops.attention.dsa_kpool_metadata.scan import bounded_scan_num_splits
|
||||
from sglang.kernels.ops.attention.dsa.kpool_metadata.scan import bounded_scan_num_splits
|
||||
|
||||
|
||||
@triton.jit(
|
||||
@@ -17,7 +17,7 @@ from sglang.kernels.ops.attention.dsa_kpool_metadata.scan import bounded_scan_nu
|
||||
"num_splits",
|
||||
]
|
||||
)
|
||||
def _fused_dsa_decode_metadata_kernel(
|
||||
def _fused_dsa_decode_metadata_kpool_kernel(
|
||||
seq_lens,
|
||||
req_pool_indices,
|
||||
req_to_token,
|
||||
@@ -122,7 +122,7 @@ def _fused_dsa_decode_metadata_kernel(
|
||||
)
|
||||
|
||||
|
||||
def fused_dsa_decode_metadata(
|
||||
def fused_dsa_decode_metadata_kpool(
|
||||
seq_lens: torch.Tensor,
|
||||
req_pool_indices: torch.Tensor,
|
||||
req_to_token: torch.Tensor,
|
||||
@@ -194,7 +194,7 @@ def fused_dsa_decode_metadata(
|
||||
num_splits = bounded_scan_num_splits(bs, num_col_blocks)
|
||||
grid = (1 + bs * num_splits,)
|
||||
|
||||
_fused_dsa_decode_metadata_kernel[grid](
|
||||
_fused_dsa_decode_metadata_kpool_kernel[grid](
|
||||
seq_lens,
|
||||
req_pool_indices,
|
||||
req_to_token,
|
||||
+4
-4
@@ -6,7 +6,7 @@ import torch
|
||||
import triton
|
||||
import triton.language as tl
|
||||
|
||||
from sglang.kernels.ops.attention.dsa_kpool_metadata.scan import bounded_scan_num_splits
|
||||
from sglang.kernels.ops.attention.dsa.kpool_metadata.scan import bounded_scan_num_splits
|
||||
|
||||
|
||||
@triton.jit(
|
||||
@@ -18,7 +18,7 @@ from sglang.kernels.ops.attention.dsa_kpool_metadata.scan import bounded_scan_nu
|
||||
"num_splits",
|
||||
]
|
||||
)
|
||||
def _fused_dsa_draft_extend_metadata_kernel(
|
||||
def _fused_dsa_draft_extend_metadata_kpool_kernel(
|
||||
seq_lens,
|
||||
extend_seq_lens,
|
||||
req_pool_indices,
|
||||
@@ -194,7 +194,7 @@ def _fused_dsa_draft_extend_metadata_kernel(
|
||||
)
|
||||
|
||||
|
||||
def fused_dsa_draft_extend_metadata(
|
||||
def fused_dsa_draft_extend_metadata_kpool(
|
||||
seq_lens: torch.Tensor,
|
||||
extend_seq_lens: torch.Tensor,
|
||||
req_pool_indices: torch.Tensor,
|
||||
@@ -270,7 +270,7 @@ def fused_dsa_draft_extend_metadata(
|
||||
num_splits = bounded_scan_num_splits(bs, num_col_blocks)
|
||||
grid = (1 + bs * num_splits,)
|
||||
|
||||
_fused_dsa_draft_extend_metadata_kernel[grid](
|
||||
_fused_dsa_draft_extend_metadata_kpool_kernel[grid](
|
||||
seq_lens,
|
||||
extend_seq_lens,
|
||||
req_pool_indices,
|
||||
+6
-6
@@ -6,7 +6,7 @@ import torch
|
||||
import triton
|
||||
import triton.language as tl
|
||||
|
||||
from sglang.kernels.ops.attention.dsa_kpool_metadata.scan import bounded_scan_num_splits
|
||||
from sglang.kernels.ops.attention.dsa.kpool_metadata.scan import bounded_scan_num_splits
|
||||
|
||||
|
||||
@triton.jit(
|
||||
@@ -17,7 +17,7 @@ from sglang.kernels.ops.attention.dsa_kpool_metadata.scan import bounded_scan_nu
|
||||
"num_splits",
|
||||
]
|
||||
)
|
||||
def _fused_dsa_target_verify_metadata_kernel(
|
||||
def _fused_dsa_target_verify_metadata_kpool_kernel(
|
||||
seq_lens,
|
||||
req_pool_indices,
|
||||
req_to_token,
|
||||
@@ -157,7 +157,7 @@ def _fused_dsa_target_verify_metadata_kernel(
|
||||
)
|
||||
|
||||
|
||||
def _prep_fused_dsa_target_verify_metadata_launch(
|
||||
def _prep_fused_dsa_target_verify_metadata_kpool_launch(
|
||||
seq_lens: torch.Tensor,
|
||||
req_pool_indices: torch.Tensor,
|
||||
req_to_token: torch.Tensor,
|
||||
@@ -265,7 +265,7 @@ def _prep_fused_dsa_target_verify_metadata_launch(
|
||||
return grid, args, constexprs
|
||||
|
||||
|
||||
def fused_dsa_target_verify_metadata(
|
||||
def fused_dsa_target_verify_metadata_kpool(
|
||||
seq_lens: torch.Tensor,
|
||||
req_pool_indices: torch.Tensor,
|
||||
req_to_token: torch.Tensor,
|
||||
@@ -291,7 +291,7 @@ def fused_dsa_target_verify_metadata(
|
||||
dsa_cu_seqlens_k[:1].zero_()
|
||||
return
|
||||
|
||||
grid, args, constexprs = _prep_fused_dsa_target_verify_metadata_launch(
|
||||
grid, args, constexprs = _prep_fused_dsa_target_verify_metadata_kpool_launch(
|
||||
seq_lens,
|
||||
req_pool_indices,
|
||||
req_to_token,
|
||||
@@ -310,4 +310,4 @@ def fused_dsa_target_verify_metadata(
|
||||
paged_mqa_ctx_lens_2d,
|
||||
index_kpool,
|
||||
)
|
||||
_fused_dsa_target_verify_metadata_kernel[grid](*args, **constexprs)
|
||||
_fused_dsa_target_verify_metadata_kpool_kernel[grid](*args, **constexprs)
|
||||
File diff suppressed because it is too large
Load Diff
@@ -6,14 +6,6 @@ import triton
|
||||
import triton.language as tl
|
||||
|
||||
|
||||
def transform_index_page_table_prefill(**kwargs):
|
||||
return transform_index_page_table_prefill_fast(**kwargs)
|
||||
|
||||
|
||||
def transform_index_page_table_decode(**kwargs):
|
||||
return transform_index_page_table_decode_fast(**kwargs)
|
||||
|
||||
|
||||
@triton.jit
|
||||
def transform_index_page_table_dcp_kernel(
|
||||
page_table_ptr,
|
||||
@@ -270,7 +262,7 @@ def transform_index_page_table_prefill_kernel(
|
||||
)
|
||||
|
||||
|
||||
def transform_index_page_table_decode_fast(
|
||||
def transform_index_page_table_decode(
|
||||
page_table: torch.Tensor,
|
||||
topk_indices: torch.Tensor,
|
||||
result: Optional[torch.Tensor] = None,
|
||||
@@ -322,7 +314,7 @@ def transform_index_page_table_decode_fast(
|
||||
return result
|
||||
|
||||
|
||||
def transform_index_page_table_prefill_fast(
|
||||
def transform_index_page_table_prefill(
|
||||
page_table: torch.Tensor,
|
||||
topk_indices: torch.Tensor,
|
||||
extend_lens_cpu: List[int],
|
||||
|
||||
@@ -1,141 +1,8 @@
|
||||
from typing import Optional, Tuple
|
||||
|
||||
import torch
|
||||
import triton
|
||||
import triton.language as tl
|
||||
|
||||
|
||||
# Triton implementation
|
||||
@triton.jit
|
||||
def _act_quant_kernel(
|
||||
X_ptr,
|
||||
Y_ptr,
|
||||
S_ptr,
|
||||
M,
|
||||
N,
|
||||
group_size: tl.constexpr,
|
||||
round_scale: tl.constexpr,
|
||||
BLOCK_M: tl.constexpr,
|
||||
BLOCK_N: tl.constexpr,
|
||||
):
|
||||
"""
|
||||
Triton kernel for activation quantization.
|
||||
|
||||
Each block processes BLOCK_M rows and group_size columns.
|
||||
"""
|
||||
# Get block IDs
|
||||
pid_m = tl.program_id(0)
|
||||
pid_n = tl.program_id(1)
|
||||
|
||||
# FP8 constants
|
||||
fp8_min = -448.0
|
||||
fp8_max = 448.0
|
||||
fp8_max_inv = 1.0 / fp8_max
|
||||
|
||||
# Calculate row and column offsets
|
||||
row_start = pid_m * BLOCK_M
|
||||
col_start = pid_n * group_size
|
||||
|
||||
# Create offset arrays
|
||||
rows = row_start + tl.arange(0, BLOCK_M)
|
||||
cols = col_start + tl.arange(0, BLOCK_N)
|
||||
|
||||
# Mask for valid rows and columns
|
||||
row_mask = rows < M
|
||||
col_mask = cols < N
|
||||
mask = row_mask[:, None] & col_mask[None, :]
|
||||
|
||||
# Load input data
|
||||
x_ptrs = X_ptr + rows[:, None] * N + cols[None, :]
|
||||
x = tl.load(x_ptrs, mask=mask, other=0.0).to(tl.float32)
|
||||
|
||||
# Compute absolute max along columns (group_size dimension) for each row
|
||||
x_abs = tl.abs(x)
|
||||
amax = tl.max(x_abs, axis=1) # Shape: (BLOCK_M,)
|
||||
|
||||
# Clamp amax to avoid division by zero
|
||||
amax = tl.maximum(amax, 1e-4)
|
||||
|
||||
# Compute scale
|
||||
if round_scale:
|
||||
# Fast round scale using bit manipulation approximation
|
||||
# This is a simplified version - the exact bit manipulation is harder in Triton
|
||||
# Using log2 + ceil + pow2 as approximation
|
||||
log_val = tl.log2(amax * fp8_max_inv)
|
||||
log_ceil = tl.ceil(log_val)
|
||||
scale = tl.exp2(log_ceil)
|
||||
else:
|
||||
scale = amax * fp8_max_inv
|
||||
|
||||
# Quantize: y = clamp(x / scale, fp8_min, fp8_max)
|
||||
scale_broadcast = scale[:, None]
|
||||
y = x / scale_broadcast
|
||||
y = tl.minimum(tl.maximum(y, fp8_min), fp8_max)
|
||||
|
||||
# Store quantized output
|
||||
y_ptrs = Y_ptr + rows[:, None] * N + cols[None, :]
|
||||
tl.store(y_ptrs, y, mask=mask)
|
||||
|
||||
# Store scales
|
||||
s_cols = pid_n
|
||||
s_ptrs = S_ptr + rows * (N // group_size) + s_cols
|
||||
s_mask = row_mask
|
||||
tl.store(s_ptrs, scale, mask=s_mask)
|
||||
|
||||
|
||||
def act_quant(
|
||||
x: torch.Tensor, block_size: int = 128, scale_fmt: Optional[str] = None
|
||||
) -> Tuple[torch.Tensor, torch.Tensor]:
|
||||
"""
|
||||
Quantizes the input tensor `x` using block-wise quantization with Triton.
|
||||
|
||||
Args:
|
||||
x (torch.Tensor): The input tensor to be quantized. Must be contiguous and its last dimension size must be divisible by `block_size`.
|
||||
block_size (int, optional): The size of the blocks to be used for quantization. Default is 128.
|
||||
scale_fmt (Optional[str], optional): The format of the scale. Default is None.
|
||||
Returns:
|
||||
Tuple[torch.Tensor, torch.Tensor]: A tuple containing:
|
||||
- The quantized tensor with dtype `torch.float8_e4m3fn`.
|
||||
- A tensor of scaling factors with dtype `torch.float32`.
|
||||
"""
|
||||
assert x.is_contiguous(), "Input tensor must be contiguous"
|
||||
assert x.size(-1) % block_size == 0, (
|
||||
f"Last dimension size must be divisible by block_size (block_size={block_size})"
|
||||
)
|
||||
|
||||
# Flatten all dims except last
|
||||
N = x.size(-1)
|
||||
x_flat = x.view(-1, N)
|
||||
M = x_flat.size(0)
|
||||
|
||||
# Allocate output tensors
|
||||
y = torch.empty_like(x, dtype=torch.float8_e4m3fn)
|
||||
y_flat = y.view(-1, N)
|
||||
s = x.new_empty(*x.size()[:-1], N // block_size, dtype=torch.float32)
|
||||
s_flat = s.view(-1, N // block_size)
|
||||
|
||||
# Launch kernel
|
||||
BLOCK_M = 32
|
||||
BLOCK_N = block_size
|
||||
grid = (triton.cdiv(M, BLOCK_M), triton.cdiv(N, block_size))
|
||||
round_scale = scale_fmt is not None
|
||||
|
||||
_act_quant_kernel[grid](
|
||||
x_flat,
|
||||
y_flat,
|
||||
s_flat,
|
||||
M,
|
||||
N,
|
||||
group_size=block_size,
|
||||
round_scale=round_scale,
|
||||
BLOCK_M=BLOCK_M,
|
||||
BLOCK_N=BLOCK_N,
|
||||
num_stages=0 if round_scale else 2,
|
||||
)
|
||||
|
||||
return y, s
|
||||
|
||||
|
||||
@triton.jit
|
||||
def _get_valid_kv_indices_kernel(
|
||||
page_table_ptr, # [bs, topk]
|
||||
|
||||
@@ -16,7 +16,7 @@ import torch
|
||||
import triton
|
||||
import triton.language as tl
|
||||
|
||||
from sglang.kernels.ops.kvcache.cache_ops import q8kv8_topk_length_from_indices
|
||||
from sglang.kernels.ops.attention.metadata import q8kv8_topk_length_from_indices
|
||||
from sglang.kernels.ops.quantization.fp8_kernel import is_fp8_fnuz
|
||||
from sglang.srt.utils import get_device_core_count, is_gfx95_supported
|
||||
|
||||
|
||||
@@ -4,25 +4,21 @@
|
||||
from .attn import (
|
||||
fused_store_cache,
|
||||
get_paged_mqa_logits_metadata,
|
||||
triton_create_paged_compress_data,
|
||||
)
|
||||
from .c128_cleanup import clear_unaccepted_c128_draft_states
|
||||
from .compress import (
|
||||
CompressorDecodePlan,
|
||||
CompressorPrefillPlan,
|
||||
compress_forward,
|
||||
compress_norm_rope_store,
|
||||
)
|
||||
from .elementwise import (
|
||||
from .norm_rope import (
|
||||
fused_k_norm_rope_flashmla,
|
||||
fused_norm_rope_inplace,
|
||||
fused_q_indexer_rope_first_quant,
|
||||
fused_q_indexer_rope_hadamard_fp4_quant,
|
||||
fused_q_indexer_rope_hadamard_quant,
|
||||
fused_q_norm_rope,
|
||||
fused_rope_inplace,
|
||||
)
|
||||
from .fp8_wo_a import sglang_per_token_group_quant_fp8_dsv4_wo_a
|
||||
from .topk import (
|
||||
plan_topk_v2,
|
||||
topk_transform_kpool_ragged_v2,
|
||||
@@ -38,19 +34,15 @@ __all__ = [
|
||||
"CompressorPrefillPlan",
|
||||
"compress_forward",
|
||||
"compress_norm_rope_store",
|
||||
"clear_unaccepted_c128_draft_states",
|
||||
"fused_norm_rope_inplace",
|
||||
"fused_store_cache",
|
||||
"fused_rope_inplace",
|
||||
"fused_q_norm_rope",
|
||||
"fused_q_indexer_rope_first_quant",
|
||||
"fused_q_indexer_rope_hadamard_fp4_quant",
|
||||
"fused_q_indexer_rope_hadamard_quant",
|
||||
"fused_k_norm_rope_flashmla",
|
||||
"sglang_per_token_group_quant_fp8_dsv4_wo_a",
|
||||
"make_name",
|
||||
"get_paged_mqa_logits_metadata",
|
||||
"triton_create_paged_compress_data",
|
||||
"topk_transform_paged",
|
||||
"topk_transform_kpool_v2",
|
||||
"topk_transform_kpool_ragged_v2",
|
||||
|
||||
@@ -1,8 +1,6 @@
|
||||
from typing import Literal, Optional, Tuple, Union
|
||||
from typing import Literal, Optional, Union
|
||||
|
||||
import torch
|
||||
import triton
|
||||
import triton.language as tl
|
||||
|
||||
from sglang.kernels.jit.utils import (
|
||||
cache_once,
|
||||
@@ -121,137 +119,3 @@ def fused_store_cache(
|
||||
if freqs_cis.is_complex():
|
||||
freqs_cis = torch.view_as_real(freqs_cis).flatten(-2)
|
||||
module.run_rope(input, cache, indices, freqs_cis.contiguous())
|
||||
|
||||
|
||||
@triton.jit
|
||||
def create_paged_compress_data_kernel(
|
||||
req_pool_indices_ptr,
|
||||
seq_lens_ptr,
|
||||
extend_seq_lens_ptr,
|
||||
req_to_token_ptr,
|
||||
full_to_swa_index_mapping_ptr,
|
||||
out_0_ptr,
|
||||
out_1_ptr,
|
||||
batch_size,
|
||||
stride_req_to_token_0,
|
||||
stride_req_to_token_1: tl.constexpr,
|
||||
stride_out_1_0,
|
||||
stride_out_1_1: tl.constexpr,
|
||||
compress_ratio: tl.constexpr,
|
||||
is_overlap: tl.constexpr,
|
||||
use_req_ring: tl.constexpr,
|
||||
swa_page_size: tl.constexpr,
|
||||
ring_size: tl.constexpr,
|
||||
BLOCK: tl.constexpr,
|
||||
) -> None:
|
||||
pid = tl.program_id(0)
|
||||
offs = pid * BLOCK + tl.arange(0, BLOCK)
|
||||
mask = offs < batch_size
|
||||
|
||||
rid = tl.load(req_pool_indices_ptr + offs, mask=mask, other=0).to(tl.int32)
|
||||
seq_len = tl.load(seq_lens_ptr + offs, mask=mask, other=0).to(tl.int32)
|
||||
extend_len = tl.load(extend_seq_lens_ptr + offs, mask=mask, other=0).to(tl.int32)
|
||||
prefix_len = seq_len - extend_len
|
||||
|
||||
cr = compress_ratio
|
||||
write_pos = ((seq_len - 1) // cr) * cr
|
||||
load_pos = ((prefix_len - 1) // cr) * cr
|
||||
write_overlap_pos = write_pos - cr
|
||||
load_overlap_pos = load_pos - cr
|
||||
v0 = tl.zeros([BLOCK], tl.int32)
|
||||
v1 = tl.zeros([BLOCK], tl.int32)
|
||||
v2 = tl.zeros([BLOCK], tl.int32)
|
||||
v3 = tl.zeros([BLOCK], tl.int32)
|
||||
|
||||
for i in tl.static_range(4):
|
||||
if i == 0:
|
||||
pos = load_pos
|
||||
elif i == 1:
|
||||
pos = write_pos
|
||||
elif i == 2:
|
||||
pos = load_overlap_pos
|
||||
else:
|
||||
pos = write_overlap_pos
|
||||
pos = tl.maximum(pos, 0)
|
||||
if compress_ratio == 128 or use_req_ring:
|
||||
state_loc = rid * ring_size + (pos % ring_size)
|
||||
else:
|
||||
loc = tl.load(
|
||||
req_to_token_ptr
|
||||
+ rid.to(tl.int64) * stride_req_to_token_0
|
||||
+ pos.to(tl.int64) * stride_req_to_token_1,
|
||||
mask=mask,
|
||||
other=0,
|
||||
).to(tl.int32)
|
||||
swa_loc = tl.load(
|
||||
full_to_swa_index_mapping_ptr + loc, mask=mask, other=0
|
||||
).to(tl.int32)
|
||||
swa_page = swa_loc // swa_page_size
|
||||
state_loc = swa_page * ring_size + (swa_loc % ring_size)
|
||||
state_loc = state_loc // cr
|
||||
if i == 0:
|
||||
v0 = state_loc
|
||||
elif i == 1:
|
||||
v1 = state_loc
|
||||
elif i == 2:
|
||||
v2 = state_loc
|
||||
else:
|
||||
v3 = state_loc
|
||||
|
||||
tl.store(out_0_ptr + offs, v1, mask=mask)
|
||||
|
||||
if is_overlap:
|
||||
base = out_1_ptr + offs * stride_out_1_0
|
||||
tl.store(base + 0 * stride_out_1_1, v2, mask=mask)
|
||||
tl.store(base + 1 * stride_out_1_1, v0, mask=mask)
|
||||
tl.store(base + 2 * stride_out_1_1, v3, mask=mask)
|
||||
tl.store(base + 3 * stride_out_1_1, write_pos.to(tl.int32), mask=mask)
|
||||
else:
|
||||
base = out_1_ptr + offs * stride_out_1_0
|
||||
tl.store(base + 0 * stride_out_1_1, v0, mask=mask)
|
||||
|
||||
|
||||
def triton_create_paged_compress_data(
|
||||
*,
|
||||
compress_ratio: int,
|
||||
is_overlap: bool,
|
||||
swa_page_size: int,
|
||||
ring_size: int,
|
||||
req_pool_indices: torch.Tensor,
|
||||
seq_lens: torch.Tensor,
|
||||
extend_seq_lens: torch.Tensor,
|
||||
req_to_token: torch.Tensor,
|
||||
full_to_swa_index_mapping: torch.Tensor,
|
||||
use_req_ring: bool = False,
|
||||
block: int = 128,
|
||||
) -> Tuple[torch.Tensor, torch.Tensor]:
|
||||
batch_size = req_pool_indices.shape[0]
|
||||
out_dim = 4 if is_overlap else 1
|
||||
device_args: dict = dict(device=req_pool_indices.device, dtype=torch.int32)
|
||||
out_0 = torch.empty((batch_size,), **device_args)
|
||||
out_1 = torch.empty((batch_size, out_dim), **device_args)
|
||||
grid = (triton.cdiv(batch_size, block),)
|
||||
create_paged_compress_data_kernel[grid](
|
||||
req_pool_indices,
|
||||
seq_lens,
|
||||
extend_seq_lens,
|
||||
req_to_token,
|
||||
full_to_swa_index_mapping,
|
||||
out_0,
|
||||
out_1,
|
||||
batch_size=batch_size,
|
||||
stride_req_to_token_0=req_to_token.stride(0),
|
||||
stride_req_to_token_1=req_to_token.stride(1), # type: ignore
|
||||
stride_out_1_0=out_1.stride(0),
|
||||
stride_out_1_1=out_1.stride(1), # type: ignore
|
||||
compress_ratio=compress_ratio, # type: ignore
|
||||
is_overlap=1 if is_overlap else 0, # type: ignore
|
||||
use_req_ring=1 if use_req_ring else 0, # type: ignore
|
||||
swa_page_size=swa_page_size, # type: ignore
|
||||
ring_size=ring_size, # type: ignore
|
||||
BLOCK=block, # type: ignore
|
||||
)
|
||||
|
||||
if not is_overlap:
|
||||
out_1.squeeze_(1)
|
||||
return out_0, out_1
|
||||
|
||||
@@ -11,7 +11,7 @@ from sglang.srt.utils import is_gfx95_supported
|
||||
|
||||
fp8_dtype = torch.float8_e4m3fnuz if is_fp8_fnuz() else torch.float8_e4m3fn
|
||||
|
||||
# v4 KV cache layout (see dsv4.index_buf_accessor._set_k_and_s_triton_kernel):
|
||||
# v4 KV cache layout (see kvcache.dsv4_main_kv_store._set_k_and_s_triton_kernel):
|
||||
# per-token: 448 fp8 nope + 64 bf16 rope (= 576 contiguous bytes) +
|
||||
# 7 ue8m0 scales padded to 8 bytes.
|
||||
# per-page: [token 0..P-1 nope+rope (P*576 bytes)] [token 0..P-1 scale (P*8 bytes)]
|
||||
@@ -24,7 +24,7 @@ NOPE_ROPE_BYTES = DIM_NOPE + DIM_ROPE * 2 # 576
|
||||
PADDED_SCALE_PER_TOKEN = NUM_SCALE_TILES + 1 # 8
|
||||
|
||||
|
||||
def dequantize_k_cache_paged(
|
||||
def dequantize_k_cache_paged_dsv4(
|
||||
quant_k_cache: torch.Tensor,
|
||||
page_table_1_flattened: torch.Tensor,
|
||||
page_size: int,
|
||||
@@ -163,7 +163,7 @@ def dequantize_k_cache_paged_v41(
|
||||
return out
|
||||
|
||||
|
||||
def gather_dequant_requant_fp8_paged(
|
||||
def gather_dequant_requant_fp8_paged_dsv4(
|
||||
quant_k_cache: torch.Tensor,
|
||||
page_table_1_flattened: torch.Tensor,
|
||||
page_size: int,
|
||||
@@ -229,74 +229,6 @@ def gather_dequant_requant_fp8_paged(
|
||||
return out
|
||||
|
||||
|
||||
def q8kv8_padded_num_heads(num_heads: int) -> int:
|
||||
"""Return a Q-head count supported by the SM90 Q8KV8 kernel."""
|
||||
if num_heads <= 0:
|
||||
raise ValueError(f"num_heads must be positive, got {num_heads}")
|
||||
if num_heads <= 64:
|
||||
return 64
|
||||
if num_heads <= 128:
|
||||
return 128
|
||||
raise ValueError(
|
||||
"DeepSeek-V4 Q8KV8 sparse prefill supports at most 128 local "
|
||||
f"query heads, got {num_heads}"
|
||||
)
|
||||
|
||||
|
||||
def cast_q_fp8_for_q8kv8_prefill(
|
||||
q: torch.Tensor,
|
||||
padded_num_heads: Optional[int] = None,
|
||||
out: Optional[torch.Tensor] = None,
|
||||
) -> tuple[torch.Tensor, torch.Tensor]:
|
||||
"""Cast DeepSeek-V4 sparse-prefill Q to the Q8KV8 kernel format.
|
||||
|
||||
The incoming Q is the model-produced BF16/FP16 tensor already shaped as
|
||||
``(num_tokens, num_heads, 512)`` after removing the singleton MQA axis.
|
||||
|
||||
The SM90 kernel processes query heads in 64-head blocks. Tensor parallelism
|
||||
commonly leaves fewer than 64 local heads, so the active heads are copied
|
||||
into a zero-padded 64/128-head FP8 tensor.
|
||||
"""
|
||||
assert q.ndim == 3
|
||||
assert q.shape[-1] == DIM_NOPE + DIM_ROPE
|
||||
|
||||
num_tokens, num_heads, head_dim = q.shape
|
||||
if padded_num_heads is None:
|
||||
padded_num_heads = q8kv8_padded_num_heads(num_heads)
|
||||
|
||||
if padded_num_heads not in (64, 128) or padded_num_heads < num_heads:
|
||||
raise ValueError(
|
||||
f"invalid padded_num_heads={padded_num_heads} for num_heads={num_heads}"
|
||||
)
|
||||
|
||||
expected_shape = (num_tokens, padded_num_heads, head_dim)
|
||||
|
||||
if out is None:
|
||||
q_fp8 = torch.zeros(
|
||||
expected_shape,
|
||||
dtype=fp8_dtype,
|
||||
device=q.device,
|
||||
)
|
||||
else:
|
||||
if (
|
||||
out.shape != expected_shape
|
||||
or out.dtype != fp8_dtype
|
||||
or out.device != q.device
|
||||
):
|
||||
raise ValueError(
|
||||
"Q8KV8 Q output must have shape/dtype/device "
|
||||
f"{expected_shape}/{fp8_dtype}/{q.device}, got "
|
||||
f"{tuple(out.shape)}/{out.dtype}/{out.device}"
|
||||
)
|
||||
q_fp8 = out
|
||||
if padded_num_heads > num_heads:
|
||||
q_fp8[:, num_heads:].zero_()
|
||||
|
||||
q_fp8[:, :num_heads].copy_(q)
|
||||
q_scale = torch.ones((), dtype=torch.float32, device=q.device)
|
||||
return q_fp8, q_scale
|
||||
|
||||
|
||||
@triton.jit
|
||||
def _dequantize_k_cache_paged_kernel(
|
||||
output_ptr,
|
||||
|
||||
@@ -7,8 +7,11 @@ from triton.language.extra import libdevice
|
||||
|
||||
from sglang.kernels.jit.utils import aligned_new_empty
|
||||
from sglang.kernels.ops.attention.deep_select import get_input_stride_alignment_bytes
|
||||
from sglang.kernels.ops.attention.dsv4.torch_quant import FP4_AMAX_FLOOR
|
||||
from sglang.srt.runtime_context import get_platform
|
||||
from sglang.kernels.ops.quantization.fp4_fake_quant import FP4_AMAX_FLOOR
|
||||
from sglang.kernels.ops.quantization.fp4_indexer_quant import (
|
||||
_ceil_ue8m0_exp,
|
||||
_fp4_e2m1_code_rne,
|
||||
)
|
||||
|
||||
INDEX_HEAD_DIM = 128
|
||||
# One index-K slot: 64 packed e2m1 bytes and four ue8m0 block exponents.
|
||||
@@ -181,349 +184,6 @@ def finish_flat_indexer_topk(
|
||||
)
|
||||
|
||||
|
||||
@triton.jit
|
||||
def _select_group_value(group, v0, v1, v2, v3):
|
||||
return tl.where(
|
||||
group == 0,
|
||||
v0,
|
||||
tl.where(group == 1, v1, tl.where(group == 2, v2, v3)),
|
||||
)
|
||||
|
||||
|
||||
@triton.jit
|
||||
def _ceil_ue8m0_exp(x):
|
||||
bits = x.to(tl.int32, bitcast=True)
|
||||
exp = (bits >> 23) & 0xFF
|
||||
mantissa = bits & 0x7FFFFF
|
||||
exp += mantissa != 0
|
||||
return tl.minimum(tl.maximum(exp, 1), 254)
|
||||
|
||||
|
||||
@triton.jit
|
||||
def _fp4_e2m1_code(x):
|
||||
ax = tl.minimum(tl.abs(x), 6.0)
|
||||
idx = (ax > 0.25).to(tl.uint8)
|
||||
idx += (ax > 0.75).to(tl.uint8)
|
||||
idx += (ax > 1.25).to(tl.uint8)
|
||||
idx += (ax > 1.75).to(tl.uint8)
|
||||
idx += (ax > 2.5).to(tl.uint8)
|
||||
idx += (ax > 3.5).to(tl.uint8)
|
||||
idx += (ax > 5.0).to(tl.uint8)
|
||||
sign = ((x < 0) & (idx != 0)).to(tl.uint8)
|
||||
return idx | (sign << 3)
|
||||
|
||||
|
||||
@triton.jit
|
||||
def _fp4_e2m1_code_rne(x):
|
||||
"""Round-to-nearest-even e2m1 code, matching the reference rounding."""
|
||||
ax = tl.minimum(tl.abs(x), 6.0)
|
||||
idx = (ax >= 0.25).to(tl.uint8)
|
||||
idx += (ax >= 0.75).to(tl.uint8)
|
||||
idx += (ax >= 1.25).to(tl.uint8)
|
||||
idx += (ax >= 1.75).to(tl.uint8)
|
||||
idx += (ax >= 2.5).to(tl.uint8)
|
||||
idx += (ax >= 3.5).to(tl.uint8)
|
||||
idx += (ax >= 5.0).to(tl.uint8)
|
||||
# Round-half-to-even: an odd index at an exact boundary drops to the even one.
|
||||
is_boundary = (
|
||||
(ax == 0.25)
|
||||
| (ax == 0.75)
|
||||
| (ax == 1.25)
|
||||
| (ax == 1.75)
|
||||
| (ax == 2.5)
|
||||
| (ax == 3.5)
|
||||
| (ax == 5.0)
|
||||
)
|
||||
idx = tl.where(is_boundary & ((idx & 1) == 1), idx - 1, idx)
|
||||
sign = ((x < 0) & (idx != 0)).to(tl.uint8)
|
||||
return idx | (sign << 3)
|
||||
|
||||
|
||||
@triton.jit
|
||||
def quantize_fp4_indexer_row(
|
||||
values,
|
||||
v0,
|
||||
v1,
|
||||
BLOCK_N: tl.constexpr,
|
||||
GROUP_N: tl.constexpr,
|
||||
RNE: tl.constexpr,
|
||||
):
|
||||
"""One row of quantize_fp4_indexer_tensor: values is the fp32 [BLOCK_N] row,
|
||||
v0 / v1 its even / odd elements as fp32 [BLOCK_N // 2]. Returns
|
||||
(packed codes [BLOCK_N // 2], packed ue8m0 exponents), the two stores of the
|
||||
standalone kernel below."""
|
||||
offs = tl.arange(0, BLOCK_N)
|
||||
abs_values = tl.abs(values)
|
||||
|
||||
amax0 = tl.max(tl.where(offs < GROUP_N, abs_values, 0.0), axis=0)
|
||||
amax1 = tl.max(
|
||||
tl.where((GROUP_N <= offs) & (offs < 2 * GROUP_N), abs_values, 0.0),
|
||||
axis=0,
|
||||
)
|
||||
amax2 = tl.max(
|
||||
tl.where((2 * GROUP_N <= offs) & (offs < 3 * GROUP_N), abs_values, 0.0),
|
||||
axis=0,
|
||||
)
|
||||
amax3 = tl.max(tl.where(3 * GROUP_N <= offs, abs_values, 0.0), axis=0)
|
||||
|
||||
sf0 = tl.maximum(amax0 / 6.0, 1.0e-4)
|
||||
sf1 = tl.maximum(amax1 / 6.0, 1.0e-4)
|
||||
sf2 = tl.maximum(amax2 / 6.0, 1.0e-4)
|
||||
sf3 = tl.maximum(amax3 / 6.0, 1.0e-4)
|
||||
|
||||
exp0 = _ceil_ue8m0_exp(sf0)
|
||||
exp1 = _ceil_ue8m0_exp(sf1)
|
||||
exp2 = _ceil_ue8m0_exp(sf2)
|
||||
exp3 = _ceil_ue8m0_exp(sf3)
|
||||
|
||||
packed_sf = exp0 | (exp1 << 8) | (exp2 << 16) | (exp3 << 24)
|
||||
|
||||
pair_offsets = tl.arange(0, BLOCK_N // 2)
|
||||
offs0 = pair_offsets * 2
|
||||
offs1 = offs0 + 1
|
||||
group0 = offs0 // GROUP_N
|
||||
group1 = offs1 // GROUP_N
|
||||
scale_exp0 = _select_group_value(group0, exp0, exp1, exp2, exp3)
|
||||
scale_exp1 = _select_group_value(group1, exp0, exp1, exp2, exp3)
|
||||
scale0 = (scale_exp0 << 23).to(tl.float32, bitcast=True)
|
||||
scale1 = (scale_exp1 << 23).to(tl.float32, bitcast=True)
|
||||
|
||||
v0 = v0 / scale0
|
||||
v1 = v1 / scale1
|
||||
if RNE:
|
||||
code0 = _fp4_e2m1_code_rne(v0)
|
||||
code1 = _fp4_e2m1_code_rne(v1)
|
||||
else:
|
||||
code0 = _fp4_e2m1_code(v0)
|
||||
code1 = _fp4_e2m1_code(v1)
|
||||
packed = (code0 & 0x0F) | ((code1 & 0x0F) << 4)
|
||||
return packed, packed_sf
|
||||
|
||||
|
||||
@triton.jit
|
||||
def _quantize_fp4_indexer_kernel(
|
||||
x,
|
||||
x_fp4,
|
||||
x_sf,
|
||||
BLOCK_N: tl.constexpr,
|
||||
GROUP_N: tl.constexpr,
|
||||
RNE: tl.constexpr,
|
||||
):
|
||||
token_id = tl.program_id(0)
|
||||
offs = tl.arange(0, BLOCK_N)
|
||||
pair_offsets = tl.arange(0, BLOCK_N // 2)
|
||||
row = x + token_id * BLOCK_N
|
||||
values = tl.load(row + offs).to(tl.float32)
|
||||
v0 = tl.load(row + pair_offsets * 2).to(tl.float32)
|
||||
v1 = tl.load(row + pair_offsets * 2 + 1).to(tl.float32)
|
||||
packed, packed_sf = quantize_fp4_indexer_row(
|
||||
values, v0, v1, BLOCK_N=BLOCK_N, GROUP_N=GROUP_N, RNE=RNE
|
||||
)
|
||||
tl.store(x_sf + token_id, packed_sf)
|
||||
tl.store(x_fp4 + token_id * (BLOCK_N // 2) + pair_offsets, packed)
|
||||
|
||||
|
||||
@triton.jit
|
||||
def _quantize_fp4_indexer_rows(
|
||||
x,
|
||||
x_fp4,
|
||||
x_sf,
|
||||
M,
|
||||
BLOCK_M: tl.constexpr,
|
||||
BLOCK_N: tl.constexpr,
|
||||
GROUP_N: tl.constexpr,
|
||||
RNE: tl.constexpr,
|
||||
):
|
||||
tl.static_assert(BLOCK_N == 128 and GROUP_N == 32)
|
||||
# Each reduction covers one scale group. Keep its values for packing,
|
||||
# avoiding four masked full-row reductions and a second input load.
|
||||
group = tl.program_id(0) * BLOCK_M * 4 + tl.arange(0, BLOCK_M * 4)
|
||||
offs = tl.arange(0, GROUP_N)
|
||||
values = tl.load(
|
||||
x + group[:, None].to(tl.int64) * GROUP_N + offs[None, :],
|
||||
group[:, None] < M * 4,
|
||||
0,
|
||||
).to(tl.float32)
|
||||
amax = tl.max(tl.abs(values), axis=1)
|
||||
exp = _ceil_ue8m0_exp(tl.maximum(amax / 6.0, 1.0e-4))
|
||||
scale = (exp << 23).to(tl.float32, bitcast=True)
|
||||
v0, v1 = tl.split(
|
||||
tl.reshape(values / scale[:, None], (BLOCK_M * 4, GROUP_N // 2, 2))
|
||||
)
|
||||
if RNE:
|
||||
code0 = _fp4_e2m1_code_rne(v0)
|
||||
code1 = _fp4_e2m1_code_rne(v1)
|
||||
else:
|
||||
code0 = _fp4_e2m1_code(v0)
|
||||
code1 = _fp4_e2m1_code(v1)
|
||||
packed = (code0 & 0x0F) | ((code1 & 0x0F) << 4)
|
||||
tl.store(
|
||||
x_fp4
|
||||
+ group[:, None].to(tl.int64) * (GROUP_N // 2)
|
||||
+ tl.arange(0, GROUP_N // 2)[None, :],
|
||||
packed,
|
||||
group[:, None] < M * 4,
|
||||
)
|
||||
# The four exponents occupy disjoint bytes, so integer sum packs them.
|
||||
shifts = tl.arange(0, 4) * 8
|
||||
packed_sf = tl.sum(
|
||||
tl.reshape(exp.to(tl.uint32), (BLOCK_M, 4)) << shifts[None, :], axis=1
|
||||
)
|
||||
token_id = tl.program_id(0) * BLOCK_M + tl.arange(0, BLOCK_M)
|
||||
tl.store(x_sf + token_id, packed_sf.to(tl.int32), token_id < M)
|
||||
|
||||
|
||||
@triton.jit
|
||||
def _store_fp4_index_k_cache_kernel(
|
||||
k_fp4,
|
||||
k_sf,
|
||||
cache,
|
||||
loc,
|
||||
page_size: tl.constexpr,
|
||||
cache_stride: tl.constexpr,
|
||||
BLOCK: tl.constexpr,
|
||||
):
|
||||
token_id = tl.program_id(0)
|
||||
offsets = tl.arange(0, BLOCK)
|
||||
cache_loc = tl.load(loc + token_id)
|
||||
page = cache_loc // page_size
|
||||
page_offset = cache_loc - page * page_size
|
||||
|
||||
k = tl.load(k_fp4 + token_id * BLOCK + offsets)
|
||||
tl.store(cache + page * cache_stride + page_offset * BLOCK + offsets, k)
|
||||
|
||||
sf = tl.load(k_sf + token_id)
|
||||
sf_offsets = tl.arange(0, 4)
|
||||
sf_bytes = (sf >> (sf_offsets * 8)) & 0xFF
|
||||
tl.store(
|
||||
cache + page * cache_stride + page_size * BLOCK + page_offset * 4 + sf_offsets,
|
||||
sf_bytes,
|
||||
)
|
||||
|
||||
|
||||
def quantize_fp4_indexer_tensor(
|
||||
x: torch.Tensor, rne: bool = False
|
||||
) -> tuple[torch.Tensor, torch.Tensor]:
|
||||
"""Per-32 ue8m0 fp4 quantize. rne=True uses round-to-nearest-even (the dsv41
|
||||
reference rounding); the default keeps ``_fp4_e2m1_code``'s thresholds."""
|
||||
assert x.shape[-1] == 128
|
||||
x = x.contiguous().view(-1, x.shape[-1])
|
||||
x_fp4 = torch.empty((x.shape[0], 64), device=x.device, dtype=torch.int8)
|
||||
x_sf = torch.empty((x.shape[0],), device=x.device, dtype=torch.int32)
|
||||
if x.shape[0] >= 4096 and get_platform().is_blackwell:
|
||||
# Independent rows share a CTA to avoid one block per 128 values.
|
||||
_quantize_fp4_indexer_rows[(triton.cdiv(x.shape[0], 8),)](
|
||||
x,
|
||||
x_fp4,
|
||||
x_sf,
|
||||
x.shape[0],
|
||||
8,
|
||||
BLOCK_N=128,
|
||||
GROUP_N=32,
|
||||
RNE=rne,
|
||||
num_warps=4,
|
||||
)
|
||||
elif x.shape[0] > 0:
|
||||
_quantize_fp4_indexer_kernel[(x.shape[0],)](
|
||||
x,
|
||||
x_fp4,
|
||||
x_sf,
|
||||
BLOCK_N=128,
|
||||
GROUP_N=32,
|
||||
RNE=rne,
|
||||
)
|
||||
return x_fp4, x_sf
|
||||
|
||||
|
||||
def store_fp4_index_k_cache(
|
||||
input: torch.Tensor,
|
||||
cache: torch.Tensor,
|
||||
loc: torch.Tensor,
|
||||
*,
|
||||
page_size: int,
|
||||
rne: bool = False,
|
||||
) -> None:
|
||||
assert input.shape[-1] == 128
|
||||
k_fp4, k_sf = quantize_fp4_indexer_tensor(input.contiguous(), rne=rne)
|
||||
n_tokens = input.numel() // input.shape[-1]
|
||||
assert k_fp4.shape == (n_tokens, 64)
|
||||
assert k_sf.shape == (n_tokens,)
|
||||
assert cache.shape[1] == page_size * INDEX_K_SLOT_BYTES
|
||||
|
||||
if n_tokens == 0:
|
||||
return
|
||||
_store_fp4_index_k_cache_kernel[(n_tokens,)](
|
||||
k_fp4.view(torch.uint8),
|
||||
k_sf,
|
||||
cache,
|
||||
loc,
|
||||
page_size,
|
||||
cache.stride(0),
|
||||
BLOCK=64,
|
||||
)
|
||||
|
||||
|
||||
@triton.jit
|
||||
def _gather_fp4_index_k_kernel(
|
||||
cache,
|
||||
cache_stride,
|
||||
slots,
|
||||
payload,
|
||||
scales,
|
||||
n,
|
||||
page_size: tl.constexpr,
|
||||
BLOCK_ROWS: tl.constexpr,
|
||||
):
|
||||
# int32 view of a page: page_size * 16 payload words, then page_size scale words.
|
||||
rows = tl.program_id(0) * BLOCK_ROWS + tl.arange(0, BLOCK_ROWS)
|
||||
valid = rows < n
|
||||
slot = tl.load(slots + rows, valid, 0).to(tl.int64)
|
||||
page = slot // page_size
|
||||
offset = slot - page * page_size
|
||||
page_base = cache + page * cache_stride
|
||||
words = tl.arange(0, 16)
|
||||
payload_words = tl.load(
|
||||
page_base[:, None] + offset[:, None] * 16 + words[None, :], valid[:, None]
|
||||
)
|
||||
tl.store(
|
||||
payload + rows[:, None].to(tl.int64) * 16 + words[None, :],
|
||||
payload_words,
|
||||
valid[:, None],
|
||||
)
|
||||
tl.store(scales + rows, tl.load(page_base + page_size * 16 + offset, valid), valid)
|
||||
|
||||
|
||||
def gather_fp4_index_k(
|
||||
cache: torch.Tensor, slots: torch.Tensor, *, page_size: int
|
||||
) -> tuple[torch.Tensor, torch.Tensor]:
|
||||
"""Packed fp4 rows at ``slots`` of the index-K ``cache`` (``[pages, page_size *
|
||||
68]`` uint8, ``store_fp4_index_k_cache``'s layout): (payload int8 ``[n, 64]``,
|
||||
ue8m0 scales packed int32 ``[n]``), the layout ``quantize_fp4_indexer_tensor``
|
||||
returns."""
|
||||
assert cache.shape[1] == page_size * INDEX_K_SLOT_BYTES and cache.stride(1) == 1
|
||||
# The kernel addresses slots as `slots + rows`, so a strided view would be read
|
||||
# as if its elements were packed.
|
||||
assert slots.is_contiguous()
|
||||
words = cache.view(torch.int32)
|
||||
n = slots.shape[0]
|
||||
payload = torch.empty((n, 16), device=cache.device, dtype=torch.int32)
|
||||
scales = torch.empty((n,), device=cache.device, dtype=torch.int32)
|
||||
if n > 0:
|
||||
block_rows = 64
|
||||
_gather_fp4_index_k_kernel[(triton.cdiv(n, block_rows),)](
|
||||
words,
|
||||
words.stride(0),
|
||||
slots,
|
||||
payload,
|
||||
scales,
|
||||
n,
|
||||
page_size,
|
||||
BLOCK_ROWS=block_rows,
|
||||
num_warps=4,
|
||||
)
|
||||
return payload.view(torch.int8), scales
|
||||
|
||||
|
||||
@triton.jit
|
||||
def _index_k_rope_pack_kernel(
|
||||
X,
|
||||
|
||||
@@ -11,16 +11,17 @@ import triton
|
||||
import triton.language as tl
|
||||
from triton.language.extra import libdevice
|
||||
|
||||
from sglang.kernels.ops.attention.dsv4.fp4_indexer import (
|
||||
_ceil_ue8m0_exp,
|
||||
quantize_fp4_indexer_row,
|
||||
quantize_fp4_indexer_tensor,
|
||||
)
|
||||
from sglang.kernels.ops.attention.dsv4.fp4_rope_fake_quant import (
|
||||
FP4_AMAX_FLOOR,
|
||||
rope_tail_fake_quant_fp4_row,
|
||||
)
|
||||
from sglang.kernels.ops.gemm.router_gemv_hip import rocm_router_gemv_split_k
|
||||
from sglang.kernels.ops.gemm.dsv4_indexer_head_weights_hip import (
|
||||
_reduce_scale_bf16_block,
|
||||
)
|
||||
from sglang.kernels.ops.quantization.fp4_indexer_quant import (
|
||||
_ceil_ue8m0_exp,
|
||||
quantize_fp4_indexer_row,
|
||||
)
|
||||
from sglang.srt.environ import envs
|
||||
|
||||
if TYPE_CHECKING:
|
||||
@@ -811,94 +812,6 @@ def aiter_k_indexer_fp4_cache_write(
|
||||
)
|
||||
|
||||
|
||||
# FlyDSL split payload / scale layout; bytes equal AITER's fp4 writer up to the sign of zero
|
||||
@triton.jit
|
||||
def _store_fp4_index_k_split_kernel(
|
||||
k_fp4,
|
||||
k_sf,
|
||||
payload,
|
||||
scale,
|
||||
loc,
|
||||
page_size: tl.constexpr,
|
||||
BLOCK: tl.constexpr,
|
||||
PAGE8: tl.constexpr,
|
||||
):
|
||||
token_id = tl.program_id(0)
|
||||
cache_loc = tl.load(loc + token_id)
|
||||
page = cache_loc // page_size
|
||||
page_offset = cache_loc - page * page_size
|
||||
|
||||
# payload [page, 1, 4 chunks, page_size, 16 bytes]: chunk c holds dims [32c, 32c + 32).
|
||||
offsets = tl.arange(0, BLOCK)
|
||||
chunk = offsets // 16
|
||||
byte = offsets - chunk * 16
|
||||
k = tl.load(k_fp4 + token_id * BLOCK + offsets)
|
||||
sf = tl.load(k_sf + token_id)
|
||||
sf_offsets = tl.arange(0, 4)
|
||||
sf_bytes = ((sf >> (sf_offsets * 8)) & 0xFF).to(tl.uint8)
|
||||
if PAGE8:
|
||||
# aiter's row-group 8-slot page: payload [t % 4][chunk][t // 4][16 B], scales [chunk][t]
|
||||
page8 = cache_loc // 8
|
||||
t = cache_loc % 8
|
||||
tl.store(
|
||||
payload
|
||||
+ page8 * (8 * BLOCK)
|
||||
+ ((t % 4) * 4 + chunk) * 32
|
||||
+ (t // 4) * 16
|
||||
+ byte,
|
||||
k,
|
||||
)
|
||||
tl.store(scale + page8 * 32 + sf_offsets * 8 + t, sf_bytes)
|
||||
return
|
||||
tl.store(
|
||||
payload
|
||||
+ page * (4 * page_size * 16)
|
||||
+ chunk * (page_size * 16)
|
||||
+ page_offset * 16
|
||||
+ byte,
|
||||
k,
|
||||
)
|
||||
# scale [page, 1, 4, page_size]: the slot axis is a 16 x 4 tile transposed (the FlyDSL K ABI)
|
||||
shuffled = (page_offset % 16) * 4 + page_offset // 16
|
||||
tl.store(
|
||||
scale + page * (4 * page_size) + sf_offsets * page_size + shuffled, sf_bytes
|
||||
)
|
||||
|
||||
|
||||
def store_fp4_index_k_cache_split(
|
||||
input: torch.Tensor,
|
||||
payload: torch.Tensor,
|
||||
scale: torch.Tensor,
|
||||
loc: torch.Tensor,
|
||||
*,
|
||||
page_size: int,
|
||||
rne: bool = False,
|
||||
page8: bool = False,
|
||||
) -> None:
|
||||
"""Quantize input [n, 128] to fp4 (per-32 ue8m0) and scatter row i to slot
|
||||
loc[i] of the split FlyDSL K layout (payload [pages, 1, 4, page_size, 16],
|
||||
scale [pages, 1, 4, page_size] with the slot axis 16 x 4 transposed); with page8
|
||||
each page holds eight 8-slot pages of aiter's row-group layout (pack_kv_cache)."""
|
||||
assert input.shape[-1] == _HEAD_DIM
|
||||
assert page_size == _KV_BLOCK_SIZE, "the 16 x 4 slot transpose spans a 64-slot page"
|
||||
assert payload.shape[1:] == (1, 4, page_size, 16), payload.shape
|
||||
assert scale.shape[1:] == (1, 4, page_size), scale.shape
|
||||
k_fp4, k_sf = quantize_fp4_indexer_tensor(input.contiguous(), rne=rne)
|
||||
n_tokens = k_fp4.shape[0]
|
||||
if n_tokens == 0:
|
||||
return
|
||||
_store_fp4_index_k_split_kernel[(n_tokens,)](
|
||||
k_fp4.view(torch.uint8),
|
||||
k_sf,
|
||||
payload.view(torch.uint8),
|
||||
scale,
|
||||
loc,
|
||||
page_size,
|
||||
BLOCK=64,
|
||||
PAGE8=page8,
|
||||
)
|
||||
|
||||
|
||||
def index_k_page8_views(
|
||||
payload: torch.Tensor, scale: torch.Tensor
|
||||
) -> Tuple[torch.Tensor, torch.Tensor]:
|
||||
@@ -911,36 +824,6 @@ def index_k_page8_views(
|
||||
)
|
||||
|
||||
|
||||
def read_fp4_index_k_split(
|
||||
payload: torch.Tensor,
|
||||
scale: torch.Tensor,
|
||||
slots: torch.Tensor,
|
||||
*,
|
||||
page_size: int,
|
||||
page8: bool = False,
|
||||
) -> Tuple[torch.Tensor, torch.Tensor]:
|
||||
"""Inverse of store_fp4_index_k_cache_split: (payload int8 [n, 64], scales
|
||||
int32 [n] with chunk c's e8m0 byte at bits 8c..8c+7), the layout of
|
||||
quantize_fp4_indexer_tensor."""
|
||||
assert page_size == _KV_BLOCK_SIZE, "the 16 x 4 slot transpose spans a 64-slot page"
|
||||
slots = slots.to(torch.int64)
|
||||
if page8:
|
||||
kv8, ks8 = index_k_page8_views(payload, scale)
|
||||
page, t = slots // 8, slots % 8
|
||||
kv8 = kv8.view(-1, 4, 4, 2, 16) # [page8, t % 4, chunk, t // 4, 16]
|
||||
rows = kv8[page, t % 4, :, t // 4, :].reshape(-1, 64).view(torch.int8)
|
||||
sf = ks8.view(-1, 4, 8)[page, :, t].to(torch.int32) # [n, 4]
|
||||
packed = sf[:, 0] | (sf[:, 1] << 8) | (sf[:, 2] << 16) | (sf[:, 3] << 24)
|
||||
return rows, packed
|
||||
page, off = slots // page_size, slots % page_size
|
||||
rows = payload.view(torch.uint8)[page, 0, :, off, :] # [n, 4, 16]
|
||||
rows = rows.reshape(-1, 64).view(torch.int8)
|
||||
shuffled = (off % 16) * 4 + off // 16
|
||||
sf = scale[page, 0, :, shuffled].to(torch.int32) # [n, 4]
|
||||
packed = sf[:, 0] | (sf[:, 1] << 8) | (sf[:, 2] << 16) | (sf[:, 3] << 24)
|
||||
return rows, packed
|
||||
|
||||
|
||||
@triton.jit
|
||||
def _quantize_fp4_query_flydsl_kernel(
|
||||
x,
|
||||
@@ -997,63 +880,6 @@ def pack_fp4_query_flydsl(q: torch.Tensor) -> Tuple[torch.Tensor, torch.Tensor]:
|
||||
return q_fp4, q_scale
|
||||
|
||||
|
||||
@triton.jit
|
||||
def _reduce_scale_bf16_block(
|
||||
block,
|
||||
part_ptr,
|
||||
out_ptr,
|
||||
M,
|
||||
stride_ps,
|
||||
stride_pm,
|
||||
stride_om,
|
||||
scale,
|
||||
N: tl.constexpr,
|
||||
SPLIT_K: tl.constexpr,
|
||||
BLOCK: tl.constexpr,
|
||||
):
|
||||
"""One BLOCK of _reduce_scale_bf16_kernel (shared with the fused index-Q launch)."""
|
||||
offs = block * BLOCK + tl.arange(0, BLOCK)
|
||||
m = offs // N
|
||||
n = offs % N
|
||||
mask = m < M
|
||||
acc = tl.load(part_ptr + m * stride_pm + n, mask=mask, other=0.0)
|
||||
for s in tl.static_range(1, SPLIT_K):
|
||||
acc += tl.load(
|
||||
part_ptr + s * stride_ps + m * stride_pm + n, mask=mask, other=0.0
|
||||
)
|
||||
# bf16(bf16(sum) * scale): the linear's bf16 output, then the aten multiply rounded to bf16
|
||||
w = acc.to(tl.bfloat16).to(tl.float32) * scale
|
||||
tl.store(out_ptr + m * stride_om + n, w.to(tl.bfloat16), mask=mask)
|
||||
|
||||
|
||||
@triton.jit
|
||||
def _reduce_scale_bf16_kernel(
|
||||
part_ptr,
|
||||
out_ptr,
|
||||
M,
|
||||
stride_ps,
|
||||
stride_pm,
|
||||
stride_om,
|
||||
scale,
|
||||
N: tl.constexpr,
|
||||
SPLIT_K: tl.constexpr,
|
||||
BLOCK: tl.constexpr,
|
||||
):
|
||||
_reduce_scale_bf16_block(
|
||||
tl.program_id(0),
|
||||
part_ptr,
|
||||
out_ptr,
|
||||
M,
|
||||
stride_ps,
|
||||
stride_pm,
|
||||
stride_om,
|
||||
scale,
|
||||
N=N,
|
||||
SPLIT_K=SPLIT_K,
|
||||
BLOCK=BLOCK,
|
||||
)
|
||||
|
||||
|
||||
@triton.jit
|
||||
def _fp4_e2m1_code_rne_lean(x):
|
||||
"""_fp4_e2m1_code_rne for every fp32 x (checked over all 2**32): within each spacing of the
|
||||
@@ -1347,29 +1173,3 @@ def index_q_rope_pack_weights_flydsl(
|
||||
num_warps=4,
|
||||
)
|
||||
return q_fp4, q_scale, weights
|
||||
|
||||
|
||||
def indexer_head_weights(
|
||||
x: torch.Tensor, weight: torch.Tensor, scale: float
|
||||
) -> torch.Tensor:
|
||||
"""bf16(bf16(x @ weight.T) * scale) as a contiguous bf16 [M, N], the layout the
|
||||
FlyDSL logits kernels take. x bf16 [M, K] and weight bf16 [N, K], with
|
||||
M <= rocm_gemv_split_k_max_tokens(n=N, k=K, weight_dtype=torch.bfloat16)."""
|
||||
partials = rocm_router_gemv_split_k(x, weight)
|
||||
split_k, M, N = partials.shape
|
||||
out = torch.empty((M, N), dtype=torch.bfloat16, device=x.device)
|
||||
block = triton.next_power_of_2(M * N)
|
||||
_reduce_scale_bf16_kernel[(triton.cdiv(M * N, block),)](
|
||||
partials,
|
||||
out,
|
||||
M,
|
||||
partials.stride(0),
|
||||
partials.stride(1),
|
||||
out.stride(0),
|
||||
float(scale),
|
||||
N=N,
|
||||
SPLIT_K=split_k,
|
||||
BLOCK=block,
|
||||
num_warps=4,
|
||||
)
|
||||
return out
|
||||
|
||||
@@ -9,7 +9,7 @@ import triton
|
||||
import triton.language as tl
|
||||
from triton.language.extra import libdevice
|
||||
|
||||
from sglang.kernels.ops.attention.dsv4.torch_quant import FP4_AMAX_FLOOR
|
||||
from sglang.kernels.ops.quantization.fp4_fake_quant import FP4_AMAX_FLOOR
|
||||
|
||||
|
||||
@triton.jit
|
||||
|
||||
+1
-41
@@ -126,21 +126,6 @@ def _jit_main_q_indexer_rope_hadamard_quant_module(dtype: torch.dtype):
|
||||
)
|
||||
|
||||
|
||||
# V3.2 lays q out as [rope | nope] (V4 is [nope | rope]) -> kRopeFirst=true, and
|
||||
# drops the Hadamard rotation (kHadamard=false).
|
||||
@cache_once
|
||||
def _jit_main_q_indexer_rope_first_quant_module(dtype: torch.dtype):
|
||||
args = make_cpp_args(dtype, is_arch_support_pdl(), True, False)
|
||||
return load_jit(
|
||||
make_name("main_q_indexer_rope_first_quant"),
|
||||
*args,
|
||||
cuda_files=["deepseek_v4/main_norm_rope.cuh"],
|
||||
cuda_wrappers=[
|
||||
("forward", f"FusedQIndexerRopeHadamardQuantKernel<{args}>::forward"),
|
||||
],
|
||||
)
|
||||
|
||||
|
||||
@cache_once
|
||||
def _jit_main_q_indexer_rope_hadamard_fp4_quant_module(dtype: torch.dtype):
|
||||
args = make_cpp_args(dtype, is_arch_support_pdl())
|
||||
@@ -171,7 +156,7 @@ def fused_rope_inplace(
|
||||
inverse: if True, apply inverse rotation (conjugate freqs)
|
||||
"""
|
||||
if _is_hip or _is_xpu:
|
||||
from sglang.kernels.ops.attention.deepseek_v4_rope import (
|
||||
from sglang.kernels.ops.attention.dsv4.rope import (
|
||||
apply_rotary_emb_triton,
|
||||
)
|
||||
|
||||
@@ -283,31 +268,6 @@ def fused_q_indexer_rope_hadamard_quant(
|
||||
return q_fp8, weights_out
|
||||
|
||||
|
||||
def fused_q_indexer_rope_first_quant(
|
||||
q_input: torch.Tensor,
|
||||
weight: torch.Tensor,
|
||||
weight_scale: float,
|
||||
cos_sin_cache: torch.Tensor,
|
||||
positions: torch.Tensor,
|
||||
) -> Tuple[torch.Tensor, torch.Tensor]:
|
||||
"""DeepSeek-V3.2 only. Indexer Q: RoPE on the leading dims + fp8 act-quant. CUDA only."""
|
||||
q_fp8 = torch.empty(q_input.shape, dtype=torch.float8_e4m3fn, device=q_input.device)
|
||||
weights_out = torch.empty(
|
||||
(*q_input.shape[:-1], 1), dtype=torch.float32, device=q_input.device
|
||||
)
|
||||
module = _jit_main_q_indexer_rope_first_quant_module(q_input.dtype)
|
||||
module.forward(
|
||||
q_input,
|
||||
q_fp8,
|
||||
weight,
|
||||
weights_out,
|
||||
float(weight_scale),
|
||||
cos_sin_cache,
|
||||
positions,
|
||||
)
|
||||
return q_fp8, weights_out
|
||||
|
||||
|
||||
def fused_q_indexer_rope_hadamard_fp4_quant(
|
||||
q_input: torch.Tensor,
|
||||
weight: torch.Tensor,
|
||||
@@ -1,13 +1,11 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
from typing import TYPE_CHECKING, Any, List, Optional
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
import torch
|
||||
|
||||
from sglang.kernels.jit.utils import cache_once, load_jit, make_cpp_args
|
||||
from sglang.kernels.ops.attention.dsv4.utils import make_name
|
||||
from sglang.srt.environ import envs
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from tvm_ffi.module import Module
|
||||
@@ -34,244 +32,84 @@ def _jit_online_c128_mtp_module(
|
||||
)
|
||||
|
||||
|
||||
@dataclass
|
||||
class _OnlineC128LayerRuntime:
|
||||
head_dim: int
|
||||
main_state: torch.Tensor
|
||||
state_dtype: torch.dtype
|
||||
state_slot_offset: int
|
||||
def online_c128_mtp_write_prefix_states(
|
||||
kv_score_input: torch.Tensor,
|
||||
seq_lens: torch.Tensor,
|
||||
req_pool_indices: torch.Tensor,
|
||||
req_to_token: torch.Tensor,
|
||||
ape: torch.Tensor,
|
||||
state: torch.Tensor,
|
||||
batch_size: int,
|
||||
num_verify_tokens: int,
|
||||
state_slot_stride: int,
|
||||
*,
|
||||
head_dim: int,
|
||||
) -> None:
|
||||
"""Stage the c128 compressor state after each verify token in that token's
|
||||
draft state slot."""
|
||||
_jit_online_c128_mtp_module(
|
||||
head_dim, seq_lens.dtype, req_pool_indices.dtype, state.dtype
|
||||
).write_prefix_states(
|
||||
kv_score_input,
|
||||
seq_lens,
|
||||
req_pool_indices,
|
||||
req_to_token,
|
||||
ape,
|
||||
state,
|
||||
batch_size,
|
||||
num_verify_tokens,
|
||||
state_slot_stride,
|
||||
)
|
||||
|
||||
|
||||
@dataclass
|
||||
class _OnlineC128VerifyContext:
|
||||
req_pool_indices: torch.Tensor
|
||||
seq_lens: torch.Tensor
|
||||
def online_c128_mtp_mark_pending(
|
||||
seq_lens: torch.Tensor,
|
||||
req_pool_indices: torch.Tensor,
|
||||
pending_seq_lens: torch.Tensor,
|
||||
batch_size: int,
|
||||
num_req_slots: int,
|
||||
*,
|
||||
head_dim: int,
|
||||
state_dtype: torch.dtype,
|
||||
) -> None:
|
||||
"""Reset all pending verify lengths, then record ``seq_lens`` for the requests
|
||||
of this verify batch."""
|
||||
_jit_online_c128_mtp_module(
|
||||
head_dim, seq_lens.dtype, req_pool_indices.dtype, state_dtype
|
||||
).mark_pending(
|
||||
seq_lens,
|
||||
req_pool_indices,
|
||||
pending_seq_lens,
|
||||
batch_size,
|
||||
num_req_slots,
|
||||
)
|
||||
|
||||
|
||||
class OnlineC128MTPController:
|
||||
def __init__(self, backend: Any):
|
||||
self.backend = backend
|
||||
self._verify_ctx: Optional[_OnlineC128VerifyContext] = None
|
||||
self._layer_runtimes: Optional[List[_OnlineC128LayerRuntime]] = None
|
||||
|
||||
def enabled(self) -> bool:
|
||||
return (
|
||||
envs.SGLANG_OPT_USE_ONLINE_COMPRESS.get()
|
||||
and envs.SGLANG_EXPERIMENTAL_ONLINE_C128_MTP.get()
|
||||
and self.backend.mtp_enabled
|
||||
)
|
||||
|
||||
def state_slot_offset(self) -> int:
|
||||
if not self.enabled():
|
||||
return 0
|
||||
return self.backend.token_to_kv_pool.get_online_c128_mtp_state_slot_offset()
|
||||
|
||||
def begin_verify(
|
||||
self,
|
||||
req_pool_indices: torch.Tensor,
|
||||
seq_lens: torch.Tensor,
|
||||
) -> None:
|
||||
if not self.enabled():
|
||||
self.clear()
|
||||
return
|
||||
|
||||
self._verify_ctx = _OnlineC128VerifyContext(
|
||||
req_pool_indices=req_pool_indices.detach(),
|
||||
seq_lens=seq_lens.detach(),
|
||||
)
|
||||
head_dim = self._head_dim()
|
||||
state_dtype = self._state_dtype()
|
||||
if head_dim is None or state_dtype is None or self._num_verify_tokens() == 0:
|
||||
return
|
||||
token_to_kv_pool = self.backend.token_to_kv_pool
|
||||
_jit_online_c128_mtp_module(
|
||||
head_dim, seq_lens.dtype, req_pool_indices.dtype, state_dtype
|
||||
).mark_pending(
|
||||
seq_lens,
|
||||
req_pool_indices,
|
||||
token_to_kv_pool.get_online_c128_mtp_pending_seq_lens(),
|
||||
min(seq_lens.shape[0], req_pool_indices.shape[0]),
|
||||
token_to_kv_pool.get_online_c128_state_num_req_slots(),
|
||||
)
|
||||
|
||||
def clear(self) -> None:
|
||||
self._verify_ctx = None
|
||||
|
||||
def prepare_forward(
|
||||
self,
|
||||
logical_forward_mode,
|
||||
req_pool_indices: torch.Tensor,
|
||||
seq_lens: torch.Tensor,
|
||||
*,
|
||||
verify_bs: Optional[int] = None,
|
||||
) -> int:
|
||||
if not self.enabled():
|
||||
self.clear()
|
||||
return 0
|
||||
if logical_forward_mode is None or logical_forward_mode.is_idle():
|
||||
self.clear()
|
||||
return 0
|
||||
|
||||
active_req_pool_indices = req_pool_indices
|
||||
active_seq_lens = seq_lens
|
||||
if logical_forward_mode.is_target_verify():
|
||||
if verify_bs is None:
|
||||
verify_bs = req_pool_indices.shape[0]
|
||||
active_req_pool_indices = req_pool_indices[:verify_bs]
|
||||
active_seq_lens = seq_lens[:verify_bs]
|
||||
if verify_bs == 0:
|
||||
self.clear()
|
||||
return 0
|
||||
|
||||
self.commit_pending(
|
||||
req_pool_indices=active_req_pool_indices,
|
||||
seq_lens=active_seq_lens,
|
||||
)
|
||||
if not logical_forward_mode.is_target_verify():
|
||||
return 0
|
||||
|
||||
self.begin_verify(
|
||||
req_pool_indices=active_req_pool_indices,
|
||||
seq_lens=active_seq_lens,
|
||||
)
|
||||
return self.state_slot_offset()
|
||||
|
||||
def write_prefix_states(
|
||||
self,
|
||||
layer_id: int,
|
||||
compressor: Any,
|
||||
kv_score_input: torch.Tensor,
|
||||
logical_forward_mode,
|
||||
) -> None:
|
||||
if (
|
||||
not self.enabled()
|
||||
or logical_forward_mode is None
|
||||
or not logical_forward_mode.is_target_verify()
|
||||
or compressor.is_in_indexer
|
||||
or compressor.ratio != 128
|
||||
or kv_score_input.numel() == 0
|
||||
):
|
||||
return
|
||||
|
||||
ctx = self._active_ctx()
|
||||
num_verify_tokens = self._num_verify_tokens()
|
||||
if ctx is None or num_verify_tokens == 0:
|
||||
return
|
||||
|
||||
token_to_kv_pool = self.backend.token_to_kv_pool
|
||||
head_dim = compressor.head_dim
|
||||
state_pool = token_to_kv_pool.get_attention_compress_states(layer_id)
|
||||
state = state_pool.kv_score_buffer.kv_score
|
||||
total_bs = kv_score_input.numel() // (num_verify_tokens * head_dim * 2)
|
||||
layer_bs = min(ctx.seq_lens.shape[0], ctx.req_pool_indices.shape[0], total_bs)
|
||||
if layer_bs <= 0:
|
||||
return
|
||||
|
||||
_jit_online_c128_mtp_module(
|
||||
head_dim, ctx.seq_lens.dtype, ctx.req_pool_indices.dtype, state.dtype
|
||||
).write_prefix_states(
|
||||
kv_score_input,
|
||||
ctx.seq_lens,
|
||||
ctx.req_pool_indices,
|
||||
self.backend.req_to_token,
|
||||
compressor.ape.reshape(128, head_dim),
|
||||
state,
|
||||
layer_bs,
|
||||
num_verify_tokens,
|
||||
state_pool.online_mtp_state_slot_offset,
|
||||
)
|
||||
|
||||
def commit_pending(
|
||||
self,
|
||||
req_pool_indices: torch.Tensor,
|
||||
seq_lens: torch.Tensor,
|
||||
) -> None:
|
||||
if self._verify_ctx is None:
|
||||
return
|
||||
if not self.enabled():
|
||||
self.clear()
|
||||
return
|
||||
if req_pool_indices.numel() == 0 or seq_lens.numel() == 0:
|
||||
return
|
||||
|
||||
num_verify_tokens = self._num_verify_tokens()
|
||||
if num_verify_tokens == 0:
|
||||
self.clear()
|
||||
return
|
||||
|
||||
backend = self.backend
|
||||
token_to_kv_pool = backend.token_to_kv_pool
|
||||
pending_seq_lens = token_to_kv_pool.get_online_c128_mtp_pending_seq_lens()
|
||||
cur_bs = min(seq_lens.shape[0], req_pool_indices.shape[0])
|
||||
|
||||
for runtime in self._iter_layer_runtimes():
|
||||
_jit_online_c128_mtp_module(
|
||||
runtime.head_dim,
|
||||
seq_lens.dtype,
|
||||
req_pool_indices.dtype,
|
||||
runtime.state_dtype,
|
||||
).commit_pending(
|
||||
seq_lens,
|
||||
req_pool_indices,
|
||||
backend.req_to_token,
|
||||
pending_seq_lens,
|
||||
runtime.main_state,
|
||||
cur_bs,
|
||||
num_verify_tokens,
|
||||
runtime.state_slot_offset,
|
||||
token_to_kv_pool.get_online_c128_state_num_req_slots(),
|
||||
)
|
||||
|
||||
self.clear()
|
||||
|
||||
def _num_verify_tokens(self) -> int:
|
||||
if not self.enabled():
|
||||
return 0
|
||||
num_verify_tokens = int(self.backend.speculative_num_draft_tokens)
|
||||
max_draft_tokens = (
|
||||
self.backend.token_to_kv_pool.get_online_c128_mtp_max_draft_tokens()
|
||||
)
|
||||
return num_verify_tokens if 0 < num_verify_tokens <= max_draft_tokens else 0
|
||||
|
||||
def _active_ctx(self) -> Optional[_OnlineC128VerifyContext]:
|
||||
ctx = self._verify_ctx
|
||||
if (
|
||||
ctx is None
|
||||
or ctx.seq_lens.numel() == 0
|
||||
or ctx.req_pool_indices.numel() == 0
|
||||
):
|
||||
return None
|
||||
return ctx
|
||||
|
||||
def _head_dim(self) -> Optional[int]:
|
||||
for runtime in self._iter_layer_runtimes():
|
||||
return runtime.head_dim
|
||||
return None
|
||||
|
||||
def _state_dtype(self) -> Optional[torch.dtype]:
|
||||
for runtime in self._iter_layer_runtimes():
|
||||
return runtime.state_dtype
|
||||
return None
|
||||
|
||||
def _iter_layer_runtimes(self):
|
||||
if self._layer_runtimes is None:
|
||||
runtimes = []
|
||||
token_to_kv_pool = self.backend.token_to_kv_pool
|
||||
for layer in self.backend.model_runner.model.model.layers:
|
||||
attn = getattr(layer, "self_attn", None)
|
||||
compressor = getattr(attn, "compressor", None)
|
||||
if compressor is None or compressor.ratio != 128:
|
||||
continue
|
||||
state_pool = token_to_kv_pool.get_attention_compress_states(
|
||||
compressor.layer_id
|
||||
)
|
||||
runtimes.append(
|
||||
_OnlineC128LayerRuntime(
|
||||
head_dim=compressor.head_dim,
|
||||
main_state=state_pool.kv_score_buffer.kv_score,
|
||||
state_dtype=state_pool.kv_score_buffer.kv_score.dtype,
|
||||
state_slot_offset=state_pool.online_mtp_state_slot_offset,
|
||||
)
|
||||
)
|
||||
self._layer_runtimes = runtimes
|
||||
return iter(self._layer_runtimes)
|
||||
def online_c128_mtp_commit_pending(
|
||||
seq_lens: torch.Tensor,
|
||||
req_pool_indices: torch.Tensor,
|
||||
req_to_token: torch.Tensor,
|
||||
pending_seq_lens: torch.Tensor,
|
||||
state: torch.Tensor,
|
||||
batch_size: int,
|
||||
num_verify_tokens: int,
|
||||
state_slot_stride: int,
|
||||
num_req_slots: int,
|
||||
*,
|
||||
head_dim: int,
|
||||
) -> None:
|
||||
"""Copy the draft state of each request's accepted prefix into its main c128
|
||||
state slot."""
|
||||
_jit_online_c128_mtp_module(
|
||||
head_dim, seq_lens.dtype, req_pool_indices.dtype, state.dtype
|
||||
).commit_pending(
|
||||
seq_lens,
|
||||
req_pool_indices,
|
||||
req_to_token,
|
||||
pending_seq_lens,
|
||||
state,
|
||||
batch_size,
|
||||
num_verify_tokens,
|
||||
state_slot_stride,
|
||||
num_req_slots,
|
||||
)
|
||||
|
||||
+2
@@ -519,6 +519,8 @@ def _fused_norm_rope_kernel(
|
||||
)
|
||||
|
||||
|
||||
# Not bit-exact with the JIT dsv4 fused_norm_rope_inplace (outputs differ by up to
|
||||
# 1 bf16 ulp), so the two are kept separate.
|
||||
def fused_norm_rope_inplace_triton(
|
||||
kv: torch.Tensor,
|
||||
weight: Optional[torch.Tensor],
|
||||
@@ -3,9 +3,18 @@
|
||||
Migrated from ``sglang.srt.layers.attention.dsv4.sparse_prefill_utils`` (RFC #29630, Phase 2.5).
|
||||
"""
|
||||
|
||||
from typing import Optional
|
||||
|
||||
import torch
|
||||
import triton
|
||||
import triton.language as tl
|
||||
|
||||
from sglang.srt.utils import ceil_align
|
||||
|
||||
# FlashMLA sparse prefill asserts ``params.topk % B_TOPK == 0``. B_TOPK is 64
|
||||
# for the h_q=64 kernel and 128 for h_q=128; pad to 128 to satisfy both.
|
||||
SPARSE_PREFILL_TOPK_ALIGNMENT = 128
|
||||
|
||||
|
||||
@triton.jit
|
||||
def _build_swa_token_ids_kernel(
|
||||
@@ -139,3 +148,181 @@ def _combine_topk_swa_indices_kernel(
|
||||
)
|
||||
|
||||
tl.store(combined_lens_ptr + token, topk_len + swa_len, mask=valid)
|
||||
|
||||
|
||||
def combined_topk_width(topk: int, window_size: int) -> int:
|
||||
"""Width of the padded combined_indices last dim that
|
||||
``combine_topk_swa_indices`` would produce for these args."""
|
||||
return ceil_align(topk + window_size, SPARSE_PREFILL_TOPK_ALIGNMENT)
|
||||
|
||||
|
||||
def combine_topk_swa_indices(
|
||||
topk_indices: torch.Tensor,
|
||||
query_start_loc: torch.Tensor,
|
||||
query_pos: torch.Tensor,
|
||||
seq_lens: torch.Tensor,
|
||||
gather_lens: torch.Tensor,
|
||||
compressed_base: torch.Tensor,
|
||||
swa_base: torch.Tensor,
|
||||
window_size: int,
|
||||
compress_ratio: int,
|
||||
topk: int,
|
||||
out_indices: Optional[torch.Tensor] = None,
|
||||
out_lens: Optional[torch.Tensor] = None,
|
||||
swa_indices: Optional[torch.Tensor] = None,
|
||||
swa_lengths: Optional[torch.Tensor] = None,
|
||||
) -> tuple[torch.Tensor, torch.Tensor]:
|
||||
"""Combine top-k and SWA indices for flash_mla_sparse_fwd.
|
||||
|
||||
Top-k indices are int32 [num_tokens, K] in each request's compressed region.
|
||||
Invalid entries within topk_len must be -1; later entries are ignored.
|
||||
query_start_loc can include a cross-chunk offset; query_pos is absolute.
|
||||
compressed_base and swa_base address the flat workspace. SWA-only layers use
|
||||
topk == 0, but compress_ratio must still be positive.
|
||||
Optional SWA indices are relative to swa_base and preserve replay floors.
|
||||
|
||||
Returns int32 indices [num_tokens, padded_topk_swa] and per-token scanned-prefix
|
||||
lengths, including -1 entries skipped by attention. Width is padded to 128.
|
||||
Rows are written whole and -1 padded, so preallocated out_indices / out_lens
|
||||
are overwritten entirely; rows past the last request are -1 with length 0.
|
||||
"""
|
||||
assert topk_indices.dtype == torch.int32
|
||||
assert query_start_loc.dtype == torch.int32
|
||||
assert query_pos.dtype == torch.int32
|
||||
assert seq_lens.dtype == torch.int32
|
||||
assert gather_lens.dtype == torch.int32
|
||||
assert compressed_base.dtype == torch.int32
|
||||
assert swa_base.dtype == torch.int32
|
||||
assert compress_ratio >= 1, "compress_ratio must be >= 1 (use topk=0 for SWA-only)"
|
||||
assert topk_indices.shape[-1] >= topk, (
|
||||
f"topk_indices width {topk_indices.shape[-1]} must be >= topk {topk}"
|
||||
)
|
||||
|
||||
num_tokens = topk_indices.shape[0]
|
||||
if swa_indices is not None:
|
||||
assert swa_indices.shape == (num_tokens, window_size)
|
||||
assert swa_lengths is not None and swa_lengths.shape == (num_tokens,)
|
||||
assert swa_indices.dtype == swa_lengths.dtype == torch.int32
|
||||
num_reqs = seq_lens.shape[0]
|
||||
combined_topk = combined_topk_width(topk, window_size)
|
||||
if out_indices is None:
|
||||
# The kernel writes whole -1 padded rows.
|
||||
combined_indices = torch.empty(
|
||||
(num_tokens, combined_topk),
|
||||
dtype=torch.int32,
|
||||
device=topk_indices.device,
|
||||
)
|
||||
else:
|
||||
assert out_indices.shape == (num_tokens, combined_topk)
|
||||
assert out_indices.dtype == torch.int32
|
||||
combined_indices = out_indices
|
||||
if out_lens is None:
|
||||
combined_lens = torch.empty(
|
||||
num_tokens, dtype=torch.int32, device=topk_indices.device
|
||||
)
|
||||
else:
|
||||
assert out_lens.shape == (num_tokens,)
|
||||
assert out_lens.dtype == torch.int32
|
||||
combined_lens = out_lens
|
||||
|
||||
if num_tokens == 0:
|
||||
return combined_indices, combined_lens
|
||||
block_t = 2 # Measured on B200 with num_warps=2, 16384 tokens, bs 1 to 64.
|
||||
_combine_topk_swa_indices_kernel[(triton.cdiv(num_tokens, block_t),)](
|
||||
combined_indices,
|
||||
combined_indices.stride(0),
|
||||
combined_lens,
|
||||
topk_indices,
|
||||
topk_indices.stride(0),
|
||||
query_start_loc,
|
||||
query_pos,
|
||||
seq_lens,
|
||||
gather_lens,
|
||||
compressed_base,
|
||||
swa_base,
|
||||
swa_indices,
|
||||
0 if swa_indices is None else swa_indices.stride(0),
|
||||
swa_lengths,
|
||||
num_tokens,
|
||||
num_reqs,
|
||||
top_k=topk,
|
||||
EXPLICIT_SWA=swa_indices is not None,
|
||||
COMPRESS_RATIO=compress_ratio,
|
||||
WINDOW_SIZE=window_size,
|
||||
WIDTH=combined_topk,
|
||||
PADDED_WIDTH=triton.next_power_of_2(combined_topk),
|
||||
BLOCK_T=block_t,
|
||||
SEARCH_STEPS=max(num_reqs, 1).bit_length(),
|
||||
num_warps=2,
|
||||
)
|
||||
return combined_indices, combined_lens
|
||||
|
||||
|
||||
def build_swa_token_ids(
|
||||
seq_lens: torch.Tensor,
|
||||
extend_seq_lens: torch.Tensor,
|
||||
req_pool_indices: torch.Tensor,
|
||||
req_to_token: torch.Tensor,
|
||||
full_to_swa: torch.Tensor,
|
||||
swa_window: int,
|
||||
total_swa: int,
|
||||
) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor]:
|
||||
"""Build a flat list of physical SWA-cache token IDs covering each
|
||||
request's positional union of every query's SWA window.
|
||||
|
||||
Per request, the union spans seq positions
|
||||
``[max(0, seq_len - extend - W + 1), seq_len)``, of length
|
||||
``min(seq_len, extend + W - 1)``. Each position is translated through
|
||||
``req_to_token`` (full kv-cache id) and then ``full_to_swa`` (SWA
|
||||
cache id) to land in the SWA-cache token-id space that
|
||||
``dequantize_k_cache_paged_dsv4`` consumes.
|
||||
|
||||
Args:
|
||||
seq_lens: (num_reqs,) int32, per-request total sequence length.
|
||||
extend_seq_lens: (num_reqs,) int32, per-request query length.
|
||||
req_pool_indices: (num_reqs,) int32, per-request row in
|
||||
``req_to_token``.
|
||||
req_to_token: (num_reqs_max, max_seq_len) int32. Full kv-cache id
|
||||
per (request, seq position).
|
||||
full_to_swa: (full_pool_size + extra,) int64. Maps full kv id to
|
||||
SWA-cache id.
|
||||
swa_window: int. SWA window size.
|
||||
total_swa: Number of token IDs to allocate, computed from CPU lengths.
|
||||
|
||||
Returns:
|
||||
swa_token_ids: (total_swa,) int32, flat physical SWA-cache token IDs.
|
||||
swa_first_pos: (num_reqs,) int32, first seq position covered per req.
|
||||
swa_gather_lens: (num_reqs,) int32, gather length per request.
|
||||
swa_offsets: (num_reqs+1,) int32, exclusive cumsum of swa_gather_lens.
|
||||
"""
|
||||
assert seq_lens.dtype == torch.int32
|
||||
assert extend_seq_lens.dtype == torch.int32
|
||||
assert req_pool_indices.dtype == torch.int32
|
||||
assert req_to_token.dtype == torch.int32
|
||||
assert full_to_swa.dtype == torch.int64
|
||||
|
||||
num_reqs = seq_lens.shape[0]
|
||||
device = seq_lens.device
|
||||
|
||||
swa_gather_lens = torch.minimum(seq_lens, extend_seq_lens + (swa_window - 1)).to(
|
||||
torch.int32
|
||||
)
|
||||
swa_first_pos = (seq_lens - swa_gather_lens).to(torch.int32)
|
||||
swa_offsets = torch.zeros(num_reqs + 1, dtype=torch.int32, device=device)
|
||||
swa_offsets[1:] = torch.cumsum(swa_gather_lens, dim=0).to(torch.int32)
|
||||
swa_token_ids = torch.empty(total_swa, dtype=torch.int32, device=device)
|
||||
if total_swa == 0:
|
||||
return swa_token_ids, swa_first_pos, swa_gather_lens, swa_offsets
|
||||
|
||||
NUM_WORKERS = 128
|
||||
_build_swa_token_ids_kernel[(num_reqs, NUM_WORKERS)](
|
||||
swa_token_ids,
|
||||
swa_first_pos,
|
||||
swa_gather_lens,
|
||||
swa_offsets,
|
||||
req_pool_indices,
|
||||
req_to_token,
|
||||
req_to_token.stride(0),
|
||||
full_to_swa,
|
||||
)
|
||||
return swa_token_ids, swa_first_pos, swa_gather_lens, swa_offsets
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
+1
-1
@@ -9,7 +9,7 @@ from typing import Optional, Tuple
|
||||
|
||||
import torch
|
||||
|
||||
from sglang.kernels.ops.attention.nsa_triton_decode.triton_mla_kernels_decode_optimized import (
|
||||
from sglang.kernels.ops.attention.dsv4.triton_sparse_decode.triton_mla_kernels_decode_optimized import (
|
||||
triton_sparse_attn_decode,
|
||||
)
|
||||
|
||||
@@ -86,7 +86,7 @@ _TARGET_WG_PER_CU = 1.5 if _is_hip else 2.0
|
||||
# halving from bf16→fp8 = 2× saving — net ~46% read bandwidth reduction).
|
||||
_FP8_GROUP_SIZE = 64
|
||||
# Match the KV-cache write format: fnuz only on AMD gfx942; Other platform e.g.
|
||||
# gfx950 store e4m3fn. See quant_k_cache.py / indexer.py FP8_DTYPE.
|
||||
# gfx950 store e4m3fn. See quantization/dsa_quant_k_cache.py / indexer.py FP8_DTYPE.
|
||||
_FP8_DTYPE = torch.float8_e4m3fnuz if is_fp8_fnuz() else torch.float8_e4m3fn
|
||||
|
||||
|
||||
@@ -120,7 +120,7 @@ _FP8_DTYPE = torch.float8_e4m3fnuz if is_fp8_fnuz() else torch.float8_e4m3fn
|
||||
_HCA_KV_SPLITS = envs.SGLANG_MLA_HCA_KV_SPLITS.get()
|
||||
|
||||
|
||||
def _kv_splits_for_stream(compress_ratio: int) -> int | None:
|
||||
def kv_splits_for_stream(compress_ratio: int) -> int | None:
|
||||
"""-> kv_splits override for a decode stream, or None to keep the heuristic.
|
||||
|
||||
Only the unclamped stream is overridden. SWA (ratio 0) is a 128-entry
|
||||
@@ -951,7 +951,7 @@ def sparse_attn_v4_paged_decode(
|
||||
will be dequantized in-kernel using 1xGROUP_SIZE (default 64) block scales.
|
||||
|
||||
``kv_splits`` overrides `_kv_splits_heuristic`. The caller knows the layer's
|
||||
``compress_ratio`` and the heuristic does not; see `_kv_splits_for_stream`.
|
||||
``compress_ratio`` and the heuristic does not; see `kv_splits_for_stream`.
|
||||
"""
|
||||
if _is_gfx1250_supported:
|
||||
# aiter ships only on ROCm, and this module is imported by a CPU-registered
|
||||
|
||||
@@ -1,8 +1,7 @@
|
||||
"""Runtime glue for the unified_kv backend.
|
||||
|
||||
Builds unified_kv-style flat ``kv_indices`` / ``kv_indptr`` from SGLang's already-computed
|
||||
DSV4 metadata, scatters SWA K into the ``unified_kv`` ring, and dispatches the
|
||||
vendored paged decode/prefill kernels.
|
||||
DSV4 metadata and dispatches the vendored paged decode/prefill kernels.
|
||||
|
||||
unified_kv[L] layout (page_size 1, row-major):
|
||||
- rows ``[0, swa_pages)`` = SWA ring (``state_slot * win + pos % win``);
|
||||
@@ -53,174 +52,6 @@ from sglang.kernels.ops.attention.dsv4.unified_kv_kernels.paged_prefill import (
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# SWA ring scatter
|
||||
# ---------------------------------------------------------------------------
|
||||
@triton.jit
|
||||
def _swa_scatter_kernel(
|
||||
kv_ptr, # [T, D] bf16
|
||||
state_slot_ptr, # [T] int
|
||||
positions_ptr, # [T] int
|
||||
final_pos_ptr, # [T] int
|
||||
unified_ptr, # [pages, D] bf16
|
||||
n_rows,
|
||||
ring_stride, # SWA ring per-slot stride
|
||||
win: tl.constexpr,
|
||||
D: tl.constexpr,
|
||||
HAS_FINAL: tl.constexpr,
|
||||
BLOCK_D: tl.constexpr,
|
||||
):
|
||||
row = tl.program_id(0)
|
||||
if row >= n_rows:
|
||||
return
|
||||
pos = tl.load(positions_ptr + row)
|
||||
if HAS_FINAL:
|
||||
fp = tl.load(final_pos_ptr + row)
|
||||
if pos <= fp - win:
|
||||
return
|
||||
s = tl.load(state_slot_ptr + row)
|
||||
loc = s * ring_stride + (pos % ring_stride)
|
||||
offs = tl.arange(0, BLOCK_D)
|
||||
mask = offs < D
|
||||
vals = tl.load(kv_ptr + row * D + offs, mask=mask, other=0.0)
|
||||
tl.store(unified_ptr + loc * D + offs, vals, mask=mask)
|
||||
|
||||
|
||||
def store_swa_into_unified(
|
||||
*,
|
||||
kv: torch.Tensor, # [T, head_dim] bf16, or [T, nope_row_bytes] packed fp8
|
||||
state_slot: torch.Tensor, # [T] int
|
||||
positions: torch.Tensor, # [T] int
|
||||
unified_kv: torch.Tensor, # [pages, ...] same dtype and row width as kv
|
||||
win: int, # SWA attention window length
|
||||
ring_stride: int, # SWA ring stride
|
||||
final_pos: Optional[torch.Tensor] = None, # [T] req's last position
|
||||
kv_rope: Optional[torch.Tensor] = None, # [T, rope_dim] bf16, fp8 layout only
|
||||
unified_kv_rope: Optional[torch.Tensor] = None, # [pages, rope_dim] bf16
|
||||
) -> None:
|
||||
"""Scatter SWA K into ring row ``state_slot * ring_stride + pos % ring_stride``.
|
||||
|
||||
Under the fp8 layout the latent is split over two pools, so ``kv`` carries the
|
||||
already-packed nope row (DSV4_FP8_NOPE_ROW_BYTES wide: values + E8M0 scales +
|
||||
pad; nothing is quantized here) and ``kv_rope`` the bf16 rope half. That width
|
||||
is a byte count that happens to equal the bf16 head_dim in elements -- the two
|
||||
are not the same thing. The scatter itself takes the row width off the tensor;
|
||||
only the pair check reads the constant.
|
||||
|
||||
Both scatters recompute the row index from the same ``state_slot`` /
|
||||
``positions``, so the two pools stay in lockstep with each other and with the
|
||||
bf16 layout. What a caller can still get wrong is passing two pools that aren't
|
||||
a pair, so the pair goes through ``check_two_pool_pair`` before the first launch
|
||||
-- shared with the fused store, which has the same coupling.
|
||||
"""
|
||||
n_rows = kv.shape[0]
|
||||
if n_rows == 0:
|
||||
return
|
||||
|
||||
has_final = final_pos is not None
|
||||
fp_arg = final_pos if has_final else positions
|
||||
assert kv.is_contiguous() and kv.dtype == unified_kv.dtype
|
||||
assert state_slot.is_contiguous() and positions.is_contiguous()
|
||||
assert fp_arg.is_contiguous()
|
||||
two_pool = kv_rope is not None
|
||||
assert two_pool == (unified_kv_rope is not None), (
|
||||
"kv_rope and unified_kv_rope come together"
|
||||
)
|
||||
if two_pool:
|
||||
assert kv_rope.is_contiguous(), (
|
||||
f"kv_rope must be contiguous, got strides {kv_rope.stride()}"
|
||||
)
|
||||
assert kv_rope.shape[0] == n_rows, (
|
||||
f"kv_rope holds {kv_rope.shape[0]} rows, kv holds {n_rows}"
|
||||
)
|
||||
check_two_pool_pair(
|
||||
unified_kv,
|
||||
unified_kv_rope,
|
||||
rope_width=kv_rope.shape[1],
|
||||
rope_dtype=kv_rope.dtype,
|
||||
)
|
||||
|
||||
def _scatter(src: torch.Tensor, dst: torch.Tensor) -> None:
|
||||
D = src.shape[1]
|
||||
assert dst.shape[1] == D, f"row width {D} does not fit pool {dst.shape[1]}"
|
||||
_swa_scatter_kernel[(n_rows,)](
|
||||
src,
|
||||
state_slot,
|
||||
positions,
|
||||
fp_arg,
|
||||
dst,
|
||||
n_rows,
|
||||
ring_stride,
|
||||
win=win,
|
||||
D=D,
|
||||
HAS_FINAL=has_final,
|
||||
BLOCK_D=triton.next_power_of_2(D),
|
||||
num_warps=8,
|
||||
)
|
||||
|
||||
if kv.element_size() == 1:
|
||||
# single-byte rows (any fp8 variant) are a pure byte move, and the E8M0
|
||||
# scale bytes aren't floats -- uint8 avoids a triton convert for `other=`
|
||||
assert unified_kv.is_contiguous()
|
||||
_scatter(kv.view(torch.uint8), unified_kv.view(torch.uint8))
|
||||
else:
|
||||
_scatter(kv, unified_kv)
|
||||
|
||||
if two_pool:
|
||||
_scatter(kv_rope, unified_kv_rope)
|
||||
|
||||
|
||||
@triton.jit
|
||||
def _scatter_loc_kernel(
|
||||
kv_ptr, # [T, D] bf16
|
||||
loc_ptr, # [T] int (unified row index; <0 => skip)
|
||||
unified_ptr, # [pages, D] bf16
|
||||
n_rows,
|
||||
D: tl.constexpr,
|
||||
BLOCK_D: tl.constexpr,
|
||||
):
|
||||
row = tl.program_id(0)
|
||||
if row >= n_rows:
|
||||
return
|
||||
loc = tl.load(loc_ptr + row).to(tl.int64)
|
||||
if loc < 0:
|
||||
return
|
||||
offs = tl.arange(0, BLOCK_D)
|
||||
mask = offs < D
|
||||
vals = tl.load(kv_ptr + row * D + offs, mask=mask, other=0.0)
|
||||
tl.store(unified_ptr + loc * D + offs, vals, mask=mask)
|
||||
|
||||
|
||||
def scatter_bf16_into_unified(
|
||||
*,
|
||||
kv: torch.Tensor, # [T, head_dim] bf16 (already norm+rope'd)
|
||||
loc: torch.Tensor, # [T] int32/int64 unified ring row; <0 => skip
|
||||
unified_kv: torch.Tensor, # [pages, head_dim] bf16
|
||||
) -> None:
|
||||
"""Scatter already-norm+rope'd bf16 K into ``unified_kv[loc]`` (skip loc < 0).
|
||||
|
||||
Companion to ``store_swa_into_unified`` for callers that already hold the
|
||||
precomputed ring row index (the DSpark draft: ``get_unified_swa_loc`` for the
|
||||
draft forward, or the commit-inject layout for target-hidden injection) and
|
||||
need per-row commit masking expressed as ``loc == -1``.
|
||||
"""
|
||||
n_rows, D = kv.shape
|
||||
if n_rows == 0:
|
||||
return
|
||||
assert kv.is_contiguous() and kv.dtype == unified_kv.dtype
|
||||
assert loc.is_contiguous()
|
||||
assert unified_kv.is_contiguous()
|
||||
_scatter_loc_kernel[(n_rows,)](
|
||||
kv,
|
||||
loc,
|
||||
unified_kv,
|
||||
n_rows,
|
||||
D=D,
|
||||
BLOCK_D=triton.next_power_of_2(D),
|
||||
num_warps=8,
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Ragged indptr helper (shared by the decode streams + prefill builders)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@@ -1,7 +1,6 @@
|
||||
# Copyright (c) 2026, Colfax International.
|
||||
|
||||
import math
|
||||
import time
|
||||
from functools import partial
|
||||
from typing import Callable, Optional
|
||||
|
||||
@@ -10,21 +9,15 @@ import cutlass
|
||||
import cutlass.cute as cute
|
||||
import cutlass.pipeline as pipeline
|
||||
import cutlass.utils.blackwell_helpers as sm100_utils
|
||||
import torch
|
||||
import torch.utils.benchmark as benchmark
|
||||
from cutlass import Boolean, Float32, Int32, Int64, Uint32, const_expr
|
||||
from cutlass.cute import FastDivmodDivisor
|
||||
from cutlass.cute.nvgpu import cpasync, tcgen05
|
||||
from cutlass.cute.runtime import from_dlpack
|
||||
from cutlass.utils import ClcDynamicPersistentTileScheduler
|
||||
from quack import copy_utils
|
||||
|
||||
import sglang.kernels.ops.attention.flash_attn.cute.blackwell_helpers as fa_sm100_utils
|
||||
from sglang.kernels.ops.attention.flash_attn.cute import utils as fa_utils
|
||||
from sglang.kernels.ops.attention.flash_attn.cute.block_info import BlockInfo
|
||||
from sglang.kernels.ops.attention.flash_attn.cute.cute_dsl_utils import (
|
||||
dump_kernel_attributes,
|
||||
)
|
||||
from sglang.kernels.ops.attention.flash_attn.cute.fa_logging import fa_log, fa_printf
|
||||
from sglang.kernels.ops.attention.flash_attn.cute.mask import AttentionMask
|
||||
from sglang.kernels.ops.attention.flash_attn.cute.named_barrier import (
|
||||
@@ -37,7 +30,6 @@ from sglang.kernels.ops.attention.flash_attn.cute.pack_gqa import (
|
||||
from sglang.kernels.ops.attention.flash_attn.cute.paged_kv import PagedKVManager
|
||||
from sglang.kernels.ops.attention.flash_attn.cute.seqlen_info import SeqlenInfoQK
|
||||
from sglang.kernels.ops.attention.flash_attn.cute.softmax import SoftmaxSm100
|
||||
from sglang.kernels.ops.attention.flash_attn.cute.testing import attention_ref
|
||||
from sglang.kernels.ops.attention.flash_attn.cute.tile_scheduler import (
|
||||
ClcState,
|
||||
ParamsBase,
|
||||
@@ -3154,784 +3146,3 @@ class FlashAttentionMLAForwardSm100:
|
||||
)
|
||||
cute.copy(thr_tmem_store, tOrO_t2r_frg, tOtO_r2t_cur)
|
||||
cute.arch.fence_view_async_tmem_store()
|
||||
|
||||
|
||||
def test_mla_kernel(
|
||||
seqlen_q=2048,
|
||||
seqlen_k=2048,
|
||||
topk_length=2048,
|
||||
nheads=1,
|
||||
batch=1,
|
||||
iter=0,
|
||||
compile_cache=dict(),
|
||||
validate=True,
|
||||
seed=0,
|
||||
gather_kv=True,
|
||||
pack_gqa=False,
|
||||
is_causal=False,
|
||||
varlen_q=False,
|
||||
varlen_k=False,
|
||||
disable_bitmask=False,
|
||||
has_qk=True,
|
||||
store_P=False,
|
||||
):
|
||||
torch.manual_seed(seed)
|
||||
hdim = 64
|
||||
hdimv = 512
|
||||
softmax_scale = 1.0 / math.sqrt(hdim + hdimv) if has_qk else 1.0 / math.sqrt(hdimv)
|
||||
|
||||
nheads_kv = 1
|
||||
qhead_per_kvhead = nheads
|
||||
seqlen_k_rounded = (seqlen_k + 128 - 1) // 128 * 128
|
||||
P_k_length = seqlen_k_rounded if not gather_kv else topk_length
|
||||
|
||||
torch_stream = torch.cuda.current_stream()
|
||||
stream = cuda.CUstream(torch_stream.cuda_stream)
|
||||
|
||||
compile_key = (
|
||||
is_causal,
|
||||
gather_kv,
|
||||
topk_length if gather_kv else None,
|
||||
pack_gqa,
|
||||
qhead_per_kvhead,
|
||||
nheads_kv,
|
||||
varlen_q,
|
||||
varlen_k,
|
||||
disable_bitmask,
|
||||
has_qk,
|
||||
)
|
||||
if compile_key not in compile_cache:
|
||||
total_q_dummy = batch * seqlen_q
|
||||
total_k_dummy = batch * seqlen_k
|
||||
|
||||
if varlen_q:
|
||||
Q = torch.randn(
|
||||
total_q_dummy, nheads, hdim, dtype=torch.bfloat16, device="cuda"
|
||||
)
|
||||
Qv = torch.randn(
|
||||
total_q_dummy, nheads, hdimv, dtype=torch.bfloat16, device="cuda"
|
||||
)
|
||||
O = torch.empty(
|
||||
total_q_dummy, nheads, hdimv, dtype=torch.bfloat16, device="cuda"
|
||||
)
|
||||
P = torch.empty(
|
||||
total_q_dummy, nheads, P_k_length, dtype=torch.bfloat16, device="cuda"
|
||||
)
|
||||
lse = torch.empty(total_q_dummy, nheads, dtype=torch.float32, device="cuda")
|
||||
row_max = torch.empty(
|
||||
total_q_dummy,
|
||||
nheads,
|
||||
P_k_length // 128,
|
||||
dtype=torch.float32,
|
||||
device="cuda",
|
||||
)
|
||||
index_topk = (
|
||||
torch.rand(total_q_dummy, topk_length, device="cuda")
|
||||
.argsort(dim=-1)
|
||||
.to(torch.int32)
|
||||
)
|
||||
cu_seqlens_q_dummy = torch.arange(
|
||||
0, (batch + 1) * seqlen_q, seqlen_q, dtype=torch.int32, device="cuda"
|
||||
)
|
||||
else:
|
||||
Q = torch.randn(
|
||||
batch, seqlen_q, nheads, hdim, dtype=torch.bfloat16, device="cuda"
|
||||
)
|
||||
Qv = torch.randn(
|
||||
batch, seqlen_q, nheads, hdimv, dtype=torch.bfloat16, device="cuda"
|
||||
)
|
||||
O = torch.empty(
|
||||
batch, seqlen_q, nheads, hdimv, dtype=torch.bfloat16, device="cuda"
|
||||
)
|
||||
P = torch.empty(
|
||||
batch, seqlen_q, nheads, P_k_length, dtype=torch.bfloat16, device="cuda"
|
||||
)
|
||||
lse = torch.empty(
|
||||
batch, seqlen_q, nheads, dtype=torch.float32, device="cuda"
|
||||
)
|
||||
row_max = torch.empty(
|
||||
batch,
|
||||
seqlen_q,
|
||||
nheads,
|
||||
P_k_length // 128,
|
||||
dtype=torch.float32,
|
||||
device="cuda",
|
||||
)
|
||||
index_topk = (
|
||||
torch.rand(batch, seqlen_q, topk_length, device="cuda")
|
||||
.argsort(dim=-1)
|
||||
.to(torch.int32)
|
||||
)
|
||||
|
||||
if varlen_k:
|
||||
K = torch.randn(
|
||||
total_k_dummy, nheads_kv, hdim, dtype=torch.bfloat16, device="cuda"
|
||||
)
|
||||
V = torch.randn(
|
||||
total_k_dummy, nheads_kv, hdimv, dtype=torch.bfloat16, device="cuda"
|
||||
)
|
||||
cu_seqlens_k_dummy = torch.arange(
|
||||
0, (batch + 1) * seqlen_k, seqlen_k, dtype=torch.int32, device="cuda"
|
||||
)
|
||||
else:
|
||||
K = torch.randn(
|
||||
batch, seqlen_k, nheads_kv, hdim, dtype=torch.bfloat16, device="cuda"
|
||||
)
|
||||
V = torch.randn(
|
||||
batch, seqlen_k, nheads_kv, hdimv, dtype=torch.bfloat16, device="cuda"
|
||||
)
|
||||
|
||||
mQ = from_dlpack(Q, assumed_align=16).mark_layout_dynamic(
|
||||
leading_dim=Q.ndim - 1
|
||||
)
|
||||
mQv = from_dlpack(Qv, assumed_align=16).mark_layout_dynamic(
|
||||
leading_dim=Qv.ndim - 1
|
||||
)
|
||||
mK = from_dlpack(K, assumed_align=16).mark_layout_dynamic(
|
||||
leading_dim=K.ndim - 1
|
||||
)
|
||||
mV = from_dlpack(V, assumed_align=16).mark_layout_dynamic(
|
||||
leading_dim=V.ndim - 1
|
||||
)
|
||||
mO = from_dlpack(O, assumed_align=16).mark_layout_dynamic(
|
||||
leading_dim=O.ndim - 1
|
||||
)
|
||||
mP = from_dlpack(P, assumed_align=16).mark_layout_dynamic(
|
||||
leading_dim=P.ndim - 1
|
||||
)
|
||||
mLSE = from_dlpack(lse, assumed_align=4).mark_layout_dynamic(
|
||||
leading_dim=lse.ndim - 1
|
||||
)
|
||||
mRowMax = from_dlpack(row_max, assumed_align=4).mark_layout_dynamic(
|
||||
leading_dim=row_max.ndim - 1
|
||||
)
|
||||
if gather_kv:
|
||||
mIndexTopk = from_dlpack(index_topk, assumed_align=16).mark_layout_dynamic(
|
||||
leading_dim=index_topk.ndim - 1
|
||||
)
|
||||
else:
|
||||
mIndexTopk = None
|
||||
|
||||
compile_kwargs = dict(mIndexTopk=mIndexTopk)
|
||||
if varlen_q:
|
||||
compile_kwargs["mCuSeqlensQ"] = from_dlpack(
|
||||
cu_seqlens_q_dummy, assumed_align=4
|
||||
)
|
||||
if varlen_k:
|
||||
compile_kwargs["mCuSeqlensK"] = from_dlpack(
|
||||
cu_seqlens_k_dummy, assumed_align=4
|
||||
)
|
||||
|
||||
if not has_qk:
|
||||
mQ = mK = None
|
||||
|
||||
if store_P is False:
|
||||
mP = mRowMax = None
|
||||
|
||||
kernel = cute.compile(
|
||||
FlashAttentionMLAForwardSm100(
|
||||
is_causal=is_causal,
|
||||
use_cpasync_load_KV=gather_kv,
|
||||
topk_length=topk_length if gather_kv else 2048,
|
||||
is_topk_gather=gather_kv,
|
||||
pack_gqa=pack_gqa,
|
||||
qhead_per_kvhead=qhead_per_kvhead,
|
||||
nheads_kv=nheads_kv,
|
||||
is_varlen_q=varlen_q,
|
||||
disable_bitmask=disable_bitmask,
|
||||
has_qk=has_qk,
|
||||
),
|
||||
mQ,
|
||||
mQv,
|
||||
mK,
|
||||
mV,
|
||||
mO,
|
||||
mLSE,
|
||||
softmax_scale,
|
||||
mP,
|
||||
mRowMax,
|
||||
**compile_kwargs,
|
||||
stream=stream,
|
||||
options="--keep-ptx --keep-cubin --generate-line-info",
|
||||
)
|
||||
dump_kernel_attributes(kernel)
|
||||
compile_cache[compile_key] = kernel
|
||||
|
||||
# ================================================================
|
||||
# ---- Generate variable seqlens for this run ----
|
||||
if varlen_q:
|
||||
torch.manual_seed(seed + 1000)
|
||||
# When causal without varlen_k, every per-batch seqlen_q must not exceed seqlen_k.
|
||||
max_seqlen_q = seqlen_k if (is_causal and not varlen_k) else seqlen_q
|
||||
seqlens_q = torch.randint(1, max_seqlen_q + 1, (batch,), dtype=torch.int32)
|
||||
cu_seqlens_q = torch.zeros(batch + 1, dtype=torch.int32, device="cuda")
|
||||
cu_seqlens_q[1:] = seqlens_q.cumsum(0).to(torch.int32).cuda()
|
||||
total_q = cu_seqlens_q[-1].item()
|
||||
else:
|
||||
seqlens_q = torch.full((batch,), seqlen_q, dtype=torch.int32)
|
||||
total_q = None # unused
|
||||
|
||||
if varlen_k:
|
||||
torch.manual_seed(seed + 2000)
|
||||
# Each batch item must have at least topk_length keys so topk gather is valid.
|
||||
min_seqlen_k = topk_length if gather_kv else 1
|
||||
seqlens_k = torch.randint(
|
||||
min_seqlen_k, seqlen_k + 1, (batch,), dtype=torch.int32
|
||||
)
|
||||
# When causal, every batch item needs seqlens_k[b] >= seqlens_q[b].
|
||||
if is_causal:
|
||||
seqlens_k = torch.maximum(seqlens_k, seqlens_q)
|
||||
cu_seqlens_k = torch.zeros(batch + 1, dtype=torch.int32, device="cuda")
|
||||
cu_seqlens_k[1:] = seqlens_k.cumsum(0).to(torch.int32).cuda()
|
||||
total_k = cu_seqlens_k[-1].item()
|
||||
else:
|
||||
seqlens_k = torch.full((batch,), seqlen_k, dtype=torch.int32)
|
||||
total_k = None # unused
|
||||
|
||||
torch.manual_seed(seed) # restore main seed before drawing actual tensors
|
||||
|
||||
# ---- Allocate Q / Qv / O / lse ----
|
||||
if varlen_q:
|
||||
Q = torch.randn(total_q, nheads, hdim, dtype=torch.bfloat16, device="cuda")
|
||||
Qv = torch.randn(total_q, nheads, hdimv, dtype=torch.bfloat16, device="cuda")
|
||||
O = torch.empty(total_q, nheads, hdimv, dtype=torch.bfloat16, device="cuda")
|
||||
P = torch.empty(
|
||||
total_q, nheads, P_k_length, dtype=torch.bfloat16, device="cuda"
|
||||
)
|
||||
lse = torch.empty(total_q, nheads, dtype=torch.float32, device="cuda")
|
||||
row_max = torch.empty(
|
||||
total_q_dummy, P_k_length // 128, nheads, dtype=torch.float32, device="cuda"
|
||||
)
|
||||
else:
|
||||
Q = torch.randn(
|
||||
batch, seqlen_q, nheads, hdim, dtype=torch.bfloat16, device="cuda"
|
||||
)
|
||||
Qv = torch.randn(
|
||||
batch, seqlen_q, nheads, hdimv, dtype=torch.bfloat16, device="cuda"
|
||||
)
|
||||
O = torch.empty(
|
||||
batch, seqlen_q, nheads, hdimv, dtype=torch.bfloat16, device="cuda"
|
||||
)
|
||||
P = torch.empty(
|
||||
batch, seqlen_q, nheads, P_k_length, dtype=torch.bfloat16, device="cuda"
|
||||
)
|
||||
lse = torch.empty(batch, seqlen_q, nheads, dtype=torch.float32, device="cuda")
|
||||
row_max = torch.empty(
|
||||
batch,
|
||||
seqlen_q,
|
||||
P_k_length // 128,
|
||||
nheads,
|
||||
dtype=torch.float32,
|
||||
device="cuda",
|
||||
)
|
||||
|
||||
# ---- Allocate K / V ----
|
||||
if varlen_k:
|
||||
K = torch.randn(total_k, nheads_kv, hdim, dtype=torch.bfloat16, device="cuda")
|
||||
V = torch.randn(total_k, nheads_kv, hdimv, dtype=torch.bfloat16, device="cuda")
|
||||
else:
|
||||
K = torch.randn(
|
||||
batch, seqlen_k, nheads_kv, hdim, dtype=torch.bfloat16, device="cuda"
|
||||
)
|
||||
V = torch.randn(
|
||||
batch, seqlen_k, nheads_kv, hdimv, dtype=torch.bfloat16, device="cuda"
|
||||
)
|
||||
|
||||
# ---- Generate index_topk with per-batch valid ranges when varlen_k ----
|
||||
# index_topk shape: (total_q, topk_length) if varlen_q else (batch, seqlen_q, topk_length)
|
||||
if gather_kv:
|
||||
topk_parts = []
|
||||
for b in range(batch):
|
||||
sl_q_b = seqlens_q[b].item()
|
||||
sl_k_b = seqlens_k[b].item()
|
||||
# Draw topk_length unique indices from [0, sl_k_b) for each query in this batch item.
|
||||
topk_b = (
|
||||
torch.rand(sl_q_b, sl_k_b, device="cuda")
|
||||
.argsort(dim=-1)[..., :topk_length]
|
||||
.to(torch.int32)
|
||||
) # (sl_q_b, topk_length), all < sl_k_b
|
||||
topk_parts.append(topk_b)
|
||||
|
||||
if varlen_q:
|
||||
index_topk = torch.cat(topk_parts, dim=0) # (total_q, topk_length)
|
||||
else:
|
||||
index_topk = torch.stack(
|
||||
topk_parts, dim=0
|
||||
) # (batch, seqlen_q, topk_length)
|
||||
else:
|
||||
index_topk = None
|
||||
|
||||
# ---- Reference computation (per-batch loop covers all four varlen combos) ----
|
||||
O_ref_list, O_pt_list, lse_ref_list, lse_pt_list = [], [], [], []
|
||||
for b in range(batch):
|
||||
qs = cu_seqlens_q[b].item() if varlen_q else b * seqlen_q
|
||||
qe = cu_seqlens_q[b + 1].item() if varlen_q else (b + 1) * seqlen_q
|
||||
ks = cu_seqlens_k[b].item() if varlen_k else b * seqlen_k
|
||||
ke = cu_seqlens_k[b + 1].item() if varlen_k else (b + 1) * seqlen_k
|
||||
|
||||
Q_b = (
|
||||
Q[qs:qe].unsqueeze(0) if varlen_q else Q[b : b + 1]
|
||||
) # (1, sl_q, nheads, hdim)
|
||||
Qv_b = (
|
||||
Qv[qs:qe].unsqueeze(0) if varlen_q else Qv[b : b + 1]
|
||||
) # (1, sl_q, nheads, hdimv)
|
||||
K_b = (
|
||||
K[ks:ke].unsqueeze(0) if varlen_k else K[b : b + 1]
|
||||
) # (1, sl_k, nheads_kv, hdim)
|
||||
V_b = (
|
||||
V[ks:ke].unsqueeze(0) if varlen_k else V[b : b + 1]
|
||||
) # (1, sl_k, nheads_kv, hdimv)
|
||||
if gather_kv:
|
||||
topk_b = (
|
||||
index_topk[qs:qe].unsqueeze(0) if varlen_q else index_topk[b : b + 1]
|
||||
)
|
||||
else:
|
||||
topk_b = None
|
||||
|
||||
O_b, _, lse_b = attention_ref(
|
||||
Q_b if has_qk else None,
|
||||
K_b if has_qk else None,
|
||||
V_b,
|
||||
qv=Qv_b,
|
||||
causal=is_causal,
|
||||
return_lse=True,
|
||||
gather_kv_indices=topk_b,
|
||||
)
|
||||
O_pt_b, _, lse_pt_b = attention_ref(
|
||||
Q_b if has_qk else None,
|
||||
K_b if has_qk else None,
|
||||
V_b,
|
||||
qv=Qv_b,
|
||||
causal=is_causal,
|
||||
upcast=False,
|
||||
reorder_ops=True,
|
||||
return_lse=True,
|
||||
gather_kv_indices=topk_b,
|
||||
)
|
||||
O_ref_list.append(O_b.squeeze(0))
|
||||
O_pt_list.append(O_pt_b.squeeze(0))
|
||||
lse_ref_list.append(lse_b.squeeze(0))
|
||||
lse_pt_list.append(lse_pt_b.squeeze(0))
|
||||
|
||||
cat_dim_o = 0 if (varlen_q) else 0 # always 0: leading token/batch dim
|
||||
cat_dim_lse = -1 if (varlen_q) else -1 # always last: token dim
|
||||
|
||||
if varlen_q:
|
||||
O_ref = torch.cat(O_ref_list, dim=0) # (total_q, nheads, hdimv)
|
||||
O_pt = torch.cat(O_pt_list, dim=0)
|
||||
lse_ref = torch.cat(lse_ref_list, dim=-1) # (nheads, total_q)
|
||||
lse_pt = torch.cat(lse_pt_list, dim=-1)
|
||||
else:
|
||||
O_ref = torch.stack(O_ref_list, dim=0) # (batch, seqlen_q, nheads, hdimv)
|
||||
O_pt = torch.stack(O_pt_list, dim=0)
|
||||
lse_ref = torch.stack(lse_ref_list, dim=0) # (batch, nheads, seqlen_q)
|
||||
lse_pt = torch.stack(lse_pt_list, dim=0)
|
||||
|
||||
rtol = 2
|
||||
atol = 2 * (O_ref + 0.3 - 0.3 - O_ref).abs().max().item()
|
||||
|
||||
# ---- CuTe tensor wrappers ----
|
||||
mQ = from_dlpack(Q, assumed_align=16).mark_layout_dynamic(leading_dim=Q.ndim - 1)
|
||||
mQv = from_dlpack(Qv, assumed_align=16).mark_layout_dynamic(leading_dim=Qv.ndim - 1)
|
||||
mK = from_dlpack(K, assumed_align=16).mark_layout_dynamic(leading_dim=K.ndim - 1)
|
||||
mV = from_dlpack(V, assumed_align=16).mark_layout_dynamic(leading_dim=V.ndim - 1)
|
||||
mO = from_dlpack(O, assumed_align=16).mark_layout_dynamic(leading_dim=O.ndim - 1)
|
||||
mP = from_dlpack(P, assumed_align=16).mark_layout_dynamic(leading_dim=P.ndim - 1)
|
||||
mLSE = from_dlpack(lse, assumed_align=4).mark_layout_dynamic(
|
||||
leading_dim=lse.ndim - 1
|
||||
)
|
||||
mRowMax = from_dlpack(row_max, assumed_align=4).mark_layout_dynamic(
|
||||
leading_dim=row_max.ndim - 1
|
||||
)
|
||||
if index_topk is not None:
|
||||
mIndexTopk = from_dlpack(index_topk, assumed_align=16).mark_layout_dynamic(
|
||||
leading_dim=index_topk.ndim - 1
|
||||
)
|
||||
else:
|
||||
mIndexTopk = None
|
||||
|
||||
run_kwargs = dict(mIndexTopk=mIndexTopk)
|
||||
if varlen_q:
|
||||
run_kwargs["mCuSeqlensQ"] = from_dlpack(cu_seqlens_q, assumed_align=4)
|
||||
if varlen_k:
|
||||
run_kwargs["mCuSeqlensK"] = from_dlpack(cu_seqlens_k, assumed_align=4)
|
||||
|
||||
if not has_qk:
|
||||
mQ = mK = None
|
||||
|
||||
if store_P is False:
|
||||
mP = mRowMax = None
|
||||
|
||||
# ---- Run kernel ----
|
||||
compile_cache[compile_key](
|
||||
mQ,
|
||||
mQv,
|
||||
mK,
|
||||
mV,
|
||||
mO,
|
||||
mLSE,
|
||||
softmax_scale,
|
||||
mP,
|
||||
mRowMax,
|
||||
**run_kwargs,
|
||||
stream=stream,
|
||||
)
|
||||
|
||||
O_ref_max = O_ref.abs().max().item()
|
||||
O_max = O.abs().max().item()
|
||||
print(f"Pytorch O max = {O_ref_max} and our O max = {O_max}")
|
||||
print(f"Pytorch max O diff: {(O_pt - O_ref).abs().max().item()}")
|
||||
print(f"Pytorch mean O diff: {(O_pt - O_ref).abs().mean().item()}")
|
||||
print(f"Max abs diff O, O_ref: {(O - O_ref).abs().max().item()}")
|
||||
print(f"Mean abs diff O, O_ref: {(O - O_ref).abs().mean().item()}")
|
||||
|
||||
lse = lse.transpose(-1, -2)
|
||||
lse_ref_max = lse_ref.abs().max().item()
|
||||
lse_max = lse.abs().max().item()
|
||||
print(f"Pytorch LSE max = {lse_ref_max} and our LSE max = {lse_max}")
|
||||
print(f"Pytorch LSE max diff: {(lse_pt - lse_ref).abs().max().item()}")
|
||||
print(f"Pytorch LSE mean diff: {(lse_pt - lse_ref).abs().mean().item()}")
|
||||
print(f"Max abs diff LSE: {(lse - lse_ref).abs().max().item()}")
|
||||
print(f"Mean abs diff LSE: {(lse - lse_ref).abs().mean().item()}")
|
||||
|
||||
if validate:
|
||||
assert (O - O_ref).abs().max().item() <= rtol * (
|
||||
O_pt - O_ref
|
||||
).abs().max().item() + atol
|
||||
varlen_tag = ""
|
||||
if varlen_q:
|
||||
varlen_tag += f", total_q:{total_q}"
|
||||
if varlen_k:
|
||||
varlen_tag += f", total_k:{total_k}"
|
||||
print(
|
||||
f"batch:{batch:3d}, nheads:{nheads:3d}, seqlen_q:{seqlen_q:5d}, seqlen_k:{seqlen_k:5d}"
|
||||
f"{varlen_tag}, iter:{iter:2d} PASSED"
|
||||
)
|
||||
else:
|
||||
print(mO)
|
||||
print(
|
||||
f"batch:{batch:3d}, nheads:{nheads:3d}, seqlen_q:{seqlen_q:5d}, seqlen_k:{seqlen_k:5d}"
|
||||
f", iter:{iter:2d} RUN (NOT TESTING CORRECTNESS)"
|
||||
)
|
||||
|
||||
return None
|
||||
|
||||
|
||||
def timeit(fn, *args, **kwargs):
|
||||
# Synchronize before timing
|
||||
torch.cuda.synchronize()
|
||||
|
||||
# Warmup
|
||||
for _ in range(10):
|
||||
fn(*args, **kwargs)
|
||||
|
||||
# Benchmark using PyTorch's Timer
|
||||
t = benchmark.Timer(
|
||||
stmt="fn(*args, **kwargs)", globals={"fn": fn, "args": args, "kwargs": kwargs}
|
||||
)
|
||||
|
||||
# Time it multiple runs
|
||||
measurement = t.timeit(20) # 20 repeats
|
||||
avg_time = measurement.mean # Average time in seconds
|
||||
|
||||
time.sleep(1)
|
||||
|
||||
return avg_time
|
||||
|
||||
|
||||
def benchmark_mla_kernel(
|
||||
batch=1,
|
||||
seqlen_q=2048,
|
||||
seqlen_k=2048,
|
||||
topk_length=2048,
|
||||
nheads=128,
|
||||
hdim=64,
|
||||
hdimv=512,
|
||||
compile_cache=dict(),
|
||||
gather_kv=True,
|
||||
is_causal=False,
|
||||
disable_bitmask=False,
|
||||
store_P=False,
|
||||
):
|
||||
assert hdim == 64, "hdim must be 64"
|
||||
assert hdimv == 512, "hdimv must be 512"
|
||||
|
||||
qhead_per_kvhead = nheads
|
||||
nheads_kv = 1
|
||||
pack_gqa = True
|
||||
softmax_scale = 1.0 / math.sqrt(hdim + hdimv)
|
||||
seqlen_k_rounded = (seqlen_k + 128 - 1) // 128 * 128
|
||||
P_k_length = seqlen_k_rounded if not gather_kv else topk_length
|
||||
|
||||
torch_stream = torch.cuda.current_stream()
|
||||
stream = cuda.CUstream(torch_stream.cuda_stream)
|
||||
|
||||
compile_key = (
|
||||
is_causal,
|
||||
gather_kv,
|
||||
topk_length if gather_kv else None,
|
||||
pack_gqa,
|
||||
qhead_per_kvhead,
|
||||
nheads_kv,
|
||||
disable_bitmask,
|
||||
)
|
||||
if compile_key not in compile_cache:
|
||||
Q = torch.randn(
|
||||
batch, seqlen_q, nheads, hdim, dtype=torch.bfloat16, device="cuda"
|
||||
)
|
||||
Qv = torch.randn(
|
||||
batch, seqlen_q, nheads, hdimv, dtype=torch.bfloat16, device="cuda"
|
||||
)
|
||||
K = torch.randn(
|
||||
batch, seqlen_k, nheads_kv, hdim, dtype=torch.bfloat16, device="cuda"
|
||||
)
|
||||
V = torch.randn(
|
||||
batch, seqlen_k, nheads_kv, hdimv, dtype=torch.bfloat16, device="cuda"
|
||||
)
|
||||
O = torch.empty(
|
||||
batch, seqlen_q, nheads, hdimv, dtype=torch.bfloat16, device="cuda"
|
||||
)
|
||||
P = torch.empty(
|
||||
batch, seqlen_q, nheads, P_k_length, dtype=torch.bfloat16, device="cuda"
|
||||
)
|
||||
index_topk = (
|
||||
torch.rand(batch, seqlen_q, topk_length, device="cuda")
|
||||
.argsort(dim=-1)
|
||||
.to(torch.int32)
|
||||
)
|
||||
|
||||
mQ = from_dlpack(Q, assumed_align=16).mark_layout_dynamic(
|
||||
leading_dim=Q.ndim - 1
|
||||
)
|
||||
mQv = from_dlpack(Qv, assumed_align=16).mark_layout_dynamic(
|
||||
leading_dim=Qv.ndim - 1
|
||||
)
|
||||
mK = from_dlpack(K, assumed_align=16).mark_layout_dynamic(
|
||||
leading_dim=K.ndim - 1
|
||||
)
|
||||
mV = from_dlpack(V, assumed_align=16).mark_layout_dynamic(
|
||||
leading_dim=V.ndim - 1
|
||||
)
|
||||
mO = from_dlpack(O, assumed_align=16).mark_layout_dynamic(
|
||||
leading_dim=O.ndim - 1
|
||||
)
|
||||
mP = from_dlpack(P, assumed_align=16).mark_layout_dynamic(
|
||||
leading_dim=P.ndim - 1
|
||||
)
|
||||
if gather_kv:
|
||||
mIndexTopk = from_dlpack(index_topk, assumed_align=16).mark_layout_dynamic(
|
||||
leading_dim=index_topk.ndim - 1
|
||||
)
|
||||
else:
|
||||
mIndexTopk = None
|
||||
|
||||
mLSE = None
|
||||
|
||||
if store_P is False:
|
||||
mP = None
|
||||
|
||||
kernel = cute.compile(
|
||||
FlashAttentionMLAForwardSm100(
|
||||
is_causal=is_causal,
|
||||
use_cpasync_load_KV=gather_kv,
|
||||
topk_length=topk_length if gather_kv else 2048,
|
||||
is_topk_gather=gather_kv,
|
||||
pack_gqa=pack_gqa,
|
||||
qhead_per_kvhead=qhead_per_kvhead,
|
||||
nheads_kv=nheads_kv,
|
||||
disable_bitmask=disable_bitmask,
|
||||
),
|
||||
mQ,
|
||||
mQv,
|
||||
mK,
|
||||
mV,
|
||||
mO,
|
||||
mLSE,
|
||||
softmax_scale,
|
||||
mP=mP,
|
||||
mIndexTopk=mIndexTopk,
|
||||
stream=stream,
|
||||
)
|
||||
compile_cache[compile_key] = kernel
|
||||
|
||||
Q = torch.randn(batch, seqlen_q, nheads, hdim, dtype=torch.bfloat16, device="cuda")
|
||||
Qv = torch.randn(
|
||||
batch, seqlen_q, nheads, hdimv, dtype=torch.bfloat16, device="cuda"
|
||||
)
|
||||
K = torch.randn(
|
||||
batch, seqlen_k, nheads_kv, hdim, dtype=torch.bfloat16, device="cuda"
|
||||
)
|
||||
V = torch.randn(
|
||||
batch, seqlen_k, nheads_kv, hdimv, dtype=torch.bfloat16, device="cuda"
|
||||
)
|
||||
O = torch.empty(batch, seqlen_q, nheads, hdimv, dtype=torch.bfloat16, device="cuda")
|
||||
P = torch.empty(
|
||||
batch, seqlen_q, nheads, P_k_length, dtype=torch.bfloat16, device="cuda"
|
||||
)
|
||||
|
||||
index_topk = (
|
||||
torch.rand(batch, seqlen_q, topk_length, device="cuda")
|
||||
.argsort(dim=-1)
|
||||
.to(torch.int32)
|
||||
)
|
||||
|
||||
mQ = from_dlpack(Q, assumed_align=16).mark_layout_dynamic(leading_dim=Q.ndim - 1)
|
||||
mQv = from_dlpack(Qv, assumed_align=16).mark_layout_dynamic(leading_dim=Qv.ndim - 1)
|
||||
mK = from_dlpack(K, assumed_align=16).mark_layout_dynamic(leading_dim=K.ndim - 1)
|
||||
mV = from_dlpack(V, assumed_align=16).mark_layout_dynamic(leading_dim=V.ndim - 1)
|
||||
mO = from_dlpack(O, assumed_align=16).mark_layout_dynamic(leading_dim=O.ndim - 1)
|
||||
mP = from_dlpack(P, assumed_align=16).mark_layout_dynamic(leading_dim=P.ndim - 1)
|
||||
if gather_kv:
|
||||
mIndexTopk = from_dlpack(index_topk, assumed_align=16).mark_layout_dynamic(
|
||||
leading_dim=index_topk.ndim - 1
|
||||
)
|
||||
else:
|
||||
mIndexTopk = None
|
||||
mLSE = None
|
||||
|
||||
if store_P is False:
|
||||
mP = None
|
||||
|
||||
exec_time_in_s = timeit(
|
||||
compile_cache[compile_key],
|
||||
mQ,
|
||||
mQv,
|
||||
mK,
|
||||
mV,
|
||||
mO,
|
||||
mLSE,
|
||||
softmax_scale,
|
||||
mP=mP,
|
||||
mIndexTopk=mIndexTopk,
|
||||
stream=stream,
|
||||
)
|
||||
|
||||
seqlen_k_eff = topk_length if gather_kv else seqlen_k
|
||||
|
||||
FLOPs = 2 * batch * nheads * seqlen_q * seqlen_k_eff * (hdim + 2 * hdimv)
|
||||
if is_causal and not gather_kv:
|
||||
FLOPs /= 2
|
||||
|
||||
TFLOPS = FLOPs / exec_time_in_s / 1e12
|
||||
|
||||
q_bytes = 2 * batch * nheads * seqlen_q * hdim
|
||||
qv_bytes = 2 * batch * nheads * seqlen_q * hdimv
|
||||
k_bytes = 2 * batch * nheads_kv * seqlen_k_eff * hdim
|
||||
v_bytes = 2 * batch * nheads_kv * seqlen_k_eff * hdimv
|
||||
o_bytes = 2 * batch * nheads * seqlen_q * hdimv
|
||||
total_bytes = q_bytes + qv_bytes + k_bytes + v_bytes + o_bytes
|
||||
TBs = total_bytes / exec_time_in_s / 1e12
|
||||
|
||||
print(
|
||||
f"batch: {batch}, seqlen_q: {seqlen_q}, seqlen_k: {seqlen_k}, nheads: {nheads}, -> {exec_time_in_s * 1e3:.2f} ms, {TFLOPS:.2f} TFLOPS, {TBs:.2f} TBs"
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
run_test = True
|
||||
run_benchmark = True
|
||||
gather_kv = True
|
||||
is_causal = False
|
||||
pack_gqa = True
|
||||
topk_length = 2048
|
||||
varlen_q = False
|
||||
varlen_k = False
|
||||
disable_bitmask = False
|
||||
validate = True
|
||||
has_qk = True
|
||||
|
||||
if run_test:
|
||||
if not gather_kv:
|
||||
seqlen_q_test_values = range(1, 4002, 400)
|
||||
seqlen_k_test_values = range(1, 4002, 400)
|
||||
else:
|
||||
seqlen_q_test_values = range(1, 1001, 200)
|
||||
seqlen_k_test_values = range(topk_length, 9001, 2000)
|
||||
seqlen_q_test_values = [4096]
|
||||
seqlen_k_test_values = [4096]
|
||||
nheads_test_values = [128]
|
||||
batch_test_values = [1]
|
||||
test_configs = [
|
||||
(
|
||||
batch,
|
||||
nheads,
|
||||
seqlen_q,
|
||||
seqlen_k,
|
||||
)
|
||||
for batch in batch_test_values
|
||||
for nheads in nheads_test_values
|
||||
for seqlen_q in seqlen_q_test_values
|
||||
for seqlen_k in seqlen_k_test_values
|
||||
]
|
||||
iters_per_config = 1
|
||||
compile_cache = dict()
|
||||
print("=" * 40)
|
||||
print("Testing MLA Kernel")
|
||||
print("=" * 40)
|
||||
for config in test_configs:
|
||||
batch, nheads, seqlen_q, seqlen_k = config
|
||||
# if is_causal and seqlen_k < seqlen_q:
|
||||
# continue
|
||||
for iter in range(iters_per_config):
|
||||
test_mla_kernel(
|
||||
seqlen_q=seqlen_q,
|
||||
seqlen_k=seqlen_k,
|
||||
topk_length=topk_length,
|
||||
nheads=nheads,
|
||||
batch=batch,
|
||||
iter=iter,
|
||||
compile_cache=compile_cache,
|
||||
validate=validate,
|
||||
seed=0,
|
||||
gather_kv=gather_kv,
|
||||
pack_gqa=pack_gqa,
|
||||
is_causal=is_causal,
|
||||
varlen_q=varlen_q,
|
||||
varlen_k=varlen_k,
|
||||
disable_bitmask=disable_bitmask,
|
||||
has_qk=has_qk,
|
||||
)
|
||||
if run_benchmark:
|
||||
if gather_kv:
|
||||
seqlen_q_benchmark_values = [1]
|
||||
seqlen_k_benchmark_values = [8192 * 2]
|
||||
nheads_benchmark_values = [128]
|
||||
batch_benchmark_values = [128]
|
||||
else:
|
||||
seqlen_q_benchmark_values = [1]
|
||||
seqlen_k_benchmark_values = [8192]
|
||||
nheads_benchmark_values = [128]
|
||||
batch_benchmark_values = [128]
|
||||
# seqlen_q_benchmark_values = [4096]
|
||||
# seqlen_k_benchmark_values = [4096]
|
||||
# nheads_benchmark_values = [16]
|
||||
# batch_benchmark_values = [8]
|
||||
benchmark_configs = [
|
||||
(
|
||||
batch,
|
||||
nheads,
|
||||
seqlen_q,
|
||||
seqlen_k,
|
||||
)
|
||||
for batch in batch_benchmark_values
|
||||
for nheads in nheads_benchmark_values
|
||||
for seqlen_q in seqlen_q_benchmark_values
|
||||
for seqlen_k in seqlen_k_benchmark_values
|
||||
]
|
||||
compile_cache = dict()
|
||||
print("=" * 40)
|
||||
print("Benchmarking MLA Kernel")
|
||||
print("=" * 40)
|
||||
for config in benchmark_configs:
|
||||
batch, nheads, seqlen_q, seqlen_k = config
|
||||
benchmark_mla_kernel(
|
||||
batch=batch,
|
||||
seqlen_q=seqlen_q,
|
||||
seqlen_k=seqlen_k,
|
||||
topk_length=topk_length,
|
||||
nheads=nheads,
|
||||
gather_kv=gather_kv,
|
||||
is_causal=is_causal,
|
||||
disable_bitmask=disable_bitmask,
|
||||
compile_cache=compile_cache,
|
||||
)
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
The public :func:`helion_fused_recurrent_kda_replayssm_decode` mirrors
|
||||
``fused_recurrent_linear_replayssm_decode(..., is_kda=True)`` from
|
||||
``sglang.kernels.ops.attention.fla.fused_recurrent_linear_replayssm``. That
|
||||
``sglang.kernels.ops.attention.linear.fused_recurrent_linear_replayssm``. That
|
||||
Triton kernel is gate-generic (GDN scalar gate / KDA per-K gate); this module
|
||||
implements the KDA path only, and additionally supports the bounded gate,
|
||||
which the Triton ReplaySSM kernel does not expose.
|
||||
|
||||
@@ -724,43 +724,3 @@ def linear_decode_forward_triton(
|
||||
# Reshape output and return
|
||||
output = rearrange(output, "b h n d -> b n (h d)")
|
||||
return output.squeeze(1).contiguous()
|
||||
|
||||
|
||||
class BailingLinearKernel:
|
||||
"""
|
||||
Linear attention kernel implementation for Bailing models.
|
||||
|
||||
This class is adapted from MiniMaxText01LinearKernel in vllm:
|
||||
https://github.com/vllm-project/vllm/blob/a9138e85b14047e06300685b48e3485b995425fb/vllm/model_executor/models/minimax_text_01.py#L289
|
||||
|
||||
The implementation maintains the same functionality while being renamed to
|
||||
match our Bailing model naming convention.
|
||||
"""
|
||||
|
||||
@staticmethod
|
||||
def jit_linear_forward_prefix(
|
||||
q: torch.Tensor,
|
||||
k: torch.Tensor,
|
||||
v: torch.Tensor,
|
||||
kv_caches: torch.Tensor,
|
||||
slope_rate: torch.Tensor,
|
||||
block_size: int,
|
||||
layer_idx: int = None,
|
||||
**kwargs,
|
||||
) -> torch.Tensor:
|
||||
|
||||
slope_rate = slope_rate.to(torch.float32)
|
||||
should_pad_dim = q.dim() == 3
|
||||
if should_pad_dim:
|
||||
q = q.unsqueeze(0)
|
||||
k = k.unsqueeze(0)
|
||||
v = v.unsqueeze(0)
|
||||
b, h, n, d = q.shape
|
||||
e = d
|
||||
kv_history = kv_caches.reshape(1, h, d, e).contiguous()
|
||||
output, kv_history = lightning_attention(
|
||||
q, k, v, slope_rate, block_size=block_size, kv_history=kv_history
|
||||
)
|
||||
kv_caches.copy_(kv_history[:, :, -1, :, :].reshape(h, d, e))
|
||||
assert output.shape[0] == 1, "batch size must be 1"
|
||||
return output.squeeze(0).transpose(0, 1).reshape([n, h * d]).contiguous()
|
||||
|
||||
@@ -1,70 +0,0 @@
|
||||
import torch
|
||||
import triton
|
||||
import triton.language as tl
|
||||
|
||||
|
||||
@triton.jit
|
||||
def _apply_log_scaling_tau_kernel(
|
||||
x_ptr,
|
||||
tau_ptr, # [rows] fp32 (flattened per-row scale)
|
||||
out_ptr, # [rows, inner] contiguous, same dtype as x
|
||||
x_row_stride,
|
||||
inner,
|
||||
total,
|
||||
BLOCK: tl.constexpr,
|
||||
):
|
||||
pid = tl.program_id(0)
|
||||
offs = pid.to(tl.int64) * BLOCK + tl.arange(0, BLOCK)
|
||||
mask = offs < total
|
||||
row = offs // inner
|
||||
col = offs % inner
|
||||
x = tl.load(x_ptr + row * x_row_stride + col, mask=mask).to(tl.float32)
|
||||
tau = tl.load(tau_ptr + row, mask=mask)
|
||||
y = x * tau
|
||||
tl.store(out_ptr + offs, y.to(out_ptr.dtype.element_ty), mask=mask)
|
||||
|
||||
|
||||
def apply_log_scaling_tau(x: torch.Tensor, tau: torch.Tensor) -> torch.Tensor:
|
||||
"""out = (x.float() * tau).to(x.dtype) with tau broadcast per leading row,
|
||||
fused into one launch. x may carry a leading-dim stride (the q slice of the
|
||||
fused qkvr output); its trailing dims must be contiguous. No dynamo: the
|
||||
torch.compile'd predecessor's call sites spanned enough rank /
|
||||
dispatch-key / 0-1 specialization variants (target + de-tied MTP heads) to
|
||||
exceed the recompile limit, which crashed (fullgraph) or wedged capture
|
||||
(raised limit)."""
|
||||
rows = x.shape[0]
|
||||
inner = x.numel() // rows if rows else 0
|
||||
inner_contiguous = x.stride(-1) == 1 and (
|
||||
x.dim() == 2 or x.stride(-2) == x.shape[-1] * x.stride(-1)
|
||||
)
|
||||
if rows == 0 or inner == 0 or not inner_contiguous:
|
||||
return (x.float() * tau).to(x.dtype)
|
||||
|
||||
if (
|
||||
x.is_cuda
|
||||
and x.dtype == torch.bfloat16
|
||||
and inner % 8 == 0
|
||||
and x.data_ptr() % 16 == 0
|
||||
and (x.stride(0) * 2) % 16 == 0
|
||||
):
|
||||
# Vectorized JIT kernel (16B loads, one row divide per vector) --
|
||||
# bit-identical output (same fp32-mul + bf16-round), ~2-3x the
|
||||
# scalar triton kernel below at every size.
|
||||
from sglang.kernels.ops.elementwise.row_scale import row_scale_bf16
|
||||
|
||||
x2d = torch.as_strided(x, (rows, inner), (x.stride(0), 1))
|
||||
return row_scale_bf16(x2d, tau.reshape(rows).float()).view(x.shape)
|
||||
|
||||
out = torch.empty(x.shape, dtype=x.dtype, device=x.device)
|
||||
total = rows * inner
|
||||
BLOCK = 1024
|
||||
_apply_log_scaling_tau_kernel[(triton.cdiv(total, BLOCK),)](
|
||||
x,
|
||||
tau.reshape(rows).to(torch.float32),
|
||||
out,
|
||||
x.stride(0),
|
||||
inner,
|
||||
total,
|
||||
BLOCK=BLOCK,
|
||||
)
|
||||
return out
|
||||
@@ -590,3 +590,51 @@ def normal_decode_set_metadata(
|
||||
num_warps=block_cols // 32,
|
||||
num_stages=3,
|
||||
)
|
||||
|
||||
|
||||
@triton.jit
|
||||
def q8kv8_topk_length_backscan_kernel(
|
||||
indices_ptr,
|
||||
out_ptr,
|
||||
stride_row,
|
||||
topk,
|
||||
BLOCK: tl.constexpr,
|
||||
):
|
||||
row = tl.program_id(0).to(tl.int64)
|
||||
base = indices_ptr + row * stride_row
|
||||
off = topk
|
||||
length = 1
|
||||
found = 0
|
||||
while (found == 0) & (off > 0):
|
||||
off -= BLOCK
|
||||
idx = off + tl.arange(0, BLOCK)
|
||||
vals = tl.load(base + idx)
|
||||
pos = tl.max(tl.where(vals >= 0, idx, -1), axis=0)
|
||||
found = tl.where(pos >= 0, 1, found)
|
||||
length = tl.where(pos >= 0, pos + 1, length)
|
||||
tl.store(out_ptr + row, length)
|
||||
|
||||
|
||||
def q8kv8_topk_length_from_indices(indices: torch.Tensor) -> torch.Tensor:
|
||||
"""Per-row valid-topk count = last non-negative position + 1 (min 1).
|
||||
|
||||
``indices``: [s_q, topk] int32 topk output whose pad slots are -1.
|
||||
Backward block scan per row: the loop exits at the first block holding a
|
||||
valid entry, so the cost is proportional to the trailing pad run — one
|
||||
block (~topk/4 elements) for rows with a full topk, which dominate long
|
||||
contexts. Semantics match the unfused ``(indices >= 0) * ramp).amax``
|
||||
derivation exactly, including all-pad rows (length 1: one pad-only block
|
||||
keeps the kernel on its clamp+mask path, contributing zero).
|
||||
"""
|
||||
s_q, topk = indices.shape
|
||||
assert indices.dtype == torch.int32 and indices.stride(1) == 1
|
||||
out = torch.empty(s_q, dtype=torch.int32, device=indices.device)
|
||||
block = 512 if topk % 512 == 0 else (256 if topk % 256 == 0 else 128)
|
||||
q8kv8_topk_length_backscan_kernel[(s_q,)](
|
||||
indices,
|
||||
out,
|
||||
indices.stride(0),
|
||||
topk,
|
||||
BLOCK=block,
|
||||
)
|
||||
return out
|
||||
|
||||
@@ -0,0 +1,554 @@
|
||||
import torch
|
||||
import triton
|
||||
import triton.language as tl
|
||||
|
||||
|
||||
@triton.jit
|
||||
def concat_and_cast_mha_k_kernel(
|
||||
k_ptr,
|
||||
k_nope_ptr,
|
||||
k_rope_ptr,
|
||||
head_cnt: tl.constexpr,
|
||||
k_stride0: tl.constexpr,
|
||||
k_stride1: tl.constexpr,
|
||||
nope_stride0: tl.constexpr,
|
||||
nope_stride1: tl.constexpr,
|
||||
rope_stride0: tl.constexpr,
|
||||
nope_dim: tl.constexpr,
|
||||
rope_dim: tl.constexpr,
|
||||
):
|
||||
pid_loc = tl.program_id(0)
|
||||
head_range = tl.arange(0, head_cnt)
|
||||
|
||||
k_head_ptr = k_ptr + pid_loc * k_stride0 + head_range[:, None] * k_stride1
|
||||
|
||||
nope_offs = tl.arange(0, nope_dim)
|
||||
|
||||
src_nope_ptr = (
|
||||
k_nope_ptr
|
||||
+ pid_loc * nope_stride0
|
||||
+ head_range[:, None] * nope_stride1
|
||||
+ nope_offs[None, :]
|
||||
)
|
||||
dst_nope_ptr = k_head_ptr + nope_offs[None, :]
|
||||
|
||||
src_nope = tl.load(src_nope_ptr)
|
||||
tl.store(dst_nope_ptr, src_nope)
|
||||
|
||||
rope_offs = tl.arange(0, rope_dim)
|
||||
src_rope_ptr = k_rope_ptr + pid_loc * rope_stride0 + rope_offs[None, :]
|
||||
dst_rope_ptr = k_head_ptr + nope_dim + rope_offs[None, :]
|
||||
src_rope = tl.load(src_rope_ptr)
|
||||
tl.store(dst_rope_ptr, src_rope)
|
||||
|
||||
|
||||
@triton.jit
|
||||
def concat_and_cast_mha_k_pad_kernel(
|
||||
k_ptr,
|
||||
k_nope_ptr,
|
||||
k_rope_ptr,
|
||||
head_cnt: tl.constexpr,
|
||||
k_stride0: tl.constexpr,
|
||||
k_stride1: tl.constexpr,
|
||||
nope_stride0: tl.constexpr,
|
||||
nope_stride1: tl.constexpr,
|
||||
rope_stride0: tl.constexpr,
|
||||
nope_dim: tl.constexpr,
|
||||
rope_dim: tl.constexpr,
|
||||
HEAD_BLOCK: tl.constexpr,
|
||||
):
|
||||
"""
|
||||
concat_and_cast_mha_k_kernel for a head count that is not a power of two.
|
||||
tl.arange needs a power-of-two extent, so walk HEAD_BLOCK == next_power_of_2(head_cnt)
|
||||
lanes and mask the tail.
|
||||
"""
|
||||
pid_loc = tl.program_id(0)
|
||||
head_range = tl.arange(0, HEAD_BLOCK)
|
||||
head_mask = head_range < head_cnt
|
||||
|
||||
k_head_ptr = k_ptr + pid_loc * k_stride0 + head_range[:, None] * k_stride1
|
||||
|
||||
nope_offs = tl.arange(0, nope_dim)
|
||||
|
||||
src_nope_ptr = (
|
||||
k_nope_ptr
|
||||
+ pid_loc * nope_stride0
|
||||
+ head_range[:, None] * nope_stride1
|
||||
+ nope_offs[None, :]
|
||||
)
|
||||
dst_nope_ptr = k_head_ptr + nope_offs[None, :]
|
||||
|
||||
nope_mask = tl.broadcast_to(head_mask[:, None], (HEAD_BLOCK, nope_dim))
|
||||
src_nope = tl.load(src_nope_ptr, mask=nope_mask, other=0)
|
||||
tl.store(dst_nope_ptr, src_nope, mask=nope_mask)
|
||||
|
||||
rope_offs = tl.arange(0, rope_dim)
|
||||
src_rope_ptr = k_rope_ptr + pid_loc * rope_stride0 + rope_offs[None, :]
|
||||
dst_rope_ptr = k_head_ptr + nope_dim + rope_offs[None, :]
|
||||
rope_mask = tl.broadcast_to(head_mask[:, None], (HEAD_BLOCK, rope_dim))
|
||||
src_rope = tl.load(src_rope_ptr)
|
||||
tl.store(
|
||||
dst_rope_ptr,
|
||||
tl.broadcast_to(src_rope, (HEAD_BLOCK, rope_dim)),
|
||||
mask=rope_mask,
|
||||
)
|
||||
|
||||
|
||||
def concat_and_cast_mha_k_triton(
|
||||
k: torch.Tensor,
|
||||
k_nope: torch.Tensor,
|
||||
k_rope: torch.Tensor,
|
||||
):
|
||||
# The source data type will be implicitly converted to the target data type.
|
||||
assert len(k.shape) == 3 and len(k_nope.shape) == 3 and len(k_rope.shape) == 3, (
|
||||
f"shape should be 3d, but got {k.shape=}, {k_nope.shape=}, {k_rope.shape=}"
|
||||
)
|
||||
assert k.shape[0] == k_nope.shape[0] and k.shape[0] == k_rope.shape[0], (
|
||||
f"invalid shape, got {k.shape=}, {k_nope.shape=}, {k_rope.shape=}"
|
||||
)
|
||||
assert k.shape[1] == k_nope.shape[1] and 1 == k_rope.shape[1], (
|
||||
f"invalid shape, got {k.shape=}, {k_nope.shape=}, {k_rope.shape=}"
|
||||
)
|
||||
assert k.shape[-1] == k_nope.shape[-1] + k_rope.shape[-1], (
|
||||
f"invalid shape, got {k.shape=}, {k_nope.shape=}, {k_rope.shape=}"
|
||||
)
|
||||
|
||||
nope_dim = k_nope.shape[-1]
|
||||
rope_dim = k_rope.shape[-1]
|
||||
grid = (k.shape[0],)
|
||||
|
||||
head_cnt = k.shape[1]
|
||||
if head_cnt != triton.next_power_of_2(head_cnt):
|
||||
# Not the power of two concat_and_cast_mha_k_kernel's tl.arange needs;
|
||||
# Kimi-K3's 96 heads give 12 at TP8.
|
||||
concat_and_cast_mha_k_pad_kernel[grid](
|
||||
k,
|
||||
k_nope,
|
||||
k_rope,
|
||||
head_cnt,
|
||||
k.stride(0),
|
||||
k.stride(1),
|
||||
k_nope.stride(0),
|
||||
k_nope.stride(1),
|
||||
k_rope.stride(0),
|
||||
nope_dim,
|
||||
rope_dim,
|
||||
HEAD_BLOCK=triton.next_power_of_2(head_cnt),
|
||||
)
|
||||
return
|
||||
|
||||
concat_and_cast_mha_k_kernel[grid](
|
||||
k,
|
||||
k_nope,
|
||||
k_rope,
|
||||
k.shape[1],
|
||||
k.stride(0),
|
||||
k.stride(1),
|
||||
k_nope.stride(0),
|
||||
k_nope.stride(1),
|
||||
k_rope.stride(0),
|
||||
nope_dim,
|
||||
rope_dim,
|
||||
)
|
||||
|
||||
|
||||
@triton.jit
|
||||
def concat_and_cast_q_fp8_pad_kernel(
|
||||
qpad_ptr, # [num_tokens, pad_heads, NOPE+ROPE] fp8 (dst; only [:, :H, :] written)
|
||||
q_nope_ptr, # [num_tokens, H, NOPE] bf16
|
||||
q_rope_ptr, # [num_tokens, H, ROPE] bf16
|
||||
qpad_s0,
|
||||
qpad_s1,
|
||||
nope_s0,
|
||||
nope_s1,
|
||||
rope_s0,
|
||||
rope_s1,
|
||||
H: tl.constexpr,
|
||||
NOPE: tl.constexpr,
|
||||
ROPE: tl.constexpr,
|
||||
):
|
||||
# One program per token: write the H active heads of the padded fp8 q buffer,
|
||||
# fusing the bf16->fp8 cast (on store) with the nope/rope concat. Bit-exact vs the
|
||||
# two strided copy_() it replaces; ~3.7x faster because copy_ into the
|
||||
# 64-head-padded buffer is strided (~4.5x off memory-bound). Strides are passed in,
|
||||
# so q_nope/q_rope may be views of a [T, H, NOPE+ROPE] q (head-stride != last-dim).
|
||||
pid = tl.program_id(0)
|
||||
hr = tl.arange(0, H)
|
||||
qpad_head = qpad_ptr + pid * qpad_s0 + hr[:, None] * qpad_s1
|
||||
no = tl.arange(0, NOPE)
|
||||
src_n = tl.load(q_nope_ptr + pid * nope_s0 + hr[:, None] * nope_s1 + no[None, :])
|
||||
tl.store(qpad_head + no[None, :], src_n)
|
||||
# NoPE models pass ROPE=0; tl.arange rejects empty ranges, so skip statically.
|
||||
if ROPE > 0:
|
||||
ro = tl.arange(0, ROPE)
|
||||
src_r = tl.load(
|
||||
q_rope_ptr + pid * rope_s0 + hr[:, None] * rope_s1 + ro[None, :]
|
||||
)
|
||||
tl.store(qpad_head + NOPE + ro[None, :], src_r)
|
||||
|
||||
|
||||
def concat_and_cast_q_fp8_pad(q_fp8_pad, q_nope, q_rope, num_heads):
|
||||
"""fused bf16->fp8 concat-cast of q_nope/q_rope into the active
|
||||
[:, :num_heads, :] slice of the padded fp8 q buffer. Bit-exact replacement for the
|
||||
two strided converting copy_() in the Q8KV8 prefill q-prep, ~3.7x faster. Requires
|
||||
num_heads / nope_dim / rope_dim to be powers of two (always true for DeepSeek: 128
|
||||
heads / any TP, 512 nope, 64 rope); rope_dim may also be 0 for NoPE models."""
|
||||
num_tokens = q_nope.shape[0]
|
||||
nope_dim = q_nope.shape[-1]
|
||||
rope_dim = q_rope.shape[-1]
|
||||
concat_and_cast_q_fp8_pad_kernel[(num_tokens,)](
|
||||
q_fp8_pad,
|
||||
q_nope,
|
||||
q_rope,
|
||||
q_fp8_pad.stride(0),
|
||||
q_fp8_pad.stride(1),
|
||||
q_nope.stride(0),
|
||||
q_nope.stride(1),
|
||||
q_rope.stride(0),
|
||||
q_rope.stride(1),
|
||||
H=num_heads,
|
||||
NOPE=nope_dim,
|
||||
ROPE=rope_dim,
|
||||
)
|
||||
|
||||
|
||||
@triton.jit
|
||||
def absorbed_bmm_concat_cast_q_fp8_kernel(
|
||||
qout_ptr, # [num_tokens, pad_heads, N+ROPE] fp8 (dst; only [:, :H, :] written)
|
||||
a_ptr, # q_nope (pre-absorb) [num_tokens, H, K] bf16
|
||||
b_ptr, # w_kc [H, K, N] bf16 (any strides; typically N-major)
|
||||
rope_ptr, # q_rope (post-rope) [num_tokens, H, ROPE] bf16
|
||||
T, # num_tokens (runtime; masked)
|
||||
qout_s0,
|
||||
qout_s1,
|
||||
a_s0,
|
||||
a_s1,
|
||||
b_s0,
|
||||
b_s1,
|
||||
b_s2,
|
||||
rope_s0,
|
||||
rope_s1,
|
||||
K: tl.constexpr,
|
||||
N: tl.constexpr,
|
||||
ROPE: tl.constexpr,
|
||||
BLOCK_M: tl.constexpr,
|
||||
BLOCK_N: tl.constexpr,
|
||||
BLOCK_K: tl.constexpr,
|
||||
K_MODE: tl.constexpr,
|
||||
):
|
||||
# One program per (token-block, head): q_out[m, h, :N] = fp8(bf16(fp32(
|
||||
# q_nope[m, h, :K] @ w_kc[h, :K, :N]))) and q_out[m, h, N:] =
|
||||
# fp8(q_rope[m, h, :ROPE]). This makes q "born fp8": the absorbed bmm,
|
||||
# the nope/rope concat, and the bf16->fp8 cast collapse into one kernel,
|
||||
# so neither the bf16 q_nope_out ([H, T, N], written by cublas and re-read
|
||||
# by the concat-cast) nor the standalone concat-cast launch exist anymore.
|
||||
#
|
||||
# K handling (K_MODE selects the codegen for the nope-gemm K dimension;
|
||||
# every mode keeps the same fp32-accumulator -> bf16 -> fp8 epilogue):
|
||||
# 0 "single": BLOCK_K == K. Preload the whole [BLOCK_M, K] a-tile once,
|
||||
# one tl.dot per N-block — identical codegen to the original
|
||||
# power-of-2-only kernel (DeepSeek K=128). For non-power-of-2 K this
|
||||
# only compiles if the Triton build allows non-power-of-2 tl.arange
|
||||
# (Triton <= 3.5.x does NOT: "arange's range must be a power of 2").
|
||||
# 1 "loop": split-K loop, K % BLOCK_K == 0, BLOCK_K power of 2 >= 16
|
||||
# (e.g. K=192 with BLOCK_K=64 -> 3 iterations). The a-tile is
|
||||
# re-loaded per (N-block, K-block); slices are L1/L2-resident after
|
||||
# the first N-block, but the load/dot interleave costs bandwidth
|
||||
# (measured ~1372 GB/s vs ~2x that for mode 0 at K=128).
|
||||
# 2 "two_dot": K = BLOCK_K + (K - BLOCK_K), both power-of-2 halves
|
||||
# (192 = 128 + 64). Both a-tiles preload once before the N-loop;
|
||||
# each N-block issues two chained tl.dot into one fp32 accumulator.
|
||||
# No K-loop, no a re-reads — the direct generalization of mode 0.
|
||||
# 3 "three_dot": K = 3 * BLOCK_K (192 = 3 x 64). Same as mode 2 with
|
||||
# three preloaded a-tiles / three chained tl.dot per N-block; the
|
||||
# hoisted-loads analogue of mode 1 (identical fp32 add order).
|
||||
# 4 "pad": BLOCK_K = next_pow2(K) > K, k-masked loads (zero fill).
|
||||
# Single tl.dot per N-block; the padded zeros are exact fp32
|
||||
# additive identities so the result matches a K-wide single dot,
|
||||
# at the cost of BLOCK_K/K (e.g. 256/192 = 1.33x) extra MMA work.
|
||||
#
|
||||
# Rounding contract: the fp32 accumulator is rounded to bf16 first (the
|
||||
# same output rounding stage as the cublas bf16 bmm) and then converted
|
||||
# bf16->fp8 by the same implicit-store conversion the fused concat-cast
|
||||
# kernel uses. The split-K accumulator stays fp32 across all K-blocks,
|
||||
# so the rounding stages are identical in both layouts. The rope half is
|
||||
# a bit-exact copy of that kernel (loads the post-rope bf16, converts on
|
||||
# store). The nope half is NOT guaranteed bit-exact vs the default path:
|
||||
# tl.dot accumulates fp32 in a different order than cublas, so last-ulp
|
||||
# fp32 differences can occasionally flip the bf16 (and hence fp8)
|
||||
# rounding.
|
||||
pid_m = tl.program_id(0)
|
||||
h = tl.program_id(1)
|
||||
offs_m = pid_m * BLOCK_M + tl.arange(0, BLOCK_M)
|
||||
m_mask = offs_m < T
|
||||
# token-row offsets in int64: T * row-stride can exceed int32 (e.g. 128
|
||||
# heads x 576 dims x tens of thousands of tokens).
|
||||
offs_m64 = offs_m.to(tl.int64)
|
||||
qout_head = qout_ptr + offs_m64[:, None] * qout_s0 + h * qout_s1
|
||||
a_row = a_ptr + offs_m64[:, None] * a_s0 + h * a_s1
|
||||
b_head = b_ptr + h * b_s0
|
||||
if K_MODE == 0:
|
||||
# single-dot path (original kernel): BLOCK_K == K, preload a once.
|
||||
offs_k = tl.arange(0, BLOCK_K)
|
||||
a = tl.load(a_row + offs_k[None, :], mask=m_mask[:, None], other=0.0)
|
||||
for nb in tl.static_range(N // BLOCK_N):
|
||||
offs_n = nb * BLOCK_N + tl.arange(0, BLOCK_N)
|
||||
b = tl.load(b_head + offs_k[:, None] * b_s1 + offs_n[None, :] * b_s2)
|
||||
acc = tl.dot(a, b) # fp32 accumulator
|
||||
val = acc.to(tl.bfloat16) # cublas-equivalent bf16 output rounding
|
||||
# implicit bf16 -> fp8 conversion on store (same as the concat-cast)
|
||||
tl.store(qout_head + offs_n[None, :], val, mask=m_mask[:, None])
|
||||
elif K_MODE == 2:
|
||||
# two-dot preload: K split as BLOCK_K + (K - BLOCK_K), no K-loop.
|
||||
offs_k0 = tl.arange(0, BLOCK_K)
|
||||
offs_k1 = BLOCK_K + tl.arange(0, K - BLOCK_K)
|
||||
a0 = tl.load(a_row + offs_k0[None, :], mask=m_mask[:, None], other=0.0)
|
||||
a1 = tl.load(a_row + offs_k1[None, :], mask=m_mask[:, None], other=0.0)
|
||||
for nb in tl.static_range(N // BLOCK_N):
|
||||
offs_n = nb * BLOCK_N + tl.arange(0, BLOCK_N)
|
||||
b0 = tl.load(b_head + offs_k0[:, None] * b_s1 + offs_n[None, :] * b_s2)
|
||||
b1 = tl.load(b_head + offs_k1[:, None] * b_s1 + offs_n[None, :] * b_s2)
|
||||
acc = tl.dot(a0, b0) # fp32 accumulator
|
||||
acc = tl.dot(a1, b1, acc) # chained: stays fp32 across both dots
|
||||
val = acc.to(tl.bfloat16) # cublas-equivalent bf16 output rounding
|
||||
# implicit bf16 -> fp8 conversion on store (same as the concat-cast)
|
||||
tl.store(qout_head + offs_n[None, :], val, mask=m_mask[:, None])
|
||||
elif K_MODE == 3:
|
||||
# three-dot preload: K = 3 * BLOCK_K, a-tiles hoisted out of the N-loop.
|
||||
offs_k0 = tl.arange(0, BLOCK_K)
|
||||
offs_k1 = BLOCK_K + offs_k0
|
||||
offs_k2 = 2 * BLOCK_K + offs_k0
|
||||
a0 = tl.load(a_row + offs_k0[None, :], mask=m_mask[:, None], other=0.0)
|
||||
a1 = tl.load(a_row + offs_k1[None, :], mask=m_mask[:, None], other=0.0)
|
||||
a2 = tl.load(a_row + offs_k2[None, :], mask=m_mask[:, None], other=0.0)
|
||||
for nb in tl.static_range(N // BLOCK_N):
|
||||
offs_n = nb * BLOCK_N + tl.arange(0, BLOCK_N)
|
||||
b0 = tl.load(b_head + offs_k0[:, None] * b_s1 + offs_n[None, :] * b_s2)
|
||||
b1 = tl.load(b_head + offs_k1[:, None] * b_s1 + offs_n[None, :] * b_s2)
|
||||
b2 = tl.load(b_head + offs_k2[:, None] * b_s1 + offs_n[None, :] * b_s2)
|
||||
acc = tl.dot(a0, b0) # fp32 accumulator
|
||||
acc = tl.dot(a1, b1, acc)
|
||||
acc = tl.dot(a2, b2, acc) # same fp32 add order as the K_MODE=1 loop
|
||||
val = acc.to(tl.bfloat16) # cublas-equivalent bf16 output rounding
|
||||
# implicit bf16 -> fp8 conversion on store (same as the concat-cast)
|
||||
tl.store(qout_head + offs_n[None, :], val, mask=m_mask[:, None])
|
||||
elif K_MODE == 4:
|
||||
# padded single dot: BLOCK_K = next_pow2(K), zero-fill the k tail.
|
||||
offs_k = tl.arange(0, BLOCK_K)
|
||||
k_mask = offs_k < K
|
||||
a = tl.load(
|
||||
a_row + offs_k[None, :],
|
||||
mask=m_mask[:, None] & k_mask[None, :],
|
||||
other=0.0,
|
||||
)
|
||||
for nb in tl.static_range(N // BLOCK_N):
|
||||
offs_n = nb * BLOCK_N + tl.arange(0, BLOCK_N)
|
||||
b = tl.load(
|
||||
b_head + offs_k[:, None] * b_s1 + offs_n[None, :] * b_s2,
|
||||
mask=k_mask[:, None],
|
||||
other=0.0,
|
||||
)
|
||||
acc = tl.dot(a, b) # fp32 accumulator (padded zeros add exactly 0)
|
||||
val = acc.to(tl.bfloat16) # cublas-equivalent bf16 output rounding
|
||||
# implicit bf16 -> fp8 conversion on store (same as the concat-cast)
|
||||
tl.store(qout_head + offs_n[None, :], val, mask=m_mask[:, None])
|
||||
else:
|
||||
# K_MODE == 1: split-K loop (K % BLOCK_K == 0, e.g. K=192, BLOCK_K=64).
|
||||
for nb in tl.static_range(N // BLOCK_N):
|
||||
offs_n = nb * BLOCK_N + tl.arange(0, BLOCK_N)
|
||||
acc = tl.zeros((BLOCK_M, BLOCK_N), dtype=tl.float32)
|
||||
for kb in tl.static_range(K // BLOCK_K):
|
||||
offs_k = kb * BLOCK_K + tl.arange(0, BLOCK_K)
|
||||
a = tl.load(a_row + offs_k[None, :], mask=m_mask[:, None], other=0.0)
|
||||
b = tl.load(b_head + offs_k[:, None] * b_s1 + offs_n[None, :] * b_s2)
|
||||
acc = tl.dot(a, b, acc) # fp32 accumulator across K-blocks
|
||||
val = acc.to(tl.bfloat16) # cublas-equivalent bf16 output rounding
|
||||
# implicit bf16 -> fp8 conversion on store (same as the concat-cast)
|
||||
tl.store(qout_head + offs_n[None, :], val, mask=m_mask[:, None])
|
||||
offs_r = tl.arange(0, ROPE)
|
||||
r = tl.load(
|
||||
rope_ptr + offs_m64[:, None] * rope_s0 + h * rope_s1 + offs_r[None, :],
|
||||
mask=m_mask[:, None],
|
||||
other=0.0,
|
||||
)
|
||||
tl.store(qout_head + N + offs_r[None, :], r, mask=m_mask[:, None])
|
||||
|
||||
|
||||
# Non-power-of-2-K variant used by variant="auto" (power-of-2 K always takes
|
||||
# the single-dot fast path). Set to the winner of the K=192 A/B in
|
||||
# benchmark/kernels/deepseek/benchmark_q8kv8_q_prep.py; "loop" = the pre-A/B split-K behavior.
|
||||
_AUTO_NONPOW2_VARIANT = "two_dot"
|
||||
|
||||
|
||||
def _qprep_env_variant():
|
||||
from sglang.srt.environ import envs
|
||||
|
||||
return envs.SGLANG_OPT_Q8KV8_QPREP_VARIANT.get()
|
||||
|
||||
|
||||
# Resolved once at import (matches the module-constant style above). "auto"
|
||||
# keeps the per-K dispatch; "cuda" routes every shape to the hand-written
|
||||
# SM90 WGMMA kernel (bitwise-identical to two_dot; 1.16-1.38x faster).
|
||||
_ENV_QPREP_VARIANT = None
|
||||
|
||||
|
||||
def absorbed_bmm_concat_cast_q_fp8(
|
||||
q_fp8_pad: "torch.Tensor",
|
||||
q_nope: "torch.Tensor",
|
||||
w_kc: "torch.Tensor",
|
||||
q_rope: "torch.Tensor",
|
||||
num_heads: int,
|
||||
block_m: int = 128,
|
||||
block_n: int = 64,
|
||||
variant: str = "auto",
|
||||
block_k: int = 0,
|
||||
num_warps: int = 8,
|
||||
num_stages: int = 0,
|
||||
):
|
||||
"""Fused absorbed-q bmm + nope/rope concat + bf16->fp8 cast ("born fp8" q).
|
||||
|
||||
Replaces ``torch.bmm(q_nope.transpose(0, 1), w_kc).transpose(0, 1)``
|
||||
followed by ``concat_and_cast_q_fp8_pad`` on the Q8KV8 sparse-prefill
|
||||
path, writing the active ``[:, :num_heads, :]`` slice of the padded fp8 q
|
||||
buffer directly. Inputs:
|
||||
|
||||
* ``q_fp8_pad``: [num_tokens, pad_heads, N + ROPE] fp8_e4m3 destination.
|
||||
* ``q_nope``: [num_tokens, H, K] bf16 pre-absorb q (strided views OK).
|
||||
* ``w_kc``: [H, K, N] bf16 absorbed weight (any strides).
|
||||
* ``q_rope``: [num_tokens, H, ROPE] bf16 post-rope q (strided views OK).
|
||||
|
||||
The rope half is bit-exact vs ``concat_and_cast_q_fp8_pad``. The nope
|
||||
half keeps the same rounding stages (fp32 accum -> bf16 -> fp8) but a
|
||||
different fp32 accumulation order than cublas, so it is near- but not
|
||||
guaranteed bit-exact; keep this path behind
|
||||
``SGLANG_ENABLE_DSA_Q8KV8_BORN_FP8_Q``.
|
||||
|
||||
K (``qk_nope_head_dim``) supports any multiple of 16 in [16, 256].
|
||||
Power-of-2 K (DeepSeek 128) always takes the preload-once
|
||||
single-``tl.dot`` path. For other K (GLM 192), ``variant`` selects the
|
||||
K-dimension codegen (every variant keeps the identical fp32 -> bf16 ->
|
||||
fp8 epilogue):
|
||||
|
||||
* ``"auto"``: the current production choice (see
|
||||
``_AUTO_NONPOW2_VARIANT``).
|
||||
* ``"loop"``: split-K accumulator loop, ``BLOCK_K`` = ``block_k`` or the
|
||||
largest power-of-2 divisor of K capped at 128 (192 -> 64 x 3).
|
||||
* ``"two_dot"``: preload a as two power-of-2 tiles (192 = 128 + 64), two
|
||||
chained ``tl.dot`` per N-block, no K-loop.
|
||||
* ``"three_dot"``: preload a as three K/3 tiles (192 = 3 x 64), three
|
||||
chained ``tl.dot`` per N-block; same fp32 add order as ``"loop"``.
|
||||
* ``"pad"``: single ``tl.dot`` with ``BLOCK_K`` = next_pow2(K) (192 ->
|
||||
256) and zero-masked k tails.
|
||||
* ``"single_k"``: single ``tl.dot`` with ``BLOCK_K`` == K. Only
|
||||
compiles if the Triton build supports non-power-of-2 ``tl.arange``
|
||||
(Triton <= 3.5.x raises "arange's range must be a power of 2").
|
||||
``block_m`` / ``block_n`` / ``num_warps`` / ``num_stages`` are tuning
|
||||
knobs for the microbench sweep (0 = Triton default for ``num_stages``).
|
||||
"""
|
||||
num_tokens, _, k_dim = q_nope.shape
|
||||
n_dim = w_kc.shape[-1]
|
||||
rope_dim = q_rope.shape[-1]
|
||||
assert q_fp8_pad.dtype == torch.float8_e4m3fn
|
||||
assert q_nope.dtype == torch.bfloat16 and w_kc.dtype == torch.bfloat16
|
||||
assert q_rope.dtype == torch.bfloat16
|
||||
assert q_nope.shape[1] == num_heads and q_rope.shape[1] == num_heads
|
||||
assert w_kc.shape[0] == num_heads and w_kc.shape[1] == k_dim
|
||||
assert q_fp8_pad.shape[0] >= num_tokens and q_fp8_pad.shape[1] >= num_heads
|
||||
assert q_fp8_pad.shape[2] == n_dim + rope_dim
|
||||
# tl.arange / tl.dot constraints
|
||||
assert k_dim % 16 == 0 and 16 <= k_dim <= 256, (
|
||||
"K must be a multiple of 16 in [16, 256]"
|
||||
)
|
||||
assert (rope_dim & (rope_dim - 1)) == 0, "ROPE must be a power of two"
|
||||
assert n_dim % block_n == 0, "N must be a multiple of block_n"
|
||||
assert q_nope.stride(2) == 1 and q_rope.stride(2) == 1
|
||||
assert q_fp8_pad.stride(2) == 1
|
||||
# Env override for production dispatch (SGLANG_OPT_Q8KV8_QPREP_VARIANT):
|
||||
# "auto" (default) keeps the per-K Triton dispatch; "cuda" routes every
|
||||
# shape to the WGMMA kernel below.
|
||||
global _ENV_QPREP_VARIANT
|
||||
if _ENV_QPREP_VARIANT is None:
|
||||
_ENV_QPREP_VARIANT = _qprep_env_variant()
|
||||
if variant == "auto" and _ENV_QPREP_VARIANT != "auto":
|
||||
variant = _ENV_QPREP_VARIANT
|
||||
# Hand-written SM90 WGMMA kernel (opt-in only; "auto" never routes here).
|
||||
# Same fp32 -> bf16 -> fp8 epilogue; bitwise identical to "two_dot" on
|
||||
# SM90. Requires K in {128, 192} and the production N-major w_kc layout
|
||||
# (see the wrapper's asserts); the Triton variants remain the
|
||||
# general-strides fallback.
|
||||
_valid = ("auto", "cuda", "loop", "two_dot", "three_dot", "pad", "single_k")
|
||||
if variant not in _valid:
|
||||
raise ValueError(
|
||||
f"unknown q-prep variant {variant!r} "
|
||||
f"(SGLANG_OPT_Q8KV8_QPREP_VARIANT); valid: {_valid}"
|
||||
)
|
||||
if variant == "cuda":
|
||||
from sglang.kernels.ops.attention.qprep_bf16_fp8_sm90 import q8kv8_qprep_fwd
|
||||
|
||||
q8kv8_qprep_fwd(q_fp8_pad, q_nope, w_kc, q_rope, num_heads)
|
||||
return
|
||||
# Resolve (K_MODE, BLOCK_K) from the variant; see the kernel's K-handling
|
||||
# comment for what each mode compiles to.
|
||||
if k_dim & (k_dim - 1) == 0:
|
||||
# power-of-2 K: every variant collapses to the single-dot fast path.
|
||||
k_mode, blk_k = 0, k_dim
|
||||
else:
|
||||
v = _AUTO_NONPOW2_VARIANT if variant == "auto" else variant
|
||||
if v == "loop":
|
||||
# Largest power-of-2 divisor of K, capped at 128 (K % 16 == 0
|
||||
# makes this >= 16), unless the caller pinned block_k.
|
||||
blk_k = block_k or min(k_dim & -k_dim, 128)
|
||||
assert k_dim % blk_k == 0 and blk_k & (blk_k - 1) == 0 and blk_k >= 16, (
|
||||
"loop needs BLOCK_K a power-of-2 divisor of K >= 16"
|
||||
)
|
||||
k_mode = 1
|
||||
elif v == "two_dot":
|
||||
blk_k = 1 << (k_dim.bit_length() - 1) # largest power of 2 < K
|
||||
k1 = k_dim - blk_k
|
||||
assert k1 & (k1 - 1) == 0 and k1 >= 16, (
|
||||
"two_dot needs K = pow2 + pow2 with both halves >= 16"
|
||||
)
|
||||
k_mode = 2
|
||||
elif v == "three_dot":
|
||||
blk_k = k_dim // 3
|
||||
assert k_dim % 3 == 0 and blk_k & (blk_k - 1) == 0 and blk_k >= 16, (
|
||||
"three_dot needs K = 3 * pow2 with pow2 >= 16"
|
||||
)
|
||||
k_mode = 3
|
||||
elif v == "pad":
|
||||
blk_k = 1 << k_dim.bit_length() # next power of 2 above K
|
||||
k_mode = 4
|
||||
elif v == "single_k":
|
||||
# Non-power-of-2 BLOCK_K == K: compiles only on Triton builds
|
||||
# that allow non-power-of-2 tl.arange (not 3.5.x).
|
||||
blk_k = k_dim
|
||||
k_mode = 0
|
||||
else:
|
||||
raise ValueError(f"unknown absorbed-bmm K variant: {variant!r}")
|
||||
extra = {"num_stages": num_stages} if num_stages else {}
|
||||
grid = (triton.cdiv(num_tokens, block_m), num_heads)
|
||||
absorbed_bmm_concat_cast_q_fp8_kernel[grid](
|
||||
q_fp8_pad,
|
||||
q_nope,
|
||||
w_kc,
|
||||
q_rope,
|
||||
num_tokens,
|
||||
q_fp8_pad.stride(0),
|
||||
q_fp8_pad.stride(1),
|
||||
q_nope.stride(0),
|
||||
q_nope.stride(1),
|
||||
w_kc.stride(0),
|
||||
w_kc.stride(1),
|
||||
w_kc.stride(2),
|
||||
q_rope.stride(0),
|
||||
q_rope.stride(1),
|
||||
K=k_dim,
|
||||
N=n_dim,
|
||||
ROPE=rope_dim,
|
||||
BLOCK_M=block_m,
|
||||
BLOCK_N=block_n,
|
||||
BLOCK_K=blk_k,
|
||||
K_MODE=k_mode,
|
||||
num_warps=num_warps,
|
||||
**extra,
|
||||
)
|
||||
+7
-7
@@ -143,7 +143,7 @@ def _triton_config(n_rows: int) -> tuple[int, int]:
|
||||
class FusedQKNorm(BaseFusedOp):
|
||||
"""Per-head RMS norm of the Q and K slices of a QKV projection output."""
|
||||
|
||||
op = "muse_spark.qk_norm"
|
||||
op = "attention.muse_spark_qk_norm_rope_store"
|
||||
priority = (KernelBackend.TRITON, KernelBackend.HELION)
|
||||
capabilities = {
|
||||
KernelBackend.HELION: {CapabilityRequirement.CUDA},
|
||||
@@ -375,10 +375,10 @@ class FusedQKNorm(BaseFusedOp):
|
||||
cos_sin_cache: Optional[torch.Tensor] = None,
|
||||
kv_store: Optional[tuple[torch.Tensor, torch.Tensor, torch.Tensor]] = None,
|
||||
) -> bool:
|
||||
from sglang.kernels.ops.muse_spark.helion_runtime import helion_supported
|
||||
from sglang.kernels.jit.utils.helion_runtime import helion_supported
|
||||
|
||||
return (
|
||||
helion_supported("qk_norm", qkv)
|
||||
helion_supported(qkv)
|
||||
and (kv_store is None or kv_store[0].dtype == qkv.dtype)
|
||||
and qkv.shape[0] > 0
|
||||
and qkv.dtype == torch.bfloat16
|
||||
@@ -405,13 +405,13 @@ class FusedQKNorm(BaseFusedOp):
|
||||
) -> tuple[torch.Tensor, torch.Tensor]:
|
||||
"""Run Helion behind the existing fused operation boundary."""
|
||||
|
||||
from sglang.kernels.ops.muse_spark.helion_qk_norm import (
|
||||
from sglang.kernels.jit.utils.helion_runtime import (
|
||||
run_pinned_helion,
|
||||
)
|
||||
from sglang.kernels.ops.attention.muse_spark_qk_norm_rope_store_helion import (
|
||||
QK_NORM_CONFIGS,
|
||||
_qk_norm_helion_kernel,
|
||||
)
|
||||
from sglang.kernels.ops.muse_spark.helion_runtime import (
|
||||
run_pinned_helion,
|
||||
)
|
||||
|
||||
if kv_store is None:
|
||||
k_cache = v_cache = cache_loc = None
|
||||
+1
-1
@@ -6,7 +6,7 @@ import helion
|
||||
import helion.language as hl
|
||||
import torch
|
||||
|
||||
from sglang.kernels.ops.muse_spark.helion_runtime import (
|
||||
from sglang.kernels.jit.utils.helion_runtime import (
|
||||
token_count_specialization_key,
|
||||
)
|
||||
|
||||
@@ -3,7 +3,7 @@
|
||||
Fuses the per-head absorbed-q bmm (q_nope [T, H, K] bf16 x w_kc [H, K, N]
|
||||
bf16, fp32 accumulate), the nope/rope concat, and the bf16 -> fp8_e4m3 cast
|
||||
into one hand-written WGMMA kernel. CUDA replacement for the Triton
|
||||
``absorbed_bmm_concat_cast_q_fp8`` (triton_ops/cache_ops.py) with the
|
||||
``absorbed_bmm_concat_cast_q_fp8`` (attention/mla_qk_prep.py) with the
|
||||
identical fp32 -> bf16 -> fp8 epilogue rounding chain; the rope half is
|
||||
bit-exact vs ``concat_and_cast_q_fp8_pad``.
|
||||
"""
|
||||
|
||||
@@ -12,6 +12,11 @@ import torch
|
||||
|
||||
from sglang.kernels.jit.utils import cache_once, load_jit
|
||||
from sglang.kernels.kernel_api_logging import debug_kernel_api
|
||||
from sglang.kernels.ops.attention.dsv4.dequant_k_cache import (
|
||||
DIM_NOPE,
|
||||
DIM_ROPE,
|
||||
fp8_dtype,
|
||||
)
|
||||
from sglang.srt.utils.custom_op import register_custom_op
|
||||
|
||||
if TYPE_CHECKING:
|
||||
@@ -524,3 +529,71 @@ def sparse_mla_q8kv8_prefill_fwd(
|
||||
)
|
||||
|
||||
return out, max_logits, lse
|
||||
|
||||
|
||||
def q8kv8_padded_num_heads(num_heads: int) -> int:
|
||||
"""Return a Q-head count supported by the SM90 Q8KV8 kernel."""
|
||||
if num_heads <= 0:
|
||||
raise ValueError(f"num_heads must be positive, got {num_heads}")
|
||||
if num_heads <= 64:
|
||||
return 64
|
||||
if num_heads <= 128:
|
||||
return 128
|
||||
raise ValueError(
|
||||
"DeepSeek-V4 Q8KV8 sparse prefill supports at most 128 local "
|
||||
f"query heads, got {num_heads}"
|
||||
)
|
||||
|
||||
|
||||
def cast_q_fp8_for_q8kv8_prefill(
|
||||
q: torch.Tensor,
|
||||
padded_num_heads: Optional[int] = None,
|
||||
out: Optional[torch.Tensor] = None,
|
||||
) -> tuple[torch.Tensor, torch.Tensor]:
|
||||
"""Cast DeepSeek-V4 sparse-prefill Q to the Q8KV8 kernel format.
|
||||
|
||||
The incoming Q is the model-produced BF16/FP16 tensor already shaped as
|
||||
``(num_tokens, num_heads, 512)`` after removing the singleton MQA axis.
|
||||
|
||||
The SM90 kernel processes query heads in 64-head blocks. Tensor parallelism
|
||||
commonly leaves fewer than 64 local heads, so the active heads are copied
|
||||
into a zero-padded 64/128-head FP8 tensor.
|
||||
"""
|
||||
assert q.ndim == 3
|
||||
assert q.shape[-1] == DIM_NOPE + DIM_ROPE
|
||||
|
||||
num_tokens, num_heads, head_dim = q.shape
|
||||
if padded_num_heads is None:
|
||||
padded_num_heads = q8kv8_padded_num_heads(num_heads)
|
||||
|
||||
if padded_num_heads not in (64, 128) or padded_num_heads < num_heads:
|
||||
raise ValueError(
|
||||
f"invalid padded_num_heads={padded_num_heads} for num_heads={num_heads}"
|
||||
)
|
||||
|
||||
expected_shape = (num_tokens, padded_num_heads, head_dim)
|
||||
|
||||
if out is None:
|
||||
q_fp8 = torch.zeros(
|
||||
expected_shape,
|
||||
dtype=fp8_dtype,
|
||||
device=q.device,
|
||||
)
|
||||
else:
|
||||
if (
|
||||
out.shape != expected_shape
|
||||
or out.dtype != fp8_dtype
|
||||
or out.device != q.device
|
||||
):
|
||||
raise ValueError(
|
||||
"Q8KV8 Q output must have shape/dtype/device "
|
||||
f"{expected_shape}/{fp8_dtype}/{q.device}, got "
|
||||
f"{tuple(out.shape)}/{out.dtype}/{out.device}"
|
||||
)
|
||||
q_fp8 = out
|
||||
if padded_num_heads > num_heads:
|
||||
q_fp8[:, num_heads:].zero_()
|
||||
|
||||
q_fp8[:, :num_heads].copy_(q)
|
||||
q_scale = torch.ones((), dtype=torch.float32, device=q.device)
|
||||
return q_fp8, q_scale
|
||||
|
||||
@@ -1,59 +1,6 @@
|
||||
import torch
|
||||
import triton
|
||||
import triton.language as tl
|
||||
|
||||
from sglang.kernels.jit.utils import is_arch_support_pdl
|
||||
from sglang.kernels.ops.attention.pad import (
|
||||
pad_sequence_with_mask as pad_sequence_with_mask,
|
||||
)
|
||||
from sglang.kernels.ops.attention.pad import (
|
||||
pad_sequence_with_mask_kernel as pad_sequence_with_mask_kernel,
|
||||
)
|
||||
from sglang.kernels.ops.attention.pad import (
|
||||
seqlens_expand_kernel as seqlens_expand_kernel,
|
||||
)
|
||||
from sglang.kernels.ops.attention.pad import (
|
||||
seqlens_expand_triton as seqlens_expand_triton,
|
||||
)
|
||||
from sglang.kernels.ops.kvcache.cache_ops import (
|
||||
concat_and_cast_mha_k_kernel as concat_and_cast_mha_k_kernel,
|
||||
)
|
||||
from sglang.kernels.ops.kvcache.cache_ops import (
|
||||
concat_and_cast_mha_k_triton as concat_and_cast_mha_k_triton,
|
||||
)
|
||||
from sglang.kernels.ops.kvcache.cache_ops import (
|
||||
concat_and_cast_q_fp8_pad as concat_and_cast_q_fp8_pad,
|
||||
)
|
||||
from sglang.kernels.ops.kvcache.cache_ops import (
|
||||
concat_and_cast_q_fp8_pad_kernel as concat_and_cast_q_fp8_pad_kernel,
|
||||
)
|
||||
from sglang.kernels.ops.kvcache.cache_ops import (
|
||||
launch_reshape_and_cache_flash as launch_reshape_and_cache_flash,
|
||||
)
|
||||
from sglang.kernels.ops.kvcache.cache_ops import (
|
||||
q8kv8_topk_length_from_indices as q8kv8_topk_length_from_indices,
|
||||
)
|
||||
from sglang.kernels.ops.kvcache.cache_ops import (
|
||||
reshape_and_cache_flash as reshape_and_cache_flash,
|
||||
)
|
||||
from sglang.kernels.ops.kvcache.kv_indices import (
|
||||
create_flashinfer_kv_indices_triton as create_flashinfer_kv_indices_triton,
|
||||
)
|
||||
from sglang.kernels.ops.kvcache.kv_indices import (
|
||||
create_flashmla_kv_indices_triton as create_flashmla_kv_indices_triton,
|
||||
)
|
||||
from sglang.kernels.ops.kvcache.kv_indices import (
|
||||
get_num_kv_index_blocks_flashmla as get_num_kv_index_blocks_flashmla,
|
||||
)
|
||||
from sglang.kernels.ops.kvcache.kv_indices import (
|
||||
get_num_page_per_block_flashmla as get_num_page_per_block_flashmla,
|
||||
)
|
||||
from sglang.kernels.ops.kvcache.kv_indices import (
|
||||
kv_indices_num_token_blocks as kv_indices_num_token_blocks,
|
||||
)
|
||||
from sglang.kernels.ops.kvcache.rope_cache import (
|
||||
fused_qk_rope_reshape_and_cache as fused_qk_rope_reshape_and_cache,
|
||||
)
|
||||
from sglang.srt.utils import is_cuda
|
||||
|
||||
_is_cuda = is_cuda()
|
||||
@@ -214,270 +161,6 @@ def concat_mla_absorb_q_general(q_nope, q_rope):
|
||||
return torch.cat([q_nope, q_rope], dim=-1)
|
||||
|
||||
|
||||
@triton.jit
|
||||
def reshape_and_cache_shuffle_5d(
|
||||
key_ptr,
|
||||
value_ptr,
|
||||
key_cache_ptr,
|
||||
value_cache_ptr,
|
||||
slot_mapping_ptr,
|
||||
swa_slot_mapping_ptr,
|
||||
key_stride_token,
|
||||
value_stride_token,
|
||||
num_heads,
|
||||
head_size,
|
||||
block_size,
|
||||
X: tl.constexpr,
|
||||
HEAD_BLOCK: tl.constexpr,
|
||||
BLOCK_D: tl.constexpr,
|
||||
HAS_SWA: tl.constexpr,
|
||||
):
|
||||
"""Scatter per-token (num_tokens, num_heads, head_size) K/V into the
|
||||
SHUFFLE 5D "vectorized" KV cache layout used by aiter CK
|
||||
`mha_batch_prefill_func` and aiter `pa_decode_gluon`.
|
||||
|
||||
K cache shape: (num_blocks, num_heads, head_size // X, block_size, X)
|
||||
V cache shape: (num_blocks, num_heads, block_size // X, head_size, X)
|
||||
where X = 16 // element_size (=8 for bf16/fp16, =16 for fp8).
|
||||
block_size must be divisible by X, and head_size must be divisible by X.
|
||||
|
||||
Each program handles one token and a HEAD_BLOCK-wide slice of heads.
|
||||
"""
|
||||
token_idx = tl.program_id(0)
|
||||
head_block_idx = tl.program_id(1)
|
||||
|
||||
slot_idx = tl.load(slot_mapping_ptr + token_idx)
|
||||
if HAS_SWA:
|
||||
slot_idx = tl.load(swa_slot_mapping_ptr + slot_idx)
|
||||
if slot_idx < 0:
|
||||
return
|
||||
|
||||
block_idx = slot_idx // block_size
|
||||
slot_in_page = slot_idx % block_size
|
||||
page_outer = slot_in_page // X
|
||||
page_inner = slot_in_page % X
|
||||
|
||||
head_idx = head_block_idx * HEAD_BLOCK + tl.arange(0, HEAD_BLOCK)
|
||||
head_mask = head_idx < num_heads
|
||||
d = tl.arange(0, BLOCK_D)
|
||||
d_mask = d < head_size
|
||||
d_outer = d // X
|
||||
d_inner = d % X
|
||||
|
||||
src_off = token_idx * key_stride_token + head_idx[:, None] * head_size + d[None, :]
|
||||
src_mask = head_mask[:, None] & d_mask[None, :]
|
||||
k = tl.load(key_ptr + src_off, mask=src_mask)
|
||||
src_off_v = (
|
||||
token_idx * value_stride_token + head_idx[:, None] * head_size + d[None, :]
|
||||
)
|
||||
v = tl.load(value_ptr + src_off_v, mask=src_mask)
|
||||
|
||||
layer_stride = num_heads * head_size * block_size
|
||||
head_stride = head_size * block_size
|
||||
|
||||
k_tgt = (
|
||||
block_idx * layer_stride
|
||||
+ head_idx[:, None] * head_stride
|
||||
+ d_outer[None, :] * block_size * X
|
||||
+ slot_in_page * X
|
||||
+ d_inner[None, :]
|
||||
)
|
||||
tl.store(key_cache_ptr + k_tgt, k, mask=src_mask)
|
||||
|
||||
v_tgt = (
|
||||
block_idx * layer_stride
|
||||
+ head_idx[:, None] * head_stride
|
||||
+ page_outer * head_size * X
|
||||
+ d[None, :] * X
|
||||
+ page_inner
|
||||
)
|
||||
tl.store(value_cache_ptr + v_tgt, v, mask=src_mask)
|
||||
|
||||
|
||||
def launch_reshape_and_cache_shuffle_5d(
|
||||
key: torch.Tensor,
|
||||
value: torch.Tensor,
|
||||
key_cache: torch.Tensor,
|
||||
value_cache: torch.Tensor,
|
||||
slot_mapping: torch.Tensor,
|
||||
swa_slot_mapping=None,
|
||||
):
|
||||
"""Launcher for reshape_and_cache_shuffle_5d.
|
||||
|
||||
Args:
|
||||
key/value: (num_tokens, num_heads, head_size) source tensors
|
||||
key_cache: (num_blocks, num_heads, head_size//X, block_size, X)
|
||||
value_cache: (num_blocks, num_heads, block_size//X, head_size, X)
|
||||
slot_mapping: per-token destination slot in [0, num_blocks*block_size)
|
||||
"""
|
||||
num_tokens, num_heads, head_size = key.shape
|
||||
assert value.shape == key.shape, "K/V must share token-major shape"
|
||||
assert key_cache.dim() == 5 and value_cache.dim() == 5
|
||||
num_blocks, kc_H, kc_D_over_X, block_size, X = key_cache.shape
|
||||
assert kc_H == num_heads and kc_D_over_X * X == head_size
|
||||
vb_blocks, vc_H, vc_page_over_X, vc_D, vc_X = value_cache.shape
|
||||
assert (
|
||||
vc_H == num_heads
|
||||
and vc_page_over_X * X == block_size
|
||||
and vc_D == head_size
|
||||
and vc_X == X
|
||||
)
|
||||
assert block_size % X == 0 and head_size % X == 0
|
||||
|
||||
HEAD_BLOCK = min(4, triton.next_power_of_2(num_heads))
|
||||
BLOCK_D = triton.next_power_of_2(head_size)
|
||||
grid = (num_tokens, triton.cdiv(num_heads, HEAD_BLOCK))
|
||||
|
||||
reshape_and_cache_shuffle_5d[grid](
|
||||
key,
|
||||
value,
|
||||
key_cache,
|
||||
value_cache,
|
||||
slot_mapping,
|
||||
swa_slot_mapping if swa_slot_mapping is not None else slot_mapping,
|
||||
key.stride(0),
|
||||
value.stride(0),
|
||||
num_heads,
|
||||
head_size,
|
||||
block_size,
|
||||
X=X,
|
||||
HEAD_BLOCK=HEAD_BLOCK,
|
||||
BLOCK_D=BLOCK_D,
|
||||
HAS_SWA=(swa_slot_mapping is not None),
|
||||
)
|
||||
|
||||
|
||||
@triton.jit
|
||||
def gather_shuffle_5d_to_linear(
|
||||
key_cache_ptr,
|
||||
value_cache_ptr,
|
||||
key_out_ptr, # (T, num_heads, head_size), store dtype
|
||||
value_out_ptr, # (T, num_heads, head_size), store dtype
|
||||
slot_mapping_ptr, # (T,) absolute pool slot id per token
|
||||
key_out_stride_token,
|
||||
value_out_stride_token,
|
||||
num_heads,
|
||||
head_size,
|
||||
block_size,
|
||||
X: tl.constexpr,
|
||||
HEAD_BLOCK: tl.constexpr,
|
||||
BLOCK_D: tl.constexpr,
|
||||
):
|
||||
"""Inverse of :func:`reshape_and_cache_shuffle_5d`.
|
||||
|
||||
Gather one token's K/V from the SHUFFLE 5D paged cache into the
|
||||
canonical (T, H, D) layout that aiter's ``mha_batch_prefill_func``
|
||||
expects in LINEAR mode. Source addressing is identical to the
|
||||
writer kernel so any bit-exact round-trip is guaranteed.
|
||||
"""
|
||||
token_idx = tl.program_id(0)
|
||||
head_block_idx = tl.program_id(1)
|
||||
|
||||
slot_idx = tl.load(slot_mapping_ptr + token_idx)
|
||||
|
||||
block_idx = slot_idx // block_size
|
||||
slot_in_page = slot_idx % block_size
|
||||
page_outer = slot_in_page // X
|
||||
page_inner = slot_in_page % X
|
||||
|
||||
head_idx = head_block_idx * HEAD_BLOCK + tl.arange(0, HEAD_BLOCK)
|
||||
head_mask = head_idx < num_heads
|
||||
d = tl.arange(0, BLOCK_D)
|
||||
d_mask = d < head_size
|
||||
d_outer = d // X
|
||||
d_inner = d % X
|
||||
|
||||
layer_stride = num_heads * head_size * block_size
|
||||
head_stride = head_size * block_size
|
||||
|
||||
src_mask = head_mask[:, None] & d_mask[None, :]
|
||||
k_src = (
|
||||
block_idx * layer_stride
|
||||
+ head_idx[:, None] * head_stride
|
||||
+ d_outer[None, :] * block_size * X
|
||||
+ slot_in_page * X
|
||||
+ d_inner[None, :]
|
||||
)
|
||||
k = tl.load(key_cache_ptr + k_src, mask=src_mask)
|
||||
v_src = (
|
||||
block_idx * layer_stride
|
||||
+ head_idx[:, None] * head_stride
|
||||
+ page_outer * head_size * X
|
||||
+ d[None, :] * X
|
||||
+ page_inner
|
||||
)
|
||||
v = tl.load(value_cache_ptr + v_src, mask=src_mask)
|
||||
|
||||
dst_k = (
|
||||
token_idx * key_out_stride_token + head_idx[:, None] * head_size + d[None, :]
|
||||
)
|
||||
tl.store(key_out_ptr + dst_k, k, mask=src_mask)
|
||||
dst_v = (
|
||||
token_idx * value_out_stride_token + head_idx[:, None] * head_size + d[None, :]
|
||||
)
|
||||
tl.store(value_out_ptr + dst_v, v, mask=src_mask)
|
||||
|
||||
|
||||
def launch_gather_shuffle_5d_to_linear(
|
||||
key_cache: torch.Tensor,
|
||||
value_cache: torch.Tensor,
|
||||
slot_mapping: torch.Tensor,
|
||||
):
|
||||
"""Inverse of :func:`launch_reshape_and_cache_shuffle_5d`.
|
||||
|
||||
Returns ``(key_out, value_out)`` each shaped
|
||||
``(T, num_heads, head_size)`` in ``key_cache.dtype`` /
|
||||
``value_cache.dtype``. The caller is responsible for passing the
|
||||
right per-tensor descales downstream when ``store_dtype`` is fp8.
|
||||
|
||||
Args:
|
||||
key_cache: (num_blocks, num_heads, head_size // X, block_size, X)
|
||||
value_cache: (num_blocks, num_heads, block_size // X, head_size, X)
|
||||
slot_mapping: (T,) per-token absolute slot id in
|
||||
``[0, num_blocks * block_size)``
|
||||
"""
|
||||
assert key_cache.dim() == 5 and value_cache.dim() == 5
|
||||
num_blocks, num_heads, kc_D_over_X, block_size, X = key_cache.shape
|
||||
vc_blocks, vc_H, vc_page_over_X, vc_D, vc_X = value_cache.shape
|
||||
assert vc_blocks == num_blocks and vc_H == num_heads
|
||||
assert vc_page_over_X * X == block_size and vc_X == X
|
||||
head_size = kc_D_over_X * X
|
||||
assert vc_D == head_size
|
||||
|
||||
num_tokens = slot_mapping.numel()
|
||||
key_out = torch.empty(
|
||||
(num_tokens, num_heads, head_size),
|
||||
dtype=key_cache.dtype,
|
||||
device=key_cache.device,
|
||||
)
|
||||
value_out = torch.empty(
|
||||
(num_tokens, num_heads, head_size),
|
||||
dtype=value_cache.dtype,
|
||||
device=value_cache.device,
|
||||
)
|
||||
|
||||
HEAD_BLOCK = min(4, triton.next_power_of_2(num_heads))
|
||||
BLOCK_D = triton.next_power_of_2(head_size)
|
||||
grid = (num_tokens, triton.cdiv(num_heads, HEAD_BLOCK))
|
||||
|
||||
gather_shuffle_5d_to_linear[grid](
|
||||
key_cache,
|
||||
value_cache,
|
||||
key_out,
|
||||
value_out,
|
||||
slot_mapping,
|
||||
key_out.stride(0),
|
||||
value_out.stride(0),
|
||||
num_heads,
|
||||
head_size,
|
||||
block_size,
|
||||
X=X,
|
||||
HEAD_BLOCK=HEAD_BLOCK,
|
||||
BLOCK_D=BLOCK_D,
|
||||
)
|
||||
return key_out, value_out
|
||||
|
||||
|
||||
def assert_buffer_fits(used: int, capacity: int, what: str, **context) -> None:
|
||||
"""Safety guard: a preallocated cuda-graph buffer must hold the runtime write.
|
||||
|
||||
|
||||
@@ -37,6 +37,8 @@ for _mod, _fn in [
|
||||
("sp_collective", "reduce_scatter_pull"),
|
||||
("sp_collective", "all_gather"),
|
||||
("sp_collective", "all_gather_direct"),
|
||||
("attn_res_collective", "attn_res_fused_pull_rs"),
|
||||
("attn_res_collective", "attn_res_fused_direct_ag"),
|
||||
]:
|
||||
register_kernel(
|
||||
KernelSpec(
|
||||
@@ -47,3 +49,47 @@ for _mod, _fn in [
|
||||
)
|
||||
)
|
||||
del _mod, _fn
|
||||
|
||||
# Muse Spark collectives. The MoE finalize op (with its optional fused TP
|
||||
# all-reduce) is built per layer: inventoried through its class methods.
|
||||
register_kernel(
|
||||
KernelSpec(
|
||||
op="communication.muse_spark_all_gather",
|
||||
backend=KernelBackend.JIT,
|
||||
target="sglang.kernels.ops.communication.muse_spark_all_gather:all_gather",
|
||||
capabilities=frozenset({CapabilityRequirement.cuda(min_sm=(9, 0))}),
|
||||
)
|
||||
)
|
||||
register_kernel(
|
||||
KernelSpec(
|
||||
op="communication.muse_spark_all_gather_norm_residual",
|
||||
backend=KernelBackend.JIT,
|
||||
target="sglang.kernels.ops.communication.muse_spark_all_gather_norm_residual:all_gather_norm_residual",
|
||||
capabilities=frozenset({CapabilityRequirement.cuda(min_sm=(10, 0))}),
|
||||
)
|
||||
)
|
||||
for _bk in ("torch", "cute_dsl", "helion"):
|
||||
register_kernel(
|
||||
KernelSpec(
|
||||
op="communication.muse_spark_moe_finalize_all_reduce",
|
||||
backend=KernelBackend(_bk),
|
||||
target="sglang.kernels.ops.communication.muse_spark_moe_finalize_all_reduce:FusedPostExpertNormWeightedSumTRTLLM."
|
||||
+ ("forward_native" if _bk == "torch" else f"forward_{_bk}"),
|
||||
capabilities=frozenset() if _bk == "torch" else CUDA_ONLY,
|
||||
)
|
||||
)
|
||||
del _bk
|
||||
|
||||
# Blackwell sequence-parallel GEMM collectives (tcgen05 CuTeDSL kernels).
|
||||
for _fn in ("all_gather_matmul", "matmul_reduce_scatter"):
|
||||
register_kernel(
|
||||
KernelSpec(
|
||||
op=f"communication.{_fn}",
|
||||
backend=KernelBackend.CUTE_DSL,
|
||||
target=f"sglang.kernels.ops.communication.sp_collective_gemm:{_fn}",
|
||||
capabilities=frozenset(
|
||||
{CapabilityRequirement.cuda(min_sm=(10, 0), max_sm=(10, 9))}
|
||||
),
|
||||
)
|
||||
)
|
||||
del _fn
|
||||
|
||||
@@ -0,0 +1,173 @@
|
||||
"""Kimi-K3 attention-residual aggregation fused with an NVLS pull reduce-scatter
|
||||
or a direct multicast all-gather (the collective variants of attn_res_fused_tma)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
import torch
|
||||
|
||||
from sglang.kernels.ops.attention.attn_res import _jit_fused_tma_module, _tuning
|
||||
from sglang.srt.utils.custom_op import register_custom_op
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from sglang.kernels.ops.communication.all_reduce import Communicator
|
||||
|
||||
_COMM_MAP: dict[int, Communicator] = {}
|
||||
|
||||
|
||||
def register_comm(comm: Communicator) -> None:
|
||||
# One communicator per world_size per process -- see the note in
|
||||
# communication/all_reduce_residual.py::register_comm. The ops key only on world_size, so an
|
||||
# overwrite here would hand the old group's callers the new group's peer
|
||||
# pointers.
|
||||
prev = _COMM_MAP.get(comm.world_size)
|
||||
assert prev is None or prev is comm, (
|
||||
f"a different communicator is already registered for world_size="
|
||||
f"{comm.world_size}"
|
||||
)
|
||||
_COMM_MAP[comm.world_size] = comm
|
||||
|
||||
|
||||
@register_custom_op(mutates_args=["out", "prefix_out"])
|
||||
def _attn_res_fused_pull_rs_op(
|
||||
world_size: int,
|
||||
input: torch.Tensor,
|
||||
residual: torch.Tensor | None,
|
||||
bank: torch.Tensor,
|
||||
cw: torch.Tensor,
|
||||
ow: torch.Tensor,
|
||||
out: torch.Tensor,
|
||||
prefix_out: torch.Tensor,
|
||||
nvb: int,
|
||||
eps: float,
|
||||
input_mc_ptr: int,
|
||||
max_blocks: int,
|
||||
chunk_rows: int,
|
||||
occupancy: int,
|
||||
consumer_regs: int,
|
||||
) -> None:
|
||||
_jit_fused_tma_module(chunk_rows, occupancy, consumer_regs).run_pull_rs(
|
||||
_COMM_MAP[world_size],
|
||||
input,
|
||||
residual,
|
||||
bank,
|
||||
cw,
|
||||
ow,
|
||||
out,
|
||||
prefix_out,
|
||||
nvb,
|
||||
eps,
|
||||
input_mc_ptr,
|
||||
max_blocks,
|
||||
)
|
||||
|
||||
|
||||
@register_custom_op(mutates_args=["bank", "out"])
|
||||
def _attn_res_fused_direct_ag_op(
|
||||
world_size: int,
|
||||
prefix_sum: torch.Tensor,
|
||||
bank: torch.Tensor,
|
||||
cw: torch.Tensor,
|
||||
ow: torch.Tensor,
|
||||
out: torch.Tensor,
|
||||
nvb: int,
|
||||
eps: float,
|
||||
output_mc_ptr: int,
|
||||
max_blocks: int,
|
||||
write_prefix: bool,
|
||||
chunk_rows: int,
|
||||
occupancy: int,
|
||||
consumer_regs: int,
|
||||
) -> None:
|
||||
_jit_fused_tma_module(chunk_rows, occupancy, consumer_regs).run_direct_ag(
|
||||
_COMM_MAP[world_size],
|
||||
prefix_sum,
|
||||
bank,
|
||||
cw,
|
||||
ow,
|
||||
out,
|
||||
nvb,
|
||||
eps,
|
||||
output_mc_ptr,
|
||||
max_blocks,
|
||||
write_prefix,
|
||||
)
|
||||
|
||||
|
||||
def attn_res_fused_direct_ag(
|
||||
world_size: int,
|
||||
prefix_sum: torch.Tensor,
|
||||
bank: torch.Tensor,
|
||||
cw: torch.Tensor,
|
||||
ow: torch.Tensor,
|
||||
out: torch.Tensor,
|
||||
nvb: int,
|
||||
eps: float,
|
||||
*,
|
||||
output_mc_ptr: int,
|
||||
max_blocks: int = 128,
|
||||
write_prefix: bool = False,
|
||||
) -> torch.Tensor:
|
||||
"""Fuse local attention-residual aggregation with direct multicast AG.
|
||||
|
||||
`out` is the full symmetric [world * local_tokens, H] output. Each rank
|
||||
aggregates its local token shard and multicast-stores normalized vectors
|
||||
directly from consumer registers into that rank's slice on every peer.
|
||||
"""
|
||||
_attn_res_fused_direct_ag_op(
|
||||
world_size,
|
||||
prefix_sum,
|
||||
bank,
|
||||
cw,
|
||||
ow,
|
||||
out,
|
||||
nvb,
|
||||
eps,
|
||||
output_mc_ptr,
|
||||
max_blocks,
|
||||
write_prefix,
|
||||
*_tuning(nvb, prefix_sum.shape[0]),
|
||||
)
|
||||
return out
|
||||
|
||||
|
||||
def attn_res_fused_pull_rs(
|
||||
world_size: int,
|
||||
input: torch.Tensor,
|
||||
residual: torch.Tensor | None,
|
||||
bank: torch.Tensor,
|
||||
cw: torch.Tensor,
|
||||
ow: torch.Tensor,
|
||||
out: torch.Tensor,
|
||||
prefix_out: torch.Tensor,
|
||||
nvb: int,
|
||||
eps: float,
|
||||
*,
|
||||
input_mc_ptr: int,
|
||||
max_blocks: int = 128,
|
||||
) -> tuple[torch.Tensor, torch.Tensor]:
|
||||
"""Fused NVLS pull RS + local residual + TMA attention aggregation.
|
||||
|
||||
`input` is the full TP-partial o_proj tensor in multicast symmetric
|
||||
memory. Every rank reduces only its contiguous token shard, optionally
|
||||
adds its already-local residual, materializes that prefix in `prefix_out`,
|
||||
and feeds it directly to the fused attention-residual aggregation/norm.
|
||||
"""
|
||||
_attn_res_fused_pull_rs_op(
|
||||
world_size,
|
||||
input,
|
||||
residual,
|
||||
bank,
|
||||
cw,
|
||||
ow,
|
||||
out,
|
||||
prefix_out,
|
||||
nvb,
|
||||
eps,
|
||||
input_mc_ptr,
|
||||
max_blocks,
|
||||
# prefix_out, not input: the shard this rank actually aggregates.
|
||||
*_tuning(nvb, prefix_out.shape[0]),
|
||||
)
|
||||
return out, prefix_out
|
||||
@@ -0,0 +1,97 @@
|
||||
"""Multicast all-gather of bf16 column slices through the CustomAllReduceV2 push workspace.
|
||||
|
||||
``out[:, rank slice] = local`` (+ b) (+ c) for every rank's ``[M, N / world]``
|
||||
slice: one kernel multicast-stores the slice into every peer's push workspace
|
||||
and Lamport-spins the full rows together. Any TP world size; needs the plane's
|
||||
multicast mapping. The column-parallel projection that produces the slices
|
||||
lives in ``sglang.srt.layers.communication.muse_spark_gemm_ag``.
|
||||
|
||||
Unlike ``communication.gemm_ag`` (Kimi-K3), whose producer fuses the GEMV for
|
||||
one fixed TP8 shape, this gathers slices an external GEMM already computed.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import TYPE_CHECKING, Optional
|
||||
|
||||
import torch
|
||||
|
||||
from sglang.kernels.jit.utils import (
|
||||
cache_once,
|
||||
is_arch_support_pdl,
|
||||
load_jit,
|
||||
make_cpp_args,
|
||||
)
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from tvm_ffi.module import Module
|
||||
|
||||
from sglang.srt.distributed.device_communicators.custom_all_reduce_v2 import (
|
||||
CustomAllReduceV2,
|
||||
)
|
||||
|
||||
SMALL_PROJECTION_MAX_TOKENS = 8
|
||||
MAX_TOKENS = 512
|
||||
_VEC = 8
|
||||
|
||||
|
||||
@cache_once
|
||||
def _jit_module(n: int, world_size: int) -> Module:
|
||||
args = make_cpp_args(n, world_size, is_arch_support_pdl())
|
||||
return load_jit(
|
||||
"muse_spark_all_gather",
|
||||
*args,
|
||||
cuda_files=["muse_spark/all_gather.cuh"],
|
||||
cuda_wrappers=[("run", f"MuseSparkAllGather<{args}>::run")],
|
||||
extra_cuda_cflags=["-O3"],
|
||||
)
|
||||
|
||||
|
||||
def gemm_ag_supported(
|
||||
comm: Optional[CustomAllReduceV2], x: torch.Tensor, weight: torch.Tensor
|
||||
) -> bool:
|
||||
from sglang.srt.distributed.device_communicators.custom_all_reduce_v2 import (
|
||||
CustomAllReduceV2,
|
||||
)
|
||||
|
||||
if (
|
||||
not isinstance(comm, CustomAllReduceV2)
|
||||
or comm.disabled
|
||||
or not comm.has_multicast
|
||||
or comm.world_size < 2
|
||||
):
|
||||
return False
|
||||
if x.dtype != torch.bfloat16 or weight.dtype != torch.bfloat16 or x.dim() != 2:
|
||||
return False
|
||||
m, k = x.shape
|
||||
n = weight.shape[0]
|
||||
return (
|
||||
1 <= m <= MAX_TOKENS
|
||||
and (
|
||||
m <= SMALL_PROJECTION_MAX_TOKENS
|
||||
or (
|
||||
comm.world_size in (4, 8)
|
||||
and n in (4096, 8192)
|
||||
and torch.cuda.get_device_capability(x.device) == (10, 3)
|
||||
)
|
||||
)
|
||||
and weight.shape[1] == k
|
||||
and n % (comm.world_size * _VEC) == 0
|
||||
and m * n * 2 <= comm.max_push_size
|
||||
and comm.config.num_push_blocks >= 2
|
||||
and weight.is_contiguous()
|
||||
)
|
||||
|
||||
|
||||
def all_gather(
|
||||
comm: CustomAllReduceV2,
|
||||
local: torch.Tensor,
|
||||
out: torch.Tensor,
|
||||
b: Optional[torch.Tensor] = None,
|
||||
c: Optional[torch.Tensor] = None,
|
||||
) -> torch.Tensor:
|
||||
"""``out[:, rank slice] = local`` gathered from every rank, (+ b) (+ c)."""
|
||||
_jit_module(out.shape[1], comm.world_size).run(
|
||||
comm.obj, local.contiguous(), b, c, out
|
||||
)
|
||||
return out
|
||||
+1
-1
@@ -12,7 +12,7 @@ from sglang.kernels.jit.utils import (
|
||||
load_jit,
|
||||
make_cpp_args,
|
||||
)
|
||||
from sglang.kernels.ops.muse_spark.norm_residual import (
|
||||
from sglang.kernels.ops.layernorm.muse_spark_norm_residual_jit import (
|
||||
norm_residual_launch_config,
|
||||
)
|
||||
|
||||
+17
-14
@@ -14,20 +14,23 @@ Backends:
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
import torch
|
||||
|
||||
from sglang.kernels.fused_op import BaseFusedOp
|
||||
from sglang.kernels.jit.utils import is_arch_support_pdl
|
||||
from sglang.kernels.ops.muse_spark import (
|
||||
fi_trtllm_moe_finalize_with_rmsnorm_cutedsl,
|
||||
)
|
||||
from sglang.kernels.ops.muse_spark.fi_trtllm_moe_finalize_with_rmsnorm_cutedsl import (
|
||||
FinalizePushWorkspace,
|
||||
from sglang.kernels.ops.communication import (
|
||||
muse_spark_moe_finalize_all_reduce_cutedsl,
|
||||
)
|
||||
from sglang.kernels.spec import CapabilityRequirement, KernelBackend
|
||||
from sglang.srt.runtime_context import get_forward
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from sglang.srt.layers.communication.muse_spark_finalize_workspace import (
|
||||
FinalizePushWorkspace,
|
||||
)
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
_missing_cutedsl_warning_logged = False
|
||||
|
||||
@@ -103,7 +106,7 @@ class FusedPostExpertNormWeightedSumTRTLLM(BaseFusedOp):
|
||||
from token count for H=4096; other shapes use a single CTA.
|
||||
"""
|
||||
|
||||
op = "muse_spark.post_expert_finalize"
|
||||
op = "communication.muse_spark_moe_finalize_all_reduce"
|
||||
priority = (KernelBackend.CUTE_DSL, KernelBackend.HELION)
|
||||
capabilities = {
|
||||
KernelBackend.CUTE_DSL: {CapabilityRequirement.CUDA},
|
||||
@@ -132,7 +135,7 @@ class FusedPostExpertNormWeightedSumTRTLLM(BaseFusedOp):
|
||||
return self._helion_eligible(*args, **kwargs)
|
||||
if backend is not KernelBackend.CUTE_DSL:
|
||||
return True
|
||||
if not fi_trtllm_moe_finalize_with_rmsnorm_cutedsl.HAS_CUTEDSL:
|
||||
if not muse_spark_moe_finalize_all_reduce_cutedsl.HAS_CUTEDSL:
|
||||
global _missing_cutedsl_warning_logged
|
||||
if not _missing_cutedsl_warning_logged:
|
||||
_missing_cutedsl_warning_logged = True
|
||||
@@ -249,7 +252,7 @@ class FusedPostExpertNormWeightedSumTRTLLM(BaseFusedOp):
|
||||
from cutlass import Int32
|
||||
from cutlass.cute.runtime import make_ptr
|
||||
|
||||
kernel_module = fi_trtllm_moe_finalize_with_rmsnorm_cutedsl
|
||||
kernel_module = muse_spark_moe_finalize_all_reduce_cutedsl
|
||||
|
||||
assert gemm2_output.is_cuda
|
||||
cutlass_dtype = kernel_module._torch_to_cutlass_dtype[gemm2_output.dtype]
|
||||
@@ -354,10 +357,10 @@ class FusedPostExpertNormWeightedSumTRTLLM(BaseFusedOp):
|
||||
*,
|
||||
ar_workspace: FinalizePushWorkspace | None = None,
|
||||
) -> bool:
|
||||
from sglang.kernels.ops.muse_spark.helion_runtime import helion_supported
|
||||
from sglang.kernels.jit.utils.helion_runtime import helion_supported
|
||||
|
||||
return (
|
||||
helion_supported("post_expert_finalize", output)
|
||||
helion_supported(output)
|
||||
and output.shape[0] > 0
|
||||
and self.topk == 8
|
||||
and self.hidden_dim == 4096
|
||||
@@ -385,13 +388,13 @@ class FusedPostExpertNormWeightedSumTRTLLM(BaseFusedOp):
|
||||
) -> None:
|
||||
"""Run Helion finalize, then a separate AR when delegated by the model."""
|
||||
|
||||
from sglang.kernels.ops.muse_spark.helion_post_expert import (
|
||||
from sglang.kernels.jit.utils.helion_runtime import (
|
||||
run_pinned_helion,
|
||||
)
|
||||
from sglang.kernels.ops.communication.muse_spark_moe_finalize_all_reduce_helion import (
|
||||
POST_EXPERT_FINALIZE_CONFIGS,
|
||||
_post_expert_finalize_helion_kernel,
|
||||
)
|
||||
from sglang.kernels.ops.muse_spark.helion_runtime import (
|
||||
run_pinned_helion,
|
||||
)
|
||||
|
||||
run_pinned_helion(
|
||||
_post_expert_finalize_helion_kernel,
|
||||
+3
-135
@@ -24,146 +24,14 @@ Algorithm for each token (1 block or DSMEM cluster per token):
|
||||
Convert the FP32 accumulator to the output dtype once.
|
||||
|
||||
The public API is ``FusedPostExpertNormWeightedSumTRTLLM`` in
|
||||
``fused_post_expert_norm_weighted_sum_trtllm.py``. This module exposes
|
||||
the CuTeDSL kernel, compilation cache, and borrowed TP push workspace.
|
||||
``muse_spark_moe_finalize_all_reduce.py``. This module exposes
|
||||
the CuTeDSL kernel and its compilation cache; the borrowed TP push workspace
|
||||
is ``sglang.srt.layers.communication.muse_spark_finalize_workspace``.
|
||||
"""
|
||||
|
||||
import functools
|
||||
|
||||
import torch
|
||||
import torch.distributed as dist
|
||||
|
||||
|
||||
class FinalizePushWorkspace:
|
||||
"""Borrow TP push buffers before capture; dependency-ordered launches only."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
communicator,
|
||||
device_group: dist.ProcessGroup,
|
||||
hidden_dim=4096,
|
||||
*,
|
||||
multicast=False,
|
||||
):
|
||||
self.communicator = communicator # Keep the allocation owner alive.
|
||||
self.group = device_group
|
||||
self.rank = communicator.rank
|
||||
self.world_size = communicator.world_size
|
||||
self.device = communicator.device
|
||||
self.sm_count = torch.cuda.get_device_properties(
|
||||
self.device
|
||||
).multi_processor_count
|
||||
self.hidden_dim = hidden_dim
|
||||
self.slot_bytes = communicator.max_push_size
|
||||
self.peers = communicator._push_workspaces
|
||||
self.counter = communicator._push_counter
|
||||
self.multicast = multicast
|
||||
self.multicast_ptr = communicator._push_multicast_ptr
|
||||
config = (
|
||||
self.controller_count,
|
||||
self.slot_bytes,
|
||||
self.sm_count,
|
||||
hidden_dim,
|
||||
multicast,
|
||||
)
|
||||
configs = [None] * self.world_size
|
||||
dist.all_gather_object(configs, config, group=communicator.group)
|
||||
if any(peer != config for peer in configs):
|
||||
raise ValueError(
|
||||
"Finalize AR workspace configuration must match across ranks"
|
||||
)
|
||||
if multicast and not communicator.has_multicast:
|
||||
raise ValueError(
|
||||
"Finalize multicast requires a multicast-capable push workspace"
|
||||
)
|
||||
if not 0 < self.controller_count <= self.sm_count:
|
||||
raise ValueError("Unsupported finalize AR controller budget")
|
||||
if hidden_dim <= 0 or hidden_dim % 256 or self.slot_bytes < hidden_dim * 2:
|
||||
raise ValueError(
|
||||
"Finalize push requires H divisible by 256 and sufficient buffer capacity"
|
||||
)
|
||||
self.max_tokens = min(384, self.slot_bytes // (hidden_dim * 2))
|
||||
self.fallback_logged = False
|
||||
self.fusion_logged = False
|
||||
|
||||
def uses_one_shot_push(self, num_tokens: int) -> bool:
|
||||
return self.communicator.uses_one_shot_push(num_tokens * self.hidden_dim * 2)
|
||||
|
||||
def can_fuse(self, tp_group, num_tokens: int) -> bool:
|
||||
"""Use custom AR storage while respecting explicit algorithm overrides.
|
||||
|
||||
Fusion has a different crossover from a standalone AR launch. Allow
|
||||
push through 384 tokens when its allocated slot fits, even if ordinary
|
||||
CUDA-graph AR would select pull above its 288-token crossover (H=4096),
|
||||
but not where the standalone AR would pick 2shot_push.
|
||||
No standalone AR thresholds or buffer allocations are changed.
|
||||
"""
|
||||
if (
|
||||
torch.compiler.is_compiling()
|
||||
or self.communicator is not tp_group.ca_comm
|
||||
or self.communicator.disabled
|
||||
or not 0 < num_tokens <= self.max_tokens
|
||||
):
|
||||
return False
|
||||
if self.world_size > 4 and not self.uses_one_shot_push(num_tokens):
|
||||
return False
|
||||
if self.communicator.uses_two_shot_push(num_tokens * self.hidden_dim * 2):
|
||||
return False
|
||||
output = (
|
||||
self.peers[self.rank]
|
||||
.view(torch.bfloat16)
|
||||
.view(-1)[: num_tokens * self.hidden_dim]
|
||||
.view(num_tokens, self.hidden_dim)
|
||||
)
|
||||
return (
|
||||
not tp_group._can_use_flashinfer_allreduce(output)
|
||||
and tp_group._resolve_outplace_all_reduce_method(output) == "ca"
|
||||
and (
|
||||
self.communicator.override_algo is None
|
||||
or self.uses_one_shot_push(num_tokens)
|
||||
)
|
||||
)
|
||||
|
||||
@property
|
||||
def controller_count(self) -> int:
|
||||
return self.counter.numel()
|
||||
|
||||
def validate(self, output: torch.Tensor, cluster_size: int = 1) -> None:
|
||||
if (
|
||||
output.device != self.device
|
||||
or output.dtype != torch.bfloat16
|
||||
or output.ndim != 2
|
||||
or output.shape[1] != self.hidden_dim
|
||||
or output.shape[0] > self.max_tokens
|
||||
or not output.is_contiguous()
|
||||
or output.data_ptr() % 16
|
||||
):
|
||||
raise ValueError("Unsupported output for finalize push workspace")
|
||||
if cluster_size not in (1, 2, 4, 8, 16) or cluster_size > self.sm_count:
|
||||
raise ValueError("Unsupported cluster size")
|
||||
|
||||
if output.shape[0] == 0:
|
||||
raise ValueError("Finalize push requires M>0")
|
||||
if cluster_size > 1 and output.shape[0] * cluster_size > self.controller_count:
|
||||
raise ValueError(
|
||||
"Single-token clusters require M * C <= controller_count; use C=1 otherwise"
|
||||
)
|
||||
|
||||
def arguments(self) -> tuple:
|
||||
import cutlass
|
||||
from cutlass.cute.runtime import make_ptr
|
||||
|
||||
return (
|
||||
tuple(make_ptr(cutlass.Uint32, p.data_ptr()) for p in self.peers),
|
||||
make_ptr(cutlass.Uint32, self.counter.data_ptr()),
|
||||
cutlass.Int32(self.rank),
|
||||
cutlass.Int32(self.slot_bytes // 4),
|
||||
make_ptr(cutlass.Uint32, self.peers[self.rank].data_ptr()),
|
||||
make_ptr(
|
||||
cutlass.Uint32, self.multicast_ptr or self.peers[self.rank].data_ptr()
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
try:
|
||||
import cuda.bindings.driver as cuda
|
||||
+28
-84
@@ -3,13 +3,25 @@
|
||||
Torch symmetric memory is used only to allocate/register NVLink-accessible
|
||||
buffers. AG dispatches between a tile-gated CUTLASS GEMM and per-shard cuBLAS
|
||||
GEMMs; RS uses a fused CUTLASS kernel.
|
||||
|
||||
The symmetric-memory workspaces are owned by
|
||||
``sglang.srt.layers.communication.sp_collective_gemm_workspace``: each entry
|
||||
point takes its getter as ``workspace`` and calls it with the tile-padded shape
|
||||
``(kind, group, rows, width, x, row_align, flag_tile_mn)``.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import TYPE_CHECKING, Callable
|
||||
|
||||
import torch
|
||||
import torch.distributed as dist
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from sglang.srt.layers.communication.sp_collective_gemm_workspace import (
|
||||
SPCollectiveGemmWorkspace,
|
||||
)
|
||||
|
||||
_TILE_M = 64
|
||||
_TILE_N = 128
|
||||
_AG_TILES = (
|
||||
@@ -18,7 +30,6 @@ _AG_TILES = (
|
||||
(64, 128, False, (1, 1)),
|
||||
)
|
||||
_RS_TILES = ((128, 256, False, (1, 1)), (64, 128, False, (1, 1)))
|
||||
_workspaces = {}
|
||||
_compiled = {}
|
||||
|
||||
|
||||
@@ -93,86 +104,10 @@ def _check_inputs(x, weight, group):
|
||||
return world, dist.get_rank(group)
|
||||
|
||||
|
||||
class _Workspace:
|
||||
def __init__(
|
||||
self,
|
||||
kind,
|
||||
group,
|
||||
rows,
|
||||
width,
|
||||
dtype,
|
||||
device,
|
||||
row_align=_TILE_M,
|
||||
flag_tile_mn=(_TILE_M, _TILE_N),
|
||||
):
|
||||
import cutlass.torch as cutlass_torch
|
||||
import torch.distributed._symmetric_memory as symm_mem
|
||||
|
||||
# Power of two >= row_align, so capacity stays a whole number of M tiles.
|
||||
self.capacity = max(row_align, 1 << (rows - 1).bit_length())
|
||||
self.world = dist.get_world_size(group)
|
||||
self.sms = torch.cuda.get_device_properties(device).multi_processor_count
|
||||
self.storage = symm_mem.empty(
|
||||
(self.capacity, width), dtype=dtype, device=device
|
||||
)
|
||||
self.handle = symm_mem.rendezvous(self.storage, group)
|
||||
if kind == "ag":
|
||||
self.gathered = torch.empty(
|
||||
(self.capacity * self.world, width), dtype=dtype, device=device
|
||||
)
|
||||
self.flags = torch.zeros(self.world, dtype=torch.int32, device=device)
|
||||
self.copy_stream = torch.cuda.Stream(device=device)
|
||||
else:
|
||||
if not self.handle.multicast_ptr:
|
||||
raise RuntimeError(
|
||||
"CUTLASS GEMM-RS requires NVSwitch multicast mapping"
|
||||
)
|
||||
self.multicast = cutlass_torch.as_tensor(
|
||||
self.handle.multicast_ptr, self.storage.shape, dtype
|
||||
)
|
||||
num_flags = (
|
||||
self.capacity // flag_tile_mn[0] * (width // flag_tile_mn[1])
|
||||
+ self.sms
|
||||
+ 8
|
||||
)
|
||||
self.flags = symm_mem.empty((num_flags,), dtype=torch.int32, device=device)
|
||||
self.flags.zero_()
|
||||
self.flag_handle = symm_mem.rendezvous(self.flags, group)
|
||||
if not self.flag_handle.multicast_ptr:
|
||||
raise RuntimeError("CUTLASS GEMM-RS requires multicast flag storage")
|
||||
self.flag_multicast = cutlass_torch.as_tensor(
|
||||
self.flag_handle.multicast_ptr, self.flags.shape, self.flags.dtype
|
||||
)
|
||||
self.peer_flags = tuple(
|
||||
self.flag_handle.get_buffer(r, self.flags.shape, self.flags.dtype)
|
||||
for r in range(self.world)
|
||||
)
|
||||
self.handle.barrier(channel=0)
|
||||
|
||||
|
||||
def _workspace(
|
||||
kind, group, rows, width, x, row_align=_TILE_M, flag_tile_mn=(_TILE_M, _TILE_N)
|
||||
):
|
||||
key = (kind, group, width, x.dtype, x.device)
|
||||
ws = _workspaces.get(key)
|
||||
if ws is None or rows > ws.capacity:
|
||||
if torch.cuda.is_current_stream_capturing():
|
||||
raise RuntimeError("Warm up CUTLASS SP workspace before CUDA graph capture")
|
||||
if ws is not None:
|
||||
# Peer mappings must stay alive until all ranks stop reading them.
|
||||
ws.handle.barrier(channel=0)
|
||||
torch.cuda.current_stream().synchronize()
|
||||
ws = _Workspace(
|
||||
kind, group, rows, width, x.dtype, x.device, row_align, flag_tile_mn
|
||||
)
|
||||
_workspaces[key] = ws
|
||||
return ws
|
||||
|
||||
|
||||
def _all_gather_matmul_cublas(
|
||||
local: torch.Tensor,
|
||||
weight: torch.Tensor,
|
||||
ws: _Workspace,
|
||||
ws: SPCollectiveGemmWorkspace,
|
||||
rank: int,
|
||||
rows: int,
|
||||
out: torch.Tensor,
|
||||
@@ -236,7 +171,14 @@ def _all_gather_matmul_cublas(
|
||||
main.wait_stream(ws.copy_stream)
|
||||
|
||||
|
||||
def all_gather_matmul(x, weight, group, *, backend: str = "auto"):
|
||||
def all_gather_matmul(
|
||||
x,
|
||||
weight,
|
||||
group,
|
||||
*,
|
||||
workspace: Callable[..., SPCollectiveGemmWorkspace],
|
||||
backend: str = "auto",
|
||||
):
|
||||
"""all_gather(x, dim=0) @ weight.T, with shape-based backend selection.
|
||||
|
||||
Backend overrides must agree across ranks and are fixed at graph capture.
|
||||
@@ -245,7 +187,7 @@ def all_gather_matmul(x, weight, group, *, backend: str = "auto"):
|
||||
import cutlass
|
||||
import cutlass.cute as cute
|
||||
|
||||
from sglang.kernels.ops.gemm.cutlass_all_gather_gemm import (
|
||||
from sglang.kernels.ops.communication.cutlass_all_gather_gemm import (
|
||||
PersistentDenseGemmKernel,
|
||||
ReleaseFlagSet,
|
||||
)
|
||||
@@ -262,7 +204,7 @@ def all_gather_matmul(x, weight, group, *, backend: str = "auto"):
|
||||
mma_m, mma_n, use_2cta, cluster = _ag_tile(n)
|
||||
local = _pad_shards(x.contiguous(), 1, mma_m)
|
||||
padded = local.shape[0]
|
||||
ws = _workspace("ag", group, padded, k, x, mma_m)
|
||||
ws = workspace("ag", group, padded, k, x, mma_m)
|
||||
gathered = ws.gathered[: padded * world]
|
||||
if backend == "cublas":
|
||||
out = x.new_empty((rows * world, n))
|
||||
@@ -328,13 +270,15 @@ def all_gather_matmul(x, weight, group, *, backend: str = "auto"):
|
||||
return _unpad_shards(out, rows, world)
|
||||
|
||||
|
||||
def matmul_reduce_scatter(x, weight, group):
|
||||
def matmul_reduce_scatter(
|
||||
x, weight, group, *, workspace: Callable[..., SPCollectiveGemmWorkspace]
|
||||
):
|
||||
"""reduce_scatter(x @ weight.T, SUM, dim=0), one fused GEMM-RS kernel."""
|
||||
import cuda.bindings.driver as cuda
|
||||
import cutlass
|
||||
import cutlass.cute as cute
|
||||
|
||||
from sglang.kernels.ops.gemm.cutlass_gemm_reduce_scatter import (
|
||||
from sglang.kernels.ops.communication.cutlass_gemm_reduce_scatter import (
|
||||
Sm100PersistentDenseGemmReduceScatterLDMCKernel,
|
||||
)
|
||||
|
||||
@@ -349,7 +293,7 @@ def matmul_reduce_scatter(x, weight, group):
|
||||
cta_m = mma_m // (2 if use_2cta else 1)
|
||||
padded_input = _pad_shards(x.contiguous(), world, cta_m)
|
||||
m = padded_input.shape[0]
|
||||
ws = _workspace("rs", group, m, n, x, cta_m, (cta_m, mma_n))
|
||||
ws = workspace("rs", group, m, n, x, cta_m, (cta_m, mma_n))
|
||||
out = x.new_empty((m // world, n))
|
||||
args = dict(
|
||||
a=_tensor(padded_input),
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user