Bump FlashInfer to 0.6.17 and remove Kimi K3 workarounds (#33997)

This commit is contained in:
Mohammad Miadh Angkad
2026-08-12 02:17:26 -07:00
committed by GitHub
parent 2d76d537e5
commit 00e57d74f0
19 changed files with 84 additions and 6496 deletions
@@ -127,16 +127,7 @@ Capacity levers, all in the Playground. Each trades precision or cache behavior
Speculation: DSPARK holds block size + 1 (= 8) intermediate states per request — the calculator folds this in — and an unset `--max-running-requests` resets to 48 under spec (the command panel reminds you; set it explicitly to raise).
**MoE runner.** Leave `--moe-runner-backend` unset on Blackwell and it resolves to FlashInfer MXFP4 (W4A8, prebuilt trtllm-gen SiTU kernels) when the cubin pool is installed, Marlin (W4A16) otherwise; H100/H200 pin Marlin. The B200 Balanced and High-Throughput cells pin `flashinfer_mxfp4` explicitly because that is the shape they were brought up on — on an install without the pool, drop the flag to fall back to Marlin. The published Docker images already provision the **SiTU cubin pool**; to install it independently, run the same flow as the Dockerfile:
```bash
wget https://github.com/sgl-project/whl/releases/download/trtllm_gen_moe_cubin_20260617/trtllm_gen_moe_cubin_pool_20260617_v0613rc1.zip
sudo mkdir -p /opt/trtllm_gen_moe_cubin_pool
sudo unzip -q trtllm_gen_moe_cubin_pool_20260617_v0613rc1.zip -d /opt/trtllm_gen_moe_cubin_pool
export SGLANG_TRTLLM_GEN_MOE_CUBIN_POOL=/opt/trtllm_gen_moe_cubin_pool/trtllm_gen_moe_cubin_pool_20260617_v0613rc1
```
Remaining kernel sources JIT once from the public `flashinfer` wheel (a few minutes, cached).
**MoE runner.** Leave `--moe-runner-backend` unset on Blackwell: FlashInfer MXFP4 (W4A8, official trtllm-gen SiTU kernels) is selected with the pinned FlashInfer 0.6.17 dependency; H100/H200 pin Marlin. The B200 Balanced and High-Throughput cells pin `flashinfer_mxfp4` explicitly because that is the shape they were brought up on. The published Docker images install the matching official `flashinfer-python`, `flashinfer-cubin`, and `flashinfer-jit-cache` packages.
**Attention backend.** Leave all three attention knobs unset on Blackwell: K3 resolves prefill, decode, and — under DSPARK — verification as a set (`trtllm_mla` across the board; `cutedsl_mla` takes decode and verification under DCP). On the non-DCP recipes, setting any one of the three cancels the auto-resolution for the others. The B200 Balanced and High-Throughput cells pin `--decode-attention-backend cutedsl_mla`, which is what auto-resolution picks for those DCP recipes anyway — it is written out because it is the shape they were brought up on, not because it changes the resolution. H100/H200 pin `flashmla` for decode.
@@ -395,7 +386,7 @@ Both presets are one click away in the [Playground above](#playground): pick a *
Decisions the preset already makes:
- **MegaMoE on `deep_gemm`** — the fastest a2a backend; needs the SiTU cubin pool ([§2](#2-configuration-tips)).
- **MegaMoE + `deep_gemm`** — the fused DeepGEMM all-to-all/MoE path used by these large-scale DP/EP throughput presets, with K3's SiTU activation.
- **SP-MoE and shared-expert overlap** engage automatically under EP a2a; the K3 all-reduce fusion does not.
- **Spec Decode follows the Deploy knob.** Acceptance thins at large batch; spec × EP × DP-attention is validated only at 8-GPU EP8 × DP2 (full GSM8K) — experimental at these scales.
+1 -1
View File
@@ -629,7 +629,7 @@ export const Playground = ({ config }) => {
"--moe-a2a-backend", "--moe-runner-backend",
]);
// Backend options may carry their own env (e.g. the FlashInfer MXFP4
// cubin-pool path): strip every backend option's env keys, then
// backend-specific path): strip every backend option's env keys, then
// re-add the selected option's.
const backendEnvKeys = [];
for (const o of (fc.backend?.options || [])) {
@@ -458,10 +458,8 @@ export const config = {
// Blackwell-only kernel-fusion path; selecting it reveals the Quantization sub-select.
{ id: "megamoe", label: "MegaMoE", flags: ["--moe-a2a-backend megamoe"],
requiresHw: ["b200", "b300", "gb200", "gb300"] },
// Blackwell-only: runs the prebuilt trtllm-gen SiTU cubins; needs the
// downloadable SiTU cubin pool unpacked and pointed to by the env var.
// Blackwell-only: runs FlashInfer's official trtllm-gen SiTU kernels.
{ id: "flashinfer_mxfp4", label: "FlashInfer (MXFP4)", flags: ["--moe-runner-backend flashinfer_mxfp4"],
env: ["SGLANG_TRTLLM_GEN_MOE_CUBIN_POOL=/path/to/trtllm_gen_moe_cubin_pool"],
requiresHw: ["b200", "b300", "gb200", "gb300"] },
{ id: "marlin", label: "Marlin (W4A16)", flags: ["--moe-runner-backend marlin"] },
],
@@ -931,10 +929,8 @@ export const config = {
"--pp-size 2",
"--dcp-size 8",
"--ep-size 8",
// Both pinned to the brought-up shape rather than left to the auto
// resolution the rest of Blackwell uses. The MXFP4 runner needs the SiTU
// cubin pool the published image ships; drop it to get the Marlin
// fallback on an install without one.
// Both backends are pinned to the validated recipe even though automatic
// resolution selects them on Blackwell.
"--moe-runner-backend flashinfer_mxfp4",
"--decode-attention-backend cutedsl_mla",
"--mem-fraction-static 0.85",