Commit Graph
16634 Commits
Author SHA1 Message Date
Cheng Wan 97279980cf config: the last runner-side instance reads read the bags
Six reads were left on `self.server_args` outside the per-instance boundary the
plan reserves for the tokenizer-manager family, and each had a different reason
to be there:

- `scheduler.process_input_requests` (`mm_feature_transport`) and
  `BaseSpecWorker._build_hicache_draft_plan` (`enable_hierarchical_cache`) are
  plain leaves -> `get_mm()` / `get_memory()`.
- `DraftBackendFactory._create_backend` read the split backend through a
  *runtime-computed name* (`getattr(self.server_args, backend_name)`) and then
  fell back to the base field by hand -- the census's documented blind spot.
  The two names it can be handed are exactly the pair `attention_backends()`
  returns with that fallback already applied, so it reads the pair and indexes
  it. The draft runner's own stamp still wins when it has one.
- `remote_instance_weight_loader_use_transfer_engine` and
  `pre_capture_activation_reserve_mb` are derived members. Both are computed
  from published leaves only, so both get a named accessor that derives from
  the bags (and therefore follows a post-publish override).

The first of those two has all its inputs in one bag, so it follows the
established shape: one `*_of(cfg)` helper in `arg_groups/overrides.py`, the
`ServerArgs` member delegating to it, and the accessor calling it on
`get_model()`. `modelexpress_transport_of` splits out the JSON parse both
sides need. The second spans four bags plus the configured parallel sizes, so
it exists twice like the mamba pair -- and `TestDerivedPredicatesAgreeAcrossTiers`
now pins both new pairs equal over their input matrices (92 subtests).

`self.server_args.X` outside the tokenizer-manager family: 11 -> 5, and the
five that remain are the documented ones (the encode server's own record, the
nixl connector's rank arithmetic, `GrammarManager`'s handed instance).

The post-capture headroom path calls the same bag-backed
`pre_capture_activation_reserve_mb` accessor the configurator uses -- the
accessor advertises override-following, and a reserve that reads the record
while its sibling reads the bags can disagree after a post-publish override.
And the conversions' orphans go with them: `RemoteInstanceWeightTransporter`
kept a `server_args` field nothing reads, and `DraftBackendFactory` parked a
record it no longer consults -- both drop the parameter, and the four factory
call sites stop threading one.
2026-08-15 00:37:06 -07:00
Cheng Wan ab810e4052 config: each runner carries its own linear-attn kernel choice
Two problems in the same family as #33312 (a per-runner decision that one
participant answered differently), one fixed here and one guarded.

**The linear-attn kernel backends were process-wide.** `attn_backend_wrapper`
rebuilt a module-level dict once per runner, from the handed record plus a local
`prefill_default`. Two things follow, and both are wrong:

- **A draft could not hold a different choice than its target.** Only the runner
  whose model is GDN gets the SM100 FlashInfer prefill default; the operator's
  explicit flag belongs to the launch. The full-attention backends already model
  this correctly -- the runner stamps `prefill_attention_backend_str` /
  `decode_attention_backend_str` and its backend objects are built from the
  stamp. Linear attn had no stamp at all.
- **The second rebuild replaced the first one's choice.** The default was also
  recorded into the process-wide config, which the record does not see, so a
  runner rebuilding without a default of its own resolved `prefill` back to the
  base backend -- silently swapping the kernel the earlier runner selected.
  Demonstrated in-process before this change: table `FLASHINFER`, then `TRITON`.

`resolve_linear_attn_backends(prefill_default=None)` returns a frozen
`LinearAttnBackends(decode, prefill, verify)` from the published `exec.mamba`
leaves; the wrapper stamps it as `runner.linear_attn_backends` before building
the backends that read it; and the three consumers (GDN, KDA, Ascend GDN) read it
off the runner they are built for. Each already took `model_runner` and cached
the result on itself, so the value now simply comes from the right place. A
backend built outside that path has no stamp and raises on the attribute, the way
the full-attention strings do -- no silent fallback to hide the wiring mistake.

The recording goes away with it. A per-runner choice in the process-wide config
has no meaning the second runner can read correctly: the leaf is how the gate
asks "did the operator name a backend", so a recorded default reads back as an
operator flag and the next runner declines its own. The leaf now keeps meaning
what was asked for at launch, and the effective choice lives in the stamp (and
in the log line the gate already emits).

Precedence is unchanged: the resolver takes the default as an argument and an
explicit `--linear-attn-prefill-backend` wins over it, with the gate declining
early so it neither probes the device nor logs.

**A draft entry class must answer the loader exactly when its target does.** The
loader asks the entry class it instantiates for the shared-experts-fusion
decision, and a draft is its own entry class. When the target family carries
auto-disable conditions and the draft's class does not expose them, the loader
installs one decision for each and the draft's weights are laid out for the wrong
one. That shipped: the DSV4 DSpark draft skipped its bundled shared-expert
tensors until #33312 gave it the gate, costing accept length 5.60 -> 2.05.

`test_fusion_gate_coverage.py` walks the same registry but asks whether an entry
class *touches* the decision -- reads the flag, names a gated class. That catches
a class once it already consumes the decision; it could not catch one that should
consume it and does not, which is what the DSpark class looked like (it built the
family's *layer* classes, so the flag reader lived in another module and its own
source named no gated class). `test_draft_entry_hook_parity.py` asks the
invariant directly: presence parity between a draft entry class and the target it
is named after. Identity is deliberately not required -- the Qwen3.5 MTP
delegates with adapted arguments (unwrapping `text_config`, using the MTP
quantization config), which is right -- and weight-name maps are out of scope,
since a draft's checkpoint has its own names.

Reverse-verified against the original defect: with the DSpark gate removed the
case names the pair and the side that is missing it; with #33312 in place it
passes.
2026-08-15 00:36:04 -07:00
6cbfa791d6 [AMD][Spec] Accelerate Qwen3.5 verification with grouped-head shared KV (#34517)
Co-authored-by: chuyeh <chuyeh@users.noreply.github.com>
Co-authored-by: Thomas Wang <1am9trash@gmail.com>
2026-08-15 00:15:52 -07:00
Liangsheng Yin 5c9ee86d90 [CI] Move the static ratchets back to CPU unit tests (#34913) 2026-08-14 23:32:00 -07:00
raghothamandehhuang 2a89c6823a fix: honor explicit model loader classes (#34880)
Co-authored-by: ehhuang <yinghai@meta.com>
2026-08-15 06:00:41 +00:00
Xinyuan Tong e99ecb6eee [CI] Path-gate Rust workspace tests in lint (#34864) 2026-08-14 22:41:54 -07:00
Zhiyao Jiang 7216a44dd8 [AMD] Broadcast the EAGLE greedy verify decision across TP ranks on ROCm (#34238) 2026-08-14 22:37:45 -07:00
ethcheandEthan Che aeee1562e6 [Kernel] Enable Helion backend for Kimi Delta-Attention (#32593)
Co-authored-by: Ethan Che <eche@meta.com>
2026-08-14 22:22:46 -07:00
Khoa PhamandClaude Opus 5 3adbbec2fd [MoE] Route every trtllm-gen MoE call site through one PDL guard (#34789)
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-14 22:09:03 -07:00
Colin Z bc7e3ba66c [AMD][Quantization] Online MXFP4 quantization 4/N - NVFP4 to MXFP4 Online Requantization on AMD GPUs (#29328) 2026-08-14 21:59:39 -07:00
5afdb1caea [AMD CI] follow the miles nightly-prefixed MI350 suite names (#34877)
Co-authored-by: Zhiyao Jiang <jessicajiang324@gmail.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Michael <michaelzhang-ai@users.noreply.github.com>
2026-08-15 12:57:12 +08:00
Michael 2012b4b196 [AMD][CI] Fix stage-b: AttributeError on multimodal embedding requests (#34769) 2026-08-14 21:35:26 -07:00
hanwlax 8720a72814 fix(scheduler): track max prefill batch size over recent real admissions (#34284) 2026-08-15 10:48:03 +08:00
weireweireandweireweire 6eb941a34c [Spec] Support MegaMoE for DSpark under dp attention (#34844)
Co-authored-by: weireweire <20922698+weireweire@users.noreply.github.com>
2026-08-14 17:20:01 -07:00
gilfordtingandClaude Fable 5 a5ba081fbb [Spec] Wire DFLASH aux-hidden capture into the Qwen3.5 text-only wrapper (#34771)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-08-14 17:10:09 -07:00
gilfordtingandJames Liu 6f005e4da1 [Fix] Require JSON booleans for response_format json_schema.strict (#34777)
Co-authored-by: James Liu <jamesl@modal.com>
2026-08-14 16:46:27 -07:00
Baizhou Zhang a1844709f1 [CI] Trim Qwen3.5 FP8 GB300 performance batches (#34882) 2026-08-14 16:45:11 -07:00
gilfordtingandJames Liu 3f64f14360 [Fix] Work around xgrammar 0.2.1 negative integer minimum in Kimi-K3 structural tags (#34778)
Co-authored-by: James Liu <jamesl@modal.com>
2026-08-14 16:41:06 -07:00
Nan Jiang be804c1b83 [RL] Expose top-p-only sampling masks (#33593) 2026-08-14 16:40:27 -07:00
Jason Mancuso 90b3db6dd8 [Fix: RL] Snapshot async state-capture outputs before overlap (#34319) 2026-08-14 16:39:35 -07:00
Baizhou Zhang 8b4faa3336 [Docs] Update Kimi-K3 installation options (#34886) 2026-08-14 16:19:23 -07:00
DanielSwift1992 17efe37428 Remove 11 dead CODEOWNERS rules (#34875) 2026-08-14 15:38:19 -07:00
Ziang Li a9654eacc1 fix(dsa): use FlashInfer fused top-k for packed PAGED rows (#33006) 2026-08-14 15:22:18 -07:00
bfb224ff01 Add Reasoning-Aware Compression (RAC) pruning recipe for reasoning models (#32414)
Co-authored-by: Ryan Lucas <ryanluc@mit.edu>
Co-authored-by: Kayhan Behdin <kbehdin@linkedin.com>
Co-authored-by: Zhipeng Wang <zwanga@wustl.edu>
2026-08-14 15:13:45 -07:00
03c1d58112 perf: add H200 Triton MoE configs for E256 N512 (#34150)
Co-authored-by: Vedant Jhaveri <vjhaveri@linkedin.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-08-14 14:37:58 -07:00
Han-Yin ChangandClaude Fable 5 22dde1dd5b [Docs] Fill GLM-5.2 H200 FP8 speed cells (low-latency, balanced); fix MTP notation (#31554)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-08-14 13:56:55 -07:00
sglang-botandsglang-bot b676793e5e docs: sync LMSYS SGLang blog cards (#32982)
Co-authored-by: sglang-bot <sglang-bot@users.noreply.github.com>
2026-08-14 13:55:52 -07:00
Sidhartha Reddy PotuandXinyuan Tong 42e8718d3d fix(muse-glimmer): parse required/named tool calls natively (#34781)
Co-authored-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
2026-08-14 20:43:48 +00:00
Mohammad Miadh Angkad d7207be156 Fix startup weight load after TorchAO removal (#34869) 2026-08-14 13:18:53 -07:00
Jackey HuaandClaude Opus 5 1af761a09a [SM12x] Default the fused MHC post+pre path on (#34019)
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-08-14 19:51:21 +00:00
41cd5a7189 [Fix] Fix Qwen3.5 MTP startup with HiCache (#34560)
Co-authored-by: hjzhang <76768149+1e4ves@users.noreply.github.com>
Co-authored-by: YAMY <74099316+YAMY1234@users.noreply.github.com>
2026-08-14 10:55:41 -07:00
Xun Sun d8399af70c fix(qwen3): support DeepEP-class backends and early EPLB state (#34810) 2026-08-15 01:27:39 +08:00
Ke Bao 7562e741e2 Retain SWA down to the last state checkpoint (#34729) 2026-08-15 01:12:38 +08:00
Ke Bao c20aceeb88 Fix mamba checkpoint depth under dcp (#34808) 2026-08-15 00:34:00 +08:00
70e291b70f [Docs] Add GB300 cells and benchmarks for Qwen3.8-27B (#34863)
Co-authored-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
Co-authored-by: yhyang201 <yhyang201@gmail.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-08-14 15:42:50 +00:00
29c6be15a4 [Docs] Add Qwen3.8-27B cookbook page (#34860)
Co-authored-by: BBuf <1182563586@qq.com>
Co-authored-by: yhyang201 <yhyang201@gmail.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-08-14 15:23:39 +00:00
Xiaoyu Zhang 9c9a3273be [diffusion] Fix Helios denoising profiler stepping (#34826) 2026-08-14 23:21:54 +08:00
Xiaoyu Zhang 5f2a6d6422 [diffusion] Fix symbolic replicated-mode counting under torch.compile (#34824) 2026-08-14 23:20:29 +08:00
Yuan Luoandluoyuan.luo b95a746948 [MoE] Fuse swiglu moe up gemm epilogue (#32944)
Co-authored-by: luoyuan.luo <luoyuan.luo@antgroup.com>
2026-08-14 22:11:34 +08:00
Mohammad Miadh Angkad 9d2f1584fa Fix MiniMax-H3 Cache-DiT BCG warning (#34848) 2026-08-14 22:07:50 +08:00
c939307e8a [MiniMax-M3] Overlap shared and routed experts (#34542)
Co-authored-by: xuebi <xuebi@minimaxi.com>
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com>
2026-08-14 22:01:48 +08:00
1a178f7c7c [EPD] Batch embedding cache host-device range copies (#31574)
Co-authored-by: 晟海 <huangtingwei.htw@antgroup.com>
Co-authored-by: liusy58 <liusy58@linux.alibaba.com>
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com>
2026-08-14 22:01:12 +08:00
Brayden ZhongandBrayden Zhong 5e65dd01a7 Remove the torchao integration (--torchao-config) (#34304)
Co-authored-by: Brayden Zhong <brayden@radixark.ai>
2026-08-14 21:49:11 +08:00
amote-i fe0c18effd [NPU] [DOC] Add Qwen3.8-Max deployment tutorial on Ascend NPUs (#34836) 2026-08-14 20:02:04 +08:00
Ke Bao 18107e38d2 Skip oow slot freeing under eagle (#34823) 2026-08-14 17:13:21 +08:00
chilltongxandchilltongx 4d94f1d310 [diffusion] fix: warn when bcg disables cache-dit (#34242)
Co-authored-by: chilltongx <284668524+chilltongx@users.noreply.github.com>
2026-08-14 16:45:36 +08:00
Liangsheng Yin f2c84de022 [Perf] Publish the WAR read-done event at DSPARK verify (#34816) 2026-08-14 00:46:52 -07:00
triple-muandMick a86edcdc0a [diffusion] feat: rebuild minimax-h3 adaln outputs on demand (#34650)
Co-authored-by: Mick <mickjagger19@icloud.com>
2026-08-14 15:33:51 +08:00
2622e013eb [Fix] has_hf_quant_config crashes on local dirs without the config (#34774)
Co-authored-by: harmya <harmya@modal.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-08-14 07:17:42 +00:00
kk 65d62109dd [AMD] Fix Triton 3.7 gfx950 extend-attention spills (#34741) 2026-08-14 00:05:42 -07:00