Commit Graph
1478 Commits
Author SHA1 Message Date
fzyzcjy c56cee0f80 Support gated launch to defer startup memory allocation (#35927) 2026-08-24 20:19:45 +08:00
092d85eb87 [Feature] Add MiniCPM-SALA support (#30360)
Co-authored-by: Alex Nails <alex.nails@radixark.ai>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-24 02:25:16 -07:00
McZyWu c439e77872 [bugfix] [NPU] fix transpose batch matmul K*B exceed 65536. (#34715) 2026-08-24 15:22:59 +08:00
Xia Weiwen 5b5b29d4e2 [XPU] Use a fused GDN kernel from sgl-kernel for Qwen3.5 (#33354) 2026-08-24 12:07:50 +08:00
Xun Sunandsiyu a90d770c40 [Weight Cache] Support static DP/EP layouts (#33684)
Co-authored-by: siyu <liusy58@linux.alibaba.com>
2026-08-23 19:57:52 -07:00
Kalyan Kumarandvshekhawat-hlab 4c02584773 Add intel_xpu to DETERMINISTIC_ATTENTION_BACKEND_CHOICES (#29143)
Co-authored-by: vshekhawat-hlab <vshekhawat@habana.ai>
2026-08-24 09:47:34 +08:00
Cheng WanandClaude Opus 5 64aa859da2 config: constructing a config no longer resolves it (#35907)
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-08-23 01:18:53 -07:00
Cheng WanandClaude Opus 5 4bc79a1b49 config: project the config bags from the resolution result (#35906)
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-08-23 01:18:24 -07:00
Cheng WanandClaude Opus 5 0e22777572 config: record resolution writes in a declaration stash (#35905)
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-08-23 01:17:27 -07:00
Lianmin ZhengandYichao Fu 6fd0384d42 Make draft attention backends extensible (#35932)
Co-authored-by: Yichao Fu <yichaofu@meta.com>
2026-08-21 23:54:44 -07:00
Jianfei Wangandmiraclezqc af39ad9349 [Model] Complete dots.note.omni support with native encoders, video preprocessing, and MTP decoding (#33829)
Co-authored-by: miraclezqc <dysania@pku.edu.cn>
2026-08-22 14:19:14 +08:00
R0CKSTARandAlex Nails d90318b3e2 [MLX] Upgrade to Torch 2.13/MLX 0.32+ and redesign the Torch-MLX tensor bridge (#32984)
Co-authored-by: Alex Nails <alex.nails@radixark.ai>
2026-08-21 18:51:42 -07:00
Baizhou Zhang 3b5909de0e [DeepSeek V4] Add W4A4 MegaMoE server flag (#35918) 2026-08-21 18:44:18 -07:00
Baizhou Zhang 60ff1e33a5 [DeepSeek V4] Default FP4 checkpoints to FlashInfer MXFP4 MoE (#35919) 2026-08-21 16:48:58 -07:00
Yanbin Jiang 7d893255c3 [Spec][LoRA] Support multi-adapter LoRA with EAGLE/NEXTN/DFLASH/DSPARK speculative decoding (#34337) 2026-08-21 14:21:53 -07:00
Zhangheng 44806dc507 Using unified radix tree by default for all case (#35081) 2026-08-21 10:45:46 +08:00
cctryandcctry 32d98aad13 [HiCache] Allow a retraction host pool smaller than the device pool (#35543)
Co-authored-by: cctry <cctry@fb.com>
2026-08-19 22:59:57 -07:00
9db4ba8da1 [DeepSeek-V4] Add Q8KV8 sparse MLA prefill runtime backend (#32327)
Co-authored-by: Ho-Ren (Jack) Chuang <horenchuang@bytedance.com>
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com>
2026-08-20 10:23:49 +08:00
Liangsheng Yin 1270204d2c Revert "[Feature] Add DeepEPv2 (ElasticBuffer) MoE A2A backend" (#35568) 2026-08-19 14:14:38 -07:00
Po-Han Huang (NVIDIA) 6f69f927da [Scheduler] Add configurable decode interval after prefill (#35017) 2026-08-19 12:01:36 -07:00
MengYuandmenyu 4f8ecf6ae9 [Feature] Add DeepEPv2 (ElasticBuffer) MoE A2A backend (#29525)
Co-authored-by: menyu <menyu@nvidia.com>
2026-08-19 11:52:45 -07:00
Khoa PhamandCursor 0e4a09480c [HiCache] Support DCP with DSpark (#35221)
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-08-19 01:42:19 -07:00
8a1e6e4e46 Qwen3.8-27B Model Support (#34859)
Co-authored-by: Jimmy Shong <69131491+Jiminator@users.noreply.github.com>
Co-authored-by: Brayden Zhong <brayden.zhong@radixark.ai>
Co-authored-by: BBuf <1182563586@qq.com>
Co-authored-by: Zijie Xia <zijie.xia@radixark.ai>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Qiaolin Yu <liin1211@outlook.com>
2026-08-19 16:31:43 +08:00
Jimmy ShongandClaude Fable 5 c863760ae1 [Fix] DCP: advertise the logical KV-event block size (#35298)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-08-18 20:55:11 -07:00
Zhiqiang Xie 977412ae61 [HiCache] Buffer-only mode for HiCache host memory layer (#34798) 2026-08-18 19:21:24 -07:00
akhilg-nvandYangmin Li 5d12280ae7 Add fmha_v2 attention backend for SM90/120 (#23112)
Co-authored-by: Yangmin Li <yangminl@nvidia.com>
2026-08-18 18:41:03 -07:00
Xingyu Liuandxingyuliu 7dcaf11987 [Fix] Select custom all-reduce v2 by topology capability (#35061)
Co-authored-by: xingyuliu <xingyuliu@fb.com>
2026-08-18 10:30:10 -07:00
paulzhang-tm 0065fbfae1 [Scheduler] Cap prefill-delayer queue target by admission capacity (#35191) 2026-08-18 21:40:13 +08:00
Siyuan Chen fcdaaf8a5d [Feature] Optimize TP LMHead with All-to-All (#32313) 2026-08-17 19:55:27 -07:00
Baizhou Zhang bc312d185d Clean deprecated DeepSeek V4 Environs (#34926) 2026-08-17 16:07:00 -07:00
cctryandcctry 2e7c85da68 [PD] Preserve decode KV across retraction in HiCache (#34801)
Co-authored-by: cctry <cctry@fb.com>
2026-08-17 08:49:11 -07:00
Lianmin Zheng af743371cc Clean up environ.py: remove dead env vars, unify deprecation handling, move examples to a unit test (#35060) 2026-08-17 06:53:34 -07:00
b83d507cd7 [NPU] Support DeepSeek-V4 DSpark and refactor DSV4 cache management (#33676)
Co-authored-by: JiaruiChang5268 <jc5268@columbia.edu>
Co-authored-by: Kelon <kelonlu@163.com>
Co-authored-by: unknown <z8ruev42yk@gmail.com>
Co-authored-by: Talantan1102 <545811257@qq.com>
Co-authored-by: Talantan1102 <44429302+Talantan1102@users.noreply.github.com>
2026-08-17 16:27:44 +08:00
Wang, FangYuan eb61cb2823 [AMD] Support prefill context parallel two batch overlap for DeepSeek V4 (#33480) 2026-08-16 22:40:29 -07:00
b6d7602914 [CPU] Add support for Gemma4 on Xeon (#22498)
Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com>
Co-authored-by: jianan-gu <jianan.gu@intel.com>
Co-authored-by: Haotong Zou <haotong.zou@intel.com>
2026-08-17 10:52:26 +08:00
Yuwei AnandClaude Opus 5 a508d60295 [BCG][6/N] Allow prefill breakable CUDA graph for the Kimi archs (#34245)
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-16 15:57:14 -07:00
Lianmin Zheng b7eccd642f Increase post-capture decode memory reserve (#34996) 2026-08-16 15:31:36 -07:00
Lianmin Zheng f61f584347 Add explicit EPLB balancedness reporting modes (#34998) 2026-08-16 15:31:11 -07:00
gilfordtingandClaude Fable 5 4a6dc267e1 [Spec] Support mamba-radix-cache-strategy extra_buffer_lazy with DFLASH (#34763)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-08-15 16:36:59 -07:00
cctryandYilong Zhao e5b3a48751 Add --http2-max-concurrent-streams server arg (#34796)
Co-authored-by: Yilong Zhao <74357408+happierpig@users.noreply.github.com>
2026-08-15 10:34:49 -07:00
Mick 35cefd1c51 feat: add safeguards for remote media URLs (#34892) 2026-08-15 18:12:15 +08:00
Cheng Wan 61908870f6 config: spell out the one dynamic config read the census could not see
`_is_dsa_active` asked `getattr(server_args, "_is_dsa_model_arch", False)`, and
that name has never existed on `ServerArgs` -- it arrived as a placeholder with
the CP strategy abstractions (#27313), so the getattr default has always decided
the predicate. A dynamic read of a name nothing sets is the one shape the config
census cannot follow, and it looked like a live decision while being dead.

Spelled as the constant it evaluates to, with the placeholder written down: what
it should ask (whether this process runs a DSA model arch) is the CP path's
call, and its only consumer, `ContextParallelStrategy.per_layer_attn_cp_comm`,
has no readers yet.

That was the sole entry in the read ratchet's `_INERT_DYNAMIC_READS`, so the
exemption list is gone with it -- there is no way to exempt a read from the
baselines any more, which is the invariant worth having. The `counted()`
indirection it existed for goes too (verified the three shapes it guarded still
report: direct, `getattr`, and an attribute-parked alias).
2026-08-15 00:38:30 -07:00
Cheng Wan 97279980cf config: the last runner-side instance reads read the bags
Six reads were left on `self.server_args` outside the per-instance boundary the
plan reserves for the tokenizer-manager family, and each had a different reason
to be there:

- `scheduler.process_input_requests` (`mm_feature_transport`) and
  `BaseSpecWorker._build_hicache_draft_plan` (`enable_hierarchical_cache`) are
  plain leaves -> `get_mm()` / `get_memory()`.
- `DraftBackendFactory._create_backend` read the split backend through a
  *runtime-computed name* (`getattr(self.server_args, backend_name)`) and then
  fell back to the base field by hand -- the census's documented blind spot.
  The two names it can be handed are exactly the pair `attention_backends()`
  returns with that fallback already applied, so it reads the pair and indexes
  it. The draft runner's own stamp still wins when it has one.
- `remote_instance_weight_loader_use_transfer_engine` and
  `pre_capture_activation_reserve_mb` are derived members. Both are computed
  from published leaves only, so both get a named accessor that derives from
  the bags (and therefore follows a post-publish override).

The first of those two has all its inputs in one bag, so it follows the
established shape: one `*_of(cfg)` helper in `arg_groups/overrides.py`, the
`ServerArgs` member delegating to it, and the accessor calling it on
`get_model()`. `modelexpress_transport_of` splits out the JSON parse both
sides need. The second spans four bags plus the configured parallel sizes, so
it exists twice like the mamba pair -- and `TestDerivedPredicatesAgreeAcrossTiers`
now pins both new pairs equal over their input matrices (92 subtests).

`self.server_args.X` outside the tokenizer-manager family: 11 -> 5, and the
five that remain are the documented ones (the encode server's own record, the
nixl connector's rank arithmetic, `GrammarManager`'s handed instance).

The post-capture headroom path calls the same bag-backed
`pre_capture_activation_reserve_mb` accessor the configurator uses -- the
accessor advertises override-following, and a reserve that reads the record
while its sibling reads the bags can disagree after a post-publish override.
And the conversions' orphans go with them: `RemoteInstanceWeightTransporter`
kept a `server_args` field nothing reads, and `DraftBackendFactory` parked a
record it no longer consults -- both drop the parameter, and the four factory
call sites stop threading one.
2026-08-15 00:37:06 -07:00
Liangsheng Yin 5c9ee86d90 [CI] Move the static ratchets back to CPU unit tests (#34913) 2026-08-14 23:32:00 -07:00
ethcheandEthan Che aeee1562e6 [Kernel] Enable Helion backend for Kimi Delta-Attention (#32593)
Co-authored-by: Ethan Che <eche@meta.com>
2026-08-14 22:22:46 -07:00
Colin Z bc7e3ba66c [AMD][Quantization] Online MXFP4 quantization 4/N - NVFP4 to MXFP4 Online Requantization on AMD GPUs (#29328) 2026-08-14 21:59:39 -07:00
Jackey HuaandClaude Opus 5 1af761a09a [SM12x] Default the fused MHC post+pre path on (#34019)
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-08-14 19:51:21 +00:00
Brayden ZhongandBrayden Zhong 5e65dd01a7 Remove the torchao integration (--torchao-config) (#34304)
Co-authored-by: Brayden Zhong <brayden@radixark.ai>
2026-08-14 21:49:11 +08:00
Xinyuan Tongandhnyls2002 85cdf1178d [CI] Prune redundant CPU test overhead (#34309)
Co-authored-by: hnyls2002 <lsyincs@gmail.com>
2026-08-13 19:51:17 -07:00
Ziang Li 9d34c2809f [FlashInfer v0.6.16] Support FlashInfer CuTe DSL NVFP4 MoE quantization (#28354) 2026-08-13 17:33:46 -07:00