Cheng Wan
bebebb8f6c
config: retire the alias-form process-global config reads
...
`sa = get_server_args()` followed by `sa.field` reads the same startup record as
the direct form; the read ratchet added in the previous slice pinned twelve of
them as the remaining surface. Eleven now read the accessor for what they
actually want:
- `is_enable_moe_cp_allgather` compares the attention-CP and MoE-DP sizes to
decide whether a forward needs an allgather, so it reads the live topology
through `get_parallel()` — the same source `get_moe_cp_size()` right above it
already uses. Both groups exist once model-parallel init has run, which is
before any forward.
- The DeepSeek MLA decode-backend gate and Inkling's attention paths read
`get_exec().kernel`; Inkling's KV-dtype checks read `get_model()`. These are
per-runner fields, and the value they get is the config published for the
runner being built — unchanged from what the alias returned.
- The int8 mamba checkpoint pool reads `get_exec().mamba`. It keeps its guard
for callers that construct the pool with no published config; that guard now
catches the namespace accessor instead of the slot.
`model_loader`'s `moe_dp_size` stays on the instance and is exempt: the dict it
belongs to already reports the live size under `"dp"`, so that entry is the
configured intent, and `get_parallel()` shadows the name with the live value.
Alias-form baseline 12 -> 0. What remains on `get_server_args()` in the package
is the derived API (properties and methods computed from several fields plus the
HF config) and four config-intent reads of live-shadowed sizes, each exempt by
name with its reason.
2026-08-05 19:29:20 -07:00
Cheng Wan
1d47952c7c
config: pass the Ray placement group as a launch argument
...
`RayEngine.__init__` set the caller's `PlacementGroup` on `ServerArgs` as an
undeclared attribute, and every consumer read it back off the config object:
the non-DP launch path took `pg` and `is_custom_pg` from it, the DP path had to
re-attach the handle after `dataclasses.replace` dropped it, the DP controller
asked the config whether the group was the caller's, and the Ray HTTP launcher
had to clear the field so a stale handle could not leak in.
A live cluster object is not config. It now travels as a `placement_group`
argument on the two launch hooks — `Engine._launch_subprocesses` and
`Engine._launch_scheduler_processes` — defaulting to `None`, with `RayEngine`
holding the caller's group in `_placement_group` and the DP path passing
`is_custom_pg` down to the controller. The public API is unchanged:
`RayEngine(placement_group=pg, ...)` still works, `pg` still falls back to the
ambient group, and the HTTP launcher gets `None` from the default.
Writer ratchet 18 -> 15.
2026-08-05 19:28:39 -07:00
Baizhou Zhang
f9b954ddb1
[CI] Temporarily disable prefill cuda graph for qwen3.5 nightly test ( #33772 )
2026-08-05 18:52:57 -07:00
Pavan Sivaram Girijala
d9b1cba255
Fix DeepSeek-OCR batching crash on variable local-crop counts ( #33214 )
2026-08-05 18:39:19 -07:00
28848bfe7c
Fix Nightly NV CI ( #33564 )
...
Co-authored-by: Brayden Zhong <brayden@radixark.ai >
Co-authored-by: Baizhou Zhang <sobereddiezhang@gmail.com >
2026-08-05 18:06:47 -07:00
Xiaoyu Zhang and Claude Fable 5
4c0a8940fa
[Kernel] Unify BaseFusedOp and MultiPlatformOp dispatch ( #33205 )
...
Co-authored-by: Claude Fable 5 <noreply@anthropic.com >
2026-08-06 08:52:09 +08:00
Mohammad Miadh Angkad
65d5a0ec25
Support ModelOpt MXFP8 checkpoints ( #32538 )
2026-08-05 17:44:32 -07:00
Liangsheng Yin
3869fe556f
[CI] Collapse the EAGLE launch matrix and the scoring engine boots on the per-commit runners ( #33756 )
2026-08-05 16:43:19 -07:00
Baizhou Zhang
070fde72bf
[CI] Remove some unneeded CP tests ( #33763 )
2026-08-05 16:07:14 -07:00
Liangsheng Yin
c0ef548eef
[misc] Unify MLA scaling init and remove dead buffer / scaling code ( #33363 )
2026-08-05 15:59:33 -07:00
YAMY
990a446773
Fix padded positions in breakable CUDA Graph attention ( #33253 )
...
TestBreakableCUDAGraphBasic and all NVIDIA CI tests pass.
2026-08-05 15:53:28 -07:00
YAMY and Chunan Zeng
7bc90ab394
Select DeepGEMM standard layouts by memory budget ( #33474 )
...
Co-authored-by: Chunan Zeng <zcnrex@gmail.com >
2026-08-05 15:50:43 -07:00
Yanbin Jiang
b9d572ee02
[test] Re-enable a pruned Inkling LoRA unit-test set (68 -> 9 cases) ( #33752 )
2026-08-05 15:06:28 -07:00
cctry
a3a1ebc7b7
Warn on risky serving-time Triton work ( #33120 )
2026-08-05 14:55:03 -07:00
Brayden Zhong and Brayden Zhong
a14c870886
Fix broken Nemotron DP attention ( #33123 )
...
Co-authored-by: Brayden Zhong <brayden@radixark.ai >
2026-08-05 14:27:05 -07:00
Hank Han
9436de717f
[Spec][PD] Enable fused TopK for GLM-5.2 MTP IndexShare ( #31477 )
2026-08-05 14:17:42 -07:00
Zhiqiang Xie
106bcc1293
Observability enhancement for HiCache ( #32388 )
2026-08-05 14:13:06 -07:00
Jialin Ouyang
717a559f02
[Scheduler] Align WAR fences with CUDA graph metadata reads ( #33587 )
2026-08-05 13:52:20 -07:00
5c4f72f92a
[Build] Add srt_empty extra group for device-agnostic install ( #31300 )
...
Co-authored-by: yixiaodapeng <yixiaodapeng@example.com >
Co-authored-by: Alex Nails <alex.nails@radixark.ai >
2026-08-05 13:17:46 -07:00
Liangsheng Yin
de34dd11e9
[CI] Fold duplicate-server suites and prune the retract matrix on 1-gpu-5090 ( #33745 )
2026-08-05 12:41:51 -07:00
Jason Mancuso
36853b8ffc
[Spec] Support logprobs with DFlash ( #33459 )
2026-08-05 12:37:12 -07:00
Liangsheng Yin
1a045669e4
[CI] Merge tokenizer worker tests and drop redundant triton attention e2e ( #33641 )
2026-08-05 11:55:11 -07:00
Xiaoyu Zhang and Claude Fable 5
3425c93666
[diffusion] Wan VAE RMSNorm+SiLU fusion behind quality=high (H200 FastWan2.2 e2e 9.611 -> 9.125 s) ( #33546 )
...
Co-authored-by: Claude Fable 5 <noreply@anthropic.com >
2026-08-05 21:33:35 +08:00
silencejade
593777c046
[FIX] [benchmark] Fix flush_cache failure after warmup by waiting for server idle ( #33527 )
2026-08-05 21:27:43 +08:00
Mick
99709f734d
[VLM] split multimodal scheduling from mm_utils ( #32415 )
2026-08-05 20:24:12 +08:00
a6e5fa7081
[Scheduler] Honor explicit min-free-slots thresholds ( #33403 )
...
Co-authored-by: hnyls2002 <lsyincs@gmail.com >
Co-authored-by: Liangsheng Yin <hnyls2002@gmail.com >
2026-08-05 01:44:18 -07:00
Liangsheng Yin
c0d5ebd6c4
[CI] Move CPU-only unit tests to the CPU suite and trim dead 5090 registrations ( #33654 )
2026-08-05 01:43:29 -07:00
059269594c
[DSV4] Add official DSV4 reasoning effort support ( #33140 )
...
Co-authored-by: Xinyuan Tong <xinyuantong.cs@gmail.com >
Co-authored-by: David Orman <ormandj@corenode.com >
2026-08-05 12:50:41 +08:00
Liangsheng Yin
198a3bc29b
[Test] Route GEMM backend UTs through real layer modules and weight loaders ( #33615 )
2026-08-04 20:53:26 -07:00
a0b04dbe4c
feat(grpc): add generation request semantics ( #32588 )
...
Signed-off-by: Connor Carpenter <connorc@nvidia.com >
Co-authored-by: ishandhanani <82981111+ishandhanani@users.noreply.github.com >
Co-authored-by: Alex Nails <alex.nails@radixark.ai >
2026-08-04 18:53:45 -07:00
Zaili Wang and Ma Mingfei
29831d58ef
fix mm-chunk-embedding test suite ( #32895 )
...
Co-authored-by: Ma Mingfei <mingfei.ma@intel.com >
2026-08-05 09:49:39 +08:00
Qiaolin Yu
9303e26f03
[ci] add qwen 3.5 mtp + replayssm + flashinfer gdn test ( #33607 )
2026-08-04 18:08:42 -07:00
211ee64249
[rotary] Rebuild the shared RoPE cache entry when its buffers are dead ( #33575 )
...
Co-authored-by: mxz <mxz@fb.com >
Co-authored-by: Lianmin Zheng <lianminzheng@gmail.com >
2026-08-04 17:24:42 -07:00
Liangsheng Yin
76dc89f5aa
[Test] Replace NVFP4 MoE runner backend e2e matrix with a layer-level unit test ( #33611 )
2026-08-04 16:03:29 -07:00
Liangsheng Yin
0d99d91e49
[CI] Make B200 base-b suites single-GPU as prep for 1-gpu B200 runners ( #33605 )
2026-08-04 15:53:01 -07:00
Liangsheng Yin
a0b3f1dde6
[Test] Replace GEMM backend e2e matrices with layer-level unit tests ( #33596 )
2026-08-04 15:50:41 -07:00
Lianmin Zheng
b0fd31ba07
Multiple flexibility fixes for DP attention ( #33537 )
2026-08-04 15:40:40 -07:00
Baizhou Zhang
6808c6d571
[Tiny] Little enhancement of Kimi-K3 test ( #33609 )
2026-08-04 15:36:41 -07:00
Filip and Claude Opus 4.8
19d3f86895
[LoRA] Laguna: per-layer LoRA hidden-dim resolution for packed attention ( #30298 )
...
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com >
2026-08-04 14:48:05 -07:00
+26
abddb1c7e9
[Kimi] Support kimi-k3 ( #32541 )
...
Co-authored-by: DarkSharpness <76582120+DarkSharpness@users.noreply.github.com >
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com >
Co-authored-by: Mick <mickjagger19@icloud.com >
Co-authored-by: Yuhao Yang <47235274+yhyang201@users.noreply.github.com >
Co-authored-by: Cheng Wan <54331508+ch-wan@users.noreply.github.com >
Co-authored-by: Ke Bao <ispobaoke@gmail.com >
Co-authored-by: Baizhou Zhang <sobereddiezhang@gmail.com >
Co-authored-by: Chunan Zeng <zcnrex@gmail.com >
Co-authored-by: Khoa Pham <khoa.pham@radixark.ai >
Co-authored-by: Ziyi Xu <ziyi.xu@radixark.ai >
Co-authored-by: Zijie Xia <37504505+zijiexia@users.noreply.github.com >
Co-authored-by: Yuwei An <ayw.sirius19@gmail.com >
Co-authored-by: zhangxiaohao <1024393531@qq.com >
Co-authored-by: Yangmin Li <yangminl@nvidia.com >
Co-authored-by: Julien Lin <jullin@nvidia.com >
Co-authored-by: Hao Phan <htphan@nvidia.com >
Co-authored-by: Thomas Wang <1am9trash@gmail.com >
Co-authored-by: RolaoDenthu <xinyisong0111@gmail.com >
Co-authored-by: pigeonsoup <32922982+pigeonsoup@users.noreply.github.com >
Co-authored-by: HaiShaw <hixiao@gmail.com >
Co-authored-by: Xinyuan Tong <115166877+JustinTong0323@users.noreply.github.com >
Co-authored-by: Pranjal Shankhdhar <pranjal.ssh@gmail.com >
Co-authored-by: Lee Nau <lee.nau@gmail.com >
Co-authored-by: HMING <126185151+Hearum@users.noreply.github.com >
Co-authored-by: elvischenv <219235043+elvischenv@users.noreply.github.com >
Co-authored-by: Byron Hsu <byronhsu1230@gmail.com >
Co-authored-by: Byron Hsu <byron+per@periodiclabs.ai >
Co-authored-by: Claude Opus 5 <noreply@anthropic.com >
Co-authored-by: Thomas Wang <thomawan@amd.com >
Co-authored-by: Xinyi Song <86638975+RolaoDenthu@users.noreply.github.com >
Co-authored-by: Mohammad Miadh Angkad <176301910+mmangkad@users.noreply.github.com >
Co-authored-by: Cheng Wan <cheng.wan@radixark.ai >
Co-authored-by: BBuf <xiaoyu.zhang@radixark.ai >
Co-authored-by: Hanming Lu <hanminglu@meta.com >
Co-authored-by: Xinyi Song <xinyis10@illinois.edu >
2026-08-04 13:22:49 -07:00
Liangsheng Yin
0753663b8e
[CI] Trim redundant B200 test registrations ( #33586 )
2026-08-04 13:22:00 -07:00
Xingyu Liu
aa06433709
Avoid TRTLLM prefill output copy ( #33306 )
2026-08-04 12:54:04 -07:00
Lianmin Zheng
4794b401d5
[Observability] Add startup, memory, and hybrid SWA diagnostics ( #33375 )
2026-08-04 12:50:09 -07:00
Lianmin Zheng and Jialin Ouyang
5081c063c0
fix(metrics): clear forward occupancy on idle ( #33562 )
...
Co-authored-by: Jialin Ouyang <Jialin.Ouyang@gmail.com >
2026-08-04 12:49:17 -07:00
Lianmin Zheng and Itai Gat
dea2be5ae3
[CUDA Graph] Allow custom decode graph runners ( #33553 )
...
Co-authored-by: Itai Gat <itaigat.mail@gmail.com >
2026-08-04 12:48:56 -07:00
e76d0acdc9
migrate NPU PR/nightly test cases to a3-560T ( #33346 )
...
Co-authored-by: Claude <noreply@anthropic.com >
Co-authored-by: sglang-npu-bot <sglangnpu@163.com >
2026-08-05 01:17:58 +08:00
Xiaoyu Zhang and Claude Fable 5
95d0e57e83
[diffusion] Fuse DiT FFN tanh-GELU into up-proj GEMM (cublasLt epilogue) behind quality=high (Qwen-Image 1024^2 denoise 12.36 -> 12.05 s on H200) ( #33536 )
...
Co-authored-by: Claude Fable 5 <noreply@anthropic.com >
2026-08-04 23:49:43 +08:00
Xiaoyu Zhang
0d0c7d853f
[diffusion] FLUX.2 VAE decoder fast path behind quality=high (H200: 1024^2 97.6->29.2 ms, 2048^2 437.2->168.5 ms) ( #33451 )
2026-08-04 23:48:19 +08:00
Lianmin Zheng and Yinghai Lu
d257b58e67
[Router] Report accelerator count in /v1/loads ( #33548 )
...
Co-authored-by: Yinghai Lu <yinghai@meta.com >
2026-08-04 05:55:35 -07:00
Lianmin Zheng
8f2a3ad6d7
[mem_cache] Label HiCache host pools and clarify post-capture KV sizing logs ( #33445 )
2026-08-04 04:21:36 -07:00
Cheng Wan