 Lianmin Zheng
|
b0fd31ba07
|
Multiple flexibility fixes for DP attention (#33537)
|
2026-08-04 15:40:40 -07:00 |
|
|
Lianmin Zhengandcctry
|
34af3ff386
|
Allow optimistic prefill with L2 hierarchical cache and write-back policy (#33545)
Co-authored-by: cctry <cctry@meta.com>
|
2026-08-04 13:23:08 -07:00 |
|
|
Lianmin Zheng
|
4794b401d5
|
[Observability] Add startup, memory, and hybrid SWA diagnostics (#33375)
|
2026-08-04 12:50:09 -07:00 |
|
|
Lianmin ZhengandJialin Ouyang
|
5081c063c0
|
fix(metrics): clear forward occupancy on idle (#33562)
Co-authored-by: Jialin Ouyang <Jialin.Ouyang@gmail.com>
|
2026-08-04 12:49:17 -07:00 |
|
|
Lianmin ZhengandItai Gat
|
dea2be5ae3
|
[CUDA Graph] Allow custom decode graph runners (#33553)
Co-authored-by: Itai Gat <itaigat.mail@gmail.com>
|
2026-08-04 12:48:56 -07:00 |
|
|
Lianmin Zheng
|
7adf2f4a9a
|
Inline _set_gc into _set_envs_and_config (#33538)
|
2026-08-04 07:31:56 -07:00 |
|
|
Lianmin ZhengandYinghai Lu
|
d257b58e67
|
[Router] Report accelerator count in /v1/loads (#33548)
Co-authored-by: Yinghai Lu <yinghai@meta.com>
|
2026-08-04 05:55:35 -07:00 |
|
|
Lianmin Zheng
|
8f2a3ad6d7
|
[mem_cache] Label HiCache host pools and clarify post-capture KV sizing logs (#33445)
|
2026-08-04 04:21:36 -07:00 |
|
|
|
b57721ccf7
|
Enable post-capture KV sizing with DP attention (#33427)
Co-authored-by: cctry <cctry@meta.com>
Co-authored-by: cctry <cctry@fb.com>
|
2026-08-04 02:20:24 -07:00 |
|
|
Lianmin Zheng
|
16d3b118a2
|
Reduce startup log noise and fix Dynamo / CUDA-graph edge cases (#33428)
|
2026-08-04 02:19:56 -07:00 |
|
|
Lianmin Zheng
|
85484c457d
|
docs: refresh README news highlights (#33347)
|
2026-08-02 22:37:01 -07:00 |
|
|
Rain Jiang
|
4af8ddb576
|
support rust sglang server (#29799)
|
2026-07-31 11:56:31 -07:00 |
|
|
Rain Jiang
|
3312645a30
|
wire the rust server modules into lib, runtime, and tokenizer manager (#32877)
|
2026-07-30 12:46:19 -07:00 |
|
|
Rain Jiang
|
047635ee35
|
add the rust server native api handlers and runtime threads (#32876)
|
2026-07-30 12:46:19 -07:00 |
|
|
Rain Jiang
|
30643f88bc
|
add the rust server api frame codec and http server entry (#32875)
|
2026-07-30 12:46:19 -07:00 |
|
|
Rain Jiang
|
4facc0e18a
|
add the rust server ingress tests, guard, and submit modules (#32874)
|
2026-07-30 12:46:18 -07:00 |
|
|
Rain Jiang
|
e2c65af229
|
add the rust server ingress request validation and api server common types (#32873)
|
2026-07-30 12:46:17 -07:00 |
|
|
Rain Jiang
|
922d6e5542
|
add the rust server tokenizer, detokenizer, and egress modules (#32872)
|
2026-07-30 12:46:17 -07:00 |
|
|
Rain Jiang
|
35f2e6ab58
|
update Cargo.lock for the rust sglang-server dependencies (#32871)
|
2026-07-30 12:32:27 -07:00 |
|
|
Lianmin Zheng
|
16a52bff23
|
[Refactor] Move sampling tokenizer validation helper (#32694)
|
2026-07-28 16:48:03 -07:00 |
|
|
Lianmin Zheng
|
9ca4023b13
|
[Core] Clean up array-like msgspec structs (#32688)
|
2026-07-28 16:25:22 -07:00 |
|
|
Lianmin Zhengandcctry
|
c19a333944
|
[mm] Handle per-item embeddings in cache misses (#32498)
Co-authored-by: cctry <cctry@meta.com>
|
2026-07-27 17:16:17 -07:00 |
|
|
Lianmin Zheng
|
34454c06b8
|
[Refactor] Tidy server_args.py section grouping and drop unused alias (#32496)
|
2026-07-27 04:09:09 -07:00 |
|
|
Lianmin Zheng
|
3863612023
|
Add oulgen to CI_PERMISSIONS.json (#32453)
|
2026-07-26 15:04:11 -07:00 |
|
|
Lianmin ZhengandAlec S
|
9989077f24
|
Use native batched llguidance mask generation (#32412)
Co-authored-by: Alec S <10566873+alecsolder@users.noreply.github.com>
|
2026-07-25 16:36:32 -07:00 |
|
|
Lianmin ZhengandXingyu Liu
|
fae84ac0f9
|
Fix token count localization for replicated attention-TP forwards (#32411)
Co-authored-by: Xingyu Liu <38244988+charlotte12l@users.noreply.github.com>
|
2026-07-25 16:36:15 -07:00 |
|
|
Lianmin Zheng
|
5f330004bd
|
Fix flaky test_sampling_mask: mask length can legitimately be top_k + 1 (#32410)
|
2026-07-25 16:05:57 -07:00 |
|
|
Lianmin Zheng
|
d3cf4dfbaa
|
Update audio container test time estimate (#32408)
Signed-off-by: Lianmin Zheng <lianminzheng@gmail.com>
|
2026-07-25 14:45:08 -07:00 |
|
|
Lianmin Zheng
|
bd3f6a7935
|
[CI] Remove redundant Rust cache save-if settings (#32243)
|
2026-07-23 18:48:54 -07:00 |
|
|
Lianmin Zheng
|
429f6b6d15
|
Turn on breakable prefill cuda graph for dp attention by default (#31682)
|
2026-07-21 00:03:43 -07:00 |
|
|
Lianmin ZhengandJason Park
|
e7e8aaa73c
|
Overlap grammar (constrained decoding) with speculative decode verify (#31488)
Co-authored-by: Jason Park <jasonjk@fb.com>
|
2026-07-20 17:36:05 -07:00 |
|
|
Lianmin Zheng
|
54aaedd76d
|
Clean up prefill CUDA graph runner (#31654)
|
2026-07-20 12:14:21 -07:00 |
|
|
Lianmin Zheng
|
7d64858093
|
Add houseroad to CI_PERMISSIONS.json (#31717)
|
2026-07-19 15:08:36 -07:00 |
|
|
Lianmin Zhengandwangwenchen0407
|
555267ed05
|
Fix ROCm fused KV and KDA paths (#31688)
Co-authored-by: wangwenchen0407 <wangwenchen@meta.com>
|
2026-07-19 15:02:35 -07:00 |
|
|
Lianmin Zheng
|
c95026aed3
|
Upgrade llguidance to 1.7.6 (#31484)
|
2026-07-17 16:31:44 -07:00 |
|
|
|
4ae9cc3c81
|
Fix gate stride for 4D decode layouts (#31231)
Co-authored-by: lmzheng <lmzheng@fb.com>
Co-authored-by: michael604work <michael604@meta.com>
|
2026-07-14 20:06:50 -07:00 |
|
|
Lianmin Zheng
|
b4fdce3b63
|
Fix post-capture KV sizing for SWA pools (#31092)
|
2026-07-14 20:06:15 -07:00 |
|
|
Lianmin ZhengandLu Fang
|
22c08a9bee
|
Preserve RMSNorm shape in batch-invariant mode (#30956)
Co-authored-by: Lu Fang <30275821+houseroad@users.noreply.github.com>
|
2026-07-12 17:51:53 -07:00 |
|
|
Lianmin Zheng
|
b94ac87e0c
|
Enable breakable prefill CUDA graph for DP attention (#30898)
|
2026-07-12 17:10:04 -07:00 |
|
|
Lianmin ZhengandJialin Ouyang
|
539253e1d5
|
Gate Rust extension builds (#30927)
Co-authored-by: Jialin Ouyang <jialino@meta.com>
|
2026-07-12 05:44:58 -07:00 |
|
|
Lianmin ZhengandYongji Wu
|
81d273f73b
|
Handle coredump dirs and cache hit updates (#30897)
Co-authored-by: Yongji Wu <30348494+libertyeagle@users.noreply.github.com>
|
2026-07-12 00:37:43 -07:00 |
|
|
Lianmin ZhengandAbhishek Parmar
|
4fcc994be1
|
Support priority request header override (#30811)
Co-authored-by: Abhishek Parmar <abhishek@orng.net>
|
2026-07-10 14:36:43 -07:00 |
|
|
Lianmin Zheng
|
bc82b06400
|
Add ZYHowell to CI_PERMISSIONS.json (#30810)
|
2026-07-10 14:09:32 -07:00 |
|
|
Lianmin Zheng
|
7998fecfd1
|
[kv canary] Support UnifiedRadixCache in kv-canary and bracket nested model.forward (#30574)
|
2026-07-10 10:58:55 -07:00 |
|
|
Lianmin ZhengandYonghao Zhuang
|
3dc93a12ca
|
Improve EPLB dispatch handling and diagnostics (#30646)
Co-authored-by: Yonghao Zhuang <yhzhuang@meta.com>
|
2026-07-10 10:40:19 -07:00 |
|
|
Lianmin Zhengandtanujtiwari1998
|
32c8973ce8
|
Configurable decode retraction order (#30573)
Co-authored-by: tanujtiwari1998 <168470992+tanujtiwari1998@users.noreply.github.com>
|
2026-07-09 17:55:26 -07:00 |
|
|
Lianmin Zhengandtanujtiwari1998
|
cfc66e05c5
|
[tokenizer] Support pluggable tokenizer worker class in multi-tokenizer mode (#30630)
Co-authored-by: tanujtiwari1998 <168470992+tanujtiwari1998@users.noreply.github.com>
|
2026-07-09 17:31:54 -07:00 |
|
|
Lianmin Zheng
|
77b7698cad
|
Add tanujtiwari1998 to CI_PERMISSIONS.json (#30649)
|
2026-07-09 17:03:46 -07:00 |
|
|
Lianmin Zheng
|
a9e804623e
|
Allow EPLB manual test to use FlashInfer A2A (#30641)
|
2026-07-09 03:46:07 -07:00 |
|
|
Lianmin Zheng
|
ca8f15cd70
|
Fix FlashInfer A2A IMA by DP-synchronizing the decode graph bucket (#30242) (#30450)
|
2026-07-08 14:34:49 -07:00 |
|
|
Lianmin Zheng
|
24c42c90be
|
Clean up ServerArgs post-init dispatch (#30186)
|
2026-07-05 23:05:07 -07:00 |
|
|
Lianmin Zheng
|
9d00385b63
|
Cleanup: relocate temp_set_env and consolidate multi-device/CUDA helpers in common.py (#30180)
|
2026-07-05 18:47:26 -07:00 |
|
|
Lianmin Zheng
|
8673e85e6c
|
Remove # fmt: off from environ.py Envs class (#30153)
|
2026-07-05 12:45:24 -07:00 |
|
|
Lianmin Zheng
|
92a1f6e06c
|
[refactor] Reorder ServerArgs sections common-first; inline LLAMA4/MIMO_V2 arch tuples (#30151)
|
2026-07-05 12:17:24 -07:00 |
|
|
Lianmin Zheng
|
1a5977d41b
|
[chore] Add no-getattr rule; refine no-dataclasses rule (#29871)
|
2026-07-01 15:31:34 -07:00 |
|
|
Lianmin Zheng
|
76d828a4e8
|
Add pranjalssh to CI_PERMISSIONS.json (#29870)
|
2026-07-01 14:52:02 -07:00 |
|
|
Lianmin Zheng
|
bb74ed4a8d
|
Replace hasattr with isinstance in SHM feature helpers (#29549)
|
2026-06-28 21:31:28 -07:00 |
|
|
Lianmin Zheng
|
f76e707f59
|
Clean up follow-ups for eagle hidden dim clean up (#29546)
|
2026-06-28 21:30:53 -07:00 |
|
|
Lianmin Zheng
|
828411e6f1
|
Fix FlashInfer A2A dispatcher during CUDA graph capture (#29461)
|
2026-06-28 02:21:26 -07:00 |
|
|
Lianmin Zheng
|
6eedc8f376
|
Fix EAGLE draft hidden dim extraction and centralize spec helpers (#29464)
|
2026-06-27 21:48:26 -07:00 |
|
|
Lianmin ZhengandYinghai Lu
|
5747ed3b19
|
Fix DP-attention SHM feature finalization race (#29543)
Co-authored-by: Yinghai Lu <yinghai@meta.com>
|
2026-06-27 21:36:58 -07:00 |
|
|
Lianmin Zheng
|
e6cbc8f5fe
|
Fix SWA cache loc slicing for all attention backends (#29460)
|
2026-06-27 20:37:58 -07:00 |
|
|
Lianmin Zheng
|
09ca4fc96b
|
Skip FlashInfer FP8 autotune for MXFP8 quantized models (#29462)
|
2026-06-26 18:15:08 -07:00 |
|
|
Lianmin Zheng
|
9c3227b689
|
Fix IPv6 wildcard bootstrap address resolution in disagg (#29459)
|
2026-06-26 16:05:36 -07:00 |
|
|
|
267d165ad0
|
shm_broadcast: retry bind on EADDRINUSE (fix dp-attention port race) (#29455)
Co-authored-by: Xingyu Liu <38244988+charlotte12l@users.noreply.github.com>
Co-authored-by: xingyuliu <xingyuliu@fb.com>
|
2026-06-26 14:49:39 -07:00 |
|
|
Lianmin Zheng
|
118d6b2e5e
|
[Cleanup] Style and type annotation improvements extracted from #28688 (#29224)
|
2026-06-25 12:51:15 -07:00 |
|
|
Lianmin Zheng
|
6c839368e0
|
[Cleanup] IPC struct renames, better typing, and SenderWrapper removal (#29214)
|
2026-06-24 17:25:24 -07:00 |
|
|
Lianmin ZhengandYinghai Lu
|
7e63feee6f
|
Add scheduler metrics extension hooks (#29207)
Co-authored-by: Yinghai Lu <yinghai@meta.com>
|
2026-06-24 15:50:05 -07:00 |
|
|
Lianmin ZhengandRain Jiang
|
fd87a85388
|
Convert SamplingParams to msgspec Struct (#29198)
Co-authored-by: Rain Jiang <96632942+rainj-me@users.noreply.github.com>
|
2026-06-24 13:42:45 -07:00 |
|
|
Lianmin Zheng
|
d5c566e59b
|
Extract profile request cleanups (#29098)
|
2026-06-24 11:22:58 -07:00 |
|
|
Lianmin ZhengandCheng Wan
|
d6aacd2801
|
Handle input-embed-only batches in eager runner (#29121)
Co-authored-by: Cheng Wan <54331508+ch-wan@users.noreply.github.com>
|
2026-06-24 10:33:52 -07:00 |
|
|
Lianmin Zheng
|
34dd9c28ca
|
[Refactor] Introduce sock_send/sock_recv wrappers for zmq IPC (#29012)
|
2026-06-23 15:54:36 -07:00 |
|
|
Lianmin Zheng
|
ecab3f322e
|
Revert "Improve MFU metrics for prefill and verify timing" (#29079)
|
2026-06-23 15:46:22 -07:00 |
|
|
Lianmin ZhengandPranjal Shankhdhar
|
b60185c41c
|
Improve MFU metrics for prefill and verify timing (#29000)
Co-authored-by: Pranjal Shankhdhar <pranjal.ssh@gmail.com>
|
2026-06-23 12:26:56 -07:00 |
|
|
Lianmin Zheng
|
b28e990161
|
Migrate all ServerArgs fields to Annotated style, reduce add_cli_args by ~2400 lines (#28919)
|
2026-06-22 08:34:37 -07:00 |
|
|
Lianmin Zheng
|
ad9723af03
|
Clean up CUDA graph capture logs (#28937)
|
2026-06-22 06:15:26 -07:00 |
|
|
Lianmin Zheng
|
886b96621d
|
Migrate more server args to annotated style (#28830)
|
2026-06-21 20:50:17 -07:00 |
|
|
Lianmin Zhengandhnyls2002
|
a4d0ff3def
|
[misc] Make NaN-logit sanitization opt-in (default off) (#28829)
Co-authored-by: hnyls2002 <lsyincs@gmail.com>
|
2026-06-21 14:31:35 -07:00 |
|
|
Lianmin Zheng
|
7942d546d1
|
Revert "[Spec] Split init_backends; account draft weights in --mem-fraction-static" (#28841)
|
2026-06-21 07:52:35 -07:00 |
|
|
Lianmin Zheng
|
3975ea5ac7
|
Fix H20 torch import reinstall fallback (#28818)
|
2026-06-21 04:50:46 -07:00 |
|
|
Lianmin Zheng
|
c9488241e9
|
[Refactor] Auto-derive CLI args from dataclass fields to eliminate duplication (#28814)
|
2026-06-21 00:51:08 -07:00 |
|
|
Lianmin Zheng
|
d331fdd2ba
|
Add project rule: prefer msgspec.Struct over dataclasses (#28816)
|
2026-06-20 18:43:16 -07:00 |
|
|
Lianmin Zheng
|
54b9b9d0c9
|
Remove threading atexit monkey patch (#28812)
|
2026-06-20 18:30:50 -07:00 |
|
|
Lianmin Zheng
|
8a3d6c3403
|
Sort pyproject dependency lists (#28811)
|
2026-06-20 17:24:57 -07:00 |
|
|
Lianmin Zheng
|
95fb1ef697
|
[CI] Remove deprecated test/srt legacy CI setup (#28810)
|
2026-06-20 15:09:33 -07:00 |
|
|
Lianmin Zheng
|
fe428dd845
|
Clean up startup log noise (#28807)
|
2026-06-20 15:02:52 -07:00 |
|
|
Lianmin ZhengandYinghai Lu
|
45d203fb08
|
Fix tokenizer state cleanup on dispatch failure (#28694)
Co-authored-by: Yinghai Lu <yinghai@meta.com>
|
2026-06-19 21:55:39 -07:00 |
|
|
Lianmin Zheng
|
7cead0fb8f
|
Add JonnyKong to CI_PERMISSIONS.json (#28550)
|
2026-06-17 11:42:45 -07:00 |
|
|
Lianmin ZhengandIan O'Connell
|
7e629a2f8c
|
Allow overriding tokenizer path in benchmark harness (#28280)
Co-authored-by: Ian O'Connell <ianoc@meta.com>
|
2026-06-15 13:07:50 -07:00 |
|
|
Lianmin Zhengandgemini-code-assist[bot]
|
3b419f66da
|
[JIT] Track angle-bracket includes in source hash (#28273)
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
|
2026-06-15 02:58:14 -07:00 |
|
|
Lianmin Zheng
|
f18d38d040
|
Revert "[AMD][Quantization] Online MXFP4 quantization 2/N - FP8 to MXFP4 requantization on AMD GPUs" (#28213)
|
2026-06-14 13:34:06 -07:00 |
|
|
Lianmin Zheng
|
b001d3e815
|
Add prajjwal1 to CI_PERMISSIONS.json (#28120)
|
2026-06-12 20:26:38 -07:00 |
|
|
Lianmin Zheng
|
165331a200
|
Share BCG output buffers across capture sizes (#27659)
|
2026-06-09 20:33:46 -07:00 |
|
|
Lianmin Zheng
|
ca716f4734
|
Add TP server GPU process regression test (#27721)
|
2026-06-09 16:25:27 -07:00 |
|
|
Lianmin Zhengandlmzheng
|
eb8dceda44
|
Defer DeepGEMM PDL setup to worker init (#27671)
Co-authored-by: lmzheng <lmzheng@fb.com>
|
2026-06-09 13:52:30 -07:00 |
|
|
Lianmin Zheng
|
fca4ef9d69
|
Fix SWA pool resolution for EAGLE draft workers (#27491)
|
2026-06-08 11:00:29 -07:00 |
|
|
Lianmin Zheng
|
bcb5645629
|
Fix TRTLLM target verify query metadata (#27495)
|
2026-06-08 10:58:48 -07:00 |
|
|
Lianmin Zheng
|
a07d813ec8
|
Revert "Fix TRTLLM target verify query metadata (#27473)" (#27494)
|
2026-06-07 02:48:52 -07:00 |
|
|
Lianmin ZhengandMing Yang
|
db58e76c33
|
Add all_to_all_single to GroupCoordinator (#27492)
Co-authored-by: Ming Yang <minos.future@gmail.com>
|
2026-06-07 02:48:47 -07:00 |
|
|
Lianmin Zheng
|
5be0b0c8c0
|
Fix TRTLLM target verify query metadata (#27473)
|
2026-06-07 02:46:01 -07:00 |
|