100 Commits
Author SHA1 Message Date
Shangming Cai 1e894eb92d [CI] disaggregation: Run the SWA decode radix cache test on Mooncake (#46126) 2026-10-24 01:41:39 +08:00
Shangming Cai 9569a36561 [Feat] scheduler: Gate --prefill-decode-interval on running batch size (#44001) 2026-10-22 14:30:12 +08:00
Shangming Cai a6a6beffac [Refactor] disaggregation: Share the flag all-reduce and host-destination predicate in the decode transfer queue (#45641) 2026-10-22 13:42:55 +08:00
Shangming Cai a175e39b84 [CI] cuda: Rebuild workspace crates when the shared Cargo target dir came from another rust/ tree (#45557) 2026-10-21 19:32:22 +08:00
Shangming Cai 43a3f956cb [Fix] sgl-router: Wait for the detached PD prefill in routing tests (#45502) 2026-10-21 19:04:26 +08:00
Shangming Cai 7554be5a69 [CI] codeowners: Add owners for the top-level sgl-router (#45564) 2026-10-21 17:28:21 +08:00
Shangming Cai be65391abd [Feat] mooncake: Enable the failed-session probe by default with backoff and idle pruning (#45338) 2026-10-21 17:16:19 +08:00
Shangming Cai cd61db49c7 [Fix] sgl-router: Bump rustls-webpki and webpki-roots for emSign advisories (#45526) 2026-10-21 15:24:11 +08:00
Shangming Cai 574f9cdb2e [CI] slash-command: Add /run-cpu-ci to run only the CPU unit suite (#45501) 2026-10-21 13:59:16 +08:00
Shangming Cai 952c63b1bb [CI] sgl-router: Gate on live run-ci label so /tag-and-rerun-ci can start the tiers (#45383) 2026-10-20 21:33:42 +08:00
Shangming Cai 0025f93944 [Fix] sgl-router: Refuse prefill workers that cannot compute before bootstrap under --decode-dispatch-limit (#45120) 2026-10-19 18:24:17 +08:00
Shangming Cai 80914b7c9b [Refactor] sgl-router: Rename --max-in-flight to --max-inflight-requests (#44950) 2026-10-18 15:33:58 +08:00
Shangming CaiandShangming Cai 1ff2c1d71e [Fix] sgl-router: Fill the media fields when converting TextGenerate to Generate (#44951)
Co-authored-by: Shangming Cai <shangming.cai@radixark.ai>
2026-10-18 13:05:45 +08:00
Shangming Cai c7fb57d990 [Fix] sgl-router: Bump rustls to 0.23.46 for RUSTSEC-2026-0346..0349 (#44509) 2026-10-16 01:28:57 +08:00
Shangming CaiandClaude Fable 5.1 4195980f5a [PD] Poison the deferred abort ack when a Mooncake send fails (#43719)
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-12 15:33:09 +08:00
Shangming CaiandKevin Flansburg e1f43c9a61 [PD][HiCache] Fold the decode restore outcome into the MIN-reduced KV poll (#43562)
Co-authored-by: Kevin Flansburg <kflansburg@cloudflare.com>
2026-10-11 10:12:34 +08:00
Shangming Cai 1814ece57f [PP+PD] Harden PPConsensusStore: msgspec wire format, drop malformed messages (#43329) 2026-10-10 12:38:38 +08:00
Shangming Cai 9578cb1c63 [CI] Don't run the full GPU suite for renderer-only Cargo.lock changes (#43025) 2026-10-08 18:25:06 +08:00
Shangming Cai 1291f3175d [rust-processor] Fix chat-template precedence wording in the processor README (#42756) 2026-10-06 19:10:26 +08:00
Shangming CaiandClaude Fable 5.1 28525b43c4 [sgl-router] Document and reject a zero request timeout (#42751)
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-06 19:07:37 +08:00
Shangming CaiandClaude Opus 5.5 ab4f8a44f4 [sgl-router] Point the design doc at the shared engine ranking (#42572)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-05 18:16:42 +08:00
Shangming CaiandClaude Opus 5 148da8e77d [PD] Validate decode state layout once at registration (#42051)
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-10-04 18:32:30 +08:00
Shangming CaiandClaude Fable 5 a71c7d19e7 [Router] Document peer bootstrap: flags, RBAC, downward API, probe implications (#42049)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-10-02 02:20:36 +08:00
Shangming CaiandClaude Fable 5 c78fa09290 [Router] Drop the snapshot client's redirect-following fallback (#42010)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-10-01 14:31:53 +08:00
Shangming CaiandClaude Fable 5 047be72a8c [Router] Harden peer-bootstrap edges: refuse redirects and metric docs (#41999)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-10-01 14:15:42 +08:00
Shangming Cai 8c75ad9e62 chore: bump mooncake version to 0.3.13.post1 (#41678) 2026-09-30 15:16:54 +08:00
Shangming CaiandClaude Fable 5 2a1c477648 [PD] Defer decode KV release on every transfer failure, not only decode-initiated aborts (#41404)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-09-29 02:19:51 +08:00
Shangming CaiandClaude Fable 5 14ab74ef95 [PD] Fan drain abort ACKs out to every decode peer of the room (#41402)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-09-28 05:16:23 +08:00
Shangming CaiandClaude Fable 5 710a41ba57 [sgl-router] Book the input_ids forwarding outcome only for built bodies (#41342)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-09-26 19:39:01 +08:00
Shangming CaiandClaude Fable 5 77173ffb08 [HiCache] Demote SWA KV to host on write_back eviction instead of dropping it (#40712)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-09-24 23:56:17 +08:00
Shangming CaiandClaude Opus 5 cd11037009 [PD] Enable deferred decode-side KV release by default (#41023)
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-24 22:35:11 +08:00
Shangming Cai 6f4c2b9b91 [HiCache] Demote internal-node mamba states on write_back eviction (#40680) 2026-09-23 02:18:58 +08:00
Shangming Cai 861b11f087 [PD] Simplify late-abort quiescent ack branch to else (#40711) 2026-09-22 19:08:47 +08:00
630b1ef322 [sgl-router] Fix reorg admission proxy test build after BucketResolver::new (#40537)
Co-authored-by: Kan Wu <wukanustc@gmail.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-21 16:42:29 +08:00
Shangming Cai 70b5b03e78 [NPU][CI] Fix paths-filter negation that makes every PR run the NPU tier (#40549) 2026-09-21 16:07:27 +08:00
Shangming Cai 745de73ba3 Add CODEOWNERS entry for sglang-renderer (#40483) 2026-09-21 01:37:50 +08:00
Shangming CaiandClaude Opus 5 3f8eb35ead [PD] Do not admit intake-rejected requests to a PD handoff (#38935)
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 12:44:49 +08:00
Shangming CaiandClaude Opus 5 47a157f257 [CI][Disaggregation] Fix EADDRINUSE flake in test_disaggregation_dwdp_gpt_oss (#39611)
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-15 23:05:05 +08:00
Shangming Cai 4b186cfea5 [CI][PD] Skip the flaky decode HiCache file-backend disaggregation test (#39368) 2026-09-15 11:19:03 +08:00
Shangming CaiandXinyuan Tong 14a131ad5b [PD][OpenAI] Gate /v1/responses persistence behind --enable-response-store, default off (#39122)
Co-authored-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
2026-09-13 21:20:47 +08:00
Shangming CaiandClaude Fable 5 981b947568 [Fix] Seed raw tokenizer_path for smg-grpc-servicer in gRPC mode (#39105)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-09-12 14:41:15 +08:00
Shangming Cai 2c10f87991 [PD] Read nixl TransferInfo.is_dummy as a field in unit tests (#39100) 2026-09-11 23:20:02 +08:00
Shangming Cai a4ff5634b8 [CI] Add CI permissions for PP contributor stepinto (#39077) 2026-09-11 18:04:57 +08:00
97ba99067d Publish per-scheduler load on a dedicated socket for load-aware routers (#34608)
Co-authored-by: Kangyan Zhou <zky314343421@gmail.com>
Co-authored-by: Zhangheng <hzh0425@apache.org>
2026-08-27 19:33:18 +08:00
Shangming Cai c8b56b1f44 chore: bump mooncake version to 0.3.13 (#36493) 2026-08-26 22:21:15 +08:00
Shangming Cai edff717ef0 fix(disagg): snapshot affected rooms before iterating outside the lock (#36351) 2026-08-26 03:31:10 +08:00
Shangming Cai f464e77d17 fix(mini-lb): forward the flush_cache timeout param to workers (#36150) 2026-08-24 15:32:57 +08:00
Shangming Cai 852b04b358 fix(xpu): read enable_deterministic_inference from the config bag (#36149) 2026-08-24 13:56:41 +08:00
Shangming Cai 8270ac4621 refactor(disagg): move _is_watermark_ready into StagingManagerMixin (#36030) 2026-08-23 14:18:41 +08:00
Shangming Cai 849ce71976 refactor(disagg): dedupe mooncake failure_exception into a mixin (#36031) 2026-08-23 14:17:44 +08:00
Shangming Cai bd72190728 refactor(disagg): register SGLANG_ENCODER_MM_LOAD_WORKERS in Envs (#36006) 2026-08-23 12:47:38 +08:00
Shangming Cai cce0a1244b refactor(disagg): hoist staging helper imports out of the bootstrap loops (#35980) 2026-08-23 01:24:58 +08:00
Shangming Cai 3c69a4c744 fix(test): unbreak test_kv_transfer_replica_metric after #35950 (#35974) 2026-08-22 19:23:46 +08:00
Shangming Cai 5c03069d4b refactor(disagg): drop dead placeholder overrides in Common KV sender/receiver (#35950) 2026-08-22 15:45:18 +08:00
Shangming Cai 15a4398320 refactor(disagg): hoist duplicated _handle_staging_req into a mixin (#35948) 2026-08-22 15:35:55 +08:00
Shangming Cai ac179eec11 fix(disagg): PD transfer-failure injection was silently inert (#35890) 2026-08-22 11:51:16 +08:00
Shangming Cai 729a050ea3 refactor(disagg): extract _all_reduce_polls helper (#35886) 2026-08-22 02:30:33 +08:00
Shangming Cai a41da991c8 refactor(disagg): collapse duplicated branches in get_kv_class (#35847) 2026-08-21 22:11:31 +08:00
Shangming Cai 4f343abc13 refactor(disagg): remove unreferenced dead code (#35838) 2026-08-21 22:09:25 +08:00
Shangming CaiandClaude Opus 5 dad6fd0f04 refactor(disagg): remove dead get_embedding_port (#35844)
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-21 18:12:23 +08:00
Shangming Cai e7a37c8550 refactor(disagg): remove dead build_and_send_encode_request (#35843) 2026-08-21 17:58:35 +08:00
Shangming Cai 23cb04093c fix(multimodal): keep LLaVA image fetch off the CPU-preprocess timeout budget (flaky test_mixed_batch) (#35700) 2026-08-21 01:26:30 +08:00
Shangming Cai 628674a5c7 Remove unused MOONCAKE_COMPILE_ARG argument from Dockerfile (#35649) 2026-08-20 14:19:52 +08:00
Shangming Cai adca19c497 [PD] Deferred decode-side KV release for the NIXL backend (#35360) 2026-08-19 17:29:45 +08:00
Shangming CaiandClaude Opus 4.8 97dedd1ce9 [PD] Deferred decode-side KV release for aborts mid-transfer (#35049)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-08-18 23:56:33 +08:00
Shangming Cai 0772e79ee7 [CI][PD] Pin nccl rendezvous port per side to fix flaky disaggregation tests (#34755) 2026-08-14 01:44:20 +08:00
Shangming Cai f6ff5e8bb0 [PD] Handle abort requests in PP mode (#32797) 2026-07-30 14:39:44 +08:00
1d350aaad3 fix(reasoning): let --enable-strict-thinking works for DeepSeek-V4 (#32400)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
2026-07-27 18:15:47 +08:00
Shangming Cai bdd0698541 chore: bump mooncake version to 0.3.12.post1 (#32302) 2026-07-25 14:12:46 +08:00
Shangming Cai fa243fee40 [CI] Skip flaky test in CI for disaggregation group (#32324) 2026-07-24 21:59:20 +08:00
Shangming Cai 34d02bae47 Add CI permissions for Elastic EP contributor UNIDY2002 (#32306) 2026-07-24 17:20:15 +08:00
Shangming Cai 93926b9a95 Add Qwen3 MoE tests for PP compatibility with CP and DP (#29640)
Signed-off-by: Shangming Cai <csmthu@gmail.com>
2026-06-29 18:52:03 +08:00
Shangming Cai 0460f277b7 Fix manual chunked-prefill test to use req.fill_len after fill_ids refactor (#29032) 2026-06-23 17:52:26 +08:00
Shangming Cai a65c68c1fb Revert "[CI] Fix flaky optimistic test by adding contention handling" (#29023) 2026-06-23 16:53:32 +08:00
Shangming Cai 43b1fb95f6 [CI] Optimize nixl dependency installation script (#28950)
Signed-off-by: Shangming Cai <csmthu@gmail.com>
2026-06-23 01:58:22 +08:00
Shangming Cai ab21dc984a [CI] Fix flaky optimistic test by adding contention handling (#28947) 2026-06-23 00:35:13 +08:00
Shangming Cai 06e001347a [CI] Update nixl installation to include nixl-cu13 for h20 runner (#28930) 2026-06-22 20:35:14 +08:00
Shangming Cai b8e64fb56d [CI] Modify nixl installation to force reinstall (#28927) 2026-06-22 19:26:26 +08:00
Shangming Cai 378e66d248 [PD] Remove outdated backend whitelist for decode radix cache (#28238)
Signed-off-by: Shangming Cai <csmthu@gmail.com>
2026-06-15 22:36:05 +08:00
Shangming CaiandAbatom 6a012fbb2d [PD] Fix ZMQ stale socket reconnection in PD disaggregation (#27796)
Signed-off-by: Shangming Cai <csmthu@gmail.com>
Co-authored-by: Abatom <abzhonghua@gmail.com>
2026-06-11 19:51:12 +08:00
Shangming Cai 6394a8b381 [EPD] Dynamic encoder registration cleanup (#27542) 2026-06-08 19:08:29 +08:00
Shangming Cai 8ff0c9fef9 [PD] Downgrade propagated rank failure logs from error to debug (#27534)
Signed-off-by: Shangming Cai <csmthu@gmail.com>
2026-06-08 16:56:46 +08:00
Shangming Cai 57909f731b [PD] Fix KV cache corruption on abort by notifying ongoing prefill (#27372)
Signed-off-by: Shangming Cai <csmthu@gmail.com>
2026-06-06 00:56:13 +08:00
Shangming Cai f3acb6d4de [CI] Fix multimodal-gen path filter for shared trace code (#27222)
Signed-off-by: Shangming Cai <csmthu@gmail.com>
2026-06-04 14:20:27 +08:00
Shangming Cai afd2d0b2f4 [PP][Bugfix] Handle input_ids assignment in prepare_for_extend (#26883) 2026-06-01 14:43:50 +08:00
Shangming Cai fba083c80f [CI] Split PP tests into base and extra suites (#26648)
Signed-off-by: Shangming Cai <csmthu@gmail.com>
2026-05-29 12:15:27 +08:00
Shangming Cai c8c1aed5e9 [PD] Fix cross-rank queue divergence by gating metadata readiness before all-reduce (#26394)
Signed-off-by: Shangming Cai <csmthu@gmail.com>
2026-05-26 21:59:42 +08:00
Shangming Cai 2aa6995308 [CI] Enable EPD CI for EPD architecture enhancements (#26281) 2026-05-25 23:52:58 +08:00
Shangming Cai bc8d64bf36 [CI] Align score threshold in dsv4 disaggregation test (#26268) 2026-05-25 17:48:45 +08:00
Shangming Cai 7f2829af39 chore: bump mooncake version to 0.3.11.post1 (#25989) 2026-05-25 16:51:03 +08:00
Shangming Cai a241659d18 [PD] Consolidate shared logic into common backend (#25979)
Signed-off-by: Shangming Cai <csmthu@gmail.com>
2026-05-23 10:41:20 +08:00
Shangming Cai fbebdd5105 [CI] Enable nixl disaggregation test for decode radix cache (#25990) 2026-05-21 19:29:24 +08:00
Shangming Cai 1fbee74fb6 [PD] Clean early abort logic in PD module (#25677)
Signed-off-by: Shangming Cai <csmthu@gmail.com>
2026-05-20 11:17:42 +08:00
Shangming Cai 86c6c77f2f [Bugfix] Fix missing group arg in get dp buffer (#25585) 2026-05-19 02:08:10 +08:00
Shangming Cai f04c522534 [PD] Add conclude_state to fake KV backend (#25599)
Signed-off-by: Shangming Cai <csmthu@gmail.com>
2026-05-18 19:56:42 +08:00
Shangming CaiandHanHan009527 4984552cc9 Fix tests for decode radix cache (#25145)
Co-authored-by: HanHan009527 <hanhan7630@outlook.com>
2026-05-13 20:31:44 +08:00
Shangming Cai 5227b07669 [CI] Add DSV4 Flash disaggregation test (#24973)
Signed-off-by: Shangming Cai <csmthu@gmail.com>
2026-05-13 18:42:42 +08:00
Shangming Cai d102b0cbc3 Add CODEOWNERS for EPD module (#24942) 2026-05-11 14:42:48 +08:00
Shangming Cai e264b5785d [PD] Centralize per-room cleanup in common backend (#24601)
Signed-off-by: Shangming Cai <csmthu@gmail.com>
2026-05-07 18:47:55 +08:00
Shangming Cai 32d9998b9d [PD] Prevent update_status to Failed from cleared entries (#24539)
Signed-off-by: Shangming Cai <csmthu@gmail.com>
2026-05-06 23:32:04 +08:00