Commit Graph
10913 Commits
Author SHA1 Message Date
Mohammad Miadh Angkad ec741e4161 Fix DSpark loading for hybrid DSV4 NVFP4 (#33276) 2026-08-02 18:05:09 +00:00
Yuwei AnandClaude Opus 5 1a3bea77f2 [Feat] DCP + HiCache L2 Support (ported from kimi-k3) (#33112)
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-02 23:35:27 +08:00
f8e62a9224 [NPU] Add PR test cases (#32392)
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: sglang-npu-bot <sglangnpu@163.com>
2026-08-02 23:00:02 +08:00
70fe2e0dd5 [diffusion] model: support minimax-h3 (#33275)
Co-authored-by: zhenaozhenfu <zhenaozhenfu@minimaxi.com>
Co-authored-by: BBuf <1182563586@qq.com>
Co-authored-by: andyluo7 <andy.luo@amd.com>
Co-authored-by: Zijie Xia <zijie_xia@icloud.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: chao-xue <877184285@qq.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-08-02 22:32:37 +08:00
Liangsheng Yin 558c9bdcc2 [misc] Improve benchmark determinism and dataset API coverage (#33255) 2026-08-02 01:39:50 -07:00
Alison Shao 43be25b2b7 [CI] Graceful teardown for kv_canary and EAGLE spec fixtures (#32829) 2026-08-02 01:09:59 -07:00
8d106c3d79 [NPU] Enable automatic ascend_attn selection for vision attention and graph runners (#31948)
Co-authored-by: litao.dream <litao.dream@bytedance.com>
Co-authored-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
2026-08-02 15:09:51 +08:00
Tao Liandlitao.dream a0b7bcf592 [Feature] Support return_hidden_states="last" (#30177)
Co-authored-by: litao.dream <litao.dream@bytedance.com>
2026-08-02 15:09:33 +08:00
sonle5 1685d29f21 [AMD] Fix DeepSeek-V4 fused-RMS FP8 scale metadata on gfx950 (#31727) 2026-08-02 00:06:09 -07:00
Bingxu Chen 37be4e9247 [AMD] Speed up DSV4 MoE weight loading from mmap views (#32315) 2026-08-01 23:47:47 -07:00
Thomas Wang 7e509f690e [AMD] Fix DeepSeek-V4 FP4 MoE expert memory bloat (#31450)
AITER specific
2026-08-01 23:42:21 -07:00
056474cdb0 feat: Session-reference-aware Unified Radix Cache for agentic multi-turn workloads (#29173)
Co-authored-by: Ishan Dhanani <ishandhanani@gmail.com>
Co-authored-by: hzh0425 <hzh0425@apache.org>
Co-authored-by: ispobock <ispobaoke@gmail.com>
2026-08-02 13:43:55 +08:00
131bd51b01 [Spec] Add trtllm_mha support for Gemma 4 MTP draft attention backend (#25545)
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-01 16:16:04 -07:00
e2cf21b9e5 [Kimi K3] Add reasoning, tool-call, and OpenAI serving support (#33025)
Co-authored-by: hnyls2002 <lsyincs@gmail.com>
Co-authored-by: Liangsheng Yin <hnyls2002@gmail.com>
Co-authored-by: A-transformer <cl5743590921@gmail.com>
2026-08-01 14:57:23 -07:00
Zhangheng f1b41a5b3d [CP]: FIx some issue for glm5.2 cp v2 (#33100) 2026-08-01 13:50:10 -07:00
Brayden Zhong e0ba311026 Disable breakable CUDA graph for NemotronH (#33130) 2026-08-01 20:48:58 +00:00
Jackey HuaandClaude Opus 5 00a219f6c9 [Quant] Keep the flashinfer_deepgemm FP8 GEMM to 1 <= M < 32 (#32843)
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-08-02 00:36:13 +08:00
Cheng Wan 7071cfb873 runtime_context: per-role namespace enforcement behind SGLANG_ROLE_NAMESPACES (#33172)
publish(role=...) has recorded provenance since the namespace split; this
wires the enforcement the role was reserved for. SGLANG_ROLE_NAMESPACES
selects the mode:

- off (default): no bookkeeping; the mode gate in config_bag stays a
  dead-branch-prunable check under dynamo (bag reads run inside compiled
  forwards — pinned by a fullgraph test).
- record: audit mode — collect (role, namespace) pairs per process and
  persist each new pair immediately to SGLANG_ROLE_NAMESPACES_OUT (worker
  teardown skips atexit), plus a per-process stderr summary at exit.
- enforce: a bag read outside the role's ROLE_NAMESPACE_SETS entry fails
  closed with an actionable error; None entries mean full tree.

Sets are filled only where audits back them: dp_controller reads only
exec (record-mode plain + DP-attention smokes agree with the module's
static read set — the elastic-EP gate). tokenizer observed zero bag
reads (per-instance managers read self.server_args by design) but keeps
the full tree until the multi-tokenizer disagg shape (TokenizerWorker's
get_disagg read) is audited; encoder / expert_backup /
weight_cache_daemon likewise await their deployment shapes.

Verified end-to-end: DP-attention smoke under enforce boots and serves
with zero violations.
2026-08-01 08:59:49 -07:00
Cheng Wan 47d8b5b749 config: route parallel config-leaf reads through get_parallel() (#33170)
The parallel namespace joins the accessor migration: 106 config-leaf reads
(enable_dp_lm_head, enable_dp_attention, pp_async_batch_depth, dp_size,
ep_join_rank_offset, dwdp_size, ...) flip from get_server_args()/
self.server_args to get_parallel(), which serves config leaves from the
published parallel bag via __getattr__.

- ParallelContext.__getattr__ is restructured to stay dynamo-traceable
  (object.__getattribute__ graph-breaks): gate helpers such as
  enable_moe_dense_fully_dp() run inside compiled model forwards. A
  fullgraph regression test pins the pattern.
- The five live-shadowed topology sizes (tp/pp/dcp/attn_cp/moe_dp_size)
  keep their server_args reads: the live @property wins on the accessor,
  and conditionally-initialized groups would fail loud at unconditional
  call sites.
- Elastic-EP scale writers (ep_size/dp_size x4 in model_runner) reroute
  to get_context().override together with their remaining instance
  readers (expert_location gpus-per-node paths); the ServerArgs.override
  ratchet drops 39 -> 35.
- The expert placement helpers (compute_logical_to_rank_dispatch_
  physical_map, _compute_logical_to_all_physical_map,
  _prefer_same_node_experts) now read everything from the bags and drop
  their server_args parameter; their unit tests publish the config they
  need instead of stubbing it.
2026-08-01 08:58:39 -07:00
Cheng Wan df55e911d6 Fix the chunked-prefix-cache gate writing config the backends never read (#33168)
The load-time gate (maybe_disable_chunked_prefix_cache) wrote its
ServerArgs instance while every reader has moved to the published
config: the attention backends assert / branch on
get_schedule().disable_chunked_prefix_cache when they initialize, so the
flip never reached them and a backend outside
CHUNKED_PREFIX_CACHE_SUPPORTED_ATTENTION_BACKENDS kept chunked prefix
enabled.

Reroute the writer through get_context().override (which writes the
published bags) and flip the two remaining instance reads — the gate's
own log check and the prefill cuda-graph runner's capture flag — to the
bag. A regression test pins the three contracts: the gate lands on the
bag, the pristine ServerArgs instance stays untouched, and the
draft-worker guard never writes.

The ServerArgs.override call-site ratchet drops 39 -> 38.
2026-08-01 08:57:28 -07:00
JemmaFan 2fd78ec2d7 [AMD] Derive AITER verify tokens-per-req from input shape (#31221) 2026-08-01 00:29:25 -07:00
33ecf4bcd8 Add pr tests (#31952)
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: sglang-npu-bot <sglangnpu@163.com>
Co-authored-by: Cherry_ming <136634645@qq.com>
2026-08-01 15:03:21 +08:00
Sugar920 fd96a35fb0 add NPU GSM8K accuracy tests for 7 models (#32649) 2026-08-01 14:22:57 +08:00
Mohammad Miadh Angkadandhnyls2002 c0d06a6547 [CI] Fix runtime context setup in flat logprob tests (#33179)
Co-authored-by: hnyls2002 <lsyincs@gmail.com>
2026-07-31 22:46:24 -07:00
fb207b72b0 feat(kernels): port standalone Kimi K3 kernels (#32890)
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: hnyls2002 <lsyincs@gmail.com>
Co-authored-by: zhangxiaolei <zhangxiaolei.666@bytedance.com>
2026-08-01 13:26:47 +08:00
jacky.cheng 0d186f49be [AMD][Fix] Restore aiter-padded MoE weight dims for serialized checkpoints (#33090)
AMD only change
2026-07-31 22:09:37 -07:00
bae8eb8d6c [minimax-m3] fp8 attention GEMMs on SM100 (fp8_e4m3 KV + trtllm_mha) (#30971)
Co-authored-by: qiuyue <qiuyue@minimaxi.com>
Co-authored-by: xuebi <xuebi@minimaxi.com>
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com>
2026-08-01 09:39:52 +08:00
Andrew Tulloch e6a4cefc69 perf(startup): skip unused PyTorch headers for KV VMM allocator stub (#33126) 2026-07-31 18:32:36 -07:00
Cheng Wan 934a13ce3e [Inkling] Hold the short-conv per-step state on one metadata struct (#33116) 2026-07-31 18:11:35 -07:00
Sam Shleifer 58974ca16c [perf] Assemble flat prompt top logprobs scheduler-side as numpy arrays (#32223) 2026-07-31 18:00:51 -07:00
Liangsheng Yin ca07917c58 [Fix] Bound FULL_MASK verify-mask reuse by the captured max_bs (#33127) 2026-07-31 17:50:40 -07:00
Khoa Pham 1496bfee93 [Kimi] Support DCP + DSpark (ported from kimi-k3 branch) (#32828) 2026-07-31 17:39:00 -07:00
Yuwei AnandClaude Fable 5 3e0f7c3f30 [BCG][3/N] Enable bcg on dsa & deepep a2a backend (#31987)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-31 16:45:18 -07:00
Ethan (Yusheng) Su 3c5f115741 Split #32584 into 2/2: [LoRA] Shard attention LoRA by attn-TP and allow dynamic LoRA with dp attention (#32708) 2026-07-31 15:37:13 -07:00
Cheng Wan 55b6769b0e config: read resolved config via namespace accessors (#33013) 2026-07-31 15:06:59 -07:00
Cheng Wan 4862edc85f runtime_context: record the publishing process role (#33012) 2026-07-31 15:06:22 -07:00
Cheng Wan ab2553284a config: preserve resolved config across nested publishes + mutation ratchets (#33011) 2026-07-31 15:05:41 -07:00
Xinyuan Tong 4480e2a051 [Fix] Repair verify mask test fixture (#33087) 2026-07-31 14:48:30 -07:00
Ethan (Yusheng) Su 7e996a5d0d Split #32584 into 1/2: [LoRA] Guard DP-attention idle forwards against stale LoRA batch state (#32707) 2026-07-31 14:23:01 -07:00
Po-Han Huang (NVIDIA) 5df193b4ac [Speculative Decoding] Fix GPT-OSS EAGLE3 hidden states (#32334) 2026-07-31 11:58:49 -07:00
Rain Jiang 4af8ddb576 support rust sglang server (#29799) 2026-07-31 11:56:31 -07:00
Cheng Wan 77c77a3da8 feat(inkling): migrate short convs onto the ShortConv attention backend (#33023) 2026-07-31 11:52:12 -07:00
Cheng Wan d3222bcc3a [unified-memory] Support fa3, the default MLA backend on pre-Blackwell hosts (#33046) 2026-07-31 11:46:46 -07:00
luchangli 26486a957d Fix --hicache-size allocating ~2x host memory on hybrid Mamba (#32915) 2026-08-01 02:37:57 +08:00
Nan Jiang 89f4a80c1f Support fastsafetensors no-GDS loading and page-cache release (#31859) 2026-07-31 23:12:32 +08:00
Danila ShtanandDanila Shtan 5f9b0db18c Fix async loading of RunAI-streamed tensors (#32896)
Co-authored-by: Danila Shtan <dan@nebius.com>
2026-07-31 21:46:33 +08:00
luoroger37andXinyuan Tong 690de097c4 [fix]reject media input for text-only models (#32914)
Co-authored-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
2026-07-31 12:43:25 +00:00
Peng Wu e3d4f48e55 [Fix] missing max_context_len on HybridAttnBackend (#32690) 2026-07-31 19:43:09 +08:00
MickandClaude Fable 5 754b692afc [diffusion] optimization: support cuda-ipc zero-staging all-to-all for 2-rank Ulysses (#31854)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-31 19:35:48 +08:00
Khoa PhamandYangmin Li 2573190b93 feat: support Kimi Linear PD disaggregation with DCP (#32837)
Co-authored-by: Yangmin Li <yangminl@nvidia.com>
2026-07-31 02:14:09 -07:00