Commit Graph
131 Commits
Author SHA1 Message Date
Cheng Wan 701213c867 [Refactor] Retire the layer facade and simplify boundary internals (#41554) 2026-09-28 20:17:33 -07:00
YAMY f209e67767 [Spec] Model-agnostic last-stage draft embedding under pipeline parallelism (#39643) 2026-09-28 10:22:40 -07:00
Rain Jiang 6c4f36e47a Rust server unify datapath for mm and generate requests (#39679) 2026-09-27 23:36:07 -07:00
kk 515f5be77e [AMD] Integrate Aiter MegaMoEv2 for DeepSeek-V4 (#35619) 2026-09-25 00:47:08 -07:00
cctryandcctry efff836911 [PD] Add decode host receive for custom transfer backends (#40238)
Co-authored-by: cctry <cctry@fb.com>
2026-09-23 21:57:33 -07:00
inkcherryandDuyi-Wang 58f622ebc4 Reduce decode bootstrap latency with request-owned speculative KV (#38978)
Signed-off-by: inkcherry <mingzhi.liu@amd.com>
Co-authored-by: Duyi-Wang <duyi.wang@amd.com>
2026-09-23 02:24:53 -07:00
Liangsheng Yin 78980a3b0b [misc] Remove deprecated endpoints, env vars and aliases past two releases (#40795) 2026-09-22 18:14:45 -07:00
jain-ria 01275aab6b fix(grpc): expose native response timeout as a server argument (#40644)
Signed-off-by: jain-ria <riajain@NVIDIA.com>
2026-09-22 14:57:38 -07:00
YAMYandzhangxiaolei 6fd98c98b9 [Qwen3.8-Next] Pipeline-parallel serving and PD-prefill MTP for Qwen4-Exp (#40501)
Co-authored-by: zhangxiaolei <zhangxiaolei.666@bytedance.com>
2026-09-22 08:46:44 -07:00
Xueshen Liu ab03a8e7eb [Perf] Fork-safe import: no CUDA context at import time, lighter argument parsing (#40201) 2026-09-21 10:48:35 +08:00
Qiaolin Yu f4c256354c [kimi k3][pd disagg] support pp prefill + dcp decode with dspark (#40045) 2026-09-20 00:15:28 -07:00
huangtingweiandZhangheng e9300f643e [Unified Cache][9/N] add opt-in MLA load deduplication for Mooncake Linker (#39565)
Co-authored-by: Zhangheng <hzh0425@apache.org>
2026-09-20 09:56:56 +08:00
YAMY 9cc7da2ab0 [MegaMoE] Wire Qwen MoE blocks to DeepGEMM MegaMoE (MXFP4 and NVFP4 experts) (#38080) 2026-09-19 16:00:38 -07:00
3a64faa1f2 Fix disagg PP MTP for GLM-5.2 (#39378)
Co-authored-by: Julien Lin <jullin@nvidia.com>
Co-authored-by: YAMY1234 <74099316+YAMY1234@users.noreply.github.com>
Co-authored-by: Yangmin Li <yangminl@nvidia.com>
2026-09-19 13:55:07 -07:00
Khoa PhamandCursor 10b0bcfd18 [PD] Allow decode radix cache and HiCache L1/L2 with DCP (#40263)
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-09-18 17:54:35 -07:00
Zhiqiang Xieandcctry ceb1d2e580 [PD] Enable optimistic prefill with buffer-only L3 write-through HiCache (#40043)
Co-authored-by: cctry <csycfl@gmail.com>
2026-09-18 16:04:15 -07:00
Sam (Kesen Li) d346b214fb feat(kv-cache): support SM100 NVFP4 GenMHA and speculative decoding (#36340) 2026-09-18 14:50:46 -07:00
Siyuan ChenandYuwei An c9a8fba991 [DSV4][BCG] Optimize the heavy memory use of C4 Indexer when BCG is enabled (#36534)
Co-authored-by: Yuwei An <ayw.sirius19@gmail.com>
2026-09-15 17:23:11 -07:00
Cheng Wan 406c9c71d8 [Feature] Support BF16 and batch-invariant inference with DeepEP v2 (#38160) 2026-09-15 14:30:50 -07:00
Lianmin Zheng a2b4e8888f Allow CUDA VMM feature transport with the Rust frontend (#39347) 2026-09-14 17:09:49 -07:00
Cheng Wan 6804eeaabe config: an out-of-tree replacement point for every resolution-pipeline step (#39134) 2026-09-12 17:26:54 -07:00
Xiaoyu ZhangandCursor 0a57403468 [Perf] Optimize Qwen3-VL unique-image serving on H100 (#36411)
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-09-12 15:16:59 +08:00
Khoa PhamandClaude Fable 5.1 6ba96d329f [DCP] Resolve --dcp-comm-backend to fi_a2a/a2a by default for every model (#39165)
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-11 23:48:42 -07:00
Hanming Lu 45715e7f20 [Fix] Disable NCCL graph buffer registration for the TP LM-head all-to-all (pure-DP decode hang under request bursts) (#38936) 2026-09-11 13:10:48 -07:00
fd7743e0e1 [Sampling] Support sampling masks with overlap scheduling (#36631)
Co-authored-by: ByronHsu <ByronHsu@users.noreply.github.com>
Co-authored-by: root <root@slurm-h200-208-179.slurm-compute.tenant-slurm.svc.cluster.local>
Co-authored-by: Byron Hsu <byron+per@periodiclabs.ai>
2026-09-10 14:52:44 -07:00
1b77f498a0 [NVIDIA] Support flashinfer Mega Moe (#31470)
Co-authored-by: djns99 <40156487+djns99@users.noreply.github.com>
Co-authored-by: 云挚 <ningyunxiao.nyx@antgroup.com>
Co-authored-by: Yangmin Li <yangminl@nvidia.com>
Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com>
2026-09-10 00:22:47 -07:00
Cheng Wan f1a512c51c [Config] msgspec.Struct for the config tier (#38753) 2026-09-09 19:41:19 -07:00
Cheng Wan 53dc77ff4e [Config] One writer for the declaration stash; no exception to the write seal (#38752) 2026-09-09 19:22:06 -07:00
Cheng Wan db272201a2 [Config] Retire get_global_server_args, and clear the deprecated flags that have a replacement (#38375) 2026-09-08 16:42:12 -07:00
+3 52fecfdf09 support qwen 3.8 flash next (#37500)
Co-authored-by: ch-wan <54331508+ch-wan@users.noreply.github.com>
Co-authored-by: ispobock <26454835+ispobock@users.noreply.github.com>
Co-authored-by: JustinTong0323 <115166877+JustinTong0323@users.noreply.github.com>
Co-authored-by: samuellees <26428561+samuellees@users.noreply.github.com>
Co-authored-by: YAMY1234 <74099316+YAMY1234@users.noreply.github.com>
Co-authored-by: yhyang201 <47235274+yhyang201@users.noreply.github.com>
Co-authored-by: yizhang2077 <25844240+yizhang2077@users.noreply.github.com>
Co-authored-by: zijiexia <37504505+zijiexia@users.noreply.github.com>
Co-authored-by: Shinto C V <cshintov@gmail.com>
Co-authored-by: Julian Huang <huangzhilin.hzl@antgroup.com>
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com>
Co-authored-by: yhyang201 <yhyang201@gmail.com>
2026-09-08 13:56:21 -07:00
Baizhou Zhang 85d39401c8 [CP V1 Deprecation 3.5/5] Deprecate HIP/NPU/MUSA prefill CP and remove legacy implementation (#38293) 2026-09-07 15:39:34 -07:00
Baizhou Zhang b6c31b155c [CP V1 Deprecation 3/5] Remove generic prefill CP v1 runtime (#36228) 2026-09-06 21:53:53 -07:00
Cheng Wan 98f69ccbf3 [Config] Round 6.3: the record remembers how it was asked for, and is sealed while resolution runs (#38048)
Third of four; stacked on #38047. Two small changes, both about the same thing:
the record holds the operator's input, and nothing else should be true of it.

## `/server_info` can answer what was actually typed

It reports `resolved_dict()` -- what resolution decided. There was no way to ask
the other question, and the two are not derivable from each other: a field
nobody set reads the same as one set to the value resolution would have picked
anyway.

The launcher stores the arguments it parsed and the in-process `Engine` stores
the call that built the record. All three readbacks report it beside the
resolved values, so both surfaces come back in one request: HTTP `/server_info`,
`Engine.get_server_info`, and the gRPC bridge's -- the last one builds from
`resolved_dict()` and would otherwise have been the one surface of the three
that answers only "what resolution decided".
It rides on the record rather than in a field -- it describes how the
configuration was asked for, so it is not part of the configuration: no CLI
flag, no namespace, not in the bags. Being on the record is what lets a
subprocess copy answer the same question the launcher can, and
`replace_resolved` carries it because a copy was launched by whatever launched
its parent.

The crash dump already collected all four surfaces (`server_args`,
`config_updates`, `resolved_config`, `launch_command`); this is the one that
`/server_info` was missing.

## The record is sealed for the length of resolution

The read-only guard armed on `_resolution_finished`, so for the whole run of the
pipeline nothing stopped a resolver from assigning a field. Nothing in `srt/`
does -- 0 assignments statically, and 0 writes observed across the launch-shape
matrix with a watching `__setattr__` -- but that was a convention, and the
defect it permits is invisible: a value a resolver wrote onto the record is
indistinguishable from a value the operator typed, which is the one distinction
the record exists to preserve.

It now arms when resolution starts. A resolver that assigns a field fails at
boot with a message naming `declare_resolution`, which is where the decision
belongs: the stash carries a source and leaves the input intact.

`declare_direct_writes` asks for the seal by name through `record_writable`. It
hands the record to an out-of-tree platform plugin that sets fields on it; those
implementations cannot be converted by editing a resolver here, so the write
stays and the diff is captured into the stash afterwards. Naming the exception
is the point -- an in-tree resolver reaching for it is doing something it should
be declaring.

## Verification

Costs nothing: the 211 test-side assignments all happen before `resolve_once`,
which a post-resolution write already refused. A full registered-unit sweep
(648 files) against the stack's merge-base: 19 failures on both sides, the same
19, none of them config. Driving a
deliberate write into a real handler produces the new error, so the seal is
tested by more than its own unit test.
2026-09-06 21:41:14 -07:00
Cheng Wan 45c24444b1 [Config] Round 6.1: "unset" gets its own spelling, and the declaration says what it means (#38046)
First of five. The stack continues a series that moved configuration out of
`ServerArgs` and into the runtime context's namespace bags. This one fixes
something that was actually broken, and gives the fix its other half.

## "Unset" gets its own spelling on two ratio fields

`swa_full_tokens_ratio` and `mamba_full_memory_ratio` carried real values as
their class defaults (0.8, 0.9), so a model family with an opinion had to ask
"is this field still equal to the class default?" to find out whether the
operator had set it. That question has two wrong answers: it says "the operator
set it" as soon as any earlier pass declares the field, and it says "the
operator did not set it" when the operator types the default value.

Both become `Optional[float] = None`. The record carries what the operator typed
and nothing else, and the family test becomes `is None`.

`mamba_radix_cache_strategy` keeps `"auto"`: unlike the ratios it already has a
spelling for "unset" that an operator can type and that means exactly that --
only its comparison changes, from the class default to the token itself, which
is the fix the comment at that site already prescribed. With that, neither
family module imports `ServerArgs` any more.

## And the declaration says what the field means when nobody answers

Making the default `None` leaves a hole: something has to supply the generic
value. `Arg(fallback=...)` supplies it from the declaration.

```python
swa_full_tokens_ratio: A[
    Optional[float],
    Arg(help="...", resolvable=True, fallback=0.8),
    NS("schedule"),
] = None
```

The dataclass default stays `None`. A fallback is not a default: the record is
the wire format, and a child process has to keep being able to tell "unset" from
"set to the value resolution would have picked anyway".

### Which surface it lives on is the whole design

Precedence becomes **override -> decision -> input -> fallback**, applied in
`resolution_result` -- which the projection, `/server_info` and every config bag
read through.

Deliberately **not** in `resolving_view` / `resolved_view`. Those are the
decision-over-input surface a pass reads *while it is deciding*, and two model
families branch on exactly this:

```python
# model_overrides/inkling.py, and the same shape in deepseek_v4.py
if cfg.swa_full_tokens_ratio is None:
    overrides["swa_full_tokens_ratio"] = 0.1
```

A fallback answering there is not "the generic value, later" -- a `__getattr__`
layer is read-time, so there is no later. Every read during resolution would
already get 0.8 and the branch would never fire. Running `_inkling_overrides`
against both versions:

```
--- fallback on the effective surface only (this PR) ---
  cfg.swa_full_tokens_ratio during resolution = None
  family declared swa = 0.1   mamba = 0.1
--- fallback also on the view a pass reads ---
  cfg.swa_full_tokens_ratio during resolution = 0.8
  family declared swa = None  mamba = None      <- the key never lands
```

So "resolution first, then the fallback" holds -- not because a step is appended
to the pipeline, but because of which surface the value lives on. Exactly one
reader consults the effective surface during resolution: the range check on the
ratio, which wants the value the pools will be sized against. It asks
`resolution_result` directly -- what its comment already claimed it was doing --
and it runs after the model families.

### The alternative, and why not

A pass that fills the field in when nothing claimed it needs a slot (after the
families, or it beats them), a second call site (the dummy-model short circuit
returns long before that slot), an idempotence requirement so the second call is
harmless, and the value written twice -- once as a literal, once as prose in the
help (`"Unset means 0.8"`). An earlier revision of this series did exactly that
and deleted it four PRs later. A declaration needs none of it, and `pipeline.py`
is untouched by the whole series as a result.

### What may be declared this way, and what may not

Across every hook, `if x is None: x = ...` appears at **55 sites over 29
fields**. They are not one thing:

| | count | examples | declarable |
|---|---|---|---|
| unconditional constant | 5 | the two ratios, `grammar_backend="xgrammar"`, `mm_process_config={}`, `custom_weight_loader=[]` | **yes** |
| unconditional, computed from another field | 4 | `tokenizer_path=model_path`, `device=get_device()`, `served_model_name`, `speculative_draft_model_quantization` | needs a `fallback="dotted.path"` form; not here |
| **conditional decision** | ~20 | `chunked_prefill_size` across seven memory tiers, `max_bs` across eight, `max_running_requests` at 48 or 256 by model family | **no, and it should not be** |

Only a value fixed for the life of the configuration belongs in a declaration.
One that depends on the machine, on another field, or on anything impure
(`random_seed = random.randint(...)`) is a decision, and decisions stay in a hook
where their order is visible. This PR converts the two ratios only.

## Verification

- `resolve_once` ends with the same effective values: the resolution result is
  identical across 24 launch shapes x 489 fields except for the two intended
  ratio changes. Separately, 16 launch shapes resolved on both sides, real model
  and dummy: 7,904 field readings, and the only difference is `random_seed`, a
  fresh `random.randint` per process.
- The CLI registers the same 507 options with the same choices and actions; only
  the two defaults move.
- `test_declared_fallbacks.py`, 17 cases. One pins the inverse of the dead branch
  above: what a pass sees while deciding is still `None`.
- The whole series was swept over all 648 registered unit-test files against its
  merge-base: 19 failures on both sides, the same 19, none of them config.


---
### CI States

Latest PR Test (Base): <!-- slot:pr-test:start -->❌ [Run #34083705463](https://github.com/sgl-project/sglang/actions/runs/34083705463)<!-- slot:pr-test:end -->
Latest PR Test (Extra): <!-- slot:pr-test-extra:start -->❌ [Run #34083705284](https://github.com/sgl-project/sglang/actions/runs/34083705284)<!-- slot:pr-test-extra:end -->
Latest PR Test (AMD ROCm 7.2): <!-- slot:pr-test-amd-rocm720:start -->❌ [Run #34083705383](https://github.com/sgl-project/sglang/actions/runs/34083705383)<!-- slot:pr-test-amd-rocm720:end -->
<!-- pr-states:end -->
2026-09-06 21:38:29 -07:00
sglang-botandsglang-bot 6252993afe chore: update CI test est_time values (#38238)
Co-authored-by: sglang-bot <sglang-bot@users.noreply.github.com>
2026-09-06 17:49:41 -07:00
55bf3380e0 Support Hy4-preview (#36805)
Co-authored-by: BBuf <1182563586@qq.com>
Co-authored-by: alphabetc1 <2508695655@qq.com>
2026-09-04 18:03:49 -07:00
Baizhou Zhang ff1285cc28 [CP V1 Deprecation 2/5] Make strategy prefill CP canonical (#36223) 2026-09-03 20:24:19 -07:00
junduandMa Mingfei 8770c1db1f [CPU][CI]: rename Xeon CPU CI suites to stage-*-intel (#37395)
Co-authored-by: Ma Mingfei <mingfei.ma@intel.com>
2026-09-04 10:08:41 +08:00
Baizhou Zhang 07c8f7294d Bump sgl-deep-gemm to 0.1.7 (#37279) 2026-08-31 17:16:34 -07:00
9cf157c252 [Radix Cache] Add Rust TreeCore backend with shared parity tests (#32710)
Co-authored-by: alphabetc1 <2508695655@qq.com>
Co-authored-by: ispobock <ispobaoke@gmail.com>
2026-09-01 00:26:20 +08:00
Lianmin ZhengandMing Yang 3ed3326631 Decouple speculative draft capacity from runtime state (#36897)
Co-authored-by: Ming Yang <minos.future@gmail.com>
2026-08-30 23:31:25 -07:00
YAMYandgithub-actions[bot] b77cac06a9 [PP] Support prefill CUDA graph proxy tensors (#36248)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-08-30 23:11:45 -07:00
Cheng WanandClaude Opus 5 e51a3ae65e [Config] Round 5.2: the per-model declarations get their own modules (#37087)
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-08-30 02:24:35 -07:00
Cheng Wan 7e751153eb [Config] Round 5.1: the published-side readers ask the bags, and a platform fact gets one address (#37086) 2026-08-30 02:18:33 -07:00
Cheng WanandClaude Opus 5 4d53767b09 config: the lazy imports that buy nothing become eager (#36975)
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-08-29 04:21:54 -07:00
Cheng WanandClaude Opus 5 b65e677e48 config: the resolution callbacks into the record go to zero (#36972)
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-08-29 04:18:05 -07:00
Cheng WanandClaude Opus 5 ef20fab38a config: the forwarding slots go; the dispatcher calls the family directly (#36792)
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-08-28 10:26:29 -07:00
Cheng WanandClaude Opus 5 c2928e86d7 config: the resolution pipeline moves out of the record (#36789)
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-08-28 10:17:24 -07:00
Cheng WanandClaude Opus 5 6ff2a20ccf config: the record is not an object that gets passed around (#36622)
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-08-27 12:57:10 -07:00
Cheng WanandClaude Opus 5 7c3b5a6732 config: every handler declares its cuda-graph decisions (#36725)
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-08-27 12:55:34 -07:00