Commit Graph
172 Commits
Author SHA1 Message Date
Cheng Wan 701213c867 [Refactor] Retire the layer facade and simplify boundary internals (#41554) 2026-09-28 20:17:33 -07:00
YAMY f209e67767 [Spec] Model-agnostic last-stage draft embedding under pipeline parallelism (#39643) 2026-09-28 10:22:40 -07:00
Rain Jiang 6c4f36e47a Rust server unify datapath for mm and generate requests (#39679) 2026-09-27 23:36:07 -07:00
kk 515f5be77e [AMD] Integrate Aiter MegaMoEv2 for DeepSeek-V4 (#35619) 2026-09-25 00:47:08 -07:00
cctryandcctry efff836911 [PD] Add decode host receive for custom transfer backends (#40238)
Co-authored-by: cctry <cctry@fb.com>
2026-09-23 21:57:33 -07:00
Cheng Wan 0010f56a0d [Test] Remove obsolete configuration migration guards (#40976) 2026-09-23 15:33:54 -07:00
inkcherryandDuyi-Wang 58f622ebc4 Reduce decode bootstrap latency with request-owned speculative KV (#38978)
Signed-off-by: inkcherry <mingzhi.liu@amd.com>
Co-authored-by: Duyi-Wang <duyi.wang@amd.com>
2026-09-23 02:24:53 -07:00
Liangsheng Yin 78980a3b0b [misc] Remove deprecated endpoints, env vars and aliases past two releases (#40795) 2026-09-22 18:14:45 -07:00
jain-ria 01275aab6b fix(grpc): expose native response timeout as a server argument (#40644)
Signed-off-by: jain-ria <riajain@NVIDIA.com>
2026-09-22 14:57:38 -07:00
YAMYandzhangxiaolei 6fd98c98b9 [Qwen3.8-Next] Pipeline-parallel serving and PD-prefill MTP for Qwen4-Exp (#40501)
Co-authored-by: zhangxiaolei <zhangxiaolei.666@bytedance.com>
2026-09-22 08:46:44 -07:00
Cheng Wan acac4dd9d9 [Refactor] Clean up parallel runtime comments (#40632) 2026-09-21 14:32:22 -07:00
Cheng Wan 73f071db52 Deprecate the parallel getters the context answers, and ratchet them shut (#40342) 2026-09-21 12:25:32 -07:00
Xueshen Liu ab03a8e7eb [Perf] Fork-safe import: no CUDA context at import time, lighter argument parsing (#40201) 2026-09-21 10:48:35 +08:00
Qiaolin Yu f4c256354c [kimi k3][pd disagg] support pp prefill + dcp decode with dspark (#40045) 2026-09-20 00:15:28 -07:00
huangtingweiandZhangheng e9300f643e [Unified Cache][9/N] add opt-in MLA load deduplication for Mooncake Linker (#39565)
Co-authored-by: Zhangheng <hzh0425@apache.org>
2026-09-20 09:56:56 +08:00
YAMY 9cc7da2ab0 [MegaMoE] Wire Qwen MoE blocks to DeepGEMM MegaMoE (MXFP4 and NVFP4 experts) (#38080) 2026-09-19 16:00:38 -07:00
3a64faa1f2 Fix disagg PP MTP for GLM-5.2 (#39378)
Co-authored-by: Julien Lin <jullin@nvidia.com>
Co-authored-by: YAMY1234 <74099316+YAMY1234@users.noreply.github.com>
Co-authored-by: Yangmin Li <yangminl@nvidia.com>
2026-09-19 13:55:07 -07:00
Khoa PhamandCursor 10b0bcfd18 [PD] Allow decode radix cache and HiCache L1/L2 with DCP (#40263)
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-09-18 17:54:35 -07:00
Cheng Wan 81421b91e9 One read path for every parallel name (#40069) 2026-09-18 17:43:56 -07:00
Zhiqiang Xieandcctry ceb1d2e580 [PD] Enable optimistic prefill with buffer-only L3 write-through HiCache (#40043)
Co-authored-by: cctry <csycfl@gmail.com>
2026-09-18 16:04:15 -07:00
Sam (Kesen Li) d346b214fb feat(kv-cache): support SM100 NVFP4 GenMHA and speculative decoding (#36340) 2026-09-18 14:50:46 -07:00
vstone-w a9fb1c3238 dsv4(npu): support prefill context parallelism with interleave and zigzag (#39427) 2026-09-17 19:59:41 +08:00
Xiaoyu Zhang 537ac52c5b [SRT] Clean up no-op compiler pass, dead helpers, and migration tests (#39295) 2026-09-16 12:04:21 +08:00
Siyuan ChenandYuwei An c9a8fba991 [DSV4][BCG] Optimize the heavy memory use of C4 Indexer when BCG is enabled (#36534)
Co-authored-by: Yuwei An <ayw.sirius19@gmail.com>
2026-09-15 17:23:11 -07:00
Cheng Wan 406c9c71d8 [Feature] Support BF16 and batch-invariant inference with DeepEP v2 (#38160) 2026-09-15 14:30:50 -07:00
Lianmin Zheng a2b4e8888f Allow CUDA VMM feature transport with the Rust frontend (#39347) 2026-09-14 17:09:49 -07:00
Cheng Wan 6410800af9 [unified-memory] Enable prefill cuda-graph capture (#37418) 2026-09-13 19:17:19 -07:00
Cheng Wan 6804eeaabe config: an out-of-tree replacement point for every resolution-pipeline step (#39134) 2026-09-12 17:26:54 -07:00
Xiaoyu ZhangandCursor 0a57403468 [Perf] Optimize Qwen3-VL unique-image serving on H100 (#36411)
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-09-12 15:16:59 +08:00
Khoa PhamandClaude Fable 5.1 6ba96d329f [DCP] Resolve --dcp-comm-backend to fi_a2a/a2a by default for every model (#39165)
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-11 23:48:42 -07:00
Hanming Lu 45715e7f20 [Fix] Disable NCCL graph buffer registration for the TP LM-head all-to-all (pure-DP decode hang under request bursts) (#38936) 2026-09-11 13:10:48 -07:00
fd7743e0e1 [Sampling] Support sampling masks with overlap scheduling (#36631)
Co-authored-by: ByronHsu <ByronHsu@users.noreply.github.com>
Co-authored-by: root <root@slurm-h200-208-179.slurm-compute.tenant-slurm.svc.cluster.local>
Co-authored-by: Byron Hsu <byron+per@periodiclabs.ai>
2026-09-10 14:52:44 -07:00
1b77f498a0 [NVIDIA] Support flashinfer Mega Moe (#31470)
Co-authored-by: djns99 <40156487+djns99@users.noreply.github.com>
Co-authored-by: 云挚 <ningyunxiao.nyx@antgroup.com>
Co-authored-by: Yangmin Li <yangminl@nvidia.com>
Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com>
2026-09-10 00:22:47 -07:00
Brayden ZhongandBrayden Zhong c0b790cf7f Delete cutlass_mla, non-Marlin GPTQ, AWQ AOT kernel, and Dual Chunk Flash Attention (#32114)
Co-authored-by: Brayden Zhong <brayden@radixark.ai>
2026-09-10 15:12:01 +08:00
Cheng Wan f1a512c51c [Config] msgspec.Struct for the config tier (#38753) 2026-09-09 19:41:19 -07:00
Cheng Wan 53dc77ff4e [Config] One writer for the declaration stash; no exception to the write seal (#38752) 2026-09-09 19:22:06 -07:00
Cheng Wan db272201a2 [Config] Retire get_global_server_args, and clear the deprecated flags that have a replacement (#38375) 2026-09-08 16:42:12 -07:00
+3 52fecfdf09 support qwen 3.8 flash next (#37500)
Co-authored-by: ch-wan <54331508+ch-wan@users.noreply.github.com>
Co-authored-by: ispobock <26454835+ispobock@users.noreply.github.com>
Co-authored-by: JustinTong0323 <115166877+JustinTong0323@users.noreply.github.com>
Co-authored-by: samuellees <26428561+samuellees@users.noreply.github.com>
Co-authored-by: YAMY1234 <74099316+YAMY1234@users.noreply.github.com>
Co-authored-by: yhyang201 <47235274+yhyang201@users.noreply.github.com>
Co-authored-by: yizhang2077 <25844240+yizhang2077@users.noreply.github.com>
Co-authored-by: zijiexia <37504505+zijiexia@users.noreply.github.com>
Co-authored-by: Shinto C V <cshintov@gmail.com>
Co-authored-by: Julian Huang <huangzhilin.hzl@antgroup.com>
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com>
Co-authored-by: yhyang201 <yhyang201@gmail.com>
2026-09-08 13:56:21 -07:00
Baizhou Zhang 85d39401c8 [CP V1 Deprecation 3.5/5] Deprecate HIP/NPU/MUSA prefill CP and remove legacy implementation (#38293) 2026-09-07 15:39:34 -07:00
Cheng Wan b5766336d4 [Perf] Unified memory: close the DCP decode gap on Blackwell (#37926) 2026-09-07 01:10:44 -07:00
Xiaoyu ZhangandMick Qian 4d23a4fa6d [Test] Consolidate test cleanup and CI taxonomy (net -11.4K lines) (#37436)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-09-07 15:13:59 +08:00
Baizhou Zhang b6c31b155c [CP V1 Deprecation 3/5] Remove generic prefill CP v1 runtime (#36228) 2026-09-06 21:53:53 -07:00
Cheng Wan aaf9a95763 [Config] Round 6.5: a namespace declares what it derives, next to what it derives it from (#38113)
Fifth of five; stacked on #38049. The split gave every namespace a file, but
only for the half an operator types. This is the other half.

## The parallel quotients are declared, not written out

`attn_tp_size` and its five siblings were sixty lines of near-identical
properties in the runtime context, a file away from the leaves they are
quotients of, so reading `parallel.py` told you what you could set and nothing
about what that decides.

They are declared in `Parallel` now, in the same class as those leaves. They
carry no annotation, so they are not dataclass fields and
`collect_input_fields` never puts them on the record -- the same mechanism that
already keeps `_NS_PATH` off it. That is the right exclusion: a quotient has no
operator input to preserve, and the record is what crosses a process boundary,
where a stamped width is one an elastic scale-up will not refresh.

## A quotient is a value in the bag, like every other derived one

`_derived_width` answered from a stamp or, failing that, a live process group.
The group read could never disagree with the stamp:

- `initialize_model_parallel` stamps all six as its last statement,
  unconditionally;
- an elastic scale-up restamps `attn_dp_size` through
  `update_dp_attention_post_scale` -- the comment claiming it does *not* was
  wrong;
- no hardware backend builds groups of its own;
- `multimodal_gen`, which has its own `initialize_model_parallel` and does not
  stamp, never reads a quotient.

So a built group was always already stamped, and the group read goes -- and with
it the last reason for a quotient to be resolved on every read.

Every input to `derive_parallel_widths` is a record field. `dcp_enabled` is
`decode_context_parallel_size > 1`, not a fact about a built group; it was
spelled `_DCP is not None`, which is a longer way to say the same thing. So the
six are fixed once the configuration is fixed -- the same test every other
`Derived(fn=...)` in this PR passes. They are declared the same way and computed
the same way: once, at publish, into ordinary bag leaves.

What remains is override -> stamp -> published leaf. The stamp stays above the
leaf because an elastic scale-up restamps `attn_dp_size`; the override stays on
top because that is how a test names a width.

## One answer for the config-derived predicates

`enable_mamba_extra_buffer` and its lazy variant, `is_ep_joiner`,
`is_ep_scale_joiner`, `is_startup_weight_load_overlap`: each existed as a
`ServerArgs` member for the resolution pipeline and, for most of them, again as
a `runtime_context` function for readers after publish. Three places to keep
saying the same thing.

A `Derived(fn=...)` is a pure function of the published configuration, so
`publish` computes it once and stores it as an ordinary bag leaf -- a plain
attribute load, which is what a read inside compiled model code needs. The
function is handed the whole resolved config rather than the bag it lands in,
because a derivation is free to span namespaces and the mamba one does: it
reads `memory.disable_radix_cache` alongside its own `exec.mamba` strategy,
which is why it could never have been a method on either bag.

The pre-publish helpers stay -- resolution needs the predicate before there is
a bag to read -- and three readers keep them, because they run before their own
process publishes: `initialize_dp_attention`, which the weight-cache daemon
calls while building its groups thirty lines before its `publish`, and
`PortArgs.init_new`, a factory handed the record that already reads eighteen
other fields off it.

## Notes for a reviewer

**Overriding a leaf does not move its quotient.** `override(tp_size=2)` leaves
`attn_tp_size` where the published config put it, because nothing is recomputed
on read. A test states a topology by publishing a config -- which is what a
real process does -- or by naming the width it wants, `override(attn_tp_size=2)`.
Six tests say it that way now. This is the price of having one answer computed
once, and it is the same price every other derived value in the config already
carries.

A caller that reads a quotient without publishing or overriding now gets an
explicit error naming the field, instead of a default that an uninitialised
group happened to supply. One fixture was in that state --
`TestMlaWriteDoorsUnderDcp` built a bare pool and asked whether DCP was on --
and it publishes a config now, which is what the process it stands in for
does.

Eighteen sites read these predicates without calling them. That is correct --
they are properties -- but it is worth saying they were checked, because a
census that assumes otherwise reports eighteen always-true conditions.

## The skill that documents this subsystem is updated with it

`.claude/rules/modify-component-must-read.md` points at
`.claude/skills/sglang-runtime-context/SKILL.md` before anyone touches these
files, so a stale sentence there is a wrong instruction rather than a stale
note. Four of its load-bearing statements stopped being true across this series
and are corrected here: `NS(...)` is no longer how a field states its namespace
(the declaring class is); the DCP degrade rule is gone, because the quotients
are not live reads; `mamba_extra_buffer_enabled()` and the other predicate
functions it named as the shape to copy no longer exist; and the
namespace-coverage ratchet is described in terms of the marker. The docstring of
`test_server_args_namespaces.py` said the same thing and is fixed too.

The consequence a test author actually trips over is stated there as well:
overriding a leaf no longer moves its quotient, so a topology is stated by
publishing a config or by naming the width.

## Verification

A full registered-unit sweep (648 files) against this stack's merge-base:
19 failures on both sides, the same 19 -- AMD `gfx950`, `modelopt`,
`cuda_vmm`, `weight_checker` and friends, none of them config. The narrower 139-file config sweep used earlier in this series
does not contain the files this change reaches -- `test_kv_index_translator`
never names `get_parallel()`, it constructs an object that does -- which is why
the baseline differential over everything is what is quoted here.
2026-09-06 21:44:24 -07:00
Cheng Wan 98f69ccbf3 [Config] Round 6.3: the record remembers how it was asked for, and is sealed while resolution runs (#38048)
Third of four; stacked on #38047. Two small changes, both about the same thing:
the record holds the operator's input, and nothing else should be true of it.

## `/server_info` can answer what was actually typed

It reports `resolved_dict()` -- what resolution decided. There was no way to ask
the other question, and the two are not derivable from each other: a field
nobody set reads the same as one set to the value resolution would have picked
anyway.

The launcher stores the arguments it parsed and the in-process `Engine` stores
the call that built the record. All three readbacks report it beside the
resolved values, so both surfaces come back in one request: HTTP `/server_info`,
`Engine.get_server_info`, and the gRPC bridge's -- the last one builds from
`resolved_dict()` and would otherwise have been the one surface of the three
that answers only "what resolution decided".
It rides on the record rather than in a field -- it describes how the
configuration was asked for, so it is not part of the configuration: no CLI
flag, no namespace, not in the bags. Being on the record is what lets a
subprocess copy answer the same question the launcher can, and
`replace_resolved` carries it because a copy was launched by whatever launched
its parent.

The crash dump already collected all four surfaces (`server_args`,
`config_updates`, `resolved_config`, `launch_command`); this is the one that
`/server_info` was missing.

## The record is sealed for the length of resolution

The read-only guard armed on `_resolution_finished`, so for the whole run of the
pipeline nothing stopped a resolver from assigning a field. Nothing in `srt/`
does -- 0 assignments statically, and 0 writes observed across the launch-shape
matrix with a watching `__setattr__` -- but that was a convention, and the
defect it permits is invisible: a value a resolver wrote onto the record is
indistinguishable from a value the operator typed, which is the one distinction
the record exists to preserve.

It now arms when resolution starts. A resolver that assigns a field fails at
boot with a message naming `declare_resolution`, which is where the decision
belongs: the stash carries a source and leaves the input intact.

`declare_direct_writes` asks for the seal by name through `record_writable`. It
hands the record to an out-of-tree platform plugin that sets fields on it; those
implementations cannot be converted by editing a resolver here, so the write
stays and the diff is captured into the stash afterwards. Naming the exception
is the point -- an in-tree resolver reaching for it is doing something it should
be declaring.

## Verification

Costs nothing: the 211 test-side assignments all happen before `resolve_once`,
which a post-resolution write already refused. A full registered-unit sweep
(648 files) against the stack's merge-base: 19 failures on both sides, the same
19, none of them config. Driving a
deliberate write into a real handler produces the new error, so the seal is
tested by more than its own unit test.
2026-09-06 21:41:14 -07:00
Cheng Wan 45c24444b1 [Config] Round 6.1: "unset" gets its own spelling, and the declaration says what it means (#38046)
First of five. The stack continues a series that moved configuration out of
`ServerArgs` and into the runtime context's namespace bags. This one fixes
something that was actually broken, and gives the fix its other half.

## "Unset" gets its own spelling on two ratio fields

`swa_full_tokens_ratio` and `mamba_full_memory_ratio` carried real values as
their class defaults (0.8, 0.9), so a model family with an opinion had to ask
"is this field still equal to the class default?" to find out whether the
operator had set it. That question has two wrong answers: it says "the operator
set it" as soon as any earlier pass declares the field, and it says "the
operator did not set it" when the operator types the default value.

Both become `Optional[float] = None`. The record carries what the operator typed
and nothing else, and the family test becomes `is None`.

`mamba_radix_cache_strategy` keeps `"auto"`: unlike the ratios it already has a
spelling for "unset" that an operator can type and that means exactly that --
only its comparison changes, from the class default to the token itself, which
is the fix the comment at that site already prescribed. With that, neither
family module imports `ServerArgs` any more.

## And the declaration says what the field means when nobody answers

Making the default `None` leaves a hole: something has to supply the generic
value. `Arg(fallback=...)` supplies it from the declaration.

```python
swa_full_tokens_ratio: A[
    Optional[float],
    Arg(help="...", resolvable=True, fallback=0.8),
    NS("schedule"),
] = None
```

The dataclass default stays `None`. A fallback is not a default: the record is
the wire format, and a child process has to keep being able to tell "unset" from
"set to the value resolution would have picked anyway".

### Which surface it lives on is the whole design

Precedence becomes **override -> decision -> input -> fallback**, applied in
`resolution_result` -- which the projection, `/server_info` and every config bag
read through.

Deliberately **not** in `resolving_view` / `resolved_view`. Those are the
decision-over-input surface a pass reads *while it is deciding*, and two model
families branch on exactly this:

```python
# model_overrides/inkling.py, and the same shape in deepseek_v4.py
if cfg.swa_full_tokens_ratio is None:
    overrides["swa_full_tokens_ratio"] = 0.1
```

A fallback answering there is not "the generic value, later" -- a `__getattr__`
layer is read-time, so there is no later. Every read during resolution would
already get 0.8 and the branch would never fire. Running `_inkling_overrides`
against both versions:

```
--- fallback on the effective surface only (this PR) ---
  cfg.swa_full_tokens_ratio during resolution = None
  family declared swa = 0.1   mamba = 0.1
--- fallback also on the view a pass reads ---
  cfg.swa_full_tokens_ratio during resolution = 0.8
  family declared swa = None  mamba = None      <- the key never lands
```

So "resolution first, then the fallback" holds -- not because a step is appended
to the pipeline, but because of which surface the value lives on. Exactly one
reader consults the effective surface during resolution: the range check on the
ratio, which wants the value the pools will be sized against. It asks
`resolution_result` directly -- what its comment already claimed it was doing --
and it runs after the model families.

### The alternative, and why not

A pass that fills the field in when nothing claimed it needs a slot (after the
families, or it beats them), a second call site (the dummy-model short circuit
returns long before that slot), an idempotence requirement so the second call is
harmless, and the value written twice -- once as a literal, once as prose in the
help (`"Unset means 0.8"`). An earlier revision of this series did exactly that
and deleted it four PRs later. A declaration needs none of it, and `pipeline.py`
is untouched by the whole series as a result.

### What may be declared this way, and what may not

Across every hook, `if x is None: x = ...` appears at **55 sites over 29
fields**. They are not one thing:

| | count | examples | declarable |
|---|---|---|---|
| unconditional constant | 5 | the two ratios, `grammar_backend="xgrammar"`, `mm_process_config={}`, `custom_weight_loader=[]` | **yes** |
| unconditional, computed from another field | 4 | `tokenizer_path=model_path`, `device=get_device()`, `served_model_name`, `speculative_draft_model_quantization` | needs a `fallback="dotted.path"` form; not here |
| **conditional decision** | ~20 | `chunked_prefill_size` across seven memory tiers, `max_bs` across eight, `max_running_requests` at 48 or 256 by model family | **no, and it should not be** |

Only a value fixed for the life of the configuration belongs in a declaration.
One that depends on the machine, on another field, or on anything impure
(`random_seed = random.randint(...)`) is a decision, and decisions stay in a hook
where their order is visible. This PR converts the two ratios only.

## Verification

- `resolve_once` ends with the same effective values: the resolution result is
  identical across 24 launch shapes x 489 fields except for the two intended
  ratio changes. Separately, 16 launch shapes resolved on both sides, real model
  and dummy: 7,904 field readings, and the only difference is `random_seed`, a
  fresh `random.randint` per process.
- The CLI registers the same 507 options with the same choices and actions; only
  the two defaults move.
- `test_declared_fallbacks.py`, 17 cases. One pins the inverse of the dead branch
  above: what a pass sees while deciding is still `None`.
- The whole series was swept over all 648 registered unit-test files against its
  merge-base: 19 failures on both sides, the same 19, none of them config.


---
### CI States

Latest PR Test (Base): <!-- slot:pr-test:start -->❌ [Run #34083705463](https://github.com/sgl-project/sglang/actions/runs/34083705463)<!-- slot:pr-test:end -->
Latest PR Test (Extra): <!-- slot:pr-test-extra:start -->❌ [Run #34083705284](https://github.com/sgl-project/sglang/actions/runs/34083705284)<!-- slot:pr-test-extra:end -->
Latest PR Test (AMD ROCm 7.2): <!-- slot:pr-test-amd-rocm720:start -->❌ [Run #34083705383](https://github.com/sgl-project/sglang/actions/runs/34083705383)<!-- slot:pr-test-amd-rocm720:end -->
<!-- pr-states:end -->
2026-09-06 21:38:29 -07:00
sglang-botandsglang-bot 6252993afe chore: update CI test est_time values (#38238)
Co-authored-by: sglang-bot <sglang-bot@users.noreply.github.com>
2026-09-06 17:49:41 -07:00
Liangsheng Yin d50e9a9756 [Test] Prune redundant unified-memory allocator and pool tests (#38093) 2026-09-04 20:47:01 -07:00
55bf3380e0 Support Hy4-preview (#36805)
Co-authored-by: BBuf <1182563586@qq.com>
Co-authored-by: alphabetc1 <2508695655@qq.com>
2026-09-04 18:03:49 -07:00
Baizhou Zhang ff1285cc28 [CP V1 Deprecation 2/5] Make strategy prefill CP canonical (#36223) 2026-09-03 20:24:19 -07:00
junduandMa Mingfei 8770c1db1f [CPU][CI]: rename Xeon CPU CI suites to stage-*-intel (#37395)
Co-authored-by: Ma Mingfei <mingfei.ma@intel.com>
2026-09-04 10:08:41 +08:00