Commit Graph
16664 Commits
Author SHA1 Message Date
Raiden MakotoandRaiden-Makoto 4c0e85524d [AMD] [GLM5] Enable dense-MHA short-context prefill fallback on gfx950 (#30808)
Co-authored-by: Raiden-Makoto <Raiden-Makoto@users.noreply.github.com>
2026-08-15 19:30:14 -07:00
Mick d269a28b47 [diffusion] refactor: route minimax h3 vae attention through native backends (#34949) 2026-08-16 10:07:26 +08:00
Mick 4f9da62547 [diffusion] chore: use native hunyuan3d paint and delight models (#34980) 2026-08-16 10:03:48 +08:00
Mick d106e8b23a [diffusion] chore: use native ernie prompt enhancer (#34951) 2026-08-16 09:59:51 +08:00
Mick 19e3bd6391 [diffusion] chore: use native qwen3-vl vision encoder (#34945) 2026-08-16 09:58:46 +08:00
Mick eb6b773149 [diffusion] chore: use native qwen2.5-vl generation (#34896) 2026-08-16 09:57:39 +08:00
gilfordtingandClaude Fable 5 4a6dc267e1 [Spec] Support mamba-radix-cache-strategy extra_buffer_lazy with DFLASH (#34763)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-08-15 16:36:59 -07:00
Mohammad Miadh AngkadandMohammad Angkad 3802a725ac [CI] Pin the allowed_media_domains supplied-instance reads in the step-12 ratchet (#34961)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-08-15 16:35:19 -07:00
d22c4cc177 [AMD] perf(sgl-kernel): default block_quota=16 for MLA page_first KV gather… (#30024)
Co-authored-by: Niko Ma <nima@amd.com>
Co-authored-by: figo <fizhang@amd.com>
Co-authored-by: AMD-yanfeiwang <yanfei.wang@amd.com>
2026-08-15 16:05:40 -07:00
Liangsheng Yin 0f7aaceda5 [misc] Rename the WAR read-done fastpath to shared-read-done (#34916) 2026-08-15 15:02:02 -07:00
Thomas Wang 4d0c5a89af [AMD] Add concat_and_cast_mha_k_pad_kernel to support 12-head and enable K3 aiter prefill kernel (#34837) 2026-08-15 14:39:30 -07:00
Ke Bao dd458f3212 Fix dsv4 kl test timeout (#34963) 2026-08-16 01:57:40 +08:00
cctryandYilong Zhao e5b3a48751 Add --http2-max-concurrent-streams server arg (#34796)
Co-authored-by: Yilong Zhao <74357408+happierpig@users.noreply.github.com>
2026-08-15 10:34:49 -07:00
huangtingwei cef8a32b9d update codeowner (#34866) 2026-08-16 00:37:18 +08:00
Shenxiu Liu fb97be4359 Fix Whisper transcription for audio over 30 seconds (#33604) 2026-08-15 23:51:05 +08:00
Mick 4beb157e87 [diffusion] doc: define native diffusion model integration contract (#34952) 2026-08-15 23:37:34 +08:00
5c0ace30c0 [diffusion] model: support ltx-2.5 (#34471)
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Mick <mickjagger19@icloud.com>
2026-08-15 23:36:02 +08:00
Mick e331baaaa8 [diffusion] chore: scope attention backend fallback (#34891) 2026-08-15 22:00:46 +08:00
Baizhou Zhang e161bd1265 Fix Python packaging shadowing in DeepEP wheel builds (#34937) 2026-08-15 03:27:32 -07:00
Mick a64791b312 test: restore GLM-4.1V nightly latency threshold (#34811) 2026-08-15 18:13:19 +08:00
Mick 35cefd1c51 feat: add safeguards for remote media URLs (#34892) 2026-08-15 18:12:15 +08:00
Xiaoyu Zhang 0c072235f4 [diffusion] Bound overlong weight lock filenames (#34825) 2026-08-15 17:20:30 +08:00
Baizhou Zhang 8d44091326 Update sgl-deep-ep release workflow for DeepEP v2 (#34914) 2026-08-15 02:08:28 -07:00
Baizhou Zhang 7769f54feb [Kimi-K3] Use explicit SiTU activation for MegaMoE (#34883) 2026-08-15 01:41:59 -07:00
Cheng Wan c87a2ced12 test(step-12): state the bag contract as what resolution produced, and the skill rule that goes with it
`test_bag_values_match_server_args` asserted `bag == field`. That holds today
only because construction resolves in place; step 12 keeps the record raw, and
the plan doc calls this test out as one that becomes **false by design** for
every field resolution fills in.

Rewritten against the resolved projection, which is the half that survives: the
bag carries what resolution produced. The `bag == field` assertion stays as one
line at the end, labelled as the tripwire -- when it starts failing for a
resolution-written leaf, the flip has landed and the bag is the only place the
effective value lives.

The reference is an independent resolution of the same raw input (a fresh,
never-published record) rather than `resolved_server_args_dict()`, which reads
`vars(server_args)` back and therefore only restates the published instance.
And the record goes through the real pipeline on a real mini config, published
through `publish()`: the dummy-model path returns at the dummy boundary with
every sampled leaf still raw, so the old comparison was raw==raw and vacuous
(both Codex catches). Reproducibility (#34094) licenses the sibling as a
stand-in for the pipeline output.

The sample admits only leaves resolution writes on this input on both CI
device shapes (attention_backend, page_size, chunked_prefill_size,
mem_fraction_static), and the raw-differs guard asserts it per leaf -- a
default-count threshold let supplied inputs like `model_path` (no dataclass
default, so any path "differs") stand in for resolution work. Passthrough
leaves (host, hicache_ratio, moe_runner_backend, model_path) move to a
separate projection smoke that claims only what it checks: publish projected
an unchanged field into its namespace. Between the two resolutions the test
restores environ and the EnvField none-flags, so the sibling resolves the
same pristine input rather than the first resolution's leftovers.

And the class runs its body exactly once, like the other dual-resolve
harnesses: a CI retry re-enters after the first attempt leaked process
state, which is the hazard the pristine snapshot exists to rule out.


docs(skill): a supplied-instance read is not automatically safe

The whole-object rule said "keep the supplied-instance contract; don't rewrite
the parameter reads unless the field is runtime-mutated". That is the right rule
for the *object* and the wrong stopping point for the *field*: after step 12 the
record carries the user's raw input, so `server_args.page_size` inside a
runner-owned constructor reads the CLI default rather than the effective value.

The rule now names that second case as step-12 debt with a guard attached
(`test_supplied_instance_exposure_ratchet.py` fails on a new pair, so the
decision is made when the read is written), and names the two shapes that stay
parameter-form on purpose: a helper the resolution pipeline calls with a
`resolved_view`, and a factory whose contract is "build X from the record you
are handed".
2026-08-15 00:40:37 -07:00
Cheng Wan d804b6bd98 config: pin the step-12 debt on the supplied-instance surface
A callee that takes `server_args` keeps the supplied-instance contract, so no
ratchet counts its reads -- and that is right for the *object*. What it does not
cover is what the object will carry after step 12: the instance stays at the
user's raw input, so a callee reading a field resolution fills in starts seeing
the CLI default instead of the effective value.

Measured, not guessed: **297 distinct file/field pairs read one of the 125
fields resolution can write** -- what remains after the earlier members'
conversions (the census found 314 across the package; the page_size,
chunked_prefill_size and graph/limit families were converted by the members
below this one, so the list lands at the remaining debt with no churn). The
census counts three spellings of the read: `server_args.field` off the
parameter, `getattr(server_args, "field", default)` with a literal name, and
the *parked* form -- `self.x = server_args` in a method that takes the
parameter, read as `self.x.field` anywhere in the class. Parking under a
different object, a container, or a computed name stays invisible, like in
every census of this family.

The written-field set is derived in-test from resolved configs against the
dataclass defaults -- the same matrix the context repo's audit tool uses -- and
the union is only as complete as the matrix: fields a matrix entry passes in
are excluded, so each entry must let resolution make the decision the entry is
about. The DWDP shape is the loudest example: `_handle_dwdp` writes `dp_size`,
`enable_dp_attention`, `ep_size` and friends itself, and without a
`{tp_size: 2, dwdp_size: 2}` entry the whole DP/EP topology family (37 pairs
across the launcher, the controllers, the tokenizer and the spec workers)
never entered the written set at all. Multi-item scoring is the same shape in
miniature -- `_handle_multi_item_scoring` writes `disable_radix_cache` itself,
and the `{enable_mis, attention_backend=flashinfer}` entry (the backend is
passed because the handler asserts rather than switches) pins the radix-cache
builder and its friends.

The written set carries **may-write semantics**, statically collected from
every mechanism that can put a resolved value on the record, unioned with the
matrix (construct-and-diff still catches value-level writes statics cannot
name). The enumeration approach kept losing to review -- the DFLASH hook hole,
then the declarative registry (`MODEL_OVERRIDES` forces dtype for two arches
through a setattr applier no assignment scan sees), then the deprecated-alias
loop that writes through a name tuple -- because each round found one more
*mechanism*, not one more field. So all of them are collected now: hook
assignments under `arg_groups/`, the record's own method assignments (the
mooncake layout rewrite, the deepseek-EP mode defaults, the seed fill that
only fires when the caller did NOT supply one -- which construct-and-diff can
never see, since measuring requires supplying), the declarative override
registry, the alias-normalization tuple (drift-guarded), and the
late-resolution keywords. A statically-collected write site that can never
fire is a dead branch to delete upstream, not a reason to shrink the census
(maintainer's ruling). And the pin is split by host: `_EXPOSED` is asserted everywhere,
`_EXPOSED_CUDA_ONLY` (empty today) is where a capability-gated write's
readers go -- one shared exact list cannot hold such a pair at all, since
pinning it fails CPU as "gone" and omitting it fails CUDA as "new".

The resolve-once shape also undercounts on axes a construction call never
sees (each of these was a review catch, and each added pinned families):
resolution branches on the *environment*, so every matrix entry resolves under
the plain env and under the CI shape (`SGLANG_IS_IN_CI`), with the pristine
process state (environ plus the `EnvField` descriptor flags) restored between
entries -- that is where `soft_watchdog_timeout`'s four readers come from. Some
fields resolve *late*, at validation rather than construction
(`declare_late_resolution`): those writers are collected statically by keyword
-- `lora_paths`, `reasoning_parser`, `tool_call_parser` -- with the one dynamic
`**detected` site spelled out in a table guarded against drift. Fields holding
only a `default_factory` are materialized rather than skipped, and
`tokenizer_path` / `served_model_name` -- always filled from `model_path` --
leave the passed-inputs exemption and pin their twelve readers. A module the
census cannot parse fails the test instead of shrinking it, which immediately
caught a BOM-carrying file every previous census had silently skipped (the
scans read `utf-8-sig` now). And the test registers on the CUDA runner besides
the CPU suite, because capability-gated writes only open on real hardware;
AMD is intentionally not registered -- an exact pin cannot be verified from
any pinning host -- with the reasoning in the header.

**A second axis is already wrong today**, independent of step 12. Some config is
decided after publish and recorded with `get_context().override(...)` -- elastic
EP resizing `ep_size`, a weight update rewriting `model_path` / `load_format`,
HiCache attach naming a storage backend, adaptive speculative decoding moving
`speculative_num_steps`. That write reaches the bags and never the record, so a
supplied-instance read of one of those fields answers with the startup value from
the moment the override lands. **73 pairs over 13 fields** are in that position -- including the overrides
that arrive as `**kwargs`: the collector statically resolves dict-literal
expansions (the HiCache attach shape, whose write/read pairs on
`hicache_write_policy` / `hicache_storage_prefetch_policy` were invisible
before) and fails loudly on anything it cannot resolve, with
`update_server_args` exempted by name because its key set is the API's
caller's, not this file's.
Whether each is a defect depends on ordering -- a value copied at construction,
before any override, is fine -- so the axis is pinned as a measurement with the
same growth guard, not as a list of bugs.

One of them *was* a defect and is fixed at the base of this stack: the
linear-attn dispatch table rebuilt itself from the record after the SM100 GDN
prefill decision had been recorded in the bag, so a second runner's rebuild
dropped it. That choice is a per-runner stamp now and is not recorded
process-wide at all, which is why neither the read nor the field appears on this
axis.

The list is pinned both ways, on both axes. A new pair fails, because the moment
to decide where a resolved value comes from is when the read is written, not
during the flip; a disappeared pair fails too, naming the entry to delete, so the
registry stays a measurement rather than a memory of one. Both axes
reverse-verified: a new read of a written field is reported by file and field.

Per-field dispositions live in the plan doc; several of these are "should this
callee take a config at all?", which is a design call rather than a sweep.


test(step-12): tripwire on the EPD guard that a raw record would silence

`_reject_missing_dispatched_encoder_embedding` is one of the two reads the
step-12 audit calls a blocker: it keys on `encoder_transfer_backend`, a field
resolution fills in, off a handed record. Today that record carries the
resolved value; after the flip it stays at the argument default `"auto"`
(`ENCODER_TRANSFER_BACKEND_CHOICES[0]`) for every auto-resolved launch and the
503 stops firing -- a guard that goes quiet, which no existing case notices.

The tripwire resolves a real language-only Kimi-K3 TP2 launch (a mini config,
the shape whose auto pick is `"zmq_to_tokenizer"`) and asserts the guard
rejects with the record resolution produced. A fixed double cannot trip on the
flip -- it would keep handing the guard the resolved value by construction --
so the record has to come from resolution itself: when step 12 lands, this
same launch hands the guard `"auto"`, the rejection silently stops, and this
test fails, which is exactly the signal that this reader needs the resolved
value from somewhere else (the per-engine overlay or the bag).

The launch pins `mamba_radix_cache_strategy=no_buffer` (+ the overlap-off it
requires): resolution's hybrid state-cache sizing branches on the host device
and asserts a GPU stack for extra_buffer, which a CPU CI runner does not have,
while the guard under test reads a field independent of that branch. The case
restores env *and* the EnvField descriptor flags -- a real resolution leaves
state os.environ does not carry.
2026-08-15 00:40:07 -07:00
Cheng Wan 1ab713c334 config: the post-publish consumers of the supplied-instance surface read the bags
config: the speculative workers take page_size from the bags

Seven worker constructors stored `self.page_size = server_args.page_size` off
the handed record. They all run after publish and all keep a copy of a
process-level value, which is the first row of the plan doc's supplied-instance
disposition table -- so they read `get_schedule().page_size`, and a post-publish
override now reaches them like it reaches every other consumer.

The supplied-instance census named the seven pairs; the exposure ratchet in the
next member pins what remains after this batch of conversions.


config: the post-publish chunked_prefill_size consumers read the bags

Four of the ten supplied-instance `chunked_prefill_size` reads are plain
post-publish consumers -- the EPLB recorder's buffer sizing, the deep-gemm
compile warmup (five reads), the KV-cache builder's effective size, and the
ngram embedding manager's assert. All are reached from runner init, so they read
`get_schedule()`.

Two are deliberately left: `create_kt_config_from_server_args` builds a config
*from a supplied record* by name and contract, and `CanaryLaunchCapacities.from_args`
is the same shape. Converting those would change what the function is, not where
it reads -- the plan doc's disposition table says so per field.


config: the remaining post-publish graph/limit consumers read the bags

Three more of the census's supplied-instance debts are plain post-publish reads: the dspark worker's
cuda-graph decode sizes, the dspark planner's SPS table bound
(`max_running_requests`), and the LoRA manager's cuda-graph moe buffers. The
dspark worker is the clearest of them -- it already read
`get_exec().graph.cuda_graph_config.decode.bs` thirty lines below the instance
read, so the file disagreed with itself about where the same value comes from.

Left where the function's contract is "build a config from the record you are
handed" rather than "read this process's config":
`create_kt_config_from_server_args`, `DllmConfig.from_server_args`,
`CanaryLaunchCapacities.from_args`, `build_compilation_config`. Changing those
would change what the function is.


config: the runner, scheduler and offload manager take page_size from the bags

The same `self.page_size = server_args.page_size` shape as the speculative
workers, in the three remaining process-owned constructors: `ModelRunner`,
`Scheduler`, and the decode-side KV offload manager. The scheduler process
publishes before any of them run. The one path that did not is `ModelRunner`
constructed standalone -- `python -m sglang.benchmark.one_batch` and the manual
runner tests build it with no prior publish, and the constructor's own publish
sat below this read -- so that publish moves above the constructor's first bag
read instead of leaving a window where the runner half-exists unpublished.

Left where the read belongs to something else: `utils/common`'s predicates are
called only from the resolution pipeline with a `resolved_view`,
`allocation_sizing` takes the config its callers supply by contract, and
`CudaVmmFeatureTransport` is tokenizer-owned -- one per tokenizer worker, which
is the per-instance boundary.

The conversion left the offload manager parking a record it no longer
reads; the parked copy goes with the read (the constructor parameter stays
-- its hicache sizing still reads it directly).
2026-08-15 00:39:35 -07:00
Cheng Wan f2ab6e306b config: the alias form of the runner-side instance read
The previous batch counted `self.server_args.X` and called the runner surface
done. It was not: the same read spelled through a local alias --
`server_args = model_runner.server_args` (or `sa = kvc.server_args`, `args = ...`)
followed by `server_args.leaf` -- is the same process-global read wearing a
local name, and the AST census counts **57 of them** across eleven files that
the grep never saw. Census per function, following the alias.

52 were leaves and go to their bag (`spec` 11, `schedule` 9, `memory` 7,
`exec.graph` 5, `exec.moe` 5, `parallel` 4, `disagg` 4, `model` 3,
`exec.mamba` 2, `exec.overlap` 2). Five were not leaves:
three derived members on the eager runner --
`max_speculative_num_draft_tokens` and `enable_mamba_extra_buffer` already had
accessors, and `max_prefill_buffer_tokens` gets one (all its inputs are `schedule`
leaves plus the configured PP size, so it derives from the bags and follows a
post-publish override; `TestDerivedPredicatesAgreeAcrossTiers` pins it against
the member over a 48-case matrix) -- plus `get_attention_backends()`, which the
same commit routes through `attention_backends()`, and a dict that merely shares
the name (`server_args_dict.items`). That dict is the one read left behind.

`build_attention_backends` also stops resolving the pair from the record: it
runs after publish, so it asks `attention_backends()` like every other consumer.
The draft override on the runner still wins first.

`dispatch_event_loop`'s three PP checks read the *configured* PP size, not the
live topology: the MLX runner stub never initializes torch.distributed, so the
live property asserts before the MLX event loop can start (a Codex catch). The
configured leaf answers the same value wherever the live groups exist.

`flashinfer_gdn_prefill_default`'s guard is the one read here that asks what the
*operator* named rather than what the config resolved to, and the bag leaf now
answers exactly that: the per-runner auto-default is stamped on the runner and
deliberately never recorded process-wide, so nothing writes that leaf after
launch and reading it back cannot mistake another runner's default for a flag.

Three test doubles injected a `SimpleNamespace`/`MagicMock` record for exactly
these reads and now publish instead (pool configurator, cache registry, GDN
prefill policy) -- the fixture publishes what the case configures and hands the
published instance to the whole-object contracts that still take one.

The functions this sweep partially converted stop mixing sources (review
catches): the flash-attention constructor's remaining seed reads
(`speculative_eagle_topk`, `speculative_algorithm`, both deterministic gates)
read their bags next to the leaves already converted;
`_should_disable_scheduler_metadata_precompute` reads the parallel config
leaves itself instead of taking the record (its alias binding was the last
use); and the autotune gates (`disable_flashinfer_autotune`, deterministic,
`flashinfer_autotune_skip_ops`) join the moe leaves the same function already
reads from the bags. The pool-configurator fixture drops a parameter nothing
published or read.
2026-08-15 00:39:03 -07:00
Cheng Wan 61908870f6 config: spell out the one dynamic config read the census could not see
`_is_dsa_active` asked `getattr(server_args, "_is_dsa_model_arch", False)`, and
that name has never existed on `ServerArgs` -- it arrived as a placeholder with
the CP strategy abstractions (#27313), so the getattr default has always decided
the predicate. A dynamic read of a name nothing sets is the one shape the config
census cannot follow, and it looked like a live decision while being dead.

Spelled as the constant it evaluates to, with the placeholder written down: what
it should ask (whether this process runs a DSA model arch) is the CP path's
call, and its only consumer, `ContextParallelStrategy.per_layer_attn_cp_comm`,
has no readers yet.

That was the sole entry in the read ratchet's `_INERT_DYNAMIC_READS`, so the
exemption list is gone with it -- there is no way to exempt a read from the
baselines any more, which is the invariant worth having. The `counted()`
indirection it existed for goes too (verified the three shapes it guarded still
report: direct, `getattr`, and an attribute-parked alias).
2026-08-15 00:38:30 -07:00
Cheng Wan d13d5c03ab config: decisions keyed on the attention backend read the configured pair
`--attention-backend` is one field of three: a launch that sets only
`--prefill-attention-backend` or `--decode-attention-backend` leaves the base
field at `None`. Seven decisions read that base field alone and therefore
answered from a field the operator never set. `attention_backends()` is the
pair with the base-field fallback already applied, so each site now asks it for
the half it actually needs:

- `inkling_common/attn` assembles backend-specific kwargs (rel_bias / score
  mods) and gates its fused prologue; the backend those describe is the one
  `self.attn` dispatches to, so `serving_attention_backend()` selects the pair
  member by `forward_batch.forward_mode`, mirroring
  `HybridAttnBackend._select_backend` exactly -- draft-extend routes through
  the prefill branch like the dispatcher does -- and preferring the
  runner-stamped pair, so a draft runner answers with its own backend. That
  preference only works if every backend that can enter a ForwardContext
  carries the stamp, so `DraftBackendFactory._create_backend` now stamps its
  products with the backend it resolved (draft override first), and the
  draft-extend conv-sidecar wrapper copies the wrapped backend's stamp -- the
  replacement backends the spec workers install had no stamp at all and fell
  back to the target's configured pair.
- The chunked-prefix-cache gate is a *prefill* feature -> prefill half. Reading
  the base field switched the feature off for every prefill-only configuration.
- `init_deterministic_inference_config` maps *prefill* knobs
  (SPLIT_TILE / PREFILL_TRUNCATION_ALIGN) -> prefill half; the map missed and
  left truncation unset.
- `two_batch_overlap` computes extend positions -> prefill half.
- mrope's interleaved-rope kernel runs in both phases -> both halves must
  support triton. This one is not conservative when it misreads:
  `support_triton(None)` answers **True**, so a `--prefill-attention-backend
  torch_native` launch took the triton path.
- The req-to-token writer has one caller, `alloc_for_extend` -> prefill half;
  its fallback pays several `.item()` syncs per request, so gating it on the
  decode half too would send every extend of a mixed launch through the slow
  path. `get_last_loc` (the spec-decode allocator's helper) keeps the
  both-halves reading: verify tokens are served by either half depending on
  `speculative_attention_mode`.
- The flashinfer version floor is a guard; it never fired for a launch that
  pinned flashinfer through a split field.

One more site the census found is not converted here: `gpt_oss` derives its
`sinks` parameter dtype from the backend, and a single parameter dtype cannot
serve a split pair (FA4 asserts bfloat16, trtllm_mha consumes float32), so
that one is a behaviour question rather than a config-source one and is fixed
in its own PR.

`test_split_attention_backend_decisions.py` pins the callable decisions by
calling them under a split-only publish, and pins the remaining ones
statically -- the file/why map fails if any of them goes back to the base field
(reverse-verified). It also asserts the `support_triton(None) is True` trap the
sweep exists for.

The stamp comes from the constructor, not the request: every factory leaf
answers ("effective_name", backend), because several map entries do not build
what their key says -- cutedsl_mla draft-extend builds the trtllm-mla backend,
"nsa" is a deprecated alias building dsa, and the hybrid-linear entries pick
fa3/intel_amx/triton by host, which no static rename table can express (a
review catch: on Blackwell the alias stamp reached Inkling's per-forward
kwargs assembly, which asserts a concrete kernel name, and crashed the first
draft-extend forward). The stamping is pinned by unit tests, not only by a
spec e2e: removing the child-stamping loop, stamping an alias from a leaf, or
dropping the wrapper copy goes red (reverse-verified), and a static guard
walks the factory source asserting no leaf answers an alias name. The child loop states its contract explicitly --
`create_decode_backend` passes `stamps_children=True` because its products
are per-step containers by construction, so a container without
`attn_backends` raises instead of being silently skipped by a defensive
probe. The `_version` invalidation names its contract (autograd's in-place
counter: private, chosen because it is the only per-tensor signal that ticks
on copy_-style updates; removal fails loudly). The version-floor guard's file
joins the pair-reader ratchet, and the one runner-seed chain read sharing the
backend's __init__ (`speculative_eagle_topk`) reads the spec bag.
2026-08-15 00:38:00 -07:00
Cheng Wan 97279980cf config: the last runner-side instance reads read the bags
Six reads were left on `self.server_args` outside the per-instance boundary the
plan reserves for the tokenizer-manager family, and each had a different reason
to be there:

- `scheduler.process_input_requests` (`mm_feature_transport`) and
  `BaseSpecWorker._build_hicache_draft_plan` (`enable_hierarchical_cache`) are
  plain leaves -> `get_mm()` / `get_memory()`.
- `DraftBackendFactory._create_backend` read the split backend through a
  *runtime-computed name* (`getattr(self.server_args, backend_name)`) and then
  fell back to the base field by hand -- the census's documented blind spot.
  The two names it can be handed are exactly the pair `attention_backends()`
  returns with that fallback already applied, so it reads the pair and indexes
  it. The draft runner's own stamp still wins when it has one.
- `remote_instance_weight_loader_use_transfer_engine` and
  `pre_capture_activation_reserve_mb` are derived members. Both are computed
  from published leaves only, so both get a named accessor that derives from
  the bags (and therefore follows a post-publish override).

The first of those two has all its inputs in one bag, so it follows the
established shape: one `*_of(cfg)` helper in `arg_groups/overrides.py`, the
`ServerArgs` member delegating to it, and the accessor calling it on
`get_model()`. `modelexpress_transport_of` splits out the JSON parse both
sides need. The second spans four bags plus the configured parallel sizes, so
it exists twice like the mamba pair -- and `TestDerivedPredicatesAgreeAcrossTiers`
now pins both new pairs equal over their input matrices (92 subtests).

`self.server_args.X` outside the tokenizer-manager family: 11 -> 5, and the
five that remain are the documented ones (the encode server's own record, the
nixl connector's rank arithmetic, `GrammarManager`'s handed instance).

The post-capture headroom path calls the same bag-backed
`pre_capture_activation_reserve_mb` accessor the configurator uses -- the
accessor advertises override-following, and a reserve that reads the record
while its sibling reads the bags can disagree after a post-publish override.
And the conversions' orphans go with them: `RemoteInstanceWeightTransporter`
kept a `server_args` field nothing reads, and `DraftBackendFactory` parked a
record it no longer consults -- both drop the parameter, and the four factory
call sites stop threading one.
2026-08-15 00:37:06 -07:00
Cheng Wan ab810e4052 config: each runner carries its own linear-attn kernel choice
Two problems in the same family as #33312 (a per-runner decision that one
participant answered differently), one fixed here and one guarded.

**The linear-attn kernel backends were process-wide.** `attn_backend_wrapper`
rebuilt a module-level dict once per runner, from the handed record plus a local
`prefill_default`. Two things follow, and both are wrong:

- **A draft could not hold a different choice than its target.** Only the runner
  whose model is GDN gets the SM100 FlashInfer prefill default; the operator's
  explicit flag belongs to the launch. The full-attention backends already model
  this correctly -- the runner stamps `prefill_attention_backend_str` /
  `decode_attention_backend_str` and its backend objects are built from the
  stamp. Linear attn had no stamp at all.
- **The second rebuild replaced the first one's choice.** The default was also
  recorded into the process-wide config, which the record does not see, so a
  runner rebuilding without a default of its own resolved `prefill` back to the
  base backend -- silently swapping the kernel the earlier runner selected.
  Demonstrated in-process before this change: table `FLASHINFER`, then `TRITON`.

`resolve_linear_attn_backends(prefill_default=None)` returns a frozen
`LinearAttnBackends(decode, prefill, verify)` from the published `exec.mamba`
leaves; the wrapper stamps it as `runner.linear_attn_backends` before building
the backends that read it; and the three consumers (GDN, KDA, Ascend GDN) read it
off the runner they are built for. Each already took `model_runner` and cached
the result on itself, so the value now simply comes from the right place. A
backend built outside that path has no stamp and raises on the attribute, the way
the full-attention strings do -- no silent fallback to hide the wiring mistake.

The recording goes away with it. A per-runner choice in the process-wide config
has no meaning the second runner can read correctly: the leaf is how the gate
asks "did the operator name a backend", so a recorded default reads back as an
operator flag and the next runner declines its own. The leaf now keeps meaning
what was asked for at launch, and the effective choice lives in the stamp (and
in the log line the gate already emits).

Precedence is unchanged: the resolver takes the default as an argument and an
explicit `--linear-attn-prefill-backend` wins over it, with the gate declining
early so it neither probes the device nor logs.

**A draft entry class must answer the loader exactly when its target does.** The
loader asks the entry class it instantiates for the shared-experts-fusion
decision, and a draft is its own entry class. When the target family carries
auto-disable conditions and the draft's class does not expose them, the loader
installs one decision for each and the draft's weights are laid out for the wrong
one. That shipped: the DSV4 DSpark draft skipped its bundled shared-expert
tensors until #33312 gave it the gate, costing accept length 5.60 -> 2.05.

`test_fusion_gate_coverage.py` walks the same registry but asks whether an entry
class *touches* the decision -- reads the flag, names a gated class. That catches
a class once it already consumes the decision; it could not catch one that should
consume it and does not, which is what the DSpark class looked like (it built the
family's *layer* classes, so the flag reader lived in another module and its own
source named no gated class). `test_draft_entry_hook_parity.py` asks the
invariant directly: presence parity between a draft entry class and the target it
is named after. Identity is deliberately not required -- the Qwen3.5 MTP
delegates with adapted arguments (unwrapping `text_config`, using the MTP
quantization config), which is right -- and weight-name maps are out of scope,
since a draft's checkpoint has its own names.

Reverse-verified against the original defect: with the DSpark gate removed the
case names the pair and the side that is missing it; with #33312 in place it
passes.
2026-08-15 00:36:04 -07:00
6cbfa791d6 [AMD][Spec] Accelerate Qwen3.5 verification with grouped-head shared KV (#34517)
Co-authored-by: chuyeh <chuyeh@users.noreply.github.com>
Co-authored-by: Thomas Wang <1am9trash@gmail.com>
2026-08-15 00:15:52 -07:00
Liangsheng Yin 5c9ee86d90 [CI] Move the static ratchets back to CPU unit tests (#34913) 2026-08-14 23:32:00 -07:00
raghothamandehhuang 2a89c6823a fix: honor explicit model loader classes (#34880)
Co-authored-by: ehhuang <yinghai@meta.com>
2026-08-15 06:00:41 +00:00
Xinyuan Tong e99ecb6eee [CI] Path-gate Rust workspace tests in lint (#34864) 2026-08-14 22:41:54 -07:00
Zhiyao Jiang 7216a44dd8 [AMD] Broadcast the EAGLE greedy verify decision across TP ranks on ROCm (#34238) 2026-08-14 22:37:45 -07:00
ethcheandEthan Che aeee1562e6 [Kernel] Enable Helion backend for Kimi Delta-Attention (#32593)
Co-authored-by: Ethan Che <eche@meta.com>
2026-08-14 22:22:46 -07:00
Khoa PhamandClaude Opus 5 3adbbec2fd [MoE] Route every trtllm-gen MoE call site through one PDL guard (#34789)
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-14 22:09:03 -07:00
Colin Z bc7e3ba66c [AMD][Quantization] Online MXFP4 quantization 4/N - NVFP4 to MXFP4 Online Requantization on AMD GPUs (#29328) 2026-08-14 21:59:39 -07:00
5afdb1caea [AMD CI] follow the miles nightly-prefixed MI350 suite names (#34877)
Co-authored-by: Zhiyao Jiang <jessicajiang324@gmail.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Michael <michaelzhang-ai@users.noreply.github.com>
2026-08-15 12:57:12 +08:00
Michael 2012b4b196 [AMD][CI] Fix stage-b: AttributeError on multimodal embedding requests (#34769) 2026-08-14 21:35:26 -07:00
hanwlax 8720a72814 fix(scheduler): track max prefill batch size over recent real admissions (#34284) 2026-08-15 10:48:03 +08:00
weireweireandweireweire 6eb941a34c [Spec] Support MegaMoE for DSpark under dp attention (#34844)
Co-authored-by: weireweire <20922698+weireweire@users.noreply.github.com>
2026-08-14 17:20:01 -07:00
gilfordtingandClaude Fable 5 a5ba081fbb [Spec] Wire DFLASH aux-hidden capture into the Qwen3.5 text-only wrapper (#34771)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-08-14 17:10:09 -07:00
gilfordtingandJames Liu 6f005e4da1 [Fix] Require JSON booleans for response_format json_schema.strict (#34777)
Co-authored-by: James Liu <jamesl@modal.com>
2026-08-14 16:46:27 -07:00
Baizhou Zhang a1844709f1 [CI] Trim Qwen3.5 FP8 GB300 performance batches (#34882) 2026-08-14 16:45:11 -07:00
gilfordtingandJames Liu 3f64f14360 [Fix] Work around xgrammar 0.2.1 negative integer minimum in Kimi-K3 structural tags (#34778)
Co-authored-by: James Liu <jamesl@modal.com>
2026-08-14 16:41:06 -07:00
Nan Jiang be804c1b83 [RL] Expose top-p-only sampling masks (#33593) 2026-08-14 16:40:27 -07:00
Jason Mancuso 90b3db6dd8 [Fix: RL] Snapshot async state-capture outputs before overlap (#34319) 2026-08-14 16:39:35 -07:00