minke.yu
8b67636010
Merge b300/integration into main: 2026-11-17 c580 baseline
...
CI Model Inventory / build-inventory (push) Failing after 41s
PR Benchmark (SMG Components) / Benchmark - Manual Policy (push) Skipped
PR Benchmark (SMG Components) / Benchmark - Request Processing (push) Skipped
PR Test (Arm64) / check-changes (push) Successful in 14s
PR Test (Xeon) / pr-gate (push) Successful in 2s
PR Test (SMG) / k8s-integration (push) Failing after 12m21s
PR Test (Rust extensions) / sglang-mm-unit (push) Failing after 12s
PR Benchmark (SMG Components) / Benchmark Compilation Check (push) Failing after 1m45s
PR Test (Arm64) / pr-gate (push) Successful in 2s
PR Test (sglang-processor) / ${{ github.event.action == 'labeled' && github.event.label.name != 'run-ci' && 'ignored-label-sgl-processor-gate' || 'sgl-processor-gate' }} (push) Successful in 2s
Seed Rust Ext Cache / seed (push) Failing after 1m53s
PR Benchmark (SMG Components) / Benchmark Summary (push) Skipped
PR Test (SMG) / docker-build-test (push) Failing after 18m37s
pr-test-xpu.yml / pr-gate (push) Successful in 2s
PR Test (sglang-processor) / ${{ github.event.action == 'labeled' && github.event.label.name != 'run-ci' && 'ignored-label-sgl-processor-lint' || 'sgl-processor-lint' }} (push) Failing after 10s
PR Test (SMG) / finish (push) Skipped
Lint / lint (push) Failing after 2m40s
pr-test-arm64.yml / pr-gate (push) Successful in 2s
PR Test (XPU) / multimodal-gen-test-1-gpu-xpu (push) Canceled after 0s
PR Test (XPU) / stage-a-test-1-gpu-xpu (push) Canceled after 0s
Release SGLang Kernels / build-rocm-matrix (3.10, ${{ fromJson(inputs.target == 'rocm720' && '["720"]' || inputs.target == 'rocm1000' && '["1000"]' || '["720", "1000"]') }}) (push) Canceled after 0s
Release SGLang Kernels / build-musa43 (43, 3.10) (push) Canceled after 0s
PR Test (Xeon) / build-test (spr1, 1, 0, 4, spr, xeon-spr, stage-a-test-cpu-intel,stage-b-test-cpu-intel) (push) Canceled after 0s
PR Test (Xeon) / build-test (spr2, 3, 2, 4, spr, xeon-spr, stage-a-test-cpu-intel,stage-b-test-cpu-intel) (push) Canceled after 0s
Seed Rust Ext Cache / seed-aarch64 (push) Canceled after 27h16m58s
PR Test (Arm64) / build-test (push) Canceled after 0s
Release SGLang Kernels / release-musa43 (push) Canceled after 0s
PR Test (XPU) / finish (push) Canceled after 0s
Release SGLang Kernels / release-cu130 (push) Canceled after 0s
Release SGLang Kernels / release-rocm720 (push) Canceled after 0s
Release SGLang Kernels / release-rocm1000 (push) Canceled after 0s
PR Test - Build Rust Extensions / Collect Rust Ext (push) Canceled after 0s
PR Test (Rust workspace) / cargo-workspace (push) Failing after 2m27s
PR Test (Rust workspace) / sglang-radix-tree (push) Successful in 2m45s
PR Test (SMG) / unit-tests (push) Failing after 1m47s
PR Test (sglang-processor) / ${{ github.event.action == 'labeled' && github.event.label.name != 'run-ci' && 'ignored-label-sgl-processor-changes' || 'sgl-processor-changes' }} (push) Successful in 14s
PR Test (sgl-router) / changes (push) Successful in 13s
PR Test (sgl-router) / gate (push) Successful in 2s
PR Test (XPU) / pr-gate (push) Successful in 2s
PR Test (Xeon) / build-test (gnr, gnr, xeon-gnr, stage-a-tp-test-cpu-intel) (push) Canceled after 0s
Release SGLang Kernels / build-cu130-matrix (aarch64, 13.0, 3.11, arm-kernel-build-node) (push) Canceled after 0s
Release SGLang Kernels / build-cu130-matrix (x86_64, 13.0, 3.11, x64-kernel-build-node) (push) Canceled after 0s
PR Test - Build Rust Extensions / Restore Rust Ext (push) Successful in 5s
PR Test - Build Rust Extensions / Build Rust Ext (3.12) (push) Canceled after 0s
PR Test (SMG) / build-wheel (push) Failing after 1m38s
pr-test-sgl-processor.yml / pr-gate (push) Successful in 2s
pr-test-sgl-router.yml / pr-gate (push) Successful in 2s
pr-test-xeon.yml / pr-gate (push) Successful in 2s
PR Test (sglang-processor) / ${{ github.event.action == 'labeled' && github.event.label.name != 'run-ci' && 'ignored-label-sgl-processor-test' || 'sgl-processor-test' }} (push) Skipped
PR Test (sgl-router) / tier-1 — lint (push) Failing after 55s
PR Test (sgl-router) / tier-3 — e2e (push) Skipped
PR Test (sglang-processor) / ${{ github.event.action == 'labeled' && github.event.label.name != 'run-ci' && 'ignored-label-sgl-processor-finish' || 'sgl-processor-finish' }} (push) Failing after 1s
PR Test (SMG) / python-unit-tests (push) Skipped
PR Test (SMG) / benchmarks (push) Skipped
PR Test (SMG) / chat-completions (push) Skipped
PR Test (SMG) / chat-completions-4gpu (push) Skipped
PR Test (SMG) / e2e (push) Skipped
PR Test (SMG) / summarize-benchmarks (push) Skipped
PR Test (Xeon) / check-changes (push) Successful in 14s
PR Test (XPU) / check-changes (push) Successful in 20s
PR Test (sgl-router) / tier-2 — build + test (push) Skipped
PR Test (sgl-router) / tier-3 — docker (placeholder) (push) Skipped
PR Test (sgl-router) / tier-3 — k8s integration (push) Skipped
PR Test (sgl-router) / finish (push) Failing after 2s
Promote the validated ds41-gist-exp baseline:
- image recipe-ns-l3ssd-433257c7b-baked-20261115 (SGLang + Mooncake 0.3.14 + L3 SSD)
- P: cache / no-optimistic / fastokens / tokenizer-worker 16 / L3 SSD 4T
- D: fastokens / tokenizer-worker 16
- gateway: normal health/circuit, no P admission caps
- c580 baseline 24.045 rps; c620 and buffer_only results recorded
- Feishu baseline doc and deploy manifests included
This is the base for the next Dynamo layer.
2026-11-17 20:08:08 +08:00
minke.yu
310ad35430
deploy: record 2026-11-17 c580 baseline (D16+fastokens) and experiments
2026-11-17 20:05:11 +08:00
minke.yu
bdbd8dffb8
deploy(ds41-gist-exp): P buffer_only + optimistic, D tokenizer 16
2026-11-16 22:03:20 +08:00
minke.yu
f363a6ab7b
deploy: record 11/16 pre-optimistic stable config
2026-11-16 22:03:11 +08:00
minke.yu
e4c7e540be
deploy(g9i): T26 qps20 + P queue32 production candidate
2026-11-16 00:42:44 +08:00
minke.yu
b8b205b367
deploy(g9i): merged ns+td gateway, 32-request P queue cap (p95 5.70s)
2026-11-15 22:25:33 +08:00
minke.yu
b3d6d19c7d
deploy(g9i): set regular P queue reject 32s (experiment)
2026-11-15 21:45:51 +08:00
minke.yu
8f53c1fb73
deploy(l3ssd): offload_on_evict=false, 4TB cap, L3 reset script
2026-11-15 20:33:38 +08:00
minke.yu
42f228bfb8
docs(deploy): fresh-instance multi-round tokenizer A/B (B2 TTFT p95 < 5s)
2026-11-15 18:20:53 +08:00
minke.yu
242a7ad23a
docs(deploy): record L3 SSD tokenizer A/B (fastokens vs huggingface)
2026-11-15 15:49:56 +08:00
minke.yu
2eb163ad73
fix(deploy): point sglang cache at the shared warm cache
2026-11-15 14:23:52 +08:00
minke.yu
d3240c745e
deploy(b300): L3 Mooncake SSD baseline with mooncake 0.3.14
2026-11-15 14:10:30 +08:00
minke.yu
433257c7b0
fix(recipe): drop the pre-48928 _fast_prompt_tokenizer fallback stub
2026-11-15 10:53:15 +08:00
leon
7980a09c0a
[Perf] DeepSeek-V4.1-Flash: Use segmented fastokens for prompt encoding ( #48928 )
...
(cherry picked from commit 219b7e7a688363bac9ef09eacf80e44eef9efe4b)
2026-11-15 10:51:40 +08:00
minke.yu
a7c668a936
fix(chat): move namespace-tool flattening validator into ChatCompletionRequest
2026-11-14 14:10:07 +08:00
minke.yu
0c6849e378
chat: flatten namespace tools to ns__function for Chat Completions
2026-11-14 14:09:36 +08:00
minke.yu
85605b8c72
fix(protocol): ResponseTool accepts extra fields (tool_search execution)
2026-11-13 23:25:50 +08:00
minke.yu
b6a5d70cba
fix(responses): tolerate tools without model_extra in native validation
2026-11-13 23:24:00 +08:00
minke.yu
cac60f7c2e
fix(protocol): effective_tool_choice preserves namespace/type
2026-11-13 23:22:21 +08:00
minke.yu
68c2af7957
fix(protocol): Function.strict is Optional[bool] for recipe tool schemas
2026-11-13 23:21:04 +08:00
minke.yu
497aeff5f6
fix(recipe): define _fast_prompt_tokenizer fallback on this base
2026-11-13 22:44:35 +08:00
minke.yu
5fa8c23227
vendor: DeepSeek-recipe namespace tools ( #46370 + #49625 )
...
Backport the upstream namespace-tool stack onto the B300 recipe base:
- #46370 : Function.namespace / ChatMessage.action protocol fields.
- #49625 : Responses/Messages native recipe rendering (convert_native),
tool_search/defer_loading, namespace identity in/out/history replay.
Adapted to this base: skips serving_kimi_k3.py (not present), drops the
Minimax/Beam/wire-only dependencies added between 10-23 and the PR base,
and keeps the existing tool-parsing fallback where the newer
`parses_without_tools()` helper is unavailable.
Also includes the PR's Anthropic tool-reference plumbing and unit tests.
2026-11-13 22:43:37 +08:00
minke.yu
f4de9f2141
observability: ignore sub-tolerance CPU>wall skew in accounting anomaly
...
A 2 ms phase whose CPU clock reads 0.1 us over its wall clock is clock
granularity, not an accounting bug; the anomaly flag fired on nearly every
short phase in the live verification run.
Only flag (and record non_cpu_raw_ms) when the deficit exceeds both 100 us and
1% of the phase wall. Larger real anomalies keep the previous behaviour.
2026-11-12 12:30:58 +08:00
minke.yu
7f42a0806a
observability: per-rank request counter for P/D RPS
...
The frontend sglang:num_requests_total has no dp_rank label (verified from the
series registry), so per-rank RPS was not derivable from any existing metric.
Add sglang:batch_requests_total{phase,dp_rank}, incremented by the scheduler
process that actually completed the request:
phase=prefill prefill compute finished and the request entered the PD
transfer queue (not a guarantee that the transfer succeeded)
phase=decode request finished decoding
Only the canonical scheduler rank reports (attn_tp_rank==0, attn_cp_rank==0,
last PP stage) — every TP/CP rank runs a scheduler and sees the same request,
so an ungated counter would be multiplied by TP*CP in the multiprocess
registry. Both hooks count each request exactly once: the decode path counts
after _handle_finish_state_updated_req and on the sampling-mask abort branch,
skipping rows that finished on an earlier step; the prefill path counts after
the request enters the disagg inflight queue (aborted/retried requests never
reach it). The per-request finished() check is skipped entirely when
SGLANG_BATCH_TELEMETRY is off.
2026-11-12 10:30:56 +08:00
minke.yu
17b922c84f
test: prefill_batch_init detail and eviction semantics
...
Covers the design's unit-level acceptance points:
* disabled/unsampled loops never read the thread clock;
* wall/CPU/non-CPU semantics, unmeasured != zero, partial measurement omits
non-CPU, busy vs sleeping windows are distinguishable;
* CPU > wall flags an accounting anomaly with a raw negative delta;
* child spans carry the CPU split;
* repeated phases (prefix indices) stay attribute-only, inclusive phases are
not summed, and eviction demand/shortfall/plan plus Mamba plan/zero results
are recorded as separate counters.
2026-11-12 10:30:47 +08:00
minke.yu
5ad9ff6a5f
observability: prefill_batch_init wall+CPU, sub-phases and eviction coverage
...
Parent span scheduler.loop.prefill_batch_init records wall + the scheduler
thread's CPU at both existing close points (empty admission and normal
return); the CPU clock is only read when loop detail is enabled and sampled,
through the safe batch_telemetry.thread_cpu_ns() wrapper so a missing clock
degrades to 'not measured' instead of failing the scheduling path.
Fixed children, in call order:
schedule_batch_init, ready_to_load_host_cache, prepare_for_extend,
alloc_for_extend (+ prefix_device_indices attribute-only, sampling_batch_info),
evict_to_free_tokens, evict_for_alloc, prefill_stats (+ pending_tokens),
mixed_chunk_tail.
* prefill_stats wraps the whole assignment so _get_num_pending_tokens() is
included; mixed_chunk_tail only exists when that branch is taken.
* prefill-only shape counters (n_reqs/n_new/n_cached) are snapshotted before
mixed-chunk folding; a mixed batch additionally publishes decode_reqs and
merged totals under explicit *_merged_* names.
* eviction is timed at the call site (outside the allocator wrapper) so
rank-consensus waits are included, and covers base, SWA (x2), UnifiedSWA
(x3) and the extend Mamba reclaim through evict_for_alloc_observed().
Demand, shortfall and plan are separate counters; Mamba plan and realized
counts are both recorded (including zero).
All of it is gated by detail on a sampled loop.
2026-11-12 10:30:38 +08:00
minke.yu
fec10023e0
observability: optional thread-CPU recording for detail phases
...
Adds an optional thread CPU clock next to the existing wall clock:
* detail_begin/end record time.thread_time_ns() only when detail is on and the
loop is sampled; detail_add() accepts an explicit thread_cpu_ns for callers
that already hold absolute start/end points (the prefill_batch_init parent).
* phase records carry [wall, count, start_wall, end_wall, cpu_sum, cpu_count].
thread_cpu_ms and thread_cpu_measured_count are emitted whenever any
occurrence was measured; non_cpu_ms only when every occurrence was, so a
partial measurement never attributes unmeasured wall to non-CPU time. A
phase whose CPU exceeds its wall keeps the clamped value, adds
non_cpu_raw_ms and accounting_anomaly, and bumps a per-loop anomaly count.
* new detail_attr_sum()/detail_attr_setdefault() accumulate low-cardinality
numeric loop attributes (eviction tokens, free-token snapshots).
* prefix_device_indices joins the attribute-only prefixes so per-request
repeats never create spans.
* child spans carry phase_thread_cpu_ms / phase_cpu_measured_count /
phase_non_cpu_ms, so the CPU split is visible without the loop attributes.
Default-off path is unchanged: no thread clock, no new dicts, no spans.
2026-11-12 10:30:29 +08:00
minke.yu
9f9dce1ff6
test: restore _emit_pass_span residual default
...
The detail-phase refactor made residual a required positional argument while
test_loop_span_accounting.py still calls _emit_pass_span with four arguments,
so that test failed on the base commit. Default it to 0.0.
Split out of the prefill_batch_init observability series to keep that review
scope focused.
2026-11-12 10:30:17 +08:00
minke.yu
c401c614fa
fix(dp): rank uncached candidates by published backlog only
...
Deriving each candidate's uncached estimate from its own recent hit rate is
self-reinforcing: a rank that falls behind sees its hit rate drop (fewer
chances to hit), gets charged more for every new request, and falls further
behind. Measured on c580: one rank ended 16% below the others, 30 s cross-rank
dispersion max 952x, and 9.6% lower rps than the total_tokens arm.
Charge the fleet-average uncached size instead and rank by the published
num_waiting_uncached_tokens, which already carries each rank's exact prefix
match.
2026-11-11 02:06:59 +08:00
minke.yu
24b284d4e7
feat(dp): uncached_tokens load-balance method
...
DP balancing on total_tokens charges each request's whole prompt, but with
prefix caching only the unmatched part costs prefill compute. On a workload
with a 95% hit rate the prompt is ~95% noise: per request the uncached part
is a few percent of the prompt yet varies by more than an order of magnitude
(p50 1.9k / p99 59k tokens against a 95k prompt), so one cold request
outweighs many hot ones and per-rank prefill work drifts apart.
Add --load-balance-method uncached_tokens: DPBudget tracks the published
LoadSnapshot.num_waiting_uncached_tokens (prefix match minus accepted storage
hits) plus the rank's recent cache_hit_rate for a newly dispatched request.
2026-11-11 00:03:55 +08:00
QNan0513
fef023b64f
[Fix] scheduler: Count all pending work in DP load snapshots
...
total_tokens DP balancing replaces its per-rank estimate with the rank's
latest load snapshot, so a snapshot that misses work already sent to the
rank lets the controller keep sending to it. Close three gaps:
- count grammar-queue requests in num_waiting_reqs and num_total_tokens
- count the in-progress chunked request's unprefilled tokens
- publish a snapshot once newly received requests are queued, since the
one published after a forward predates the requests sent during it
2026-11-10 23:58:23 +08:00
ymk
15218b2f65
deploy: bake the live hostPath overrides into the tree
...
Copies the three files that the live cw mounts over the image:
- managers/load_snapshot.py: queued_total_input_tokens + drain_rate_tokens_per_s
- managers/scheduler_components/load_inquirer.py: fills the two new fields
- models/deepseek_v4.py: env-gated MM trace (SGLANG_MM_TRACE, off by default)
After this commit every runtime file the P/D pods need is in the tree, so the
hostPath file mounts can be dropped once the image is rebuilt.
2026-11-09 06:31:48 +00:00
minke.yu
883f7d08e1
fix(mooncake): use enable_client_http_server (not enable_http_server)
2026-11-09 11:04:01 +08:00
minke.yu
e40c6dfe83
observability(mooncake): opt-in client metrics HTTP server per rank
...
SGLANG_MOONCAKE_CLIENT_METRICS_PORT_BASE=<base> makes MooncakeStore.setup()
pass enable_http_server=True and client_http_port=base+dp_rank, so each DP
rank's embedded Mooncake client exposes /metrics (needs MC_STORE_CLIENT_METRIC=1).
Off by default.
2026-11-09 10:55:17 +08:00
minke.yu
32e5e0f30f
observability: GPU-window probe + forward/prefill sub-phase timing
...
Phase 1/3 of the async-backup handoff instrumentation plan:
* new fwd_timeline probe (SGLANG_FWD_TIMELINE=1): one CUDA event pair around
the model execute call per forward, resolved without blocking and written as
wall-clock JSONL per DP rank. Joined with loop spans it separates host time
spent while the GPU is busy (overlapped) from GPU-idle host stalls.
* detail phases: fwd_init_new, fwd_prepare, fwd_execute, fwd_graph_load,
fwd_graph_replay, fwd_post, fwd_runner_tail,
prefill_policy, prefill_adder_init, prefill_scan, prefill_batch_init.
All spans ride the existing SGLANG_TRACE_LOOP_DETAIL path; the GPU probe is
off by default.
2026-11-08 01:19:09 +08:00
minke.yu
a5fbe834e8
observability: split get_next residual into its four phases
...
Adds the Phase A instrumentation from the async-backup handoff (§7.1):
* PassTimings: pending_abort / chunk / new_batch / set_sched (+ chunk_count)
* prefill.get_next_disagg_prefill_batch_to_run: time
process_pending_chunked_abort, process_prefill_chunk,
get_new_batch_prefill and set_schedule_time_batch
* scheduler.prefill.pass gains sglang.{pending_abort,chunk,new_batch,set_sched}_ms,
sglang.get_next_residual_ms (= other - these four) and sglang.chunk_count;
the existing sglang.other_ms is unchanged for dashboard compatibility.
~5 monotonic calls per pass; enabled by SGLANG_BATCH_STAGE_TIMING (already on).
2026-11-08 00:20:55 +08:00
minke.yu
d6f1e1efbd
rebase(hicache): per-op async backup on top of #46884 + #48782
...
Net async change (from exp-async-backup-perop-20261107) applied onto
b300-pick-46884-20261107. Runtime files applied cleanly; the new tests and
the timeline probe are added as-is.
Interaction to validate on hardware: #46884 publishes prefill read-done before
the replay, so the scheduler-stream fence that the index copy waits on is
shorter; the async worker's wait should shrink and the copy may overlap more of
the in-flight forward. Re-baseline the 4-arm A/B on this branch.
2026-11-07 22:04:33 +08:00
alphabetc1
520ca548ea
[Fix] hicache: Wait for overlapping write-through acknowledgements
...
(cherry picked from commit 032725770c33c61c236f2884ce7e7ff1fa99cef9)
2026-11-07 14:37:24 +08:00
metamergebot
5f7be7fab4
[Perf] hicache: Publish prefill read-done when offload copies are fenced on the forward stream
...
The prefill shared-read-done fastpath lets the schedule stream run ahead of
the forward that writes a stashed chunk's KV instead of waiting for the whole
forward. It was disabled whenever hierarchical cache or a storage backend was
enabled, because host and storage offload copies used to be ordered only
behind the schedule stream and could have read unfinished KV.
The cache controller now orders its device-to-host copies behind the forward
stream through write_fence_stream, and its load-back H2D copies through
load_fence_stream, so that reason no longer holds when every KV offload copy
carries that fence. The scheduler records on the model runner whether that is
the case: hierarchical cache with a cache controller and no decode offload
manager, which has no such fence. The publish helper keeps the coarse
whole-forward barrier for LMCache and for unfenced offload, and publishes
read-done otherwise.
With the coarse barrier, a synchronous page-index copy issued while preparing
the next batch inherited the wait on the entire in-flight forward, leaving the
GPU idle between batches. With read-done published, that copy only waits for
the shared-table reads to finish; KV ordering is unchanged because the backup
copies still wait on the forward stream.
Tests: the CPU runner test covers the fenced hierarchical-cache and storage
cases publishing, and LMCache keeping the barrier even when hierarchical cache
is fenced. The GPU ordering test checks that a write-through backup of a chunk
stashed before its forward finished still copies the final KV.
Co-authored-by: ZYHowell <ZYHowell@users.noreply.github.com >
(cherry picked from commit bb1cb2dded6fb9352d98076280342b10684e061b)
2026-11-07 14:36:38 +08:00
minke.yu
97c087778b
observability: harden flush span attributes
...
* Normalize the host-pool layout to a string (a None attribute would make
the OTel SDK drop the whole flush span).
* Drop any remaining None-valued flush attributes for the same reason.
2026-11-07 11:35:03 +08:00
minke.yu
53267ee18f
observability: time loop-other inflight poll and release/insert
...
Adds loop-detail phases outside run_batch/process_result/get_next:
* inflight_poll: process_disagg_prefill_inflight_queue() (overlap loop)
* inflight_release: release_kv_cache(checkpoint=True) + tree_cache.finish()
inside it, so the checkpoint/insert cost is separated from the poll
* launch_sample: launch_batch_sample_if_needed()
The get_next side already publishes match/adder/hicache/bootstrap/mlp_sync
and a pass residual (sglang.other_ms) on scheduler.prefill.pass.
2026-11-07 09:26:46 +08:00
minke.yu
aa53670fcd
observability: split flush_pending_backups host chain (off by default)
...
Adds the phase-3/phase-4 instrumentation from
handoff/prefill-flush-backups-validation-20261107.md:
* process_batch_result outer phases: flush_trace, load_snapshot,
flush_backups, step_counters, metrics_report. process_residual now
subtracts them, so the 37.5% residual is attributable.
* SGLANG_TRACE_BACKUP_DETAIL=1 (default 0) traces one flush:
merge_ops / move_write (anchor + per-sidecar indices) / indices_cpu /
index_sort / index_select / write_fence / build_l2 / submit_d2h
(start-event wait, prepare, per-pool launch, ack events) / ack_enqueue,
plus token/op/pool/device dimensions and ack queue depth.
* Per-flush resolution: one scheduler.cache_flush_backup span per
non-empty flush, child of the loop span. Sub-phases never enter the
loop aggregate, so the residual identity cannot double count.
* Empty vs non-empty flushes are counted on the loop.
All timings are scheduler host wall, not device transfer time.
2026-11-07 09:23:44 +08:00
ymk
7b3f34cbc0
test: cover async SWA state-payload call chain; clarify CP gate
...
Review follow-up for the v2 async state payload:
- P2: new tests use CustomTestCase (with a local fallback when sglang is not
importable), matching the disaggregation/Mooncake test convention.
- P3: CP gate docstring now says all-CP-rank transfer *without DSA cache
layer split* keeps the synchronous path.
- Add real call-chain coverage:
* _send_kv_chunk -> sender.send -> add_transfer_request -> transfer_worker
-> materialize_state_indices -> maybe_send_extra, with DSV4's four
components (DSV4_REQUEST_STATE, REQUEST_ROW, SWA, SWA);
* the worker materializes pinned tensors only after indices_event and
fails loudly when a tensor has no event;
* a second materialization without an event (staging retry) is a no-op;
* real CUDA ordering: the pinned helper returns before a long kernel and
the event-protected values match kv_to_page_indices.
- Verified on b300 (P container, overlay source): 9 passed in
test_async_state_payload.py, 2 passed in the new
TestMooncakeAsyncStateMaterialization class; the old batching suite has
11 pre-existing failures that also fail on the image source.
The test-only batching suite requires the detail envs to be off when run in
the live pod (the inherited SGLANG_TRACE_MOONCAKE_DETAIL=1 exposes an existing
test fixture omission); noted, not changed here.
2026-11-07 00:22:57 +08:00
ymk
3b0baa3455
prefill: async SWA state-payload D2H (first version)
...
Implements the v2 plan in handoff/prefill-state-payload-async-review-20261106.md.
- mem_cache/common.py: add kv_to_page_indices_pinned() (pinned CPU buffer,
non-blocking copy; no synchronous .cpu()).
- prefill.py: gate via should_async_state_payload() (last chunk, independent
supports_async_state_indices, kv_shard_size==1, CP guard); only the
_swa_payload request_window-is-None hot branch uses the pinned helper.
Final segment asserts async state has an indices_event. Other state
builders (mamba/full/DSA) stay synchronous.
- mooncake/conn.py: MooncakeKVSender advertises
supports_async_state_indices; the worker materializes pinned state tensors
to numpy only after indices_event.synchronize(), and fails loudly if a
tensor arrives without an event.
- tests: guard/event/materialize/pinned-body unit tests + CUDA ordering test
(skipped unless torch+CUDA+sglang are available).
No scheduler/transfer semantics change; rollback is the helper switch or
reverting this commit. 49 tests pass, 1 skipped locally.
2026-11-06 23:00:22 +08:00
ymk
fe9c6574fe
observability: trace KV transfer chunks end-to-end (M2)
...
Implements M2 of handoff/prefill-cpu-kv-overlap-design-20261106.md, behind
SGLANG_TRACE_MOONCAKE_DETAIL=1 (default off; reuses the existing trace stack).
- TransferKVChunk gains optional observation fields: chunk_seq, loop_seq,
send_kind, enqueue/last_enqueue monotonic+wall stamps, queue_wait_accum_ns,
queue_id, requeue_count, num_pages, approx_bytes, observe
- chunk_seq and enqueue timestamps are assigned at the real queue.put()
point (after every early return), so skipped paths cannot leave holes
- Mooncake transfer worker measures queue wait, source-event wait, indices
wait, KV/state/aux transfer, worker busy and residual; emits one
mooncake.kv_chunk span per attempt (+ queue/source/indices wait children)
and updates a per-room join summary
- request completion emits kv.request_join with chunk_count,
last_chunk_queue_wait/worker, final_exposed_ms, completion_poll_ms,
request_join_ms and first-enqueue-to-completion. First enqueue is stamped
by the Mooncake manager at the real queue.put(); no ReqTimeStats field is
added and the legacy queue_entry semantics are unchanged.
- nixl/mori add_transfer_request accept the observation kwarg (no-op there)
No scheduling/transfer semantics change; queue wait is excluded from worker
busy, and worker_busy excludes source/indices event waits. Unit tests: test_mooncake_chunk_detail.py (15 cases, including the
abort/worker interleaving and attempt-vs-total queue wait windows).
2026-11-06 20:01:56 +08:00
ymk
72e556ec81
observability: split prefill run_batch/process_result into host phases (M1)
...
Implements M1 of handoff/prefill-cpu-kv-overlap-design-20261106.md:
per-loop detail phases behind SGLANG_TRACE_LOOP_DETAIL=1 (requires
SGLANG_BATCH_STAGE_TIMING=1 + SGLANG_TRACE_LOOP_SPAN=1, only sampled loops).
- run_batch: early_send_host, resolve_inputs, model_forward_host,
copy_result_submit (+ run_residual_ms)
- process_result: copy_done_wait, auxiliary_and_logprob, per_request_result,
kv_state_payload/page_indices/sender_send by send_kind, report_stats
(+ process_residual_ms)
- war_barrier is measured as its own loop phase (outside run_batch's span)
- repeated/per-request phases are summed attributes; single phases are child
spans, so span count never grows with n_reqs/n_segments
- DP-skew join fields: pass_end_wall_ns, arrival_est_wall_ns = pass_end -
mlp_sync, previous-loop phase summary (sglang.prev_*, published only for
consecutive loop_seq; prev_detail_missing marks a gap)
No CUDA syncs added; when disabled every hook is a module-level bool check.
Unit tests: test_loop_detail_phases.py (10 cases).
2026-11-06 20:01:56 +08:00
minke.yu
23053730fc
observability: count SWA gate rejections per prefill admission
...
Review follow-up. match_kv_cache also runs on waiting-queue probes
(calc_priority), PD decode _match_prefix_and_lock and HiCache rematches.
Emitting the counter there counted the same gap once per scheduling round
and attributed "re-prefilled" tokens to decode engines.
- match_kv_cache now only records req.full_kv_hit_length (plus the match-time
gap); it no longer emits.
- the metric is emitted once in PrefillAdder._commit_prefill_admission, where
the accepted prefix is final after any host load-back.
- rename the request counter to swa_gate_reject_admissions_total and document
that a retracted and re-admitted request is counted again (it is re-prefilled).
- pre-create zero-valued children so gate-off deployments expose the series
before the first rejection.
- tests: repeated match does not emit, admission emits once, a retry counts
again, and host load-back closes the gap.
Not runtime verified (no GPU run on this build).
2026-11-06 13:18:07 +08:00
ymk
bf8cf7f1dc
hicache: pipeline prefetch/backup IO lanes + stage timing
...
- prefetch_io_aux_func: allow up to SGLANG_HICACHE_PREFETCH_IO_THREADS
transfers in flight (default 1 = legacy behaviour); PrefetchAck is still
emitted strictly FIFO so the per-rank ack stream (and the TP all_reduce in
prefetch_sync_thread_func) is unchanged.
- backup_thread_func: same pipelining for L2->L3 writes via
SGLANG_HICACHE_BACKUP_IO_THREADS.
- SGLANG_HICACHE_TIMING=1: emit [HC_TIMING] prefetch/backup queue, inflight,
p50/p95 per 100 ops (the prefetch/backup lanes were previously single
threaded; this is what caps L3<->L2 at ~1 op at a time).
2026-11-06 13:18:07 +08:00
minke.yu
c911d1b572
observability: expose SWA prefix-match gate rejections
...
The unified-cache SWA component gates prefix matching on a live sliding
window: when the window is missing, the match stops before an otherwise
available Full-KV prefix. The existing hit counters do not show the tokens
lost to that gate, which makes the gate-on/gate-off trade-off invisible.
Record the gap between the longest Full-KV match and the accepted prefix
(full_kv_hit_length - (device_prefix_len + host_hit_length)) on the Req and
export it as:
sglang:swa_gate_reject_tokens_total
sglang:swa_gate_reject_reqs_total
This is the token volume that will be re-prefilled because a component
validator (SWA window; Mamba where present) rejected the match. With the
gate enabled it measures the hit length lost to a missing window; with the
gate disabled it only captures the host-only / load-back-gap fallback.
Includes unit tests for the gap and the host-hit case. Not runtime verified
yet (no GPU run on this build).
2026-11-06 10:33:52 +08:00
minke.yu
145e05c44d
observability: fix scheduler no-batch accounting
2026-11-05 16:42:39 +08:00
minke.yu
5d5870f3ea
observability: add scheduler loop span and no-batch accounting
...
Adds an opt-in scheduler.loop span (SGLANG_TRACE_LOOP_SPAN) with recv /
get_next / run_batch / process_result / kv_send children. The span is only
created for sampled loops that have pending work; fully idle loops stay in
the counters so the 714 pass/s idle spin never floods Jaeger.
No-batch accounting is always on with SGLANG_BATCH_STAGE_TIMING:
- sglang:scheduler_no_batch_{seconds,total}{dp_rank,reason,forward_overlap}
- sglang:scheduler_dp_lockstep_idle_seconds_total{dp_rank,forward_overlap}
with the design's reason priority (result_backpressure, deferred_input,
waiting_bootstrap, waiting_prefetch, admission_blocked, waiting_pd,
fully_idle, no_ready_other) and local_batch_kind/global_outcome from the DP
sync. DP lockstep idle is counted separately from pure no-batch wait.
loop gpu_ms reuses the existing DeviceTimer intervals (no new CUDA events);
the forward.gpu lane is optional and approximate. Pass/recv spans become
children of the loop span when it is active.
Tests: test_loop_span_accounting (5) and the existing stage-timing suite (8)
pass.
2026-11-05 15:00:23 +08:00
minke.yu
309af6e6e1
observability: record prefetch ready latency on the L3 hit success path
...
_handle_prefetch_result() retires the operation with a direct
del ongoing_prefetch[request] instead of _retire_ongoing_prefetch(), so
the ready-latency hook only ever saw miss/revoke paths: with a cache-hit
workload storage_prefetch_ready_seconds{result="hit"} stayed at 0 even
though storage_prefetch_hit_tokens_total was moving ~312k tok/s.
Observe right after the del, classifying by loaded_from_storage.
2026-11-04 19:14:02 +08:00
minke.yu
1a274df27e
observability: fix prefetch ready latency to cover fast lookups
...
The first version stamped the request on its first in-flight
check_prefetch_progress() call. A fast L3 query can be drained by
_process_hicache_events before that check runs (the op is already gone
from ongoing_prefetch), so those requests produced no sample -- exactly
the common case.
Stamp at issue time in prefetch_from_storage() and observe once at
_retire_ongoing_prefetch(), classifying by operation.storage_hit_count.
Retire is the single exit for hit completion, below-threshold revoke,
anchor loss and host-capacity fallback, so hit and miss are both covered.
No behavior change; the hook is still behind SGLANG_BATCH_STAGE_TIMING.
2026-11-04 18:04:34 +08:00
minke.yu
cb7988f849
observability: measure storage prefetch ready latency (hit/miss)
...
Under --hicache-storage-prefetch-policy wait_complete a request is not
admitted until its L3 lookup resolves, and a miss pays the same query round
trip as a hit. The scheduler only exposed a skip counter, so the actual wait
was unknown.
Add sglang:storage_prefetch_ready_seconds{dp_rank, result=hit|miss}:
* stamp cache_request_handle on its first in-flight check,
* observe at the moment check_prefetch_progress() turns usable, split by
whether prefetch_loaded_tokens_by_reqid ended up > 0,
* clear the stamp on revoke / abort / accounting discard.
Buckets cover 1 ms .. 30 s. Gated by SGLANG_BATCH_STAGE_TIMING (off by
default). The measurement is ~2 dict ops + one observe per request, not per
pass.
Tests: test/registered/unit/observability/test_batch_telemetry_stage_timing.py
adds the hit/miss split assertion (9 tests, all passing).
2026-11-04 17:26:41 +08:00
minke.yu
0866012b80
observability: split scheduler recv_requests into pull/broadcast/unwrap
...
Runtime accounting on the prefill service (64 x 14.6k-token requests,
8-way concurrency) showed the non-forward wall time is dominated by the
scheduler receive path, not by radix match:
forward (extend+idle) 58.2% | get_next 6.9% | recv_requests 25.6%
process_batch_result 1.8% | run_batch overhead 0.9% | other ~6%
The existing pass span only covers get_next, so it cannot explain that
25.6%. Split it with the same opt-in gate:
* sglang:recv_requests_seconds{dp_rank, part} (total/pull/broadcast/
unwrap/mm), 5 new series per rank
* sglang:recv_requests_msgs_total, sglang:recv_requests_prompt_tokens_total
* OTLP span scheduler.recv_requests with the same breakdown, sampled
deterministically by a per-process counter (SGLANG_RECV_SPAN_SAMPLE,
default 0.05; SGLANG_RECV_SPAN=0 disables the span only)
Note on scope: tokenization itself does NOT run in the scheduler. The
tokenizer workers (--tokenizer-worker-num) produce TokenizedGenerateReqInput
and the scheduler receives the already-tokenized ids. What this measures is
ZMQ pull + decode, the cross-rank broadcast of the pickled objects, and
unwrap_pickle_fields() of the token-id buffers -- which is why large prompts
make recv_requests expensive.
Tests: test/registered/unit/observability/test_batch_telemetry_stage_timing.py
now also covers the recv histogram parts, counters, zero-message passes and
deterministic recv-span sampling (8 tests total, all passing).
2026-11-04 16:11:13 +08:00
minke.yu
85f4f9576c
scheduler: add opt-in per-pass prefill stage timing (SGLANG_BATCH_STAGE_TIMING)
...
Motivation: on the PD prefill node the wall-clock budget is dominated by
get_next_batch_to_run (~50% of the hour on 2026-11-02 16:00-17:00), and
forward_overlap=partial is only 21-87 s/h, i.e. it does not overlap the GPU
forward. We know the total but not which segment inside it costs: radix
match, PrefillAdder admission, the HiCache event collective, the bootstrap
all-reduce, or the DP metadata gather.
Measurement design (one sample per scheduling pass, including passes that
produce no batch; see handoff/prefill-batch-stage-timing-20261103.md v2/v3):
L1 7 per-pass histograms:
sglang:batch_prefill_{get_next,match,adder,hicache_events,
bootstrap,mlp_sync,other}_seconds
other = total - (match + adder + hicache + bootstrap + mlp_sync),
clamped to >= 0. Nested stages are not double counted.
L3 5 per-pass counters:
candidates / skipped_prefetch / admitted / resumed_chunk / new_chunk
L2 a dedicated per-pass span scheduler.prefill.pass (not the batch span),
emitted from end_pass() on every pass. Its sampling decision is a pure
function of mlp_sync_seq, so a sampled gather round is sampled on all
DP ranks. Idle ranks and ranks without a local batch still emit, which
is exactly what the cross-rank skew analysis needs. No links.
Hooks are installed in both scheduling loops:
* get_next_disagg_prefill_batch_to_run() for --disaggregation-mode prefill
(hicache / bootstrap / mlp_sync segments), and
* get_next_batch_to_run() for standalone prefill (hicache / mlp_sync),
so the instrumentation can be validated on ds41-prefill-dp4 without a decode
peer. match/adder/counters live in the shared get_new_batch_prefill path.
Semantics:
* match times only init_next_round_input; check_prefetch_progress counts as
skipped_prefetch, never as match.
* bootstrap and hicache_events are cross-rank sync points measured at the
outer call, not attributed to local CPU work.
* dp_gather_ms comes from the existing MLPSyncBatchInfo.sync_wait_seconds;
sglang:dp_attention_sync_wait_seconds is reused, no duplicate metric.
The value is cleared before each outer call and mlp_sync_seq only
increments when a gather actually ran, so stale values cannot leak.
* resumed vs new chunked requests are captured separately: had_chunked is
recorded at pass start (before process_prefill_chunk/admission can
consume the last chunk) and new_chunk from adder.new_chunked_req. This
fixes both the first-chunk and last-chunk mislabels.
Opt-in:
* SGLANG_BATCH_STAGE_TIMING=1 (default 0);
SGLANG_BATCH_PASS_SPAN=1 / SGLANG_BATCH_PASS_SPAN_SAMPLE=0.05.
* When off every hook is one module-level bool check; no logging, no CUDA
calls, no new collectives.
Tests: test/registered/unit/observability/test_batch_telemetry_stage_timing.py
covers first/middle/last chunk labelling, all-prefetch-skip passes without a
batch, stale-gather clearing, rank-deterministic pass-span sampling, and the
disabled no-op path. py_compile passes on all touched files.
2026-11-04 14:53:29 +08:00
minke.yu
2434d09848
observability: commit the in-image per-batch telemetry baseline (btelem4-6)
...
These three files were already running in the gist production image
(gist-nvlink-ee9170e7bb-pzcfix) but were never committed: the image was
produced by docker-commit with the btelem4/5/6 files overlaid on top of
the built source tree.
Port them so the tree matches the image before layering new instrumentation:
* observability/batch_telemetry.py: per-batch histograms, per-DP-step
shape histograms and the sampled OTLP span scheduler.batch.{prefill,decode}.
* managers/scheduler_components/metrics_reporter.py: observe_prefill/
observe_decode call sites and the batch-trace log suffix.
* managers/scheduler_components/dp_attn.py: observe_dp_step().
Content is identical to the image; only line endings were normalised to LF
(the image copy was mixed CRLF/LF). No behavior change when
SGLANG_BATCH_TELEMETRY is unset (default off).
2026-11-04 14:48:38 +08:00
Florian Wunderlich
d47a99b2ad
test: import SWA eviction helper directly
...
(cherry picked from commit 5729f61e175bde3ef2b67dc3a95abb1042f4bf1b)
2026-11-04 10:41:52 +08:00
Florian Wunderlich
2ba9b8431a
Fix SWA eviction behind the replay-trimmed prefix frontier
...
(cherry picked from commit f399d1e9a2bf7c33c7244ecf405e756292ddaf2b)
2026-11-04 10:41:52 +08:00
zicai.pu
ee9170e7bb
fix(mooncake): bind IntraNode batches to the bootstrap CUDA device
...
Sender and executor threads can enter synchronous IntraNode transfers with
another current CUDA device. Keep the explicit bootstrap ordinal, check it
against the published model device, and scope the native batch call to that
device while restoring the calling thread on success or failure.
Ordinary transports, force-TCP and non-CUDA devices keep their existing
path. Missing or inconsistent CUDA identity is rejected before native IO.
Add 13 CPU regressions importing the production wrapper, including real
constructor calls with a mocked SDK, concurrent thread-local device states,
error restoration and transport/device compatibility. All 13 pass; two
selected tests fail on the unchanged wrapper at behavior assertions. Native
and CUDA boundaries are mocked, so this does not certify physical CUDA IPC
or attribute the separate container-PID collision to this change.
2026-10-30 11:42:19 +08:00
zicai.pu
17505e6cf6
hicache: align prefetch and host restore boundaries
...
Use the request radix-key cap for L3 prefetch so bounded replay cannot
split a queried trailing-state checkpoint at the later rematch. Reject
Host-only SWA-gap candidates before they mask a complete ancestor, while
preserving device matching and the final load-back completeness guard.
Add CPU regressions that import the real scheduler, cache and TreeCore
methods. All 13 test methods pass; the same tests against the unmodified
scheduler/SWA code fail six methods (ten assertions including subtests).
The checks use synthetic cache state and controller boundaries, not GPU
or live Store execution.
2026-10-30 11:42:19 +08:00
zicai.pu
81f7398855
fix(hicache): reject unproven trailing restore endpoints
...
Do not invent shorter checkpoints when a trailing pool has no restorable-prefix inventory. Reject partial allocation and vote no common endpoint while preserving the collective shape and plain-KV contiguous behavior.
Add negative controls for missing inventory and retain pure-KV coverage. Validation: 33 cache/PP CPU tests passed with an external Mac native-hash import adapter using the repository Python hash oracle; repository pre-commit checks passed.
2026-10-30 11:42:19 +08:00
zicai.pu
0a56a5f6e7
fix(hicache): preserve sparse restore checkpoints under host pressure
...
Choose a common validated trailing-state endpoint before partial host allocation, retain that endpoint set across query synchronization, and size auxiliary staging from the selected length. Assumed trailing hits prove only their original endpoint.
Add controller/cache CPU regressions and a PP checkpoint-vote control. Validation: 14 targeted CPU tests and repository pre-commit checks passed.
2026-10-30 11:35:29 +08:00
minke.yu
7f6fe22445
mem_cache: fail-soft on SWA load-back gaps instead of asserting
...
With long prompts and L3 under pressure, a matched prefix can contain a node
whose SWA component has neither a device nor a host value (host tombstone
evicted between match and load-back). build_hicache_transfers asserted on
this and killed the prefill scheduler.
Add SWAComponent.has_load_back_gap() and have build_load_back_spec return an
empty transfer spec when a gap is found, so the caller backs off and the
request is recomputed. Measured on peak15-v6 (1M, cap 64): 0 scheduler
exceptions, hit 59.4%->94.0%, TTFT p95 47.0s->3.4s with the SWA prefix-match
gate disabled.
2026-10-26 13:23:17 +00:00
minke.yu
f6fc6ac3e7
mem_cache: allow disabling the DSv4.1 SWA prefix-match gate via env
...
The unified-cache SWA component gates prefix matching on a live SWA window,
so a long prefix only matches up to the last sliding window and the tail is
recomputed. Add SGLANG_DSV4_SWA_GATES_PREFIX_MATCH (default True) so the gate
can be turned off for measurement; the SWA window is then rebuilt/recovered
downstream as for the req-ring paths. Measured on 4xB300 DP4 at 160k prefix:
c32 hit tokens 40.6% -> ~100% (with HiCache L2/L3 enabled), needle test
(cold vs hit) identical.
2026-10-26 04:06:25 +00:00
minke.yu
0a3c370451
mooncake: probe pointers via cuda.bindings instead of ctypes libcudart
...
The ctypes-based probing loaded a second libcudart copy into the process,
which corrupted CUDA/runtime state and segfaulted the scheduler processes
shortly after registration (reproducible on prefill ranks, intermittent on
decode). cuda.bindings.runtime.cudaPointerGetAttributes is a properly
typed binding and boots cleanly on both roles.
2026-10-25 14:04:56 +00:00
minke.yu
b1b303c9f2
mooncake: support host-memory payloads with intra-node NVLink transport
...
IntraNodeNvlinkTransport can only register/reach device memory. sglang's
register list includes host regions (aux buffers, some state components);
one host region failed the whole registerLocalMemoryBatch and the engine
rolled back every region, leaving the segment descriptor empty so all KV
transfers failed with 'Requested address ... not found'.
When the intra-node NVLink transport is active (MC_INTRANODE_NVLINK etc.):
- register only device-memory regions (probe via cudaPointerGetAttributes)
- route transfer blocks whose local source is host memory over the ordered
per-endpoint zmq channel (new STATE_DATA message) instead of the engine;
the receiver validates and writes the block into its buffer, relying on
the same ordering guarantee as the aux TCP path
- force aux over the TCP socket path automatically
2026-10-25 14:04:52 +00:00
Liangsheng Yin
7d996300b9
[Perf] dsv41: Use the fused ratio-2 compressor in the prefill graph ( #46230 )
2026-10-23 21:33:42 -07:00
Liangsheng Yin
54df642fe6
[Perf] dsv41: Translate SWA locations once per prefill graph step ( #46229 )
2026-10-23 21:33:15 -07:00
Liangsheng Yin
564aec61c8
[Perf] dsv41: Feed wo_b the fused MXFP8 attention output in the prefill graph ( #46231 )
2026-10-23 21:31:33 -07:00
Yihao Wang and Claude Opus 5.5
a7183a8246
[Feat] spec: Let out-of-tree platforms provide EAGLE draft graph runners ( #45145 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-23 21:28:56 -07:00
Yihao Wang and Claude Opus 5.5
af3f9e4c8f
[Feat] spec: Let out-of-tree platforms provide their own speculative worker ( #45144 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-23 21:24:58 -07:00
Mick and Mick Qian
f09cec65e9
[Fix] diffusion/storage: Join uploads before request cleanup ( #46068 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
2026-10-24 12:19:10 +08:00
Mick and Mick Qian
f4ecaae888
[Fix] diffusion/mesh: Guard asynchronous job publication and defaults ( #46081 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
2026-10-24 12:18:25 +08:00
Mick and Mick Qian
cdab6c1bcc
[Fix] diffusion/loader: Preserve checkpoints on read failures ( #46095 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
2026-10-24 12:17:46 +08:00
f2c634961b
[Perf] diffusion: Project LTX-2 prompt K/V once per denoising loop ( #46105 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-24 12:17:04 +08:00
Liangsheng Yin
3b95120f00
[Fix] dsv41: Revert MegaGate routing in prefill ( #46242 )
2026-10-23 21:15:38 -07:00
55f23a8b59
[Fix] [AMD] dsv4.1: Fix decode graph capture with quantized low-ratio activations ( #46151 )
...
Co-authored-by: Kevin Mi <kevin.mi@radixark.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-23 20:55:13 -07:00
Liangsheng Yin
3c6a6c8068
[Perf] dsv41: Run MegaGate routing in prefill ( #46232 )
2026-10-23 20:31:50 -07:00
Liangsheng Yin
17d92ea671
[Feat] openai: Parse DeepSeek text for the native endpoints through deepseek-recipe ( #46219 )
2026-10-23 20:17:37 -07:00
Richard Wang and Mick Qian
7cefa60765
[Perf] diffusion: Wait on the socket when the scheduler is idle ( #46007 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
2026-10-24 11:02:24 +08:00
25ba24e9a7
[Perf] diffusion: Build one LTX-2.3 AdaLN row per sample when every token shares the sigma ( #46109 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-24 11:00:14 +08:00
42ac81c655
[Perf] diffusion: Route dense attention with a sub-tile side to FA4 on Blackwell ( #46107 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-24 10:59:25 +08:00
jambow0320 and Shangming Cai
de7055f596
[Refactor] pd: Consolidate sender and receiver implementations ( #41140 )
...
Co-authored-by: Shangming Cai <csmthu@gmail.com >
2026-10-24 10:50:10 +08:00
jacky.cheng
88ac7c02a6
[Fix] [AMD] diffusion: Fix the first-step denoise regression from compiling ENABLE_CK=0 BF16 attention ( #45838 )
2026-10-23 19:42:54 -07:00
Jackey Hua and hnyls2002
a77998e157
[Feat] openai: parse DeepSeek output through deepseek-recipe ( #45907 )
...
Co-authored-by: hnyls2002 <lsyincs@gmail.com >
2026-10-23 19:23:22 -07:00
Liangsheng Yin and kpham-sgl
79701a2549
[Perf] dsv41: Read Engram rows from peer device shards ( #46180 )
...
Co-authored-by: kpham-sgl <khoa.pham@radixark.ai >
2026-10-23 19:01:26 -07:00
Yihao Wang and Claude Opus 5
327c906764
[Perf] [XPU] diffusion: Avoid QKV copies in flash attention ( #41944 )
...
Co-authored-by: Claude Opus 5 <noreply@anthropic.com >
2026-10-24 09:33:26 +08:00
Yihao Wang and Claude Opus 5
5bef70c0a2
[Perf] flux2: Build the single-block output projection without a cat ( #41943 )
...
Co-authored-by: Claude Opus 5 <noreply@anthropic.com >
2026-10-24 09:33:26 +08:00
Mick and Mick Qian
1d4dcc81cc
[Perf] diffusion/registry: Reduce model discovery work at startup ( #46043 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
2026-10-24 09:30:04 +08:00
47c0791ad1
[Perf] diffusion: Write uncompressed PNG responses without Pillow's row filters ( #46099 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-24 09:28:55 +08:00
bbc77e7db4
[Perf] diffusion: Cut LTX-2.3 per-block elementwise work (contiguous AdaLN values, gate-folded masks, vectorized head gate) ( #46110 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-24 09:27:39 +08:00
4e2d5b17ad
[Perf] diffusion: Keep every component resident in auto mode when memory allows ( #45978 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-24 09:26:13 +08:00
be8600d02d
[Fix] diffusion: Step MOVA latents outside the serial-CFG branch ( #45993 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-24 09:23:54 +08:00
a08bb0ec9e
[Perf] diffusion: Run LTX-2 stage 2 on a premerged resident DiT ( #46108 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-24 09:23:08 +08:00
e66c9e981d
[Perf] diffusion: Turn on breakable CUDA graphs by default for LTX-2/LTX-2.3 two-stage serving ( #45977 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-24 09:22:06 +08:00
2e6d87e2ca
[Perf] diffusion: Serve FA4 head sizes on Blackwell instead of falling back to sm80 SDPA ( #46102 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-24 09:20:32 +08:00
386fa9ecfd
[Perf] diffusion: Drop LTX-2's all-valid prompt mask under lossless quality ( #46106 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-24 09:19:12 +08:00
5b8955e70f
[Perf] diffusion: Shard LTX-2 stage 2 over the CFG ranks as Ulysses ( #45965 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-24 09:17:41 +08:00
738e4535ea
[Perf] diffusion/qwen-image-2.1: Keep an image prefix's K/V in the packed buffer across steps ( #46101 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-24 09:14:10 +08:00
3712b5944d
[Perf] diffusion: Patchify Wan latents straight into tokens ( #46148 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-24 09:12:49 +08:00
Mick and Mick Qian
2d5ad3243f
[Fix] diffusion/http: Allow bounded graceful shutdown ( #46169 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
2026-10-24 09:11:15 +08:00
Jackey Hua and hnyls2002
e22a530859
[Feat] openai: Render DeepSeek-V4/V4.1 chat prompts with deepseek-recipe ( #45315 )
...
Co-authored-by: hnyls2002 <lsyincs@gmail.com >
2026-10-23 18:05:35 -07:00
weireweire and weireweire
49a2fbf817
[Fix] docker: Pin sgl-nvidia-mega to 3333e4c so the megakernel starts on main ( #46030 )
...
Co-authored-by: weireweire <20922698+weireweire@users.noreply.github.com >
2026-10-23 17:58:04 -07:00
Khoa Pham and Claude Fable 5.1
ee2dd727a3
[Perf] dsv41: Let Mega-mHC write the MegaMoE input under MegaGate ( #46163 )
...
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com >
2026-10-23 17:54:03 -07:00
Liangsheng Yin
79e5c4bb9b
[Perf] dsv41: Trim per-step DSpark decode overhead ( #46176 )
2026-10-23 17:24:06 -07:00
Qiaolin Yu
5994c5597d
[Perf] spec-decoding: fuse simulated speculative acceptance tensor preparation ( #41169 )
2026-10-23 17:15:21 -07:00
Baizhou Zhang and sid-rp
6a319e7b34
[Fix] muse: Support Responses namespace tools and defer_loading ( #44685 )
...
Co-authored-by: sid-rp <sidrp@meta.com >
2026-10-23 16:48:08 -07:00
0d592c475b
[Fix] rust-server: Keep a transfer backend's own bootstrap port on prefill ( #46066 )
...
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com >
Co-authored-by: shodoco <shodoco@users.noreply.github.com >
2026-10-23 16:47:14 -07:00
Cheng Wan
855c092deb
[Fix] inkling: Keep BF16 LM-head operands with FP32 output ( #45335 )
2026-10-23 15:51:44 -07:00
Ziang Li
197be2203a
[Doc] quantization: Link W4A16 design from environment reference ( #46170 )
2026-10-23 15:49:18 -07:00
Liangsheng Yin
f05ad5ec01
[Perf] logits: Cast bf16 logits into the fp32 buffer with a Triton kernel ( #46167 )
2026-10-23 15:06:09 -07:00
Rohan Varma and Jun Wan
15dc838f9b
[Fix] http_server: Apply SGLANG_TIMEOUT_KEEP_ALIVE to Granian HTTP/1.1 connections ( #45760 )
...
Signed-off-by: Rohan Varma <rohanv@nvidia.com >
Co-authored-by: Jun Wan <8323392+junwan01@users.noreply.github.com >
2026-10-23 14:43:54 -07:00
Baizhou Zhang
ea226cf782
[Feat] glm5.3flash: Support padded CP BCG and prefix replay with PD/DFlash2 ( #43774 )
2026-10-23 14:21:31 -07:00
Yuhao Yang and Claude Opus 5.5
5f7542d762
[Perf] dsv41: Build the MegaGate row masks once per forward ( #46071 )
...
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
2026-10-23 14:15:01 -07:00
27ac614f05
[Feat] parser: Add checkpoint-driven response-template adapters ( #40479 )
...
Signed-off-by: Yoni Gozlan <yonigozlan@users.noreply.github.com >
Co-authored-by: Yoni Gozlan <yonigozlan@users.noreply.github.com >
Co-authored-by: Xinyuan Tong <xinyuantong.cs@gmail.com >
Co-authored-by: Jimmy Shong <69131491+Jiminator@users.noreply.github.com >
2026-10-23 14:05:54 -07:00
em-dreyer
4542fc8f59
[Fix] mistral3: Subclass LlavaForConditionalGeneration instead of wrapping it ( #45656 )
2026-10-23 14:05:32 -07:00
Michael
194abb8e54
[CI] [AMD] diffusion: Drain the Cosmos Dreams decode stream after freeing its outputs ( #45983 )
2026-10-23 13:26:56 -07:00
1d605b7ece
[CI] vlm: Harden fixture cleanup after setup and shutdown failures ( #45461 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Alison Shao <a.shao@wustl.edu >
2026-10-23 13:20:06 -07:00
005333d029
[Refactor] radix-tree: Make each cache action a typed CacheAction implementation ( #45444 )
...
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com >
Co-authored-by: Jialin <Jialin@users.noreply.github.com >
2026-10-23 12:49:30 -07:00
Cheng Wan
d0022e1df3
[Fix] lora: Preserve MoE projection identity and routing lifetimes ( #45681 )
2026-10-23 12:32:55 -07:00
Cheng Wan
ac3db731b3
[Fix] moe: Preserve CUTLASS routing metadata and execution budgets ( #45917 )
2026-10-23 12:32:05 -07:00
Cheng Wan
8b9b992e41
[Perf] moe: Preserve Marlin workspace and route-count budgets ( #45462 )
2026-10-23 12:31:17 -07:00
0931dfedc9
[Fix] [AMD] dsa: Fix AITER prefill with an FP8 KV cache ( #39083 )
...
Co-authored-by: Pawel Kowalski <pawel.kowalski@silo.ai >
Co-authored-by: Thomas Wang <thomawan@amd.com >
2026-10-23 12:30:42 -07:00
Cheng Wan
e6451c1a95
[Fix] gguf: Preserve MoE routing, weighting and allocation budgets ( #45916 )
2026-10-23 12:29:51 -07:00
Cheng Wan
1340044e97
[Fix] deepseek-v4: Preserve decoder-tail token-count ownership ( #45915 )
2026-10-23 12:28:43 -07:00
9d7ad308bd
[Fix] [AMD] Quark: fix Quark block-FP8 loading in DeepSeek loader ( #40776 )
...
Co-authored-by: clintg6 <7388379+clintg6@users.noreply.github.com >
Co-authored-by: Thomas Wang <thomawan@amd.com >
Co-authored-by: Bowen Bao <bowenbao@amd.com >
Co-authored-by: Hubert Lu <Hubert.Lu@amd.com >
2026-10-23 12:11:21 -07:00
Alex Nails and Claude Opus 5.5
baf80c61b7
[Feat] platforms: Let out-of-tree platforms bind group devices and custom ops ( #45897 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-23 12:02:42 -07:00
Shangming Cai
1e894eb92d
[CI] disaggregation: Run the SWA decode radix cache test on Mooncake ( #46126 )
2026-10-24 01:41:39 +08:00
Peng Wu and Claude Opus 5.5
16aa816060
[Fix] disaggregation: Restart a lost prefill's heartbeat miss count ( #45922 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-24 01:26:10 +08:00
74c08f3dc5
[Feat] rust-server: Let a multimodal processor finalize item layout after caller hashes ( #45914 )
...
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com >
Co-authored-by: Lu Fang <30275821+houseroad@users.noreply.github.com >
2026-10-23 09:26:52 -07:00
30dc2f9430
[Feat] benchmark: Add --input-len-list to bench_one_batch_server for per-request input lengths ( #45982 )
...
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com >
Co-authored-by: wpc <wpc@users.noreply.github.com >
2026-10-23 09:04:59 -07:00
5a86d6ee17
[Fix] rust-server: Deliver scheduler-side aborts to Rust frontend clients ( #45899 )
...
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com >
Co-authored-by: shodoco <shodoco@users.noreply.github.com >
2026-10-23 09:04:54 -07:00
eb8ef80842
[Feat] rust-server: Serve /abort_request from the Rust frontend ( #45888 )
...
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com >
Co-authored-by: shodoco <shodoco@users.noreply.github.com >
2026-10-23 09:04:49 -07:00
Avaya Aggarwal
efd2ce8a9d
[Feat] comfyui: video-to-video input and prompt enhancement for server-mode nodes ( #43673 )
2026-10-23 23:09:30 +08:00
c677171c1d
[Feat] hicache: Support MLA DCP with HiCache L3(file/mooncake) ( #41187 )
...
Co-authored-by: Codex <codex@openai.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: alphabetc1 <2508695655@qq.com >
Co-authored-by: Venkat Raman <venkat.raman@verda.com >
2026-10-23 23:09:04 +08:00
02b983603d
[Fix] diffusion: Keep dynamic LoRA deltas on fused linear-GELU sites ( #45943 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-23 21:19:53 +08:00
0e34962884
[Perf] diffusion: Cut LTX-2 denoising-step host overhead ( #45966 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-23 21:18:57 +08:00
82ff16dad0
[Perf] diffusion: Skip the residency calibration probe in auto mode until a planner consumes it ( #45979 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-23 21:18:06 +08:00
3e5f5505ed
[Perf] diffusion: Serve image responses from in-memory encoded bytes ( #45987 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-23 21:16:44 +08:00
2fbd094a8c
[Fix] diffusion: Offset LTX-2.3 BCG video RoPE by the SP shard start ( #45995 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-23 21:14:37 +08:00
pllimax
9b46f27090
[Fix] [NPU] tests: Restore evalscope GSM8K baseline via NPUGSM8KMixin backend ( #45732 )
2026-10-23 21:06:21 +08:00
shun8686
2a46e32fbc
[CI] [NPU] precise-test: skip recommend-tests-from-coverage when no NPU paths changed ( #46055 )
2026-10-23 21:01:42 +08:00
86933b1f33
[Fix] deps: Pin python-dateutil<3 to keep botocore resolvable ( #45857 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Po-Han Huang <pohanh@nvidia.com >
2026-10-23 20:57:44 +08:00
shun8686
0347b2bb4e
[CI] [NPU] precise-test: gate precision on PR judgeability, not bucket count ( #46042 )
2026-10-23 20:41:22 +08:00
Jimmy Shong
4b1941e889
[CI] diffusion: Run the FLUX.2 XPU norm-fusion test on the XPU suite only ( #46040 )
2026-10-23 19:54:03 +08:00
chuyeh and Cursor
f967418a35
[Doc] cookbook: Qwen3.5 MI355X FP8 agentic adds scheduler/tokenizer tuning from the InferenceX recipe ( #46032 )
...
Co-authored-by: Cursor <cursoragent@cursor.com >
2026-10-23 19:12:05 +08:00
shun8686
535aeae2a7
[CI] [NPU] precise-test: run only the recommended multimodal_gen cases ( #46014 )
2026-10-23 18:06:58 +08:00
chuyeh and Cursor
69f830756e
[Doc] cookbook: Qwen3.5 MI355X FP8 agentic uses HiCache write_through on the v0.5.22 image ( #46017 )
...
Co-authored-by: Cursor <cursoragent@cursor.com >
2026-10-23 17:47:52 +08:00
Sage
611a703061
[Chore] api: add fields to generate request ( #45164 )
...
Signed-off-by: Sage Ahrac <sagiahrak@gmail.com >
2026-10-23 17:10:23 +08:00
494eeecb6c
[Fix] diffusion: Keep the linear-GELU verdict cache out of torch.compile traces ( #45944 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-23 17:01:51 +08:00
f9129915a3
[Fix] diffusion: Give every SP rank the same Wan TI2V timestep ( #45994 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-23 17:00:40 +08:00
Xinyuan Tong
8b9f72417f
[Feat] Inkling: render and parse chat via tml-renderers ( #45340 )
2026-10-23 16:43:38 +08:00
Michael
bb72b47daf
[Test] [AMD] deepseek-v4.1: Give the encoder-replay test's request window a zero_row ( #45728 )
2026-10-23 01:38:54 -07:00
Shuwen Wang
77bc8cab18
[Fix] hicache/pd: Promise prefill only the L3 span the decode prefetch takes ( #43548 )
2026-10-23 16:32:41 +08:00
Michael
7d1a16d3eb
[CI] [AMD] diffusion: Skip the LTX-2.5 diffusion decoder case on the 2-GPU lane ( #45861 )
2026-10-23 01:28:12 -07:00
2b8e4cde95
[Doc] [CPU] Cookbook: Update to the cookbook for models supported by Intel Xeon CPUs ( #40092 )
...
Co-authored-by: Niu Ziru <niuziru@a4bf018d3341.jf.intel.com >
Co-authored-by: Niu Ziru <niuziru@gnr799217.jf.intel.com >
2026-10-23 16:22:50 +08:00
Avaya Aggarwal
4b273537de
[Fix] diffusion: runtime utils bug fixes ( #44887 )
2026-10-23 16:08:13 +08:00
Po-Han Huang (NVIDIA) and Claude Opus 5.5
02d2445138
[Doc] cookbook: Refresh SemiAnalysis AgentX launches with seven open InferenceX PRs ( #46005 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-23 16:04:01 +08:00
shun8686
472208fec0
[CI] [NPU] precise-test: recommend multimodal_gen cases at test-case granularity ( #45955 )
2026-10-23 15:57:31 +08:00
Yao Matrix and Alex Nails
6d6c9cd4ae
[Fix] offloader: Preserve cached weights during CPU offload ( #44139 )
...
Co-authored-by: Alex Nails <alex.nails@radixark.ai >
2026-10-23 00:50:24 -07:00
Yihao Wang and Claude Opus 5
6f466f655a
[Perf] [XPU] diffusion/flux2: Fuse bit-exact LayerNorm and gated residual norm ( #41942 )
...
Co-authored-by: Claude Opus 5 <noreply@anthropic.com >
2026-10-23 15:38:12 +08:00
8708d2ee99
[Fix] diffusion/ltx: Keep tiled VAE decode off the sharded convs under parallel decode ( #45942 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-23 15:32:03 +08:00
4f47c48b33
[Fix] diffusion: Keep the N-rank IPC all-to-all's flags and staging out of expandable segments ( #45948 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-23 15:31:19 +08:00
1cd0ebb3d3
[Perf] diffusion: Remove per-step and per-request host syncs in Cosmos3 and Qwen-Image-2.1 ( #45967 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-23 15:30:09 +08:00
Avaya Aggarwal
4da6f585f3
[Fix] diffusion: OpenAI API hardening ( #43585 )
2026-10-23 15:22:00 +08:00
Shunkangz and Zhangheng
79ab8f6d7c
[Feat] KVShard: Support kv cache sharding for MTP ( #40929 )
...
Co-authored-by: Zhangheng <hzh0425@apache.org >
2026-10-23 15:20:19 +08:00
Banruo Liu and Claude Opus 5.5
6e667c4310
[Fix] disaggregation: Poison the deferred abort ack when a NIXL or Mori write fails ( #45973 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-23 14:47:59 +08:00
Liangsheng Yin
6c844eb12e
[Perf] dsv4: Replay the V4.1 prefill graph past 16K contexts on SM90 ( #45941 )
2026-10-22 23:47:17 -07:00
ashwini rathi
6dd0dfabf0
[CI] [Intel] docker/xpu: Tag nightly images with main-whl and main-main kernel variant ( #45854 )
2026-10-23 14:44:55 +08:00
Jimmy Shong
2be6577ba9
[Doc] beam: Offer Python and Docker commands in the cookbook ( #45991 )
2026-10-22 23:39:10 -07:00
Chetan Kumar Verma and Claude Opus 5.5
76c382e80f
[Perf] [XPU] diffusion: Use the AOT SYCL fused residual-norm-scale-shift kernel ( #41401 )
...
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
2026-10-23 14:32:25 +08:00
Ziru and Niu Ziru
904691e0b5
[Feat] [CPU] model: Support Cosmos3 series models on CPU ( #45478 )
...
Co-authored-by: Niu Ziru <niuziru@gnr799217.jf.intel.com >
2026-10-23 14:24:28 +08:00
vikram singh shekhawat
3f36fc0833
[Fix] [XPU] runtime: Fix XPU rotary embeddings and tokenizer padding ( #41213 )
2026-10-23 14:09:59 +08:00
Liangsheng Yin
0e338777b0
[Perf] dsv4: Let the fused MXFP8 producers run inside the breakable prefill graph ( #45980 )
2026-10-22 23:00:10 -07:00
Pinjie Xu
7c680a0d45
[Perf] diffusion/cosmos3: optimize v2v preprocess performance ( #45398 )
...
Signed-off-by: pinjie <pinjiex@nvidia.com >
2026-10-23 13:57:38 +08:00
Michael
7baacd1d1b
[CI] [AMD] docker: Add a build-only dispatch option to the ROCm nightly image workflows ( #45882 )
2026-10-23 13:41:02 +08:00
Khoa Pham
c7994e7076
[Fix] dsv41: Keep MegaGate out of prefill graphs by gating on the forward mode ( #45969 )
2026-10-22 22:11:18 -07:00
Khoa Pham and hnyls2002
5bdbec0d48
[Perf] dsv41: Drop the MegaGate padded-row copy on sgl-deep-gemm 0.2.3.post2 ( #45933 )
...
Co-authored-by: hnyls2002 <lsyincs@gmail.com >
2026-10-22 22:10:57 -07:00
e83f585ae6
[Perf] dsv41: Serve token-interleaved V4.1 KV caches with sgl-flash-mla ( #45893 )
...
Co-authored-by: kpham-sgl <khoa.pham@radixark.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-22 22:02:49 -07:00
Liangsheng Yin
49933325cc
[Perf] dspark: Combine the draft's mHC streams from bf16 without an fp32 copy ( #45964 )
2026-10-22 21:44:01 -07:00
Mick
f4a2df3dac
[CI] diffusion: Calibrate single-GPU shard scheduling from CI ( #45831 )
2026-10-23 12:03:24 +08:00
Avaya Aggarwal and niehen6174
fa628c5359
[Fix] comfyui: Fix GGUF detection, import guard and VRAM accounting in SGLDiffusion plugin ( #43536 )
...
Co-authored-by: niehen6174 <1639206518@qq.com >
2026-10-23 11:58:23 +08:00
em-dreyer and Claude Opus 5.5
79eb10e182
[CI] [NVIDIA] gb300: Restore the aarch64 rust-ext cache on ARM runners ( #45920 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-22 20:01:42 -07:00
Wookje Han
f2873f1613
[Perf] [NVIDIA] muse-spark-open: Fuse pre-expert RMSNorm with static-scale NVFP4 quantization ( #44554 )
2026-10-22 20:00:37 -07:00
Baizhou Zhang
698651acd0
[Feat] HiCache: Support fused DFLASH KV in unified hybrid SWA pools ( #45700 )
2026-10-22 20:00:28 -07:00
chenyang08056032
ce2261863f
[CI] [NPU] precise-test: add empty-recommendation skip-all rule, slim baseline artifacts, and move setup-covstub off the NPU pool ( #45571 )
2026-10-23 10:52:14 +08:00
Liangsheng Yin
a4838160b3
[Perf] dsv4: Write V4.1 K rows with one warp per token in the fused k-norm-rope store ( #45934 )
2026-10-22 19:50:50 -07:00
Liangsheng Yin
c4fa35b829
[Perf] dsv4: Map V4.1 prefill index selections to pool slots in one kernel ( #45938 )
2026-10-22 19:50:31 -07:00
Liangsheng Yin
2da6c0c237
[Perf] dsv4: Combine the final mHC streams from bf16 without an fp32 copy ( #45940 )
2026-10-22 19:43:46 -07:00
Mick and Mick Qian
363e6c6b4d
[Feat] diffusion/disaggregation: split FLUX.3 prompt and image encoders ( #45309 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
2026-10-23 10:35:41 +08:00
Lianmin Zheng
5daa1352ba
[CI] tests: Retire test/srt and reject legacy test paths ( #42567 )
2026-10-22 19:15:55 -07:00
Khoa Pham and Claude Opus 5.5
e5dc0c363a
[Perf] dsv41: Integrate DeepGEMM MegaGate routing ( #43262 )
...
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
2026-10-22 19:08:39 -07:00
YAMY and Oasis-Git
c82f7141ad
[Perf] [NVIDIA] dspark: Move the host work between DSv4.1 decode graphs into the graphs for the megakernel path ( #45450 )
...
Co-authored-by: Oasis-Git <ayw.sirius19@gmail.com >
2026-10-22 19:08:06 -07:00
metamergebot and Jialin
94e4611eac
[Refactor] unified-cache: Factor match validators and match completion into helpers ( #45714 )
...
Co-authored-by: Jialin <Jialin@users.noreply.github.com >
2026-10-22 18:58:15 -07:00
Xinyuan Tong
92e57d5de7
[Perf] request_logger: skip per-token watermark redaction walk ( #44692 )
2026-10-22 18:57:19 -07:00
Liangsheng Yin
b588aa39e9
[Perf] dsv4: Optionally stage V4.1 engram rows from the shared host table on the scheduler ( #45921 )
2026-10-22 18:35:55 -07:00
zijiexia and Claude Opus 5.5
f36ee5de93
[Doc] Cookbook: add MiniMax-M3.1 ( #45924 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-23 09:07:35 +08:00
26396bf566
[Fix] [NVIDIA] deepseek-v4.1: Fix the breakable prefill CUDA graph in the late-layer tail on SM90 ( #45407 )
...
Co-authored-by: JohnZhao01 <325926370+JohnZhao01@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
Co-authored-by: hnyls2002 <lsyincs@gmail.com >
2026-10-22 17:57:16 -07:00
Michael
a95ae5f1c5
[CI] [AMD] docker: Start the ROCm 10 nightly at 10:00 UTC and ROCm 7.2 at 11:00 UTC ( #45848 )
2026-10-23 08:52:06 +08:00
Liangsheng Yin
97b79f0796
[Fix] prefill-graph: Keep every DP rank eager when the replay tail captures behind rank barriers ( #45881 )
2026-10-22 17:36:00 -07:00
Liangsheng Yin
9c7dd3c45f
[Perf] dsv4: Dequantize each shared sparse-prefill KV source once per forward ( #45909 )
2026-10-22 17:08:40 -07:00
Liangsheng Yin
ae4258715c
[Perf] dsv4: Keep V4.1 image rows inside the Engram gate kernel ( #45904 )
2026-10-22 17:05:01 -07:00
HEJIAN SANG and Claude Opus 5.5
77f90ade97
[Fix] weight_checker: Make the checksum device-agnostic ( #45918 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-22 16:57:49 -07:00
bb9e9d5632
[Perf] mlp: Fuse dense SwiGLU with block-FP8 quantization on decode ( #45219 )
...
Co-authored-by: Jayden Yu <jaydenyu@fb.com >
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com >
2026-10-23 07:49:39 +08:00
Chunan Zeng and Claude Opus 5.5
5849ff5337
[Fix] [NVIDIA] jit: Build sm_100f on CC 10.7 until the bundled CUTLASS supports SM107 ( #45911 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-22 16:42:00 -07:00
Baizhou Zhang
8d47d515e4
[CI] deepgemm: Remove obsolete ref_mhc preflight check ( #45910 )
2026-10-22 16:27:40 -07:00
Michael
035ed73b9d
[Fix] [AMD] kernels: Import the HIP launch helpers from jit.utils.hipcc in smallm_fp8_blockscale_gfx950 ( #45676 )
2026-10-22 16:19:34 -07:00
Alex Nails
c394a90c2b
[Perf] gemma4: Capture single-step frozen-KV drafts ( #45771 )
2026-10-22 15:37:19 -07:00
Liangsheng Yin
f9b7225f2e
[Perf] dsv4: Use the fused MXFP8 producers in eager prefill ( #45903 )
2026-10-22 15:29:09 -07:00
Liangsheng Yin
9c0f6d6a61
[Perf] dsv4: Write the MegaAttention output straight into the prefill graph buffer ( #45901 )
2026-10-22 14:56:08 -07:00
Liangsheng Yin
818d7932d7
[Perf] dsv4: Drop local references that keep residual-stream tensors alive through the prefill step ( #45900 )
2026-10-22 14:55:49 -07:00
3b0003dac1
[Fix] HiCache: stale host KV from write-through copies racing the in-flight forward ( #42701 )
...
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
Co-authored-by: Zhiqiang Xie <xiezhq@stanford.edu >
2026-10-22 14:46:51 -07:00
Alison Shao
a904e4d31a
[Fix] ci: Reduce runner utilization scan overhead and bound API retries ( #45873 )
2026-10-22 14:34:28 -07:00
Liangsheng Yin
840a6313c9
[Perf] dsv4: Score the V4.1 index logits in bf16 on SM100 ( #45880 )
2026-10-22 14:29:08 -07:00
25ad569ff9
[Feat] speculative: Seed rejection-sampling draft proposals for deterministic inference ( #33395 )
...
Co-authored-by: Ajit Mistry <ajmistry@nvidia.com >
Co-authored-by: Ajit Mistry <ajmistry@amistry.nvidia.com >
Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com >
Co-authored-by: Po-Han Huang <pohanh@nvidia.com >
2026-10-22 14:04:32 -07:00
Alex and Jimmy Shong
e5db41b329
[Fix] nemotron_vl: Pad visual spans using processor offsets ( #44581 )
...
Co-authored-by: Jimmy Shong <69131491+Jiminator@users.noreply.github.com >
2026-10-22 13:58:47 -07:00
786cdfc040
[Fix] nccl: Disable graph buffer registration whenever a captured graph can hold a collective ( #43216 )
...
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
Co-authored-by: Jimmy Shong <69131491+Jiminator@users.noreply.github.com >
2026-10-22 13:56:19 -07:00
Yanbin Jiang
13999673ef
[Feat] lora: Add plan-driven dense and MoE LoRA engines ( #45892 )
2026-10-22 13:40:56 -07:00
Shunkangz
154e29c1ec
[Perf] CP: QSA Context Parallel Optimization ( #43792 )
2026-10-22 13:33:04 -07:00
Liangsheng Yin
ba83415eea
[Perf] engram: Read host-table rows in address order on GPUs without host page tables ( #45879 )
2026-10-22 13:13:43 -07:00
bf0e193bfd
[Refactor] radix-tree: Move insert overlap handling into a resumable OVERLAP phase ( #45702 )
...
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com >
Co-authored-by: Jialin <Jialin@users.noreply.github.com >
2026-10-22 13:12:37 -07:00
aae2145eb2
[Feat] radix-tree: Report the last visited node and a shorter FULL device match in match results ( #45703 )
...
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com >
Co-authored-by: Jialin <Jialin@users.noreply.github.com >
2026-10-22 13:11:33 -07:00
357323e912
[Refactor] hicache: Route buffer-mode backup admission keys and FULL transfers through pipeline hooks ( #45701 )
...
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com >
Co-authored-by: Jialin <Jialin@users.noreply.github.com >
2026-10-22 13:10:18 -07:00
Liangsheng Yin
4f96807ea8
[CI] radix-cache: Build unified radix cache test states through production paths ( #45334 )
2026-10-22 13:04:23 -07:00
YAMY
d67d1ae02c
[Fix] logits: Preserve lm_head precision with shared buffers ( #45855 )
2026-10-22 12:46:27 -07:00
Liangsheng Yin
0d6b283d7e
[Refactor] dsv4: Keep only the two-level V4.1 indexer in the prefill graph ( #45783 )
2026-10-22 12:26:43 -07:00
WenhaoZhang
8d8916815b
[Fix] ci: Add pytest entry point for Muse Spark router tests ( #45843 )
2026-10-22 11:39:34 -07:00
Zhangheng
913e91ed51
[Fix] CI: add missing entry point to Muse Spark router test ( #45845 )
2026-10-23 00:45:41 +08:00
Aarne Talman
c80285ca9a
[Fix] [AMD] deterministic: Refuse RCCL all-reduce in deterministic mode above TP2 ( #43809 )
...
Signed-off-by: Aarne Talman <aarne.talman@amd.com >
2026-10-22 09:31:25 -07:00
Shunkangz and Claude Opus 5.5
cc811ff0c3
[Feat] KVShard: Add attention and indexer support for page-sharded DSA caches ( #44280 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-23 00:16:35 +08:00
Shuwen Wang
8cd3c494a3
[Fix] mem_cache: Skip linker L3 lookup on PD decode ( #45538 )
2026-10-22 22:36:32 +08:00
Wenbo Ji 嵇文博
afd0d7b47b
[Fix] diffusion/flux: use true_cfg_scale as the FLUX CFG weight ( #44260 )
2026-10-22 21:30:45 +08:00
Mick
0f7d3a0451
[CI] diffusion: target thirty-minute CI shards ( #45761 )
2026-10-22 21:19:42 +08:00
Sun and Mick
dcf05724f7
[Perf] diffusion: accumulate chunked LoRA merges with addmm ( #37620 )
...
Co-authored-by: Mick <mickjagger19@icloud.com >
2026-10-22 21:18:39 +08:00
trebladev
698fa949ec
[Perf] diffusion: fuse FLUX3 SwiGLU with rowwise FP8 quantization ( #42758 )
2026-10-22 21:17:56 +08:00
Song Bian and kernel-design-agents
635836129b
[Perf] [KDA] muse_spark: use BLOCK_M 240 for BF16 MegaMoE prefill shards ( #45637 )
...
Co-authored-by: kernel-design-agents <kernel-design-agents@users.noreply.github.com >
2026-10-22 20:57:30 +08:00
gjsheu
a9902f47e3
[Fix] [NPU] qwen: adapt pcp-v2 ( #41564 )
2026-10-22 20:54:11 +08:00
Song Bian and kernel-design-agents
851419717a
[Perf] [KDA] muse_spark: Run the one-token router GEMV on CuTe DSL ( #45508 )
...
Co-authored-by: kernel-design-agents <kernel-design-agents@users.noreply.github.com >
2026-10-22 20:49:29 +08:00
Trevor Morris
91fc46a017
[Fix] [NVIDIA] docker: Disable torch 2.15 nccl2 in CUDA 13.4 container to fix EPLB rebalance crash ( #45456 )
2026-10-22 20:46:32 +08:00
Song Bian and kernel-design-agents
0ec3fee429
[Perf] [KDA] cookbook: enable custom all-reduce for GB300 BF16 Muse Spark ( #45429 )
...
Co-authored-by: kernel-design-agents <kernel-design-agents@users.noreply.github.com >
2026-10-22 20:46:28 +08:00
Song Bian and kernel-design-agents
e43f6128a0
[Fix] [KDA] cookbook: add MoE runner backend to Muse Spark BF16 recipes ( #45422 )
...
Co-authored-by: kernel-design-agents <kernel-design-agents@users.noreply.github.com >
2026-10-22 20:38:35 +08:00
Jzz1943
f4ccfd4183
[Perf] moe: Add H200 block-FP8 Triton configurations ( #45591 )
2026-10-22 20:37:54 +08:00
Monishver and Xiaoyu Zhang
178febef5d
[Perf] qknorm: run head_dim-128 rows two per warp with 16B lanes, bit-identical ( #45214 )
...
Signed-off-by: Monishver Chandrasekaran <monishverchandrasekaran@gmail.com >
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com >
2026-10-22 20:34:16 +08:00
81841d6bdc
[Fix] hicache: Fail fast on a null pointer from the Mooncake host allocator ( #40565 )
...
Co-authored-by: huangtingwei <141888744+huangtingwei9988@users.noreply.github.com >
Co-authored-by: Teng Ma <sima.mt@alibaba-inc.com >
2026-10-22 20:12:25 +08:00
Jzz1943
8a6770d566
[Fix] NIXL: Preserve explicit UCX network device selection ( #45722 )
2026-10-22 20:07:27 +08:00
weireweire and weireweire
96f013b452
[Doc] Cookbook: update GB300 DSV4.1 disagg AgentX launches from InferenceX #4303 ( #45495 )
...
Co-authored-by: weireweire <20922698+weireweire@users.noreply.github.com >
2026-10-22 19:18:55 +08:00
Alex Nails and Claude Opus 5.5
d0a7565436
[Fix] diffusion: Drop stale mismatch_msg from FLUX 3 video VAE gate call ( #45778 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-22 01:31:54 -07:00
Liangsheng Yin
3f6c313389
[Fix] diffusion: Restore the kernel registry import after #41894 ( #45775 )
2026-10-22 01:25:13 -07:00
ashwini rathi
5c2cb60790
[Fix] [Intel] diffusion: Restore copy offload for FLUX.2-dev nightly ( #45725 )
2026-10-22 16:24:41 +08:00
Michael
0a87f163dc
[Test] [AMD] deepseek-v4.1: Fix the HIP breakable-graph unit tests for the default prefill context limit and the DSpark draft ring ( #45507 )
2026-10-22 01:14:14 -07:00
kangwangamd
1c967f5043
[Fix] [AMD] test: Recalibrate MiniMax-M2.7 GSM8K threshold 0.93->0.91 for MI300 runners ( #34241 )
2026-10-22 01:08:54 -07:00
jacky.cheng
b5137693a5
[Fix] [AMD] diffusion: Enable --quantization fp8 on gfx1250 (torch.compile lowering + per-channel weight scales) ( #45304 )
2026-10-22 01:05:36 -07:00
WenhaoZhang and Mick Qian
24e45961c6
[Perf] diffusion: fuse QKV norm + RoPE, cached RoPE tables for FLUX 3 video VAE ( #41894 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
2026-10-22 15:44:54 +08:00
James Xu and Cursor
c64886f9e3
[Perf] diffusion: fuse Wan VAE DupUp3D shortcut add ( #32860 )
...
Co-authored-by: Cursor <cursoragent@cursor.com >
2026-10-22 15:38:51 +08:00
Ankit Dhall and Ankit Dhall
f727076ed0
[Feat] diffusion: support wan-animate-2 distilled model ( #44085 )
...
Co-authored-by: Ankit Dhall <8938083+ankitdhall@users.noreply.github.com >
2026-10-22 15:37:08 +08:00
Shuwen Wang
9d5386fd13
[Perf] dsv41: Fuse request-window layout construction ( #45726 )
2026-10-22 00:13:33 -07:00
shun8686
33a0734ca1
[CI] [NPU] precise-test: collect multimodal_gen coverage per test case ( #45553 )
2026-10-22 15:11:47 +08:00
黄孝君
569dda1e61
[CI] [NPU] npu: Update SGLang kernel tag to 2026.9.0.post10 ( #45288 )
2026-10-22 14:55:56 +08:00
Baizhou Zhang
779adb4b82
[Perf] cuda-graph: Raise MLA prefill capture default to chunked size capped at 8192 ( #45565 )
2026-10-21 23:49:25 -07:00
f68e82e7bc
[Fix] epd: Accept tokens-in requests on scheduler-side transfer backends ( #41376 )
...
Signed-off-by: Krishnan Prashanth <kprashanth@nvidia.com >
Co-authored-by: Shangming Cai <csmthu@gmail.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-22 14:45:38 +08:00
Liangsheng Yin
82f7249ce6
[Perf] dsv4: Gather V4.1 index-K slots once per prefill graph replay ( #45741 )
2026-10-21 23:34:17 -07:00
Ziyi Xu and Claude Opus 5.5
a56024c267
[CI] flash-mla: Add the sgl-flash-mla release workflow ( #45651 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-21 23:34:14 -07:00
Shangming Cai
9569a36561
[Feat] scheduler: Gate --prefill-decode-interval on running batch size ( #44001 )
2026-10-22 14:30:12 +08:00
Richard Wang
ce66b3ab59
[Feat] systemone: serve Kev pointer head decision models ( #45413 )
2026-10-22 14:16:52 +08:00
Po-Han Huang (NVIDIA) and Claude Opus 5.5
12370da115
[Doc] cookbook: Refresh SemiAnalysis AgentX launches with five open InferenceX PRs ( #45749 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-22 14:15:19 +08:00
Shangming Cai
a6a6beffac
[Refactor] disaggregation: Share the flag all-reduce and host-destination predicate in the decode transfer queue ( #45641 )
2026-10-22 13:42:55 +08:00
WenhaoZhang and Mick
d9ce511646
[Feat] diffusion: support FastH3 Omni (Ref2VA 8-step PDD) ( #45530 )
...
Co-authored-by: Mick <mickjagger19@icloud.com >
2026-10-22 13:07:13 +08:00
Mick and Mick Qian
3472e16f96
[Refactor] diffusion: remove the deprecated kitchen_int8 alias ( #45706 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
2026-10-22 13:06:07 +08:00
Shuwen Wang
80263b5580
[Fix] sgl-router: Do not abort forwarded requests on reqwest's default 30 s TCP_USER_TIMEOUT ( #45696 )
2026-10-22 12:43:54 +08:00
zijie and Zijie Chen
f91b81b82a
[Doc] [AMD] cookbook: Add Muse Spark Open MI355X AgentX recipe ( #45689 )
...
Co-authored-by: Zijie Chen <300606707+zijiecode@users.noreply.github.com >
2026-10-21 21:12:52 -07:00
Apeiria and Rain Jiang
8d7df07a56
[Feat] rust-server: Add HiCache storage management endpoints ( #44726 )
...
Signed-off-by: 0130w <1157374677@qq.com >
Co-authored-by: Rain Jiang <96632942+rainj-me@users.noreply.github.com >
2026-10-22 04:01:35 +00:00
Liangsheng Yin
17babd67b4
[Fix] dsv4: Never read attention metadata for idle SWA stores ( #45699 )
2026-10-21 20:56:40 -07:00
WenhaoZhang
45de4adde9
[Feat] diffusion: Add worker-side ComfyUI compatibility patches ( #44841 )
2026-10-21 20:41:51 -07:00
Thomas Wang
9ee9858805
[Doc] [AMD] dsv41: Update amd cookbook 1021 ( #45693 )
2026-10-21 20:33:34 -07:00
Khoa Pham
f1e26dc658
[Perf] logits: Write lm_head logits straight into the fp32 logits buffer ( #45631 )
2026-10-21 20:32:16 -07:00
WenhaoZhang
1962cbeebe
[Perf] diffusion: convrot_int8 comfy_kitchen issues one GEMM per layer by default ( #44179 )
2026-10-22 11:30:32 +08:00
Viraat Chandra and Claude Opus 5.5
5f62e23d30
[Doc] cookbook: Add Muse Spark Open GB300 Dynamo + SGLang deployments measured with AA AgentPerf ( #45629 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-22 11:24:58 +08:00
Shuwen Wang
6929afb528
[Fix] hisparse: Keep PD destination host pages until the decode KV release ( #45108 )
2026-10-22 11:19:21 +08:00
8dc7f98005
[Perf] pd: Send prefill KV chunks without a host sync on the scheduler thread ( #45492 )
...
Co-authored-by: alphabetc1 <2508695655@qq.com >
Co-authored-by: Shuwen Wang <47200617+alphabetc1@users.noreply.github.com >
2026-10-21 20:11:16 -07:00
3468a65b51
[CI] labeler: Add a mooncake PR label ( #45578 )
...
Co-authored-by: Cursor Agent <cursoragent@cursor.com >
Co-authored-by: Teng Ma <stmatengss@users.noreply.github.com >
2026-10-22 11:09:48 +08:00
Mick and Mick Qian
780a5b1acc
[Doc] diffusion: document prompt enhancer tasks and link Cosmos recipes ( #45705 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
2026-10-22 10:28:58 +08:00
Tamir Baydasov and Claude Opus 5.5
b057d5b3f1
[Fix] [NPU] mla-pool: DCP allocator page and index-K bound, short-pool warning ( #42976 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-22 05:24:14 +03:00
b453996b7d
[Doc] diffusion/minimax-h3: Refresh VDN-H3 B200 benchmarks, fix 8-step bench presets and popular-models snippet ( #45267 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: zijiexia <37504505+zijiexia@users.noreply.github.com >
2026-10-22 10:09:13 +08:00
em-dreyer and Claude Opus 5.5
3d508d614f
[CI] [NVIDIA] gb300: Run DSV4.1-Flash tests on 2 GPUs with engram host tables ( #45294 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-21 18:53:24 -07:00
wutao
67bbfcee3c
[Perf] [NPU] scheduler: Remove blocking seq_lens D2H from graph replay ( #45170 )
2026-10-22 09:50:04 +08:00
Liangsheng Yin
d61f618276
[Perf] spec: Let DSPARK prefill publish read-done by placing its draft KV inject before the target forward ( #45682 )
2026-10-21 18:40:44 -07:00
4049941a82
[Fix] [NPU] scope: fix dtype error in sampling backend ascend ( #39609 )
...
Co-authored-by: ronnie_zheng <zl19940307@163.com >
Co-authored-by: Артем Савкин <58187114+OrangeRedeng@users.noreply.github.com >
2026-10-22 04:34:29 +03:00
Khoa Pham and Claude Opus 5.5
aa10927271
[Perf] dsv41: Mega-mHC and the wo_a einsum write MXFP8 for wqkv_a and wo_b ( #45632 )
...
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
2026-10-21 18:27:21 -07:00
Baizhou Zhang
59149553b9
[Refactor] attention: centralize linear and draft TP group selection ( #45684 )
2026-10-21 18:25:40 -07:00
Rohan Varma and Yangmin Li
9ec74a53cc
[Fix] [NVIDIA] docker: Build sgl-deep-gemm 0.2.3 for cu134 to match the sglang pin ( #45521 )
...
Co-authored-by: Yangmin Li <yangminl@nvidia.com >
2026-10-22 09:01:05 +08:00
Jimmy Shong
9301ca37a0
[Fix] modelopt: Correct NVFP4 TRT-LLM clamps for untied gate/up scales ( #45386 )
2026-10-21 17:37:07 -07:00
Michael
5fff58a169
[CI] [AMD] miles: Fix the ROCm 10 nightly suites that never finish ( #45469 )
2026-10-21 17:24:51 -07:00
eacd0a07d7
[Feat] bench: support --tokenize-prompt for generated-shared-prefix ( #40900 )
...
Co-authored-by: Yihao Wang <42559837+AgainstEntropy@users.noreply.github.com >
Co-authored-by: chaoyujiang <chaoyujiang@hygon.com >
Co-authored-by: Shangming Cai <csmthu@gmail.com >
2026-10-22 08:17:15 +08:00
Khoa Pham
ed89693c4a
[Perf] dsv41: Fuse the q_norm MXFP8 quantize for wq_b at every row count ( #45633 )
2026-10-21 16:56:45 -07:00
Liangsheng Yin
da7b97a737
[Perf] dsv4: Publish prefill read-done under the request window and on V4.1 ( #45582 )
2026-10-21 16:29:28 -07:00
Liangsheng Yin
3438ed660d
[Test] cpu: prune mirror and tautology cases from the per-commit CPU suite ( #45672 )
2026-10-21 16:02:47 -07:00
ed1f56fead
[Fix] [Apple] mlx: Give the stub model a shared-expert layout ( #45419 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: R0CKSTAR <yeahdongcn@gmail.com >
2026-10-21 15:54:49 -07:00
Baizhou Zhang
f4035cab0b
[Feat] cp: Enable GLM-5.3 Flash interleave BCG with batched prefill and DFlash2 ( #43182 )
2026-10-21 15:51:28 -07:00
42db2db4a8
[Fix] rust-server: Check only a client-sent max_new_tokens against the context window ( #45460 )
...
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com >
Co-authored-by: shodoco <shodoco@users.noreply.github.com >
2026-10-21 15:50:13 -07:00
991b846227
[Perf] speculative: Parallelize EAGLE spec-decode KV index building over token blocks (flashinfer draft decode; verify/draft-extend for flashinfer, flashinfer_mla, aiter) ( #42803 )
...
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com >
Co-authored-by: YAMY1234 <74099316+YAMY1234@users.noreply.github.com >
Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com >
Co-authored-by: Po-Han Huang <pohanh@nvidia.com >
2026-10-21 15:26:35 -07:00
maocheng23
4b9936b5e3
[Fix] deepseek_v4: Fix mHC prewarm when chunked prefill is disabled ( #43448 )
2026-10-21 15:21:31 -07:00
Liangsheng Yin
99cf403c9e
[Fix] dsv4: Write idle SWA stores to an idle request-window layout ( #45586 )
2026-10-21 15:18:34 -07:00
em-dreyer
aa5880e2b8
[Fix] bailing: Renormalize top-k routing weights by default for Ling-2.5/2.6 ( #45662 )
2026-10-21 14:53:58 -07:00
Michael
51baa1f6e9
[Test] [AMD] debug-utils: Opt TestCrashDump into expect_server_exit ( #45408 )
2026-10-21 14:49:53 -07:00
Liangsheng Yin
f3f0f7b68c
[Perf] dsv4: Let the decoder tail leave the prefill graph under DP attention with MegaMoE ( #45581 )
2026-10-21 14:44:21 -07:00
Liangsheng Yin
f5eca05b05
[Fix] [SM100] quantization: Honor deep_gemm for 32x32 block-FP8 linears ( #45585 )
2026-10-21 14:42:50 -07:00
29e78045fb
[Fix] dsv4: Skip DP-attention pad rows in the engram hasher and low-ratio sources ( #40424 )
...
Co-authored-by: jiashuai <jiashuai@baidu.com >
Co-authored-by: hnyls2002 <lsyincs@gmail.com >
2026-10-21 14:37:55 -07:00
Liangsheng Yin
68623cae92
[Fix] test: Gather Mamba track indices from track_position in the mixed-batch track test ( #45670 )
2026-10-21 14:33:48 -07:00
Liangsheng Yin
f7d4f06242
[Perf] dsv4: Compute request-window history rows once per batch ( #45579 )
2026-10-21 14:01:19 -07:00
Liangsheng Yin
bc412213c1
[Perf] dsv4: Gather V4.1 index-K slots once per forward and cache the prefill index plan ( #45577 )
2026-10-21 14:00:35 -07:00
Liangsheng Yin
620718c8ea
[Fix] moe: Keep the DeepSeek V4.1 vision router off the packed-ids path under routing simulation ( #45575 )
2026-10-21 13:58:44 -07:00
Liangsheng Yin
97945831f9
[Perf] dsv4: Restore sparse prefill for request-window rows ( #45573 )
2026-10-21 13:54:32 -07:00
Liangsheng Yin and lixiufei-leo
2c7a9e3b50
[Fix] radix-cache: Order linker offloads behind the forward that writes the pages ( #45559 )
...
Co-authored-by: lixiufei-leo <294029388+lixiufei-leo@users.noreply.github.com >
2026-10-21 13:53:37 -07:00
Khoa Pham
028de2c5d4
[Perf] dsv41: Use tuned MXFP8 tactics at decode sizes ( #45630 )
2026-10-21 13:35:56 -07:00
Yoni Gozlan and Jimmy Shong
a2e4ffbcea
[Fix] parser: Accept whole-number floats for integer tool arguments ( #43941 )
...
Co-authored-by: Jimmy Shong <69131491+Jiminator@users.noreply.github.com >
2026-10-21 13:04:15 -07:00
Michael
7e0e259435
[CI] [AMD] disaggregation: Register Mooncake KV buffers without HIP dmabuf ( #45519 )
2026-10-21 12:47:42 -07:00
a3d00eff58
[Perf] srt: Record on the host whether an extend batch completes a prompt ( #45487 )
...
Co-authored-by: Lu Fang <116399278+luccafong@users.noreply.github.com >
Co-authored-by: shodoco <shodoco@users.noreply.github.com >
2026-10-21 11:58:25 -07:00
7e5256d1ba
[Feat] srt: Balance uvicorn tokenizer workers with per-worker SO_REUSEPORT listeners ( #45482 )
...
Co-authored-by: luccafong <116399278+luccafong@users.noreply.github.com >
Co-authored-by: joshuadeng <joshuadeng@users.noreply.github.com >
2026-10-21 11:57:26 -07:00
Baizhou Zhang
f06d1bb1f4
[Perf] cuda-graph: Capture FlashInfer KDA prefill ( #45473 )
2026-10-21 11:57:23 -07:00
740d5f9f9e
[Perf] scheduler: Plan hybrid Mamba extend batches without host-device syncs ( #45472 )
...
Co-authored-by: Lu Fang <116399278+luccafong@users.noreply.github.com >
Co-authored-by: Shiyan Deng <20773414+842974287@users.noreply.github.com >
2026-10-21 11:56:50 -07:00
5e79333c95
[Refactor] radix-tree: Reuse the insert walk, component hooks and bindings for custom tree cores ( #45439 )
...
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com >
Co-authored-by: Jialin <Jialin@users.noreply.github.com >
2026-10-21 10:53:26 -07:00
paulzhang-tm
b57b5e2286
[Fix] disaggregation: Synchronize deferred decode KV release across TP ranks ( #45453 )
2026-10-22 01:37:28 +08:00
Khoa Pham
ed268dd320
[Fix] moe: Apply SGLANG_SIMULATE_* routing on DeepSeek-V4.1 vision-biased gates ( #45628 )
2026-10-21 10:00:15 -07:00
vorapolsiloai
6e98c84c60
[Perf] [AMD] quark: Add opt-in AITER ASM FP4 GEMM for Quark dense linear ( #39598 )
2026-10-21 09:37:47 -07:00
Michael
4b1a44bc77
[Test] [AMD] diffusion: Avoid Python 3.11-only enterContext in test_usp_ring_replicated ( #45409 )
2026-10-22 00:11:08 +08:00
zijiexia
75878f27c8
[Doc] cookbook: Add Recursive Flint cookbook ( #45625 )
2026-10-21 08:58:16 -07:00
Shiyang Lyu
cdbd478422
[Doc] [NPU] cookbook: add tested A2 deployment and A3 reference for MiniMax-M2.5 ( #40120 )
2026-10-21 22:59:37 +08:00
Huang Long and Shangming Cai
af49ada16d
[Fix] pd: Allow runtime role switching with DP and DP attention ( #44215 )
...
Co-authored-by: Shangming Cai <csmthu@gmail.com >
2026-10-21 22:58:29 +08:00
Peng Xingchen
7163e97462
[Doc] [NPU] GLM: Add Ascend cookbook deployment recipes ( #45597 )
2026-10-21 22:44:08 +08:00
8e3e2980db
[Feat] [Intel] disaggregation: Support XPU for staging-buffer KV transfer ( #40861 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Shangming Cai <csmthu@gmail.com >
2026-10-21 22:09:04 +08:00
pllimax
7e5d3780ef
[Fix] [NPU] tests: Run the HiCache Mamba case with bfloat16 SSM kernels ( #45593 )
2026-10-21 21:48:12 +08:00
Shawn Yang
dd9ea06c00
[Fix] rust: Format health probe handle setup ( #45606 )
2026-10-21 21:40:08 +08:00
unclezhou
e109bc1832
[Doc] [NPU] deepseek-v4: Add A3/950 single-node cookbook recipes ( #45602 )
2026-10-21 21:16:49 +08:00
amote-i
2815eed49c
[Fix] [NPU] pre_warm_nccl: init npu backend before the parallel runtime touches a device ( #43039 )
2026-10-21 19:44:30 +08:00
amote-i
fda84f80b7
[Doc] [NPU] docs: reference released Qwen3.8-Max W4A8 checkpoint in tutorial ( #45574 )
2026-10-21 19:38:17 +08:00
Shangming Cai
a175e39b84
[CI] cuda: Rebuild workspace crates when the shared Cargo target dir came from another rust/ tree ( #45557 )
2026-10-21 19:32:22 +08:00
Avaya Aggarwal and niehen6174
62b4127668
[Feat] comfyui: Apply ControlNet residuals in Flux integrated mode ( #43698 )
...
Co-authored-by: niehen6174 <1639206518@qq.com >
2026-10-21 19:29:43 +08:00
Shangming Cai
43a3f956cb
[Fix] sgl-router: Wait for the detached PD prefill in routing tests ( #45502 )
2026-10-21 19:04:26 +08:00
pllimax
43728298fe
[Fix] [NPU] qwen3_6: Set accuracy cases to greedy decoding and tune retries ( #40889 )
2026-10-21 18:43:30 +08:00
ca72dad953
[Fix] rust-server: Answer health probes a busy scheduler skipped on the response ring ( #45475 )
...
Co-authored-by: Lu Fang <116399278+luccafong@users.noreply.github.com >
Co-authored-by: shodoco <shodoco@users.noreply.github.com >
2026-10-21 03:41:36 -07:00
Jimmy Shong
5bf10d4c79
[Fix] nemotron: Correct weight-export failures and GEMV dispatch ( #45082 )
2026-10-21 18:38:35 +08:00
Jzz1943
2e7c9d851c
[Feat] NIXL: Support per-GPU UCX device selection ( #43482 )
2026-10-21 17:54:11 +08:00
bz f
9c10e0dd61
[Doc] [NPU] cookbook: add NPU (A3 series) deployment scenarios for Qwen3.6 ( #38678 )
2026-10-21 17:37:24 +08:00
Shangming Cai
7554be5a69
[CI] codeowners: Add owners for the top-level sgl-router ( #45564 )
2026-10-21 17:28:21 +08:00
kangwangamd
131c42ba33
[Perf] [AMD] aiter: Update ROCm AITER pin to 285579b62 (gfx942/gfx950 only) ( #45498 )
2026-10-21 02:25:02 -07:00
Shangming Cai
be65391abd
[Feat] mooncake: Enable the failed-session probe by default with backoff and idle pruning ( #45338 )
2026-10-21 17:16:19 +08:00
Thomas Wang
1278087336
[Fix] [AMD] mono-kernel: Update sgl-amd-mono version ( #45546 )
2026-10-21 02:02:03 -07:00
2c9d919071
[Fix] srt: Pin the source address of TCP ZMQ connects for the peer allowlist ( #45437 )
...
Co-authored-by: Lu Fang <116399278+luccafong@users.noreply.github.com >
Co-authored-by: fei-xx <fei-xx@users.noreply.github.com >
2026-10-21 01:59:03 -07:00
21f775cd97
[Feat] rust-server: Let the launcher set the default max_new_tokens for /generate ( #45431 )
...
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com >
Co-authored-by: shodoco <shodoco@users.noreply.github.com >
2026-10-21 01:58:34 -07:00
Aurick Qiao and Xiaoyu Zhang
7a632cf998
[Fix] quantization: Preserve BF16 NaNs in per-token FP8 quantization ( #42689 )
...
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com >
2026-10-21 16:52:28 +08:00
8bd6aa9a6c
[Feat] parser: Enrich chat_parsing streaming events ( #40478 )
...
Co-authored-by: Yoni Gozlan <yonigozlan@users.noreply.github.com >
Co-authored-by: Cursor <cursoragent@cursor.com >
Co-authored-by: Xinyuan Tong <xinyuantong.cs@gmail.com >
Co-authored-by: Jimmy Shong <69131491+Jiminator@users.noreply.github.com >
2026-10-21 01:46:26 -07:00
Alex and Jimmy Shong
dbe0a9a7c9
[Fix] nemotron_vl: Preserve tokenized prompts during media expansion ( #44184 )
...
Co-authored-by: Jimmy Shong <69131491+Jiminator@users.noreply.github.com >
2026-10-21 01:40:51 -07:00
Liangsheng Yin
6efdd33d07
[Refactor] spec: Move platform dispatch and Triton kernels into kernels/ops ( #45504 )
2026-10-21 01:22:16 -07:00
6cac4dc3fe
[Refactor] diffusion: Rollout controller as a scheduler-step plugin instead of mixins ( #44603 )
...
Signed-off-by: rockdu <kangrdu@gmail.com >
Co-authored-by: WenhaoZhang <42087078+niehen6174@users.noreply.github.com >
Co-authored-by: JingwenGu0829 <75733630+JingwenGu0829@users.noreply.github.com >
2026-10-21 16:04:33 +08:00
Avaya Aggarwal and WenhaoZhang
fee32a5c1f
[Feat] ComfyUI: Add online INT8 quantization for BF16 Qwen-Image-2.1 single files ( #44865 )
...
Co-authored-by: WenhaoZhang <42087078+niehen6174@users.noreply.github.com >
2026-10-21 16:01:18 +08:00
Jzz1943
a361f0c410
[Fix] kda: Align graph tracking branch dtype with the recurrent state pool ( #44886 )
2026-10-21 15:58:34 +08:00
f7b8c6fe40
[Fix] glm-5.2: Record side-stream work right before its join to keep CUDA-graph replay on one stream ( #45349 )
...
Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com >
Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com >
2026-10-21 15:36:54 +08:00
Hongkuan Zhou and ishandhanani
8acfabbd67
[Fix] observability: Correct speculative FPM timing and decode lengths ( #38711 )
...
Signed-off-by: hongkuanz <hongkuanz@nvidia.com >
Co-authored-by: ishandhanani <82981111+ishandhanani@users.noreply.github.com >
2026-10-21 15:31:18 +08:00
Shangming Cai
cd61db49c7
[Fix] sgl-router: Bump rustls-webpki and webpki-roots for emSign advisories ( #45526 )
2026-10-21 15:24:11 +08:00
YC Yen-Ching Tseng
132836b369
[Fix] [AMD] docker: Build Triton from source for ROCm 10.1 gfx1250 ( #45329 )
2026-10-21 15:14:24 +08:00
Peng Xingchen and pengxingchen0810
4449d7ecb0
[Doc] [NPU] GLM: Add GLM-5.3 and GLM-5.3-Flash deployment tutorials ( #43372 )
...
Co-authored-by: pengxingchen0810 <pengxingchen0810@noreply.gitcode.com >
2026-10-21 15:02:58 +08:00
Junlin Wu
844d2b5dec
[Fix] [NPU] qwen3.5: Fix dense serving on Ascend 950 ( #32745 )
2026-10-21 09:50:38 +03:00
Jimmy Shong
8599b7251f
[Chore] deps: Bump the OpenAI SDK to 2.8.0 for Responses shell tools (2/8) ( #45291 )
2026-10-20 23:49:30 -07:00
wutao
fb6e6a09ce
[Fix] [NPU] disaggregation: Fix host-pool backup for decode retraction ( #43814 )
2026-10-21 14:48:01 +08:00
chenyang08056032
c3b2b55bbb
[CI] [NPU] precise-test: add recommendation summary & run artifacts, fix NPU-only coverage view, and quiet benign teardown noise ( #45374 )
2026-10-21 14:40:50 +08:00
Khoa Pham and AMD-yanfeiwang
da7e3f1c12
[Fix] cache: Accept packed draft layer mappings in the UMBP linker ( #44929 )
...
Co-authored-by: AMD-yanfeiwang <yanfei.wang@amd.com >
2026-10-20 23:27:22 -07:00
Thomas Wang
bd5a445bf7
[Fix] [AMD] dsv41: Set moe act to fp8 by default ( #45480 )
2026-10-20 23:09:27 -07:00
Shangming Cai
574f9cdb2e
[CI] slash-command: Add /run-cpu-ci to run only the CPU unit suite ( #45501 )
2026-10-21 13:59:16 +08:00
956b9a24ee
[Perf] [AMD] qwen3.5: Fuse activation quant into small-M FP8 block-scale GEMM on gfx950 ( #42229 )
...
Co-authored-by: Cursor <cursoragent@cursor.com >
Co-authored-by: jacky.cheng <yichiche@amd.com >
2026-10-20 22:32:30 -07:00
Po-Han Huang (NVIDIA) and Claude Opus 5.5
1689cb2d52
[Doc] cookbook: Add GLM-5.3 AgentX section and refresh with eight open InferenceX PRs ( #45513 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-21 13:22:38 +08:00
Aarne Talman
4b5c9c4b5e
[Perf] [AMD] attention: Restore the gfx950 extend tile for head_dim <= 128 ( #45099 )
2026-10-20 22:02:50 -07:00
32c26788f3
[Fix] mamba: Keep prefill rows' checkpoint tracking through mixed chunked prefill ( #45435 )
...
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com >
Co-authored-by: YavorGIvanov <YavorGIvanov@users.noreply.github.com >
2026-10-20 21:56:50 -07:00
Liangsheng Yin
573b5716f5
[Fix] spec: Take post-verify KV move targets from the verify window ( #45491 )
2026-10-20 21:12:41 -07:00
1c2f42470c
[Fix] disaggregation: Keep a requeued optimistic prefill request's multimodal inputs ( #45440 )
...
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com >
Co-authored-by: libertyeagle <30348494+libertyeagle@users.noreply.github.com >
Co-authored-by: fei-xx <135589532+fei-xx@users.noreply.github.com >
2026-10-20 20:27:34 -07:00
metamergebot and shodoco
6bd1c76735
[Fix] rust-server: Drop the fixed 64-item multimodal cap in favor of --limit-mm-data-per-request ( #45432 )
...
Co-authored-by: shodoco <shodoco@users.noreply.github.com >
2026-10-20 20:25:22 -07:00
Kan Wu and Claude Opus 5.5
aba892fbdf
[Refactor] sgl-router: Move experimental/sgl-router to top-level sgl-router ( #45414 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-21 11:09:13 +08:00
eb91a41f38
[Feat] unified_cache: Support FULL + MAMBA MLA models (Kimi-K3) in the direct external linker ( #44977 )
...
Co-authored-by: Yutong Wu <yutong.wu@amd.com >
Co-authored-by: Zhangheng <hzh0425@apache.org >
2026-10-21 10:47:17 +08:00
metamergebot
f1ba723c64
[Test] [AMD] moe: Expect gfx1250 W4A8 layers to build with expert parallelism ( #45476 )
2026-10-21 10:45:31 +08:00
Chunan Zeng
93088901fb
[Chore] attention: Remove the tokenspeed_mla backend and dependency ( #44803 )
2026-10-20 18:50:05 -07:00
unclezhou
c0d18c0254
[Fix] [NPU] dspark: Fix attn_tp > 1 draft wo_b all_reduce group ( #40399 )
2026-10-21 09:20:15 +08:00
c1c287a429
[Doc] cookbook: add TPU7x recipe for GLM-5.3-Flash ( #45326 )
...
Co-authored-by: Qinghan Chen <qinghan.chen@radixark.net >
Co-authored-by: zijiexia <37504505+zijiexia@users.noreply.github.com >
2026-10-20 17:10:18 -07:00
b326e8349d
[Perf] mamba: Copy tracked decode states at HBM bandwidth in one persistent launch ( #45213 )
...
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com >
Co-authored-by: YavorGIvanov <YavorGIvanov@users.noreply.github.com >
2026-10-20 16:52:15 -07:00
Baizhou Zhang
5b1dbc2c80
[Fix] [NVIDIA] modelopt-nvfp4: Keep MoE global scales in gate/up order ( #45289 )
2026-10-20 16:49:13 -07:00
kk and wunhuang
9686edcf80
[Perf] [AMD] dsv41: faster mHC boundary Triton tile for 96+ rows, bitwise unchanged ( #45124 )
...
Co-authored-by: wunhuang <wunhuang@amd.com >
2026-10-20 16:39:34 -07:00
Cheng Wan
027208ae02
[Doc] layer-boundary/moe: Correct guides and comments that drifted from the code ( #44896 )
2026-10-20 16:26:40 -07:00
Cheng Wan and Aditya Suresh
299a75ac5d
[Refactor] moe: Derive routing inputs from the layer and active forward ( #44994 )
...
Co-authored-by: Aditya Suresh <152752311+adi-suresh01@users.noreply.github.com >
2026-10-20 16:26:15 -07:00
kk and wunhuang
533020b500
[Perf] [AMD] dsv41: run gfx950 MXFP8 dense GEMMs on aiter's preshuffled FlyDSL kernel ( #45123 )
...
Co-authored-by: wunhuang <wunhuang@amd.com >
2026-10-20 16:19:32 -07:00
Cheng Wan and Aditya Suresh
4f8f06faf3
[Refactor] moe: Derive layer contracts from the selected quant method ( #44993 )
...
Co-authored-by: Aditya Suresh <152752311+adi-suresh01@users.noreply.github.com >
2026-10-20 16:18:45 -07:00
zydo and Cheng Wan
7b1d5cd02d
[Doc] agent-skills: Fix stale paths and deleted guard tests in skills and rules ( #43666 )
...
Co-authored-by: Cheng Wan <cheng.wan@radixark.ai >
2026-10-20 16:15:10 -07:00
bf145750a6
[Perf] prefill-cuda-graph: Make prefill shared-read completion explicit and default-on ( #37958 )
...
Co-authored-by: weireweire <20922698+weireweire@users.noreply.github.com >
Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com >
Co-authored-by: Po-Han Huang <pohanh@nvidia.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: hnyls2002 <lsyincs@gmail.com >
2026-10-20 16:12:55 -07:00
Aarne Talman
d5e83ea4f6
[Perf] [AMD] deterministic: Faster extend attention launch on gfx950 ( #45098 )
2026-10-20 16:11:01 -07:00
averyjones4 and averyjones4
c1e43b95f4
[Doc] cookbook: Add DeepReinforce Ornith-1.5 ( #45455 )
...
Co-authored-by: averyjones4 <averyjones4@users.noreply.github.com >
2026-10-20 16:08:53 -07:00
Cheng Wan and Aditya Suresh
f1eec1232d
[Fix] config: Preserve ServerArgs positional constructor slots ( #44991 )
...
Co-authored-by: Aditya Suresh <152752311+adi-suresh01@users.noreply.github.com >
2026-10-20 16:08:28 -07:00
Aditya Suresh and Cheng Wan
e2e1a97fc3
[Fix] moe: Map expert ids for layers whose runner needs local ids under EP ( #44480 )
...
Co-authored-by: Cheng Wan <cheng.wan@radixark.ai >
2026-10-20 16:07:48 -07:00
Baizhou Zhang
b008ec3de6
[CI] muse_spark: Move unified memory test to B300 ( #45454 )
2026-10-20 15:56:49 -07:00
Liangsheng Yin
1dd0f361b8
[Fix] moe: Run the shared-slot routing test without a published server config ( #45448 )
2026-10-20 15:03:47 -07:00
Baizhou Zhang
583cfc4d55
[CI] muse_spark: Use complete local checkpoints in model tests ( #45428 )
2026-10-20 14:57:01 -07:00
Yuwei An
f0b6590722
[Fix] DSpark: Disable simulated acceptance shortcuts by default ( #45433 )
2026-10-20 14:10:59 -07:00
YAMY
7d9bc10493
[Perf] [NVIDIA] DSpark: Enable the simulated-acceptance fast paths on CUDA ( #45421 )
2026-10-20 13:36:33 -07:00
cc7ae372ac
[Refactor] radix-cache: Route pending-backup split bookkeeping through one component action ( #45242 )
...
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com >
Co-authored-by: Jialin <Jialin@users.noreply.github.com >
2026-10-20 13:24:49 -07:00
0b536ca112
[Fix] radix-cache: Separate logical and reclaimable value lengths in tree components ( #45238 )
...
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com >
Co-authored-by: Jialin <Jialin@users.noreply.github.com >
2026-10-20 13:24:20 -07:00
metamergebot and joshuadeng
468e26f253
[Perf] detokenizer: Add SGLANG_BATCH_TOKENIZER_OUTPUTS to opt out of grouped return sends ( #45280 )
...
Co-authored-by: joshuadeng <joshuadeng@users.noreply.github.com >
2026-10-20 13:24:14 -07:00
bab4a808d8
[Perf] unified-memory: Allocate free pages from each sub-pool's own end first ( #45222 )
...
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com >
Co-authored-by: ZYHowell <ZYHowell@users.noreply.github.com >
2026-10-20 13:24:09 -07:00
0d7638b035
[Fix] scheduler: Exclude accepted storage-tier hits from the waiting prefill load ( #45158 )
...
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com >
Co-authored-by: Jialin <Jialin@users.noreply.github.com >
2026-10-20 13:24:04 -07:00
dc62c3effd
[Feat] radix-tree: Report each path node's child count in collect_path_info ( #45241 )
...
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com >
Co-authored-by: Jialin <Jialin@users.noreply.github.com >
2026-10-20 13:23:59 -07:00
metamergebot and daniel-salib
194876bea0
[Fix] srt: Admit only allowlisted peers on TCP-bound ZMQ IPC sockets ( #45250 )
...
Co-authored-by: daniel-salib <daniel-salib@users.noreply.github.com >
2026-10-20 13:23:53 -07:00
Yuwei An
afe3e1df31
[Fix] DSpark: Restore weight-loading CPU tests ( #45427 )
2026-10-20 13:13:39 -07:00
Jimmy Shong
d69966f9c0
[Chore] docker: Add Beam CUDA image (1/8) ( #45253 )
2026-10-20 13:00:46 -07:00
Baizhou Zhang
cfec1b91bb
[Fix] muse_spark: support unified KV memory with TRT-LLM ( #45356 )
2026-10-20 12:54:53 -07:00
Aarne Talman
29274b41c6
[Fix] [AMD] activation: Run ReLU2 natively on ROCm ( #44198 )
...
Signed-off-by: Aarne Talman <aarne.talman@amd.com >
2026-10-20 12:05:14 -07:00
Thomas Wang
82ac83a8d0
[Perf] [AMD] dsv41: Skip unneeded layer 20 computation under decoder-bounded-replay ( #45339 )
2026-10-20 11:49:05 -07:00
Michael and jacky.cheng
f82465718b
[Test] [AMD] diffusion: Skip MOSS-Image mismatched-operand test on ROCm ( #44596 )
...
Co-authored-by: jacky.cheng <yichiche@amd.com >
2026-10-20 11:37:32 -07:00
Khoa Pham
c2e5742464
[Perf] deepseek-v4: DSpark draft stages take the TF32 mHC statistics path from 32 rows on SM100 ( #45260 )
2026-10-20 10:12:06 -07:00
sogalin_codegen
89860fbd2e
[Fix] [AMD] gpt-oss: support EP for gfx1250 W4A8 MXFP4 MoE ( #45365 )
2026-10-20 10:08:39 -07:00
Michael
7591118794
[Fix] [AMD] grok: accept allow_patterns_overrides in presharded _prepare_weights patch ( #45217 )
2026-10-20 16:12:06 +00:00
eb4b167d27
[Feat] KVShard: PD transfer and e2e test ( #38473 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Zhangheng <hzh0425@apache.org >
2026-10-20 22:48:43 +08:00
7656071309
[Feat] platform: Add initial T-HEAD SAIL platform support [2/N] ( #43826 )
...
Co-authored-by: Zhangheng <hzh0425@apache.org >
Co-authored-by: yzshi <163985193+Yunzshi310@users.noreply.github.com >
2026-10-20 22:19:07 +08:00
pllimax
e262b44eef
[CI] [NPU] tests: Replace DeepSeek-V2-Lite-W8A8 cases with Moonlight ( #43063 )
2026-10-20 21:35:49 +08:00
Shangming Cai
952c63b1bb
[CI] sgl-router: Gate on live run-ci label so /tag-and-rerun-ci can start the tiers ( #45383 )
2026-10-20 21:33:42 +08:00
Jensen and ronnie_zheng
10d5293546
[Perf] [NPU] glm5.2: Support dynamic INT8 MLAProlog for attention ( #43469 )
...
Co-authored-by: ronnie_zheng <zl19940307@163.com >
2026-10-20 16:33:08 +03:00
Jensen
3bf4cbc6cf
[Perf] [NPU] mem_cache: Increase GLM-5.2-W4A8C8 KV cache capacity with compact DSA indexer allocation ( #44436 )
2026-10-20 16:30:01 +03:00
gjsheu
aa91108f64
[Fix] [NPU] glm5.2: support pcp v2 with interleave & zigzag ( #40165 )
2026-10-20 21:26:30 +08:00
gjsheu
ee2f95a522
[Fix] [NPU] glm5.2-mtp: L2 cache adaptation ( #41642 )
2026-10-20 21:25:37 +08:00
df13d5dce3
[Feat] sgl-router: Route prefill buckets by uncached tokens ( #45103 )
...
Co-authored-by: Xiufei Li <294029388+lixiufei-leo@users.noreply.github.com >
Co-authored-by: Shangming Cai <csmthu@gmail.com >
2026-10-20 20:56:14 +08:00
Po-Han Huang (NVIDIA) and Claude Opus 5.5
eae134d61f
[Test] kernels: Import silu_and_mul_contig_post_quant from its new home ( #45368 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-20 19:48:59 +08:00
Kurkur
07eb671b6a
[Perf] [NPU] attn: add not needs_cpu_seq_lens for dsa modules ( #38461 )
2026-10-20 19:19:34 +08:00
Shijin Zhang and Xinyuan Tong
ca473ef3e4
[Feat] glm-5.3: Render GLM-5.3 and GLM-5.3-Flash chat prompts with a Python chat template ( #45154 )
...
Signed-off-by: Shijin Zhang <75300765+Dovis01@users.noreply.github.com >
Co-authored-by: Xinyuan Tong <xinyuantong.cs@gmail.com >
2026-10-20 19:11:40 +08:00
Ho-Ren (Jack) Chuang
800e60e345
[Fix] moe: Publish a server config in the CUTLASS runner dispatch-format and placement tests ( #45376 )
2026-10-20 19:05:17 +08:00
Yuxuan Zhou
306ccf6f57
[Perf] spec: Let the FlashAttention backend capture the draft-extend CUDA graph ( #40762 )
2026-10-20 03:05:42 -07:00
Sihan Chen
9db58ab547
[Fix] eplb: Fix Qwen3 MoE output corruption after EPLB rebalance ( #39580 )
...
Signed-off-by: Spycsh <sihan.chen@intel.com >
2026-10-20 03:00:39 -07:00
a70b1b73d3
[Fix] tracing: Fix OTel tracing context lost in multi-tokenizer router processes ( #38211 )
...
Co-authored-by: Ma <T9FOPWP@cn.vwgroup.com >
Co-authored-by: Shangming Cai <csmthu@gmail.com >
2026-10-20 17:57:16 +08:00
16eb87c06f
[Fix] openai: distinguish s3 served-model-name from LoRA adapter syntax ( #44450 )
...
Signed-off-by: Muqi Li <muqi1029@gmail.com >
Signed-off-by: Xinyuan Tong <xinyuantong.cs@gmail.com >
Co-authored-by: Xinyuan Tong <xinyuantong.cs@gmail.com >
Co-authored-by: Xinyuan Tong <115166877+JustinTong0323@users.noreply.github.com >
2026-10-20 17:11:06 +08:00
Xiaoyu Zhang and Claude Opus 5.5
2229345dff
[Doc] kda_kernels: Point the SM121 QSA kernel provenance at its SGLang PR ( #45359 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-20 16:50:58 +08:00
Ho-Ren (Jack) Chuang
f8fd41c36e
[Perf] moe: Extend FlashInfer CUTLASS MoE autotune coverage to prefill chunks ( #43363 )
2026-10-20 01:41:16 -07:00
Ma Mingfei and Copilot
06314dc2e8
[CI] [Intel] tests: Remove CPU monitor and crash-dump tests from Xeon CI ( #45281 )
...
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com >
2026-10-20 16:40:13 +08:00
Xiaoyu Zhang and Claude Opus 5.5
0788f1fdd0
[Refactor] kernels: Register agent-generated kernels under their KDA / Cake backend ( #45277 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-20 16:35:11 +08:00
Xinyuan Tong
fdd89d182a
[Fix] moe: add uses_expert_location to MegaMoE test fixtures ( #45350 )
...
Signed-off-by: Xinyuan Tong <xinyuantong.cs@gmail.com >
2026-10-20 16:20:46 +08:00
Mohammad Miadh Angkad and Mohammad Angkad
78185f4b3f
[Fix] [NVIDIA] modelopt-nvfp4: Read w13 global scales in the TRT-LLM [up, gate] order ( #44858 )
...
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai >
2026-10-20 07:38:45 +00:00
XXXXRT666 and Alex Nails
1739d962a4
[Refactor] [Apple] MLX: Use Native Extension Builder for Metal Kernels ( #44367 )
...
Co-authored-by: Alex Nails <alex.nails@radixark.ai >
2026-10-20 00:35:29 -07:00
Shuwen Wang
15e8e687db
[Chore] environ: Remove retired unified radix tree environment variable ( #45090 )
2026-10-20 07:02:43 +00:00
Po-Han Huang (NVIDIA) and Claude Opus 5.5
9e23107772
[Doc] cookbook: Refresh SemiAnalysis AgentX launches with five open InferenceX PRs ( #45311 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-20 14:48:48 +08:00
1ec902e00f
[Fix] cache: Support DeepSeek-V4.1 external linker pools ( #44623 )
...
Co-authored-by: Zhangheng <hzh0425@apache.org >
Co-authored-by: Shuwen Wang <47200617+alphabetc1@users.noreply.github.com >
2026-10-19 23:40:25 -07:00
Michael
407bfc94f9
[Test] [AMD] diffusion: Make multimodal_gen unit tests ROCm-aware ( #44599 )
2026-10-20 14:24:37 +08:00
Xinyuan Tong
4b83f215f4
[Feat] entrypoints: add a chat encoding adapter registry ( #45300 )
2026-10-20 14:10:01 +08:00
lpdink and Zhangheng
4a651146de
[Feat] hicache: Notify storage backends when the initial host-pool registration burst completes ( #44030 )
...
Co-authored-by: Zhangheng <hzh0425@apache.org >
2026-10-20 14:09:13 +08:00
Michael
36978d6c4d
[Test] [AMD] diffusion: Skip the Kandinsky6 FA dispatch test on ROCm ( #44421 )
2026-10-20 13:54:23 +08:00
Duyi-Wang
50fa3727eb
[Doc] [AMD] cookbook: Fix DeepSeek-V4 PD image and c1 capacity ( #45297 )
2026-10-19 22:45:50 -07:00
Liangsheng Yin
472f0d540b
[Fix] deepseek-v4.1: Keep the padded query buffer inside decoder tail graphs ( #45301 )
2026-10-19 22:41:53 -07:00
Khoa Pham and Claude Fable 5.1
476fe86a12
[Perf] deepseek-v4.1: Score a request's verify rows as one request in the full-scan indexer ( #45251 )
...
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com >
2026-10-19 22:25:04 -07:00
35c873cff4
[Fix] observability: Log scheduler stats once per attention CP group ( #45319 )
...
Co-authored-by: Hun-ho Kim <hunho.kim@samsung.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-19 22:22:31 -07:00
weireweire and weireweire
c7687a0e7c
[Fix] deepseek-v4: Keep the draft MoE out of the target's expert placement ( #45299 )
...
Co-authored-by: weireweire <20922698+weireweire@users.noreply.github.com >
2026-10-19 22:20:26 -07:00
602c92ccd8
[Perf] DeepSeek-V4.1: Raise the SM100 swap-AB decode cap to 40 rows ( #44632 )
...
Co-authored-by: weireweire <20922698+weireweire@users.noreply.github.com >
Co-authored-by: nbommi19 <227267412+nbommi19@users.noreply.github.com >
2026-10-19 22:20:10 -07:00
553836c164
[Feat] simulator: Add baseline collection and ML latency predictor training tools ( #44299 )
...
Co-authored-by: shijunyu.sjy <shijunyu.sjy@alibaba-inc.com >
Co-authored-by: baoyunong.byn <baoyunong.byn@alibaba-inc.com >
Co-authored-by: wangyi.wy <aaronwangyi0201@gmail.com >
2026-10-20 13:02:33 +08:00
parkeychen and parkeychen
43a896933e
[Fix] deepseek-v4: Refresh attn_sink after loading weights ( #41630 )
...
Co-authored-by: parkeychen <parkeychen@tencent.com >
2026-10-19 21:59:32 -07:00
ashwini rathi and Claude Opus 5.5
d1c3346f74
[Fix] [Intel] spec: Fix XPU nightly dflash stop_regex and DSpark capture check ( #45033 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-20 12:27:36 +08:00
72d886efb8
[Perf] DeepSeek-V4.1: Run the small-batch decode fusions at TP2 ( #43838 )
...
Co-authored-by: weireweire <20922698+weireweire@users.noreply.github.com >
Co-authored-by: nbommi19 <227267412+nbommi19@users.noreply.github.com >
Co-authored-by: Po-Han Huang <pohanh@nvidia.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com >
2026-10-19 21:17:05 -07:00
Mohammad Miadh Angkad and Mohammad Angkad
fdd53418a0
[CI] [NVIDIA] gb300: Drop Kimi-K2.5 and Qwen3.5 nightly tests ( #45308 )
...
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai >
2026-10-19 21:13:00 -07:00
Shuwen Wang
c53b98d1fe
[Fix] test-utils: Allow expected server exits in canary tests ( #45172 )
2026-10-19 21:07:03 -07:00
Mohammad Miadh Angkad and Mohammad Angkad
5b683d6b72
[Fix] test: Add logical_forward_mode to the KDA prefill-graph stand-in batch ( #45218 )
...
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai >
2026-10-19 21:06:10 -07:00
Mingyi
4a0c6f2fe0
docs: promote SGLang Summit 2026 in README ( #45271 )
2026-10-20 03:36:56 +00:00
wxzhoucs and Xiaoyu Zhang
e3205080ce
[Fix] qwen3.8: Fix batch-invariant kernels for Qwen3.8 Flash-Next ( #42087 )
...
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com >
2026-10-20 11:32:40 +08:00
1140a8dfd9
[Fix] [AMD] aiter: Fix garbage decode output for non-sliding-window models on the vectorized_5d KV layout ( #43335 )
...
Co-authored-by: Cursor <cursoragent@cursor.com >
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com >
2026-10-20 11:31:20 +08:00
Cao E and Ma Mingfei
3c07ff2654
[Perf] [XPU] fp8: Optimize FP8 scale layouts ( #40052 )
...
Co-authored-by: Ma Mingfei <mingfei.ma@intel.com >
2026-10-20 11:24:16 +08:00
huangtingwei
3eb4bd598f
[Perf] hicache: Unify PP prefetch hit queries and buffer allocation ( #43344 )
2026-10-20 11:05:00 +08:00
Trevor Morris
ec1fdc22a9
[Fix] [NVIDIA] docker: Fix CUDA 13.4 container by updating torch nightly ( #45252 )
2026-10-20 10:52:42 +08:00
kpjeeja
8683f8ad4a
[Feat] [Intel] hicache: Enable L1<->L2 support on XPU ( #32503 )
2026-10-20 10:42:21 +08:00
Liangsheng Yin
5e630fa571
[Perf] deepseek-v4.1: Drop host syncs from encoder-replay folding and tail metadata ( #45275 )
2026-10-19 19:39:39 -07:00
Mohammad Miadh Angkad and Mohammad Angkad
af856717c2
[Fix] mamba: Skip zero-width conv pools in the MTP verify scatter ( #45212 )
...
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai >
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai >
2026-10-19 19:23:17 -07:00
Vladimir Serov and ronnie_zheng
55b2aef7ad
[Fix] [NPU] LoRA: Update for follow-up with changes in sgl-kernel-npu ( #40908 )
...
Co-authored-by: ronnie_zheng <zl19940307@163.com >
2026-10-20 05:16:05 +03:00
wutao
2c00b5d129
[Test] constrained: Cover JSON whitespace limits ( #43981 )
2026-10-20 10:12:22 +08:00
Gabriel Wu and number-eleven11
67d9ee156c
[Fix] [SM120] quantization: Requantize block-FP8 MoE weights to UE8M0 for DeepGEMM ( #43556 )
...
Signed-off-by: Zihua Wu <13583761+lucifer1004@users.noreply.github.com >
Co-authored-by: number-eleven11 <1635219023@qq.com >
2026-10-19 19:06:21 -07:00
Liangsheng Yin
ce329c85fa
[Perf] deepseek-v4.1: Skip the request-window history check when invariant checks are off ( #45276 )
2026-10-19 18:38:44 -07:00
79ed13fc2c
[Fix] glm47: Disable tool grammar for tool_choice="none" ( #41233 )
...
Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com >
Co-authored-by: Xinyuan Tong <115166877+JustinTong0323@users.noreply.github.com >
Co-authored-by: Po-Han Huang <pohanh@nvidia.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-20 09:36:06 +08:00
Lee Nau
1897575ce1
[Fix] runtime: Version FlashInfer caches for default-tactic tuning. ( #44514 )
2026-10-19 18:33:49 -07:00
Anihc and sunguozhong
ebc71e650e
[Perf] [NPU] mem_cache: Add optional ScatterPa KV cache writes ( #40535 )
...
Co-authored-by: sunguozhong <sun_gz2772@163.com >
2026-10-20 09:33:34 +08:00
71ee2dc950
[Fix] inkling: Return 400 instead of 500 for unfetchable/undecodable multimodal input ( #40899 )
...
Co-authored-by: apex-mochen <265981728+apex-mochen@users.noreply.github.com >
Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com >
Co-authored-by: apex-mochen <2756823972@qq.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Po-Han Huang <pohanh@nvidia.com >
2026-10-20 09:18:18 +08:00
Xiaoyu Zhang and Claude Opus 5.5
693d333306
[Refactor] kernels: Regroup operators by computation and dissolve ops/muse_spark ( #45112 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-20 08:47:57 +08:00
Liangsheng Yin
ab580b2e06
[CI] radix-cache: Split unified radix cache unit tests into topic files and a shared kit ( #45236 )
2026-10-19 17:00:42 -07:00
Cheng Wan
6b90e5716d
[Refactor] moe: Scope shared-expert configuration to each model ( #44913 )
2026-10-19 16:59:37 -07:00
Cheng Wan
964ab2b280
[Refactor] moe: Split the two meanings of num_fused_shared_experts ( #44912 )
2026-10-19 16:57:15 -07:00
Cheng Wan
48ea486928
[Fix] moe: Renormalize per-rank shared-slot routing over routed experts only ( #44904 )
2026-10-19 16:56:21 -07:00
Cheng Wan
3a059de94c
[Fix] moe: Count routed_scaling_factor once when top-k folds it ( #44901 )
2026-10-19 16:54:27 -07:00
Qiaolin Yu and zijiexia
28448a86d4
[Doc] cookbook: Default Qwen3.8-Flash-Next datacenter recipes to FP8 KV cache and QSA indexer ( #45245 )
...
Co-authored-by: zijiexia <37504505+zijiexia@users.noreply.github.com >
2026-10-19 15:57:22 -07:00
Liangsheng Yin
1238edd954
[Perf] deepseek-v4.1: Reset request-window tags without a host sync ( #45247 )
2026-10-19 15:45:17 -07:00
Mingyi
9a949403f1
[Doc] tpu: Introduce SGLang-TorchTPU in README and TPU guide ( #45246 )
2026-10-19 15:34:47 -07:00
ab258b6453
[Fix] symm-mem: Name the window registration in the scheduler watchdog dump ( #44681 )
...
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com >
Co-authored-by: hnyls2002 <lsyincs@gmail.com >
2026-10-19 15:29:35 -07:00
Gabriel Wu
05d7f7a166
[Fix] attention: Honor selected FlashAttention implementation ( #33007 )
2026-10-19 15:27:52 -07:00
Jiuqiang Tang
49766a74f9
[Fix] attn_residual: int64 program ids in the score/combine kernels (int32 overflow at 37,451 tokens) ( #43438 )
2026-10-19 15:12:12 -07:00
0z5a and Richard Wang
80ea5ee638
[Fix] dLLM: Preserve prompt tails and literal mask tokens across prefix reuse ( #39592 )
...
Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de >
Co-authored-by: Richard Wang <wangrichard08@gmail.com >
2026-10-19 14:48:37 -07:00
Liangsheng Yin
fdc2dd0d05
[Perf] dspark: Make the prefill SWA valid-from update sync-free ( #45234 )
2026-10-19 14:44:01 -07:00
Liu Ke and hnyls2002
81ac15391c
[Fix] metrics: record HTTP request metrics in tokenizer workers ( #44663 )
...
Co-authored-by: hnyls2002 <lsyincs@gmail.com >
2026-10-19 14:36:16 -07:00
Mohammad Miadh Angkad and Mohammad Angkad
7656ea0e74
[CI] python: Move the remaining CUDA build paths off Python 3.10 ( #44393 )
...
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai >
2026-10-19 14:29:00 -07:00
Liangsheng Yin
b20cb3665d
[CI] radix-cache: Drop redundant unified radix cache unit tests ( #45232 )
2026-10-19 14:20:32 -07:00
Baizhou Zhang
642a671c30
[CI] tests: Re-enable muse spark B300 PD coverage ( #44975 )
2026-10-19 13:40:02 -07:00
Liangsheng Yin
8b1d1e0d74
[Perf] deepseek-v4.1: Check request-window SWA history without per-layer host syncs ( #45223 )
2026-10-19 13:34:52 -07:00
Rain Jiang
aaa9388c91
[Fix] kda: Keep FlashInfer prefill off its grouped kernel's bf16/flat-dt_bias contract ( #45018 )
2026-10-19 13:16:54 -07:00
Liangsheng Yin
f1a2ee763d
[Perf] deepseek-v4.1: Run decoder replay tail graphs in prefill-graph steps and under DP attention ( #45107 )
2026-10-19 13:02:07 -07:00
Qiaolin Yu
5385866108
[Fix] moe: gated NVFP4 MoE intermediate alignment for FlashInfer TRTLLM ( #44138 )
2026-10-19 12:52:36 -07:00
Qiaolin Yu
9ab74b8517
[Fix] Qwen3.8next: Stop retaining per-size PLE outputs in breakable prefill CUDA graphs ( #44150 )
2026-10-19 12:41:39 -07:00
0ce41c5a06
[Fix] PD: retraction restores racing the in-flight decode forward ( #42702 )
...
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
Co-authored-by: Zhiqiang Xie <xiezhq@stanford.edu >
2026-10-19 11:55:04 -07:00
YAMY
722669e606
[Feat] [NVIDIA] megakernels: Add --enable-nvidia-megakernel and a layer for NVIDIA megakernel plugins ( #44945 )
2026-10-19 11:54:44 -07:00
AMD-yanfeiwang
678913a0f0
[Feat] sgl-model-gateway: Add --decode-dispatch-limit to pick PD decode workers near KV arrival ( #44689 )
2026-10-20 02:16:21 +08:00
jacky.cheng
80b86dbecb
[Fix] [AMD] diffusion: Accept residual and skip_bias in the ROCm Conv3d-to-Conv2d VAE patch ( #44672 )
2026-10-19 11:07:37 -07:00
jacky.cheng
5ed306bd69
[Fix] [AMD] diffusion: Keep gfx1250 FlyDSL BF16 attention out of torch.compile ( #44671 )
2026-10-19 10:36:28 -07:00
lixiufei-leo and Xiufei Li
4ef9765e76
[Feat] sgl-router: Add --pd-stream-after-prefill ( #45102 )
...
Co-authored-by: Xiufei Li <294029388+lixiufei-leo@users.noreply.github.com >
2026-10-20 01:13:25 +08:00
chenyang08056032
716c9151f2
[CI] [NPU] npu-coverage: run only coverage-recommended tests on PRs (precise test selection) ( #44019 )
2026-10-20 00:57:27 +08:00
Jan Schalkamp and Xiaoyu Zhang
8e7601ae4a
[Feat] paged-experts: Serve MoE models whose weights do not fit VRAM ( #42557 )
...
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com >
2026-10-19 23:10:04 +08:00
Monishver
806e7423af
[Perf] deepseek_v4: size the masked silu-mul-quant grid to the GPU, not the padded capacity ( #44618 )
...
Signed-off-by: Monishver Chandrasekaran <monishverchandrasekaran@gmail.com >
2026-10-19 23:07:12 +08:00
Avaya Aggarwal
c04f9a61f0
[Fix] diffusion: Fix VMoBA chunk selection bugs ( #44286 )
2026-10-19 23:04:26 +08:00
Siliang Chen and Cursor
7c4cc5a51c
[Perf] [AMD] attention: Spread uneven Triton decode batches across XCDs ( #41229 )
...
Co-authored-by: Cursor <cursoragent@cursor.com >
2026-10-19 23:03:27 +08:00
Sam Shleifer
96ba108072
[Fix] dsa-eagle-topk: Copy token keys and scales when relocating index cache ( #44564 )
2026-10-19 22:54:08 +08:00
Sam Shleifer
417606527d
[Fix] nvfp4: Own streamed tensors while waiting for paired weights ( #44500 )
2026-10-19 22:41:53 +08:00
pllimax
8c4680cb65
[CI] npu: Move llada2-mini FDFO test to full test ( #45153 )
2026-10-19 22:29:30 +08:00
lixiufei-leo and Xiufei Li
62cee1b15c
[Fix] sgl-router: Keep the kv-indexer connection through bursts of cancelled prefix queries ( #45100 )
...
Co-authored-by: Xiufei Li <294029388+lixiufei-leo@users.noreply.github.com >
2026-10-19 21:41:03 +08:00
zhaozx-cn
ed0b8f433f
[Fix] [NPU] kimi-k3: skip fused gmm1 for situ activation in Ascend runner ( #43837 )
2026-10-19 19:30:41 +08:00
Shangming Cai
0025f93944
[Fix] sgl-router: Refuse prefill workers that cannot compute before bootstrap under --decode-dispatch-limit ( #45120 )
2026-10-19 18:24:17 +08:00
Quanli Li and 全力
e45cbce599
[Fix] diffusion: Prevent int32 overflow in Qwen-Image-2.1-Turbo VAE RMSNorm ( #45118 )
...
Co-authored-by: 全力 <liquanli.lql@antgroup.com >
2026-10-19 18:01:02 +08:00
Niko Ma and inkcherry
a90e9c5dd8
[Fix] [AMD] mhc: Bound the Triton fused post+pre buffer cache ( #44443 )
...
Co-authored-by: inkcherry <mingzhi.liu@amd.com >
2026-10-19 01:37:50 -07:00
8111841103
[Perf] deepseek-v4.1: Replay the breakable prefill CUDA graph at any context length ( #43733 )
...
Co-authored-by: Oasis-Git <ayw.sirius19@gmail.com >
Co-authored-by: hnyls2002 <lsyincs@gmail.com >
2026-10-19 01:29:14 -07:00
33dd8f6a2e
[Fix] hybrid-SSM: DP attention failures at low concurrency ( #34535 )
...
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
Co-authored-by: Jimmy Shong <69131491+Jiminator@users.noreply.github.com >
Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com >
2026-10-19 16:24:41 +08:00
Thomas Wang
5d9c956f8e
[Doc] [AMD] dsv4: Update amd cookbook 1016 ( #45070 )
2026-10-19 16:11:11 +08:00
Po-Han Huang (NVIDIA) and Claude Opus 5.5
af9bb2a32d
[Doc] cookbook: Refresh SemiAnalysis AgentX launches with twelve open InferenceX PRs ( #45111 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-19 15:24:08 +08:00
Bingxu Chen
3d588779e1
[CI] [AMD] test: Run UMBP HiCache E2E through standalone DRAM server ( #40674 )
2026-10-19 00:23:52 -07:00
AMD-yanfeiwang
f86a04a331
[Feat] sgl-router: Add --decode-dispatch-limit to pick decode DP ranks near KV arrival ( #44402 )
2026-10-19 15:11:40 +08:00
a5117beb11
[Perf] mhc: Parallelize small-batch post across hidden tiles ( #44906 )
...
Co-authored-by: leonardHONG <178642616+leonardHONG@users.noreply.github.com >
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com >
2026-10-19 15:07:13 +08:00
ashwini rathi and Claude Opus 5.5
b6d3b7e163
[CI] [Intel] xpu: Disable L0 copy offload to fix first-H2D-copy hang ( #45013 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-19 14:48:45 +08:00
97857f9f96
[Doc] cookbook: GB300 full-set accuracy and tool follow-up output for Ling-3.1-flash ( #44307 )
...
Co-authored-by: andyluo7 <andy.luo@amd.com >
Co-authored-by: zijiexia <37504505+zijiexia@users.noreply.github.com >
2026-10-18 23:29:09 -07:00
desmond-intel
dc81f17b90
[Fix] [XPU] gemma3: Enable bidirectional image attention ( #44225 )
2026-10-19 14:08:24 +08:00
Liangsheng Yin
a3330237db
[Perf] deepseek-v4.1: Run decoder replay tail layers from CUDA graphs on eager prefill steps ( #45059 )
2026-10-18 23:07:50 -07:00
Trevor Morris
04c8c9703f
[Fix] docker: Install numactl in runtime images - needed for SGLANG_NUMA_BIND_V2 ( #43883 )
2026-10-18 22:55:58 -07:00
7c9a78efbe
[Feat] nemotron: Port Nemotron 3.5 Super serving changes ( #44518 )
...
Co-authored-by: BBuf <1182563586@qq.com >
Co-authored-by: elvischenv <219235043+elvischenv@users.noreply.github.com >
Co-authored-by: Jimmy Shong <jimmy.shong@mac.attlocal.net >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-19 13:27:22 +08:00
Ziru
f726461287
[Feat] [CPU] model: Support LTX and MOVA models on CPU ( #36632 )
2026-10-19 13:25:13 +08:00
Ziru and Niu Ziru
9a93a17c08
[Feat] [CPU] model: Support Muse models on CPU ( #44624 )
...
Co-authored-by: Niu Ziru <niuziru@gnr799217.jf.intel.com >
2026-10-19 13:12:32 +08:00
Mick and Mick Qian
2b75048e2a
[CI] diffusion: tighten H100 baselines and enable B200 performance checks ( #44996 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
2026-10-19 12:54:54 +08:00
Avaya Aggarwal
58ad0748a6
[Fix] diffusion: Unmerge LoRA layers the next adapter does not cover ( #44770 )
2026-10-19 12:33:40 +08:00
Xinyuan Tong and 文贤
266550048a
[Fix] bailing: MoE gate dtype follows config.router_dtype ( #44715 )
...
Co-authored-by: 文贤 <luoyuan.luo@antgroup.com >
2026-10-19 12:21:29 +08:00
Liangsheng Yin
82c9a8b037
[Perf] deepseek-v4.1: Default PD to MegaAttention, with the KV layout in the PD handshake ( #45075 )
2026-10-18 21:16:45 -07:00
Yuwei An
834bfe4ef0
[Fix] attention: Opt FlashMLA scheduler into dynamic shared memory ( #44580 )
2026-10-18 20:31:39 -07:00
Duyi-Wang
8d3eee46a7
[Perf] [AMD] deepseek-v4: Skip top-k refinement for proven positive-zero ties ( #44927 )
2026-10-19 11:27:47 +08:00
06342acad2
[Perf] deepseek-v4.1: Stage late-layer DP row counts through pinned memory ( #45062 )
...
Co-authored-by: weireweire <20922698+weireweire@users.noreply.github.com >
Co-authored-by: kpham-sgl <khoa.pham@radixark.ai >
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com >
2026-10-18 20:19:26 -07:00
f02458fff1
[Fix] [CPU] memory: Honor cgroup memory.max when sizing CPU memory ( #40066 )
...
Co-authored-by: Zhang, Mingxu <mingxu.zhang@intel.com >
Co-authored-by: MingxuZh <109504044+MingxuZh@users.noreply.github.com >
2026-10-19 11:11:44 +08:00
Alexandr
617c8b67b1
[Fix] Z-Image: Synchronize device after warmup request ( #44306 )
2026-10-19 05:36:16 +03:00
Ankur Singh and Claude Opus 5.5
7b1710324c
[Doc] cookbook: Refresh SemiAnalysis AgentX launches for Qwen3.8-Flash-Next H200 (v0.5.22, TP2 points) ( #45039 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Force merge to unblock SA AgentX submission
2026-10-19 10:31:51 +08:00
Kotthagattu Meher Sai
7f96dc6874
[Perf] [XPU] server_args: Enable fused QK-norm + RoPE by default ( #43345 )
2026-10-19 10:19:52 +08:00
642f43ce59
[Fix] diffusion: size the in-context slot from both input grids for wan-animate-2( #44090 )
...
Co-authored-by: Ankit Dhall <8938083+ankitdhall@users.noreply.github.com >
Co-authored-by: Mick <mickjagger19@icloud.com >
2026-10-19 10:11:58 +08:00
Khoa Pham
a7fd2e1158
[Feat] dsv4.1: Return prompt logprobs under encoder SWA bounded replay ( #45050 )
2026-10-18 19:08:20 -07:00
Wuhen Duan and Even Zhou
69076b9f9e
[CI] [NPU] tests: Reduce HiCache sizes and enable expandable pinned memory in focused tests ( #41675 )
...
Co-authored-by: Even Zhou <even.y.zhou@outlook.com >
2026-10-19 10:06:35 +08:00
fea48754de
[Feat] rust/grpc: Add ListModels, Detokenize, and Abort to api.v1 ( #42118 )
...
Signed-off-by: jain-ria <riajain@NVIDIA.com >
Co-authored-by: ishandhanani <82981111+ishandhanani@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Ishan Dhanani <ishandhanani@gmail.com >
2026-10-19 09:53:56 +08:00
Mohammad Miadh Angkad and Mohammad Angkad
377cd867fd
[Perf] dsa: Route the k-pool indexer top-k through the top-k v2 kernel ( #43775 )
...
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai >
2026-10-19 09:49:39 +08:00
Yuwei An
df078cc729
[Fix] DSv4.1: Support CUDA image requests with encoder bounded replay and CI Test ( #45048 )
2026-10-18 18:47:17 -07:00
Emre Albayrak and Xiaoyu Zhang
10ec53ac2f
[Perf] [NVIDIA] fp8_fa_sm120: Use the block-scaled MMA encoding ( #43390 )
...
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com >
2026-10-19 09:36:21 +08:00
Jimmy Shong
0dc6de2e74
[Doc] nemotron-3.5-super: update H200 and Spark wizard recipes ( #44587 )
2026-10-18 18:22:05 -07:00
A&W
cff88d6eac
[Fix] diffusion: Invalidate merged LoRA cache after adapter rewrites ( #44204 )
2026-10-19 01:08:57 +00:00
Liangsheng Yin
299da60131
[CI] test-utils: Abort the test run when its server dies inside a CustomTestCase method ( #45053 )
2026-10-18 18:00:06 -07:00
Xiaoyu Zhang and Claude Opus 5.5
329072d99a
[Refactor] kernels/diffusion/tests: Prune dead code and fix tests that never ran ( #44690 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-19 08:55:47 +08:00
Liangsheng Yin
59fec011dd
[Fix] deepseek-v4.1: Reject encoder replay on a chunk too small for a hit or on other models ( #45051 )
2026-10-18 17:55:35 -07:00
Liangsheng Yin
3c560089e1
[Refactor] scheduler: Bill encoder replay rows in _update_prefill_budget ( #45047 )
2026-10-18 17:17:38 -07:00
Liangsheng Yin
1637a4e305
[Fix] scheduler: Clear stale encoder_swa_reset in convert_decode_to_extend ( #45042 )
2026-10-18 17:09:19 -07:00
Liangsheng Yin
6708426728
[Fix] spec: Remove --speculative-skip-dp-mlp-sync ( #45043 )
2026-10-18 16:41:21 -07:00
Liangsheng Yin
3f13f5d966
[Fix] scheduler: Drop prefill embedding overrides in prepare_for_decode ( #45041 )
2026-10-18 15:54:20 -07:00
Liangsheng Yin
1461152cb0
[Fix] deepseek-v4.1: Leave room for encoder replay rows when sizing a prefill chunk ( #45040 )
2026-10-18 15:43:08 -07:00
Liangsheng Yin
fe89a7c890
[Fix] deepseek-v4.1: Start the PD decode draft SWA window at the prefill draft valid-from ( #45037 )
2026-10-18 15:06:20 -07:00
Liangsheng Yin
da414adda7
[Fix] engram: Clamp multimodal pad ids in host-built history rows ( #45038 )
2026-10-18 12:24:05 -07:00
Liangsheng Yin
0f4aedfecc
[Perf] deepseek-v4.1: Default decoder SWA bounded replay on for both PD roles on CUDA ( #45035 )
2026-10-18 11:53:35 -07:00
Ke Bao
4ec19ca27f
[Refactor] memory-pool: Build KV host pools from buffer info ( #40917 )
2026-10-18 23:21:16 +08:00
Ke Bao
bb1d4cddd9
[Refactor] hicache: Route Mamba host transfers from pool info ( #44968 )
2026-10-18 23:02:46 +08:00
Xinyuan Tong
d4e3c163c9
[CI] radix-cache: Register the Inkling cache linker KL test to extra-b ( #45004 )
2026-10-18 22:23:34 +08:00
e9b22a0818
[Perf] [AMD] monokernels: Let a plugin skip per-step attention metadata ( #44601 )
...
Co-authored-by: Kevin Mi <kevin.mi@radixark.ai >
Co-authored-by: kkHuang-amd <43161300+kkHuang-amd@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-18 14:04:57 +00:00
Cheng Wan
57db5089c2
[Perf] attention: Capture translated FA4 verify metadata for DSPARK ( #44973 )
2026-10-18 05:42:16 -07:00
Cheng Wan
2e9b2cacda
[Perf] speculative: Fuse draft write-window gathering and KV translation ( #44972 )
2026-10-18 05:41:34 -07:00
Cheng Wan
2a83471676
[Perf] mamba: Fuse strided state updates and replay index translation ( #44971 )
2026-10-18 05:40:57 -07:00
Cheng Wan
d318a546da
[Perf] unified-memory: Reduce allocation and SWA retirement overhead ( #44970 )
2026-10-18 05:40:25 -07:00
Cheng Wan
a5b21145e1
[Refactor] attention: Separate decode metadata from pool ownership ( #44969 )
2026-10-18 05:39:44 -07:00
3edb156dec
[Feat] unified-memory: Fuse drafts with sliding-window and recurrent-state layers ( #43326 )
...
Co-authored-by: Caihua Li <caihua.li@bytedance.com >
Co-authored-by: Cheng Wan <cheng.wan@radixark.ai >
2026-10-18 05:39:09 -07:00
caihuali95
19cb24f1e1
[Feat] unified-memory: Support asymmetric KV rows ( #43325 )
2026-10-18 05:37:50 -07:00
caihuali95 and Caihua Li
1d17066bad
[Perf] gdn: Select FlashInfer prefill for page-major pools ( #43171 )
...
Co-authored-by: Caihua Li <caihua.li@bytedance.com >
2026-10-18 05:37:24 -07:00
caihuali95 and Caihua Li
ba50ea3d77
[Perf] unified-memory: Free SWA segments without a host sync ( #43170 )
...
Co-authored-by: Caihua Li <caihua.li@bytedance.com >
2026-10-18 05:36:56 -07:00
caihuali95 and Caihua Li
47eeddcd18
[Fix] swa: Retain a whole window behind insertion pages ( #43168 )
...
Co-authored-by: Caihua Li <caihua.li@bytedance.com >
2026-10-18 05:36:02 -07:00
lixiufei-leo and Xiufei Li
fe1851c3c2
[Fix] sgl-router: Receive kv-indexer-bridge event batches outside tokio's coop budget ( #44305 )
...
Co-authored-by: Xiufei Li <294029388+lixiufei-leo@users.noreply.github.com >
2026-10-18 20:35:50 +08:00
WenhaoZhang
e17fed6ea7
[Perf] diffusion: fused gate/up + SwiGLU MLP on comfy_kitchen INT8 for Qwen-Image 2.1 ( #44196 )
2026-10-18 20:33:31 +08:00
Linda and Mick
680a2ff27e
[Perf] [NVIDIA] diffusion/FLUX.3: Run dynamic LoRA through PrimsTS GEMMs ( #44761 )
...
Co-authored-by: Mick <mickjagger19@icloud.com >
2026-10-18 20:32:19 +08:00
WenhaoZhang and Mick
1fd46ddd56
[Perf] diffusion: fused indexed modulation for FP32 AdaLN tables ( #44180 )
...
Co-authored-by: Mick <mickjagger19@icloud.com >
2026-10-18 20:31:36 +08:00
Shuwen Wang
8dee3f6bdb
[Fix] [H200] ci: Split GLM-5.3-Flash serving recipes into separate tests ( #44892 )
2026-10-18 19:26:40 +08:00
Shuwen Wang
f8e32706fc
[CI] radix-cache: Re-enable Inkling unified cache linker KL test ( #44938 )
2026-10-18 19:26:22 +08:00
AMD-yanfeiwang
a3abea3560
[Fix] sgl-router: Resend on a closed pooled connection; add --disable-circuit-breaker ( #44075 )
2026-10-18 18:23:18 +08:00
3b932cde26
[Fix] VLM: harden Kimi image cache identity across preprocessing and radix hashing ( #43043 )
...
Co-authored-by: Rahul Chalamala <22563365+rchalamala@users.noreply.github.com >
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
Co-authored-by: Liangsheng Yin <hnyls2002@gmail.com >
2026-10-18 17:39:37 +08:00
5a930cfd88
[Fix] glm4v: Clamp request video budgets to the deployment limit ( #43871 )
...
Co-authored-by: cipheraxat <cipheraxat@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-18 17:31:33 +08:00
Ilia Yastrebov and ishandhanani
6c3db0f520
[Perf] NIXL: prepared dlists for hybrid-model state transfer ( #44740 )
...
Co-authored-by: ishandhanani <82981111+ishandhanani@users.noreply.github.com >
2026-10-18 16:59:18 +08:00
Xinyuan Tong
80fef63786
[Fix] Inkling: frame constrained output in native message tokens ( #44716 )
2026-10-18 16:47:56 +08:00
Huang Long
889658a5be
[Fix] disaggregation: Reuse startup decode graphs across PD role switches ( #43760 )
2026-10-18 16:16:46 +08:00
+14
d33dd0566a
[Feat] srt: Add muse spark runtime, multimodal support and tests (3/5) ( #44666 )
...
Co-authored-by: 1am9trash <1am9trash@gmail.com >
Co-authored-by: Alex Nails <alex.nails@radixark.ai >
Co-authored-by: Chunan Zeng <zcnrex@gmail.com >
Co-authored-by: Claude Fable 5 <noreply@anthropic.com >
Co-authored-by: Ethan Che <eche@meta.com >
Co-authored-by: HAI <hixiao@gmail.com >
Co-authored-by: Jiajun Li <guapisolo@gmail.com >
Co-authored-by: Jiajun Li <jiajun.li@radixark.ai >
Co-authored-by: Jimmy Shiu <jimmysh341@gmail.com >
Co-authored-by: Jimmy Shong <69131491+Jiminator@users.noreply.github.com >
Co-authored-by: Liangsheng Yin <lsyincs@gmail.com >
Co-authored-by: Wookje Han <52435620+wookjeHan@users.noreply.github.com >
Co-authored-by: Wookje Han <wookjeh@nvl72d079-T17.cm.cluster >
Co-authored-by: Wookje Han <wookjeh@nvl72d094-T17.cm.cluster >
Co-authored-by: Wookje Han <wookjeh@nvl72d157-T16.cm.cluster >
Co-authored-by: Wookje Han <wookjeh@nvl72d170-T04.cm.cluster >
Co-authored-by: Wookje Han <wookjeh@nvl72d214-T17.cm.cluster >
Co-authored-by: Yuhao Yang <47235274+yhyang201@users.noreply.github.com >
Co-authored-by: Yuhao Yang <yhyang201@users.noreply.github.com >
Co-authored-by: Zijie Chen <300606707+zijiecode@users.noreply.github.com >
Co-authored-by: ethche <ethche@gmail.com >
Co-authored-by: jacky.cheng <yichiche@amd.com >
Co-authored-by: jianan-gu <jianan.gu@intel.com >
2026-10-18 01:00:24 -07:00
Shangming Cai
80914b7c9b
[Refactor] sgl-router: Rename --max-in-flight to --max-inflight-requests ( #44950 )
2026-10-18 15:33:58 +08:00
Yuwei An
3a6e1fcf77
[Perf] DSv4.1: Bound encoder BCG metadata memory ( #44963 )
2026-10-18 00:33:35 -07:00
Richard Wang and Mick Qian
de61ab53d4
[Perf] diffusion: Defer full GC collections while importing the runtime ( #44606 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
2026-10-18 15:31:10 +08:00
Zhaoyi Li
f1cf2e1005
[Fix] [AMD] mori: Look up elided DSA state entries by the paired dst index ( #44412 )
2026-10-18 00:29:22 -07:00
Zhaoyi Li
396e97e5dd
[CI] [AMD] ci/slurm: Add a Mooncake KV-transfer test to the MI355X disaggregation nightly ( #44409 )
2026-10-18 00:28:34 -07:00
Zhaoyi Li
8a4239fbdc
[Fix] [AMD] mooncake: Bump Mooncake to v0.3.14-rc1 for cross-node KV transfer ( #44404 )
2026-10-18 00:22:55 -07:00
Shuwen Wang
ba3fc9441d
[Fix] mamba: Release consumed lazy Mamba prefix locks before donation retry ( #43559 )
2026-10-18 15:16:31 +08:00
jacky.cheng and Mick Qian
f96a7326e6
[Fix] [AMD] diffusion: Prevent ComponentResidencyError after compile warmup ( #43952 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
2026-10-18 00:13:25 -07:00
Shuwen Wang
a11fa06be7
[Fix] hicache: Key MLA Mamba state per TP rank in the file backend ( #41907 )
2026-10-18 15:10:50 +08:00
Shuwen Wang
ffc4573c98
[Fix] hicache: Key MLA Mamba state per TP rank in the SeaweedFS backend ( #43816 )
2026-10-18 15:09:58 +08:00
Shuwen Wang
e6c2cd47a9
[Fix] hicache: Keep unified memory off the direct page_first_direct copies ( #43333 )
2026-10-18 15:09:26 +08:00
Liangsheng Yin
df966f2345
[Feat] deepseek-v4.1: Run encoder SWA bounded replay on PD prefill workers ( #44952 )
2026-10-18 00:07:55 -07:00
Ziang Li
520df8a491
[Perf] weight-sync: Allocate packed receive buffers from metadata ( #39264 )
2026-10-18 00:04:10 -07:00
Michael
00f6cdcd1b
[Fix] [AMD] kda: Reject match_cutedsl_decode on ROCm and skip its parity tests ( #44089 )
2026-10-18 00:01:40 -07:00
Michael
3626978a13
[Fix] [AMD] kda: Match the fused verify Q/K norm to its reference kernel ( #43991 )
2026-10-18 00:01:11 -07:00
Bowen Wang
3671503e44
[Feat] bench: report goodput and SLO attainment in bench_serving ( #43967 )
2026-10-18 13:31:48 +08:00
62d4261cf4
[Doc] sgl-router: Rewrite README and add architecture guide ( #43592 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Shangming Cai <csmthu@gmail.com >
2026-10-18 13:29:08 +08:00
Thomas Wang
e5dc10e73b
[Doc] [AMD] dsv41: Update amd cookbook 1016 ( #44903 )
2026-10-17 22:10:42 -07:00
Shangming Cai and Shangming Cai
1ff2c1d71e
[Fix] sgl-router: Fill the media fields when converting TextGenerate to Generate ( #44951 )
...
Co-authored-by: Shangming Cai <shangming.cai@radixark.ai >
2026-10-18 13:05:45 +08:00
Liangsheng Yin
d20d010574
[Fix] deepseek-v4.1: Exchange late-layer DP row counts before the forward ( #44948 )
2026-10-17 21:46:46 -07:00
Liangsheng Yin
6fe630ae14
[CI] deepseek-v4: Set the KV pool in the BCG replay contract test ( #44946 )
2026-10-17 20:59:03 -07:00
Baizhou Zhang
a8b939da43
Revert "[Doc] cookbook: Add Muse Spark Open SWA ratio calculator" ( #44947 )
2026-10-17 20:55:00 -07:00
140c57d5bf
[Perf] diffusion: classify residual_gate_add operands once per call ( #44478 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-18 11:51:52 +08:00
Liangsheng Yin and Vincent Zhong
ff73d63121
[Perf] deepseek-v4.1: Run encoder SWA bounded replay in the breakable prefill graph ( #44930 )
...
Co-authored-by: Vincent Zhong <207368749+vincentzed@users.noreply.github.com >
2026-10-17 20:51:18 -07:00
Chunan Zeng
a0b77db84d
[Perf] DFLASH: build the draft num_token_non_padded scalar with torch.full ( #44119 )
2026-10-17 20:40:16 -07:00
Kaushik Chatterjee
3acfa2efcc
[Feat] runtime: Support tokenized multimodal generation ( #42833 )
2026-10-18 12:35:51 +09:00
Richard Wang
01f3cba70e
[Fix] dLLM: Size decode graphs to the default max running requests ( #40941 )
2026-10-17 20:22:39 -07:00
531b426281
[Feat] diffusion/model: support Cosmos Dreams and Cosmos Multiview ( #39475 ) ( #39475 )
...
Co-authored-by: Kedi Wu <kediw@nvidia.com >
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
2026-10-18 11:15:57 +08:00
zijiexia
3b0a154e60
[Doc] cookbook: Add DeepSeek V4.1 Flash 1017 variant ( #44937 )
2026-10-17 20:04:41 -07:00
zijiexia and Claude Sonnet 5.5
df24bca4ba
[Doc] popular-models: Replace Kimi-K3 with Muse Spark Open ( #44693 )
...
Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com >
2026-10-17 19:54:45 -07:00
14ad64c411
[Perf] custom_op: call register_custom_op ops directly in plain eager execution ( #44158 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-18 10:10:10 +08:00
Mick and Claude Opus 5
d748bede96
[Feat] diffusion: selectable sequence-parallel strategies for LingBot causal attention ( #34385 )
...
Co-authored-by: Claude Opus 5 <noreply@anthropic.com >
2026-10-18 10:03:43 +08:00
8d02b69c11
[Perf] diffusion: Replay Triton launches without the runner's per-launch hook metadata ( #44471 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-18 09:58:54 +08:00
949e1a2080
[Perf] diffusion: Cache the fused linear-GELU verdict per linear ( #44643 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-18 09:55:29 +08:00
f53bd436c1
[Perf] distributed: cache the eager launch of custom all-reduce v2 per size ( #44642 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-18 09:54:58 +08:00
Yuwei An
b234b5c32e
[Refactor] TCPCG: Consolidate attention eager regions and isolate Inkling policy (4/9) ( #42288 )
2026-10-17 16:06:24 -07:00
Yuwei An
e4512a5a17
[Perf] DSv4.1: Reduce BCG metadata memory ( #44920 )
2026-10-17 16:05:28 -07:00
Liangsheng Yin
563cd60345
[Fix] [AMD] deepseek-v4.1: Reject attention DP with encoder SWA bounded replay on ROCm ( #44934 )
2026-10-17 15:47:41 -07:00
Yi (Vincent) Zhong
05ce0e980f
[Perf] deepseek-v4.1: Skip wasted layer 20 work on earlier prompt tokens under decoder bounded replay ( #42901 )
2026-10-17 14:12:00 -07:00
05c1539098
[Feat] unified_cache: Support hybrid SWA + Mamba models in the Mooncake direct linker ( #41343 )
...
Co-authored-by: Zhangheng <hzh0425@apache.org >
Co-authored-by: Teng Ma <sima.mt@alibaba-inc.com >
2026-10-17 13:24:08 -07:00
YAMY and Claude Opus 5.5
3b34d9f8b6
[Perf] [NVIDIA] qsa: Keep sparse decode scratch in FP8 for FP8 KV pools ( #44114 )
...
With an FP8 KV pool on SM10x, keep the packed sparse-decode K/V scratch in FP8 instead of dequantizing to BF16 (BF16-q + FP8-KV TRTLLM-gen decode). Tests cover FP8 pool -> FP8 scratch.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-17 11:50:38 -07:00
4eb606b2ae
[Feat] diffusion: support opt-in pi0.5 completion-boundary action batching ( #44634 )
...
Co-authored-by: Jiahua Huang <35223412+dadiaokua@users.noreply.github.com >
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
2026-10-17 20:49:44 +08:00
b15a78a881
[Fix] diffusion/lora: revalidate FLUX.2 FP8/NVFP4 fast paths after LoRA attach ( #40171 )
...
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
Co-authored-by: Mick <mickjagger19@icloud.com >
2026-10-17 20:48:14 +08:00
23f8e6f7ec
[Fix] diffusion: keep ZMQ ingress sockets on their configured port ( #43732 )
...
Co-authored-by: Sylvan Liu <syliu@cse-ai-16.amd.com >
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
Co-authored-by: Mick <mickjagger19@icloud.com >
2026-10-17 20:47:29 +08:00
+3
d4191a4624
[Fix] diffusion: stabilize FLUX.3 serving and reuse static conditioning ( #44493 )
...
Co-authored-by: Yihao Wang <42559837+AgainstEntropy@users.noreply.github.com >
Co-authored-by: Dom Brown <3886319+DomBrown@users.noreply.github.com >
Co-authored-by: niehen6174 <1639206518@qq.com >
Co-authored-by: Mick Qian <mickqian@radixark.ai >
Co-authored-by: Nikita Korobov <14355239+nekorobov@users.noreply.github.com >
Co-authored-by: Daniel Campora <961215+dcampora@users.noreply.github.com >
Co-authored-by: Richard Wang <11150595+rwang5203@users.noreply.github.com >
Co-authored-by: BBuf <1182563586@qq.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
Co-authored-by: Fabio Camaiora <fcamaiora@nvidia.com >
Co-authored-by: Linda-Stadter <lstadter@nvidia.com >
2026-10-17 20:46:13 +08:00
9771e5a4f0
[Feat] diffusion: Per-task prompt enhancer config and the FLUX.3 Klein upsampler protocol ( #44662 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-17 20:45:07 +08:00
Mick and Mick Qian
e01dfd3cbc
[Refactor] multimodal: Replace padding strategies with helpers ( #44845 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
2026-10-17 20:43:02 +08:00
Shuwen Wang
86e67f7d93
[Fix] unified_cache: Prevent misaligned Mamba checkpoints after speculative truncation ( #43387 )
2026-10-17 20:20:49 +08:00
SuperSong
025655068c
[Fix] unified-memory: Avoid gather scratch during Mamba and Conv compaction ( #44097 )
2026-10-17 04:56:41 -07:00
SuperSong
ee30dc4d40
[Fix] unified-memory: Bound joint reclaim and preserve cached prefixes ( #39294 )
...
Bound joint FULL/SWA/Mamba reclaim so feasible allocations recover capacity without evicting unnecessary cached prefixes, and impossible demands preserve the cache. Use exact unsharded paged-extend demand and update allocation regression fixtures.
2026-10-17 04:46:48 -07:00
Shuwen Wang
ae28306a74
[Fix] cuda_graph: Wait for in-flight layer-wise cache loads before prefill graph replay ( #44220 )
2026-10-17 19:13:57 +08:00
Shuwen Wang and Zhangheng
5fcb07da34
[Fix] hicache: Key MLA Mamba state per TP rank in the HF3FS backend ( #43817 )
...
Co-authored-by: Zhangheng <hzh0425@apache.org >
2026-10-17 11:08:17 +00:00
Woody-Hu and Mick
465ddf9758
[Feat] diffusion/VSA-H3: support sm80 (A100) with a raw-pointer Triton load path ( #43823 )
...
Co-authored-by: Mick <mickjagger19@icloud.com >
2026-10-17 18:47:19 +08:00
400c1dfe6e
[Perf] distributed: skip the DLPack re-wrap of custom all-reduce results ( #44251 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-17 18:44:31 +08:00
pllimax
eb5739b30e
[CI] pr-test-npu: Temporarily disable recommend-tests-from-coverage job ( #44881 )
2026-10-17 18:21:17 +08:00
Khoa Pham and Claude Fable 5.1
a126e8d640
[Perf] deepseek_v4: fold the MegaAttention query layout into wq_b for block-FP8-as-MXFP8 checkpoints ( #44838 )
...
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com >
2026-10-17 02:45:53 -07:00
Mick and Mick Qian
80f40522cd
[UX] diffusion: log LoRA merge decisions only on first evaluation ( #44376 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
2026-10-17 16:47:07 +08:00
56a20d7e34
[Perf] kernels: Enter a JIT kernel's device guard only when the device differs ( #44644 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-17 16:43:27 +08:00
f75232cb42
[Fix] hicache: Order write-back host copies after the forward that produced the KV ( #44800 )
...
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com >
Co-authored-by: xiezhq-hermann <xiezhq-hermann@users.noreply.github.com >
2026-10-17 01:31:40 -07:00
0d13efb2b8
[Feat] rust-server: Serve /v1/tokenize and /v1/detokenize from the Rust frontend ( #44767 )
...
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com >
Co-authored-by: shodoco <shodoco@users.noreply.github.com >
2026-10-17 01:30:50 -07:00
a2953d7b7f
[Fix] deepseek-v4.1: Serve prompt-logprob requests under decoder SWA bounded replay ( #44850 )
...
Co-authored-by: Pedram Ghazi <pedram.ghazi@amd.com >
Co-authored-by: David Orman <ormandj@corenode.com >
2026-10-17 01:25:33 -07:00
McZyWu
7a8804edb5
[Fix] [NPU] qwen: Fix shared-expert routing with Ascend FuseEP ( #40516 )
2026-10-17 16:19:44 +08:00
Liangsheng Yin
ef26f6ad49
[Fix] deepseek-v4.1: Keep the cached KV of folded encoder replay rows ( #44857 )
2026-10-17 00:56:18 -07:00
Khoa Pham and Claude Fable 5.1
bce7b27aa6
[Perf] deepseek_v4: run FlashInfer autotune for block-FP8 models served as MXFP8 ( #44842 )
...
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com >
2026-10-17 00:19:35 -07:00
01dc08e23f
[Refactor] dsv41: Shard Engram tables on hash-column boundaries ( #42787 )
...
Co-authored-by: Claude Fable 5 <noreply@anthropic.com >
Co-authored-by: hnyls2002 <lsyincs@gmail.com >
2026-10-17 00:15:49 -07:00
Khoa Pham and Claude Fable 5.1
14c4dda24e
[Perf] deepseek_v4: fuse block-FP8 shared experts into MegaMoE for any 32-aligned block size ( #44837 )
...
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com >
2026-10-16 23:47:41 -07:00
Mohammad Miadh Angkad and Mohammad Angkad
9c7b338a19
[CI] jit-kernel: Honor continue_on_error on scheduled runs ( #44645 )
...
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai >
2026-10-16 22:29:01 -07:00
Mohammad Miadh Angkad and Mohammad Angkad
2d8a05e719
[Test] attention: Fix replicated attention test missing full_attention_interval ( #44602 )
...
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai >
2026-10-16 22:28:37 -07:00
Liangsheng Yin
9259553f09
[Fix] megamoe: Support EPLB rebalance on SM100 ( #44847 )
2026-10-16 22:22:40 -07:00
Mohammad Miadh Angkad and Mohammad Angkad
3fc08b939a
[Feat] server-args: Add --enable-hicache alias for --enable-hierarchical-cache ( #44849 )
...
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai >
2026-10-17 13:22:34 +08:00
Mohammad Miadh Angkad and Mohammad Angkad
4fc86f73c2
[CI] glm5.2: Keep EOS in HiSparse needle test output ( #44597 )
...
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai >
2026-10-17 12:20:51 +08:00
Liangsheng Yin
4cbea9d3c7
[Doc] cookbook: Allow PD with DSpark for DeepSeek-V4.1 ( #44848 )
2026-10-16 21:17:32 -07:00
Xinyuan Tong
86cc2e02ec
[Fix] gateway: Drop unused npyz dev-dependency to unbreak builds after zip yank ( #44720 )
2026-10-16 20:56:31 -07:00
Ziang Li
29a7e19de1
[Refactor] nvfp4_online: Use explicit FlashInfer 4over6 config ( #41745 )
2026-10-16 20:56:10 -07:00
feixiangpeng and zijiexia
9590dc80cc
[Doc] hicache: Explain HiCache host pool sizing with a storage backend ( #44790 )
...
Co-authored-by: zijiexia <37504505+zijiexia@users.noreply.github.com >
2026-10-17 11:28:37 +08:00
Shuwen Wang
6a39ddf1f2
[Fix] unified_cache: Resize test allocator before allocating backed pages ( #44843 )
2026-10-16 20:21:49 -07:00
3603824541
[Perf] DSv4.1: Enable decoder bounded replay by default ( #44403 )
...
Co-authored-by: hnyls2002 <lsyincs@gmail.com >
Co-authored-by: Liangsheng Yin <hnyls2002@gmail.com >
2026-10-16 19:52:02 -07:00
McZyWu
2d3dd3168d
[Fix] [NPU] attention: Use FIA TND for MLA prefix attention and add Gemma norm fallback ( #40131 )
2026-10-17 10:12:41 +08:00
Khoa Pham
ff979e0816
[Fix] megamoe: Respect request-pool limits in decode validation ( #44816 )
2026-10-16 18:56:38 -07:00
5cbe2371a5
[Feat] rust-server: Report Python's launch-time /server_info fields from the Rust frontend ( #44764 )
...
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com >
Co-authored-by: shodoco <shodoco@users.noreply.github.com >
2026-10-16 18:14:50 -07:00
Shuwen Wang
33b5d26fa6
[CI] lora: Use the canonical Llama model ID in eviction tests ( #44771 )
2026-10-16 18:13:37 -07:00
b62a48ee15
[Feat] sglang-server: Serve /flush_cache from the Rust frontend ( #44766 )
...
Co-authored-by: fei-xx <fei-xx@users.noreply.github.com >
Co-authored-by: shodoco <shodoco@users.noreply.github.com >
2026-10-16 17:24:01 -07:00
d8dd49d21d
[Fix] srt: Apply --gc-threshold in every server process and freeze every tokenizer worker ( #44785 )
...
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com >
Co-authored-by: joshuadeng <joshuadeng@users.noreply.github.com >
2026-10-16 16:55:54 -07:00
d230f2c35b
[Feat] profiler: Add the forward pass id to detailed step span labels ( #44775 )
...
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com >
Co-authored-by: charlotte12l <charlotte12l@users.noreply.github.com >
2026-10-16 16:51:20 -07:00
36cf028346
[Perf] unified-memory: Prewarm the compaction move at startup and copy whole page envelopes by page id ( #44592 )
...
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com >
Co-authored-by: ZYHowell <ZYHowell@users.noreply.github.com >
2026-10-16 16:47:50 -07:00
Khoa Pham and Claude Fable 5.1
aaf5f397a5
[Feat] dsv4.1: Support encoder SWA bounded replay with CUDA DP attention ( #43813 )
...
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com >
2026-10-16 15:38:33 -07:00
Yuwei An
8e3f11b219
[Perf] DSv4.1: Skip discarded KV rows during window commit ( #44783 )
2026-10-16 15:05:45 -07:00
a46fe83c70
[CI] tests: Fork the gloo exchange test's ranks from a preloaded forkserver ( #44679 )
...
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com >
Co-authored-by: fei-xx <fei-xx@users.noreply.github.com >
2026-10-16 14:53:21 -07:00
Aarav and Aarav
d5d87899f9
[Fix] DSA: skip-topk pipeline handoff for owed outputs ( #43832 )
...
Co-authored-by: Aarav <211792885+AaravMehta-07@users.noreply.github.com >
2026-10-16 14:35:23 -07:00
Khoa Pham and Trevor Morris
71b60fa46c
[Fix] engram: Avoid blocking history copies during PD admission ( #44787 )
...
Co-authored-by: Trevor Morris <tmorris@nvidia.com >
2026-10-16 14:29:05 -07:00
2948662026
[Perf] dp-attention: Exchange the scheduler's MLP sync rows with gather + broadcast on gloo ( #44622 )
...
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com >
Co-authored-by: hanming-lu <hanming-lu@users.noreply.github.com >
2026-10-16 13:53:41 -07:00
metamergebot and metamergebot
f39c2693aa
[CI] rust-server: Raise the ChunkEvent frame-size cap to 152 bytes ( #44773 )
...
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com >
2026-10-16 13:47:52 -07:00
Chunan Zeng
07950be745
[Perf] [NVIDIA] custom-all-reduce: Add 2shot_push all-reduce and retune SM100 TP4/TP8 ( #42811 )
2026-10-16 13:43:18 -07:00
Mohammad Miadh Angkad and Mohammad Angkad
21791e7023
[CI] [NVIDIA] DSv4.1-Flash: Split GB300 accuracy test into one file per config ( #44696 )
...
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai >
2026-10-16 13:40:03 -07:00
metamergebot and metamergebot
a31a8c2bbd
[CI] tests: Let test_hicache_storage size its host cache to the runner ( #44729 )
...
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com >
2026-10-16 13:38:39 -07:00
Liangsheng Yin
a18a908066
[Test] mem_cache: Free SWA peers below the eviction floor in the SWA repair test ( #44777 )
2026-10-16 12:41:39 -07:00
+14
456231e251
[Feat] kernels: Add muse spark operators and collective GEMM (2/5) ( #44665 )
...
Co-authored-by: 1am9trash <1am9trash@gmail.com >
Co-authored-by: Alex Nails <alex.nails@radixark.ai >
Co-authored-by: Chunan Zeng <zcnrex@gmail.com >
Co-authored-by: Claude Fable 5 <noreply@anthropic.com >
Co-authored-by: Ethan Che <eche@meta.com >
Co-authored-by: HAI <hixiao@gmail.com >
Co-authored-by: Jiajun Li <guapisolo@gmail.com >
Co-authored-by: Jiajun Li <jiajun.li@radixark.ai >
Co-authored-by: Jimmy Shiu <jimmysh341@gmail.com >
Co-authored-by: Jimmy Shong <69131491+Jiminator@users.noreply.github.com >
Co-authored-by: Liangsheng Yin <lsyincs@gmail.com >
Co-authored-by: Wookje Han <52435620+wookjeHan@users.noreply.github.com >
Co-authored-by: Wookje Han <wookjeh@nvl72d079-T17.cm.cluster >
Co-authored-by: Wookje Han <wookjeh@nvl72d094-T17.cm.cluster >
Co-authored-by: Wookje Han <wookjeh@nvl72d157-T16.cm.cluster >
Co-authored-by: Wookje Han <wookjeh@nvl72d170-T04.cm.cluster >
Co-authored-by: Wookje Han <wookjeh@nvl72d214-T17.cm.cluster >
Co-authored-by: Yuhao Yang <47235274+yhyang201@users.noreply.github.com >
Co-authored-by: Yuhao Yang <yhyang201@users.noreply.github.com >
Co-authored-by: Zijie Chen <300606707+zijiecode@users.noreply.github.com >
Co-authored-by: ethche <ethche@gmail.com >
Co-authored-by: jacky.cheng <yichiche@amd.com >
Co-authored-by: jianan-gu <jianan.gu@intel.com >
2026-10-16 10:49:35 -07:00
ormandj and hnyls2002
80c54e6ac5
[Fix] [AMD] hicache: Serve encoder-replay prefix hits with a paged DSpark draft SWA ( #42716 )
...
Co-authored-by: hnyls2002 <lsyincs@gmail.com >
2026-10-16 10:45:16 -07:00
553068af25
[Fix] rust-server: Honour cache_salt and require_reasoning on /generate ( #44533 )
...
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com >
Co-authored-by: shodoco <shodoco@users.noreply.github.com >
2026-10-16 10:38:09 -07:00
453b251dbd
[Fix] hicache: Bound the storage prefetch timeout from the first blocked admission poll ( #44614 )
...
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com >
Co-authored-by: xiezhq-hermann <xiezhq-hermann@users.noreply.github.com >
2026-10-16 10:37:11 -07:00
metamergebot and jiayisuse
989ccf4863
[Feat] disaggregation: Let a KV transfer backend skip the checksum for an incomparable peer ( #44605 )
...
Co-authored-by: jiayisuse <jiayisuse@users.noreply.github.com >
2026-10-16 10:36:46 -07:00
63a9b7a713
[Feat] http_server: Make the multi-worker ASGI app path configurable ( #44583 )
...
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com >
Co-authored-by: joshuadeng <joshuadeng@users.noreply.github.com >
2026-10-16 10:36:35 -07:00
26a2f18c02
[Feat] sglang-server: serve /v1/loads from the Rust frontend ( #44532 )
...
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com >
Co-authored-by: shodoco <shodoco@users.noreply.github.com >
2026-10-16 10:35:37 -07:00
Xinyuan Tong
a038a0dca0
[Fix] dsv41: Preserve tool-result image order and current-turn thinking ( #44751 )
2026-10-16 23:39:50 +08:00
Duyi-Wang
ed2ebad342
[Doc] [AMD] cookbook: Refresh DeepSeek V4 AgentX PD recipe optimizations ( #44454 )
2026-10-16 23:34:01 +08:00
Jimmy Shong and Jimmy Shong
2c2986be2b
docs: add Beam 1.0 deployment cookbook ( #44694 )
...
Co-authored-by: Jimmy Shong <jimmy.shong@mac.attlocal.net >
2026-10-16 13:29:14 +00:00
Benjamin Truong and Claude Opus 5.5
7c99ace127
[Fix] dsa: Fix ragged seed indices before MTP draft decode ( #42565 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-16 19:37:41 +08:00
Po-Han Huang (NVIDIA) and Claude Opus 5.5
5e240a200d
[Doc] cookbook: Refresh SemiAnalysis AgentX launches for Qwen3.5 and DeepSeek-V4.1-Flash GB300 ( #44712 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-16 19:31:08 +08:00
Xinyi Song and HAI
4afae5568b
[Perf] [AMD] dsv4: Add a separate MegaMoE decode instance ( #44657 )
...
Co-authored-by: HAI <hixiao@gmail.com >
2026-10-16 03:11:02 -07:00
2adc76ad61
[Perf] [AMD] dsv4: reuse the q_lora fp8 quant for the C4 indexer wq_b (4/8) ( #41903 )
...
Co-authored-by: kk <43161300+kkHuang-amd@users.noreply.github.com >
Co-authored-by: Duyi-Wang <duyi.wang@amd.com >
2026-10-16 02:55:25 -07:00
YC Yen-Ching Tseng
9246acd462
[CI] [AMD] aiter: pin aiter commit to 2bb549 for rocm10.1 gfx1250 ( #44620 )
2026-10-16 17:50:20 +08:00
kk and Duyi-Wang
baf451fcd6
[Perf] [AMD] dsv4: carry aiter DSv4 GEMM tuned rows and mHC tile_m=16 until aiter ships them (7/8) ( #44008 )
...
Co-authored-by: Duyi-Wang <duyi.wang@amd.com >
2026-10-16 02:47:41 -07:00
f05da43d44
[Feat] [AMD] disagg: Support DSV4 decode radix cache on ROCm unified KV (bf16/fp8, DSpark) ( #30929 )
...
Co-authored-by: HAI <hixiao@gmail.com >
Co-authored-by: Duyi-Wang <duyi.wang@amd.com >
2026-10-16 02:39:27 -07:00
42eb556e66
[Perf] [AMD] dsv4: optional e8m0 mxscale GEMM for dense 128x128 fp8 linears (6/8) ( #44007 )
...
Co-authored-by: Duyi-Wang <duyi.wang@amd.com >
Co-authored-by: HAI <hixiao@gmail.com >
2026-10-16 02:28:31 -07:00
caihuali95
8d229d96e0
[Fix] unified-memory: enabling trtllm_mha with decode graphs and speculative decoding ( #42930 )
2026-10-16 02:24:06 -07:00
kk and Duyi-Wang
6efc96abf9
[Perf] [AMD] dsv4: FP4 indexer on aiter's row-group kernel for decode and target-verify (5/8) ( #44006 )
...
Co-authored-by: Duyi-Wang <duyi.wang@amd.com >
2026-10-16 02:16:06 -07:00
kk and Duyi-Wang
9fde8bcc46
[Perf] [AMD] moe: small-batch sort with histogram + bitonic sort for 32 < P <= 256 (3/8) ( #44004 )
...
Co-authored-by: Duyi-Wang <duyi.wang@amd.com >
2026-10-16 02:09:32 -07:00
kk and Duyi-Wang
f0ff18353d
[Perf] [AMD] dsv4: fp8 verify writes the SWA ring in the fused norm/rope kernel (2/8) ( #44003 )
...
Co-authored-by: Duyi-Wang <duyi.wang@amd.com >
2026-10-16 01:58:00 -07:00
Mingyi and Zijie Xia
56a3c4fa16
[Doc] cookbook: Add RTX Spark Qwen3.6 NVFP4 recipe ( #44707 )
...
Co-authored-by: Zijie Xia <zijie.xia@radixark.ai >
2026-10-16 01:55:41 -07:00
Liangsheng Yin
81c9d804b0
[CI] pr-check-comment: Comment on PRs when lint or title check fails ( #44708 )
2026-10-16 01:54:38 -07:00
kk and Duyi-Wang
ab5858ac8a
[Perf] [AMD] dsv4: fp8 unified_kv decode/verify on FlyDSL v4 nm decode with a latency-model split plan (1/8) ( #44016 )
...
Co-authored-by: Duyi-Wang <duyi.wang@amd.com >
2026-10-16 01:50:16 -07:00
Xinyuan Tong and zijiexia
9baace8456
[Doc] cookbook: refresh MiniCPM-V 4.7 DSPARK benchmarks at the final model PR head ( #44310 )
...
Co-authored-by: zijiexia <37504505+zijiexia@users.noreply.github.com >
2026-10-16 01:48:25 -07:00
3d61bdd2c9
[Feat] sgl-router: Serve the remaining native gRPC inference RPCs ( #42681 )
...
Co-authored-by: Kan Wu <kan.wu@radixark.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Shangming Cai <csmthu@gmail.com >
2026-10-16 16:23:17 +08:00
kk and wunhuang
536cdb4b04
[Feat] [AMD] docker: Install the sgl-amd-mono monokernel plugin in the ROCm image ( #44621 )
...
Co-authored-by: wunhuang <wunhuang@amd.com >
2026-10-16 01:20:40 -07:00
Liangsheng Yin
fad7d95a7f
[Fix] deepseek-v4.1: Keep the DSpark draft SWA out of the radix tree ( #44673 )
2026-10-16 01:18:58 -07:00
Baizhou Zhang
06b81b120e
[CI] deepgemm: Increase B200 release test timeout to 180 minutes ( #44706 )
2026-10-16 01:14:54 -07:00
kk and wunhuang
bdc4ec15ce
[Perf] [AMD] dspark: No per-step host sync in decode (1/4) ( #44405 )
...
Co-authored-by: wunhuang <wunhuang@amd.com >
2026-10-16 01:13:37 -07:00
Ziru and Niu Ziru
5dd37af412
[Doc] [XPU] Cookbook: Update the XPU Docker version to 0.5.22 ( #44647 )
...
Co-authored-by: Niu Ziru <niuziru@gnr799217.jf.intel.com >
2026-10-16 16:13:24 +08:00
jiaqiang-dot-liu
a448fdf316
[Perf] [AMD] qwen3: Fuse QK-norm + RoPE + KV cache write for dense Qwen3 decode ( #44267 )
2026-10-16 00:51:55 -07:00
amote-i
9d394dfbd7
[Doc] [NPU] docs: refresh associated dependency versions in npu docs ( #44695 )
2026-10-16 15:48:37 +08:00
Ziang Li
7098ec387c
[Feat] FlashInfer: Add NVFP4 W4A16 and SiTU support to MegaMoE ( #44036 )
2026-10-16 00:38:50 -07:00
Liangsheng Yin and Jingshu Chen
3a5d46e946
[Perf] pd: Stage decode-side KV preallocation lengths through pinned memory ( #38955 )
...
Co-authored-by: Jingshu Chen <96519051+elmo2019@users.noreply.github.com >
2026-10-16 00:34:56 -07:00
Xiaoyu Zhang
51faf52305
[Fix] monokernels: Honor resolved CUDA graph configuration ( #44630 )
2026-10-16 15:06:14 +08:00
Baizhou Zhang
dfc89580f1
[Doc] cookbook: Add Muse Spark Open SWA ratio calculator ( #44687 )
2026-10-16 00:00:12 -07:00
amote-i
92580581da
[Doc] [NPU] docs: Freeze qwen3-235b-a22b recommended image to a fixed version ( #44650 )
2026-10-16 14:14:05 +08:00
Mohammad Miadh Angkad and Mohammad Angkad
44be30d216
[Chore] deps: Bump FlashInfer to 0.7.2 ( #44591 )
...
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai >
2026-10-15 22:51:21 -07:00
Shu Wang
83940471b4
[Fix] [NVIDIA] allreduce-fusion: Decline MNNVL fusion across NVLink cliques ( #44387 )
2026-10-15 22:50:33 -07:00
+14
e1d20574da
[Chore] docker: Add muse spark CUDA image and DeepGEMM patches (1/5) ( #44664 )
...
Co-authored-by: 1am9trash <1am9trash@gmail.com >
Co-authored-by: Alex Nails <alex.nails@radixark.ai >
Co-authored-by: Chunan Zeng <zcnrex@gmail.com >
Co-authored-by: Claude Fable 5 <noreply@anthropic.com >
Co-authored-by: Ethan Che <eche@meta.com >
Co-authored-by: HAI <hixiao@gmail.com >
Co-authored-by: Jiajun Li <guapisolo@gmail.com >
Co-authored-by: Jiajun Li <jiajun.li@radixark.ai >
Co-authored-by: Jimmy Shiu <jimmysh341@gmail.com >
Co-authored-by: Jimmy Shong <69131491+Jiminator@users.noreply.github.com >
Co-authored-by: Liangsheng Yin <lsyincs@gmail.com >
Co-authored-by: Wookje Han <52435620+wookjeHan@users.noreply.github.com >
Co-authored-by: Wookje Han <wookjeh@nvl72d079-T17.cm.cluster >
Co-authored-by: Wookje Han <wookjeh@nvl72d094-T17.cm.cluster >
Co-authored-by: Wookje Han <wookjeh@nvl72d157-T16.cm.cluster >
Co-authored-by: Wookje Han <wookjeh@nvl72d170-T04.cm.cluster >
Co-authored-by: Wookje Han <wookjeh@nvl72d214-T17.cm.cluster >
Co-authored-by: Yuhao Yang <47235274+yhyang201@users.noreply.github.com >
Co-authored-by: Yuhao Yang <yhyang201@users.noreply.github.com >
Co-authored-by: Zijie Chen <300606707+zijiecode@users.noreply.github.com >
Co-authored-by: ethche <ethche@gmail.com >
Co-authored-by: jacky.cheng <yichiche@amd.com >
Co-authored-by: jianan-gu <jianan.gu@intel.com >
2026-10-15 22:46:22 -07:00
Yuwei An
06934575b9
[Fix] DSv4.1: Fix BCG image handling and add multimodal bounded replay coverage ( #44131 )
2026-10-15 22:32:55 -07:00
2a08e261ec
[Feat] MiniCPM-V-4.7: add model support ( #41993 )
...
Co-authored-by: Xinyuan Tong <115166877+JustinTong0323@users.noreply.github.com >
Co-authored-by: Xinyuan Tong <xinyuantong.cs@gmail.com >
2026-10-16 13:24:48 +08:00
a2981ab8ea
[Feat] sgl-router: Serve native gRPC ChatComplete ( #42674 )
...
Co-authored-by: Kan Wu <kan.wu@radixark.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-16 13:11:36 +08:00
Po-Han Huang (NVIDIA) and Claude Opus 5.5
fd4b75c696
[Doc] cookbook: Add SemiAnalysis AgentX launches to Muse Spark Open, refresh DeepSeek-V4.1-Flash ( #44654 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-16 12:45:07 +08:00
Kolade Fajimi and Xiaoyu Zhang
5117bfbaeb
[Fix] compressed-tensors: Serve NVFP4 W4A4 linears via FP4 Marlin when --fp4-gemm-backend marlin is set on SM100 ( #44213 )
...
Signed-off-by: kolade fajimi <fajimikolade23@gmail.com >
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com >
2026-10-16 12:36:02 +08:00
HuangJi
6d70787398
[Fix] diffusion: Align Qwen-Image 2.1 resolution and handle RGBA JPEG output ( #44275 )
2026-10-16 12:35:04 +08:00
Mick and Mick Qian
e5274752a8
[Doc] diffusion: align PR title conventions ( #44646 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
2026-10-16 11:29:20 +08:00
Mick
3420baea8c
[diffusion] CI: batch performance failure-policy runner scenarios ( #44508 )
2026-10-16 11:04:05 +08:00
Mohammad Miadh Angkad and Mohammad Angkad
6231a617d3
[Test] diffusion: Load NVFP4 block scales through the weight loader in tests ( #44595 )
...
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai >
2026-10-16 10:56:27 +08:00
ashwini rathi and Claude Opus 5.5
75e25a21bd
[Fix] [XPU] nightly: Fix rerank, zimage, flux2 and EAGLE parity ( #44100 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-16 10:27:51 +08:00
Chetan Kumar Verma and Claude Sonnet 5
75e77fca25
[Fix] wan: fix opt-in Wan2.2 progressive mode and one-step transitions ( #43768 )
...
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com >
2026-10-16 10:22:09 +08:00
Shuwen Wang
f9df33494a
[Fix] scheduler: Keep PD decode non-idle while deferred KV releases are held ( #44386 )
2026-10-15 19:17:51 -07:00
Chetan Kumar Verma and Claude Opus 5.5
1e33d2c4bd
[Perf] [XPU] usp: Enable packed Ulysses QKV all-to-all ( #39989 )
...
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
2026-10-16 10:08:50 +08:00
4e85e8ed37
[Fix] imports: Bind FlashInfer lazily at four import sites so the server starts without it ( #44430 )
...
Co-authored-by: fei-xx <fei-xx@users.noreply.github.com >
Co-authored-by: yeqcharlotte <yeqcharlotte@users.noreply.github.com >
2026-10-15 18:52:33 -07:00
shodoco and metamergebot
2957ff3165
[Fix] sglang-server: Keep the matched stop token in output_ids ( #42637 )
...
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com >
2026-10-15 18:51:47 -07:00
sglang-bot and sglang-bot
c15edf314d
[Doc] blogs: Sync LMSYS SGLang blog cards ( #44616 )
...
Co-authored-by: sglang-bot <sglang-bot@users.noreply.github.com >
2026-10-16 00:52:14 +00:00
e3961eb6fa
[Fix] utils: Allowlist torch.bcomplex32 in SafeUnpickler ( #44440 )
...
Co-authored-by: fei-xx <fei-xx@users.noreply.github.com >
Co-authored-by: zx3xyy <zx3xyy@users.noreply.github.com >
2026-10-15 17:50:43 -07:00
f110b2f801
[Feat] rust-server: Report cached_tokens and reasoning_tokens in meta_info ( #44531 )
...
Co-authored-by: fei-xx <fei-xx@users.noreply.github.com >
Co-authored-by: shodoco <shodoco@users.noreply.github.com >
2026-10-15 17:50:27 -07:00
d393298736
[Fix] tokenizer_manager: re-report only emitted customized_info keys on cumulative streams ( #44537 )
...
Co-authored-by: fei-xx <fei-xx@users.noreply.github.com >
Co-authored-by: shodoco <shodoco@users.noreply.github.com >
2026-10-15 17:49:49 -07:00
metamergebot and Ye Qi
bda6ef7e01
[Perf] [NVIDIA] scheduler: Pageable host staging and inline hidden-state/DSpark readback under Confidential Computing ( #44568 )
...
Co-authored-by: Ye (Charlotte) Qi <7269575+yeqcharlotte@users.noreply.github.com >
2026-10-15 17:48:50 -07:00
ed8c3139f1
[Fix] deepseek-v4.1: Keep the full SWA window for short bounded-replay extends ( #41303 )
...
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com >
Co-authored-by: hnyls2002 <lsyincs@gmail.com >
2026-10-15 17:32:50 -07:00
Khoa Pham
a93f16f0cf
[Perf] deepseek-v4.1: Integrate FlashMLA MegaAttention with NVFP4 KV ( #43922 )
2026-10-15 17:21:43 -07:00
Liangsheng Yin
d14fa5db9b
[Perf] pd: Copy prebuilt decode batch inputs to the GPU without host syncs ( #44598 )
2026-10-15 17:20:02 -07:00
Khoa Pham
4cca7fcd3e
chore: bump sgl-kernel version to 0.4.10 ( #44590 )
2026-10-15 15:50:51 -07:00
ec7ab9d0c2
[Feat] [AMD] monokernels: one runtime switch for vendor monokernel plugins ( #44353 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: wunhuang <wunhuang@amd.com >
2026-10-15 15:50:25 -07:00
0b1a73599a
[Feat] rust-server: Apply the x-override-* request headers ( #44312 )
...
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com >
Co-authored-by: shodoco <shodoco@users.noreply.github.com >
2026-10-15 15:50:04 -07:00
William Hu
4e0609ea27
[Fix] moe: Preserve negative routing sentinel in EP mapping ( #40498 )
2026-10-15 15:05:08 -07:00
Yuwei An
91f6ef2322
[DSv4.1] Fuse request-window KV copies ( #44555 )
2026-10-15 14:11:51 -07:00
Xinyuan Tong
877be2974a
[CI] pr-test: Let stage wait jobs reach their max-wait budget ( #44410 )
2026-10-15 13:34:49 -07:00
snadampal
aa5c7e887c
[Feat] docker: Add AWS EFA image variants built on published images ( #41006 )
...
Signed-off-by: Sunita Nadampalli <nadampal@amazon.com >
2026-10-15 13:19:22 -07:00
Ziyi Xu and Claude Opus 5.5
bc76a6b819
[DSv4] Sparse prefill: build compressed lengths without a host sync ( #44557 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-15 12:45:56 -07:00
87211088c9
perf: fix overlap scheduling for NVIDIA Confidential Computing(CC) on Blackwell ( #36810 )
...
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
Co-authored-by: Po-Han Huang <pohanh@nvidia.com >
Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com >
2026-10-15 12:09:09 -07:00
02a43eeef6
[Feat] rust-server: Attach configured metadata headers to a healthy /health_generate ( #44315 )
...
Co-authored-by: fei-xx <fei-xx@users.noreply.github.com >
Co-authored-by: shodoco <shodoco@users.noreply.github.com >
2026-10-15 12:04:23 -07:00
173e0f2e9b
[Feat] hicache: Add opt-in tail-only SWA backups in buffer mode ( #44173 )
...
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com >
Co-authored-by: xiezhq-hermann <xiezhq-hermann@users.noreply.github.com >
2026-10-15 12:02:59 -07:00
Michael
bba535ebb4
[Fix] [AMD] docker: Repair the AITER build step after the 2bb549102 pin bump ( #44534 )
2026-10-15 11:18:41 -07:00
c5a5bf0d0b
[Fix] pp: Order output send/recv by rank parity to fix cross-node hang ( #44453 )
...
Co-authored-by: chuixue <chuixue@minimaxi.com >
Co-authored-by: Shangming Cai <csmthu@gmail.com >
2026-10-16 01:46:37 +08:00
Shangming Cai
c7fb57d990
[Fix] sgl-router: Bump rustls to 0.23.46 for RUSTSEC-2026-0346..0349 ( #44509 )
2026-10-16 01:28:57 +08:00
a8aaf95b58
[Cookbook] Add cookbook for muse spark open ( #44527 )
...
Co-authored-by: thomawan <thomawan@amd.com >
Co-authored-by: HAI <hixiao@gmail.com >
Co-authored-by: jianan-gu <jianan.gu@intel.com >
Co-authored-by: Thomas Wang <1am9trash@gmail.com >
Co-authored-by: zijie <zijie.chen@amd.com >
Co-authored-by: Alex Nails <alex.nails@radixark.ai >
2026-10-15 10:20:38 -07:00
Aleksi Vesanto and jacky.cheng
5ef8319f16
[Perf] [AMD] attention: Enable diffusion BF16 attention and MXFP4 compilation ( #27702 )
...
Co-authored-by: jacky.cheng <yichiche@amd.com >
2026-10-16 00:40:41 +08:00
Yihao Wang and Claude Opus 5.5
5b12ec163c
[Doc] cookbook: Expand the FLUX.3 Image Klein model description ( #44515 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-15 08:56:54 -07:00
Jimmy Shong and Jimmy Shong
05cbe241be
[Doc] cookbook: Add Nemotron 3.5 Super VL ( #44510 )
...
Co-authored-by: Jimmy Shong <jimmy.shong@mac.attlocal.net >
2026-10-15 08:39:14 -07:00
28b98e5d13
[Doc] cookbook: Add the FLUX.3 prompt upsampler section ( #44512 )
...
Co-authored-by: Sandro Cavallari <scavallari@gmail.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-15 08:38:33 -07:00
YC Yen-Ching Tseng
e87e32b311
[Feat] [AMD] docker: Release ROCm 10.1 for gfx1250 in parallel with ROCm 10.0 ( #42699 )
2026-10-15 22:59:54 +08:00
a57707377f
[Fix] sgl-router: Send disagg_prefill_dp_rank to decode under --dp-aware ( #44072 )
...
Co-authored-by: Shangming Cai <csmthu@gmail.com >
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com >
2026-10-15 22:42:33 +08:00
qiaonan02
c4870d98cf
[Fix] glm5_next: Correct num_experts_per_tok default to 8 ( #44093 )
2026-10-15 21:53:31 +08:00
09b7383624
[diffusion] model: support FLUX.3 image ( #44418 )
...
Co-authored-by: Dom Brown <3886319+DomBrown@users.noreply.github.com >
Co-authored-by: niehen6174 <1639206518@qq.com >
Co-authored-by: Mick Qian <mickqian@radixark.ai >
Co-authored-by: Nikita Korobov <14355239+nekorobov@users.noreply.github.com >
Co-authored-by: Daniel Campora <961215+dcampora@users.noreply.github.com >
Co-authored-by: Richard Wang <11150595+rwang5203@users.noreply.github.com >
Co-authored-by: BBuf <1182563586@qq.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
2026-10-15 21:50:59 +08:00
elvischenv and Claude Opus 5.5
6fe5596ebb
Support EPLB for Nemotron-H ( #43685 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-15 13:50:09 +00:00
ormandj and Benjamin Oldenburg
c5bf3aafbc
Fix overflow in legacy and v1 DSA top-k selectors ( #37625 )
...
Co-authored-by: Benjamin Oldenburg <benjamin.oldenburg@ordis.co.th >
2026-10-15 21:44:58 +08:00
jiaqiang-dot-liu and Cursor
2f5407da28
[Fix] [AMD] qwen3: Fix KV cache write in the fused QK-norm + mRoPE decode path ( #44266 )
...
Co-authored-by: Cursor <cursoragent@cursor.com >
2026-10-15 21:22:46 +08:00
sogalin_codegen
2092160d42
[AMD][Diffusion] Route unquantized linears through aiter tgemm on gfx1250 ( #41988 )
2026-10-15 21:08:44 +08:00
amote-i
abf92826ee
[NPU] [DOC] update NPU best practices and model deployment docs to CANN 9.1.0 ( #44432 )
2026-10-15 20:22:54 +08:00
Zhang Yiyang (SII) and Mick Qian
23c2a026db
[diffusion] model: support MOSS-Image ( #44325 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
2026-10-15 19:23:56 +08:00
Cheng Wan
f7ad4345b8
[Refactor] deepseek-v4: Use explicit stage boundaries and row transport ( #43607 )
2026-10-15 02:05:28 -07:00
Cheng Wan
3a835cf417
[Refactor] inkling: Use short-convolution stage boundaries ( #43606 )
2026-10-15 02:04:46 -07:00
Cheng Wan
e1f9fce7b3
[Refactor] layer-boundary: Bind residual completion to producer updates ( #43605 )
2026-10-15 02:04:02 -07:00
Cheng Wan
d59c3448b0
[Fix] kimi-k3: Preserve pipeline graph bank inputs and bound chunk profiling ( #43604 )
2026-10-15 02:03:27 -07:00
Eddie Zhou
71a17dbc24
[AMD][CI] setup: Skip redundant pip-cache chown walk and MORI rebuild ( #42148 )
2026-10-15 16:21:46 +08:00
93dbe88183
Fuse MiMo verification RoPE and KV cache writes ( #43106 )
...
Co-authored-by: lianghougong <lianghouhong@xiaohongshu.com >
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com >
2026-10-15 16:15:24 +08:00
kangwangamd
28703575f9
[AMD] Update ROCm AITER pin to 2bb549102 ( #44379 )
2026-10-15 16:10:41 +08:00
GeorgeSherif and Xiaoyu Zhang
8f853b0041
[Doc] cookbook: Add Jais 3 deployment recipes ( #42766 )
...
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com >
2026-10-15 16:09:44 +08:00
86f4ede4fb
[Fix] dsa: Make the fused top-k exact on an overflowing threshold bin ( #37941 )
...
Co-authored-by: Cursor <cursoragent@cursor.com >
Co-authored-by: DarkSharpness <2040703891@qq.com >
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
2026-10-15 15:52:48 +08:00
9ea5472833
[HiCache] Add the page_unified host KV layout and the unified L3 key ( #39727 )
...
Co-authored-by: huangtingwei <141888744+huangtingwei9988@users.noreply.github.com >
Co-authored-by: Zhangheng <hzh0425@apache.org >
2026-10-15 15:46:13 +08:00
33618d4f00
[Feat] runtime: Include FlashInfer’s default tactic in startup tuning. ( #43927 )
...
Co-authored-by: Po-Han Huang <pohanh@nvidia.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-15 15:30:45 +08:00
Liangsheng Yin
a776851c97
[Refactor] scheduler: Check the prefill budget at the admission loop head; drop a redundant PD gate ( #44420 )
2026-10-15 00:16:56 -07:00
fa66e7d816
[NPU] [Bugfix] Fix rope sin cos pre-calculation for Qwen3.5 during Pipeline Parallelism ( #29753 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Артем Савкин <58187114+OrangeRedeng@users.noreply.github.com >
2026-10-15 09:58:47 +03:00
ashwini rathi and Claude Opus 5.5
01ac2cf36d
[CI] [XPU] nightly: Split Intel Main and Wheel workflows ( #44108 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-15 14:56:27 +08:00
Mick and Mick Qian
f2b73d4f9b
[diffusion] doc: document reference resizing, LoRA auto mode and image profiling ( #44377 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
2026-10-15 13:41:44 +08:00
Dayuxiaoshui
a50c17e979
[diffusion] perf: fuse the Wan VAE conv bias epilogue and tile the RMSNorm+SiLU kernel ( #38650 )
2026-10-15 13:39:27 +08:00
676c038aa5
[diffusion] fix: fix the load duration missing from subclassed server tests ( #40657 )
...
Co-authored-by: Cursor Agent <cursoragent@cursor.com >
Co-authored-by: Michael <michaelzhang-ai@users.noreply.github.com >
Co-authored-by: quitenode <quitenode@users.noreply.github.com >
Co-authored-by: jacky.cheng <yichiche@amd.com >
2026-10-15 13:38:23 +08:00
Liangsheng Yin
f9bdb74a5a
[Fix] scheduler: Count a request's held req_to_token row once at admission ( #44401 )
2026-10-14 22:29:10 -07:00
Yuxingwang-intel
93842f52fe
[CPU]: Add group-aware CPU SHM collective kernels ( #35435 )
2026-10-15 13:22:20 +08:00
Michael
4dc9b5c68c
[Test] [AMD] diffusion: Expect CUDA-only fast paths to be off on ROCm in kernel tests ( #44311 )
2026-10-15 13:18:01 +08:00
1d809bbbbd
[Fix] DSv4.1: Size the encoder-replay SWA pool by the DSpark draft's stages, and keep more prefix tails ( #43502 )
...
Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com >
Co-authored-by: Po-Han Huang <pohanh@nvidia.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Yuwei An <ayw.sirius19@gmail.com >
2026-10-14 22:04:58 -07:00
kk and root
e573d91275
[CI] tests: Fix stale fakes in the CP strategy and request-class override unit tests ( #44380 )
...
Co-authored-by: root <root@crsuse2-m2m-223.us-east2-a.compute.internal >
2026-10-14 21:35:20 -07:00
Khoa Pham
d4daf52369
[Fix] dsv41: Bound MoE routing to live SWA graph-tail rows ( #44347 )
2026-10-14 21:14:28 -07:00
AMD-yanfeiwang
08f15e76bb
[Feat] sgl-router: Add --dp-rank-policy round_robin for --dp-aware ( #43511 )
2026-10-15 12:03:01 +08:00
WenhaoZhang and Mick
c92108bb93
[diffusion] comfyui: grouped SGLDOptions, native model loader and model-aware preflight ( #44058 )
...
Co-authored-by: Mick <mickjagger19@icloud.com >
2026-10-15 11:59:03 +08:00
Liangsheng Yin
3b5ae4ea86
[Fix] scheduler: Drop finished requests from the running batch before every admission ( #44351 )
2026-10-14 20:40:05 -07:00
2528cd2b91
[PD] Send DSA indexer state only past decode_prefix_len ( #44140 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Shangming Cai <csmthu@gmail.com >
2026-10-15 11:25:18 +08:00
Avaya Aggarwal and niehen6174
0cf683ea8d
[diffusion] comfyui: return several server-mode images as one IMAGE batch ( #44088 )
...
Co-authored-by: niehen6174 <1639206518@qq.com >
2026-10-15 10:38:37 +08:00
Thomas Wang
4851d182a1
[Doc][AMD] Update v41 amd cookbook 1015 ( #44368 )
2026-10-14 19:33:08 -07:00
xiaofei-zheng
a12814c158
[Perf] [AMD] dsa: Enable the gfx950 fused DSA indexer for block-FP8 wq_b/wk ( #43643 )
2026-10-15 09:36:16 +08:00
Yuhao Yang and Claude Opus 5.5
9fa9308e95
[Doc] cookbook: Add Step 5 Preview cookbook ( #44357 )
...
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
2026-10-15 09:21:06 +08:00
9d68ecce2e
[DeepSeek-V4.1] DeepSelect cluster top-k for long rows and bf16 index logits ( #44051 )
...
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com >
2026-10-15 09:00:19 +08:00
xiaofei-zheng
c5c4bbbd25
[Perf] [AMD] dsa: Run DSA prefill under breakable CUDA graphs on HIP ( #43642 )
2026-10-15 08:59:34 +08:00
Michael and jacky.cheng
8333ccfccb
[Fix] [AMD] diffusion: Compare ReduceOp by identity so the compiled TP all-reduce stays one graph ( #43770 )
...
Co-authored-by: jacky.cheng <yichiche@amd.com >
2026-10-14 17:40:07 -07:00
Yuwei An
64a0a08f0a
[DSv4.1] Support TRT-LLM decoder bounded replay with BCG ( #44326 )
2026-10-14 17:39:52 -07:00
YAMY and Yuwei An
c891c5c7ca
[DSv4.1] Fold encoder-SWA replay into the extend forward ( #43166 )
...
Co-authored-by: Yuwei An <ayw.sirius19@gmail.com >
2026-10-14 17:27:07 -07:00
Shuwen Wang
3c28614f12
[Fix] hicache: Defer query beliefs until the rank-synchronized hit drain ( #43108 )
2026-10-14 17:25:08 -07:00
Aditya Singh and hnyls2002
a6a8c56015
[Fix] detokenizer router: Route by rid without tokenizer worker ipcs and broadcast logging config ( #33715 )
...
Co-authored-by: hnyls2002 <lsyincs@gmail.com >
2026-10-14 17:20:21 -07:00
0536cec73d
[Feat] metrics: Expose multi-detokenizer router CPU usage ( #43323 )
...
Co-authored-by: raghul chandramouli <raghul@c1xind0180-0243.local >
Co-authored-by: Xinyuan Tong <xinyuantong.cs@gmail.com >
Co-authored-by: Xinyuan Tong <115166877+JustinTong0323@users.noreply.github.com >
Co-authored-by: hnyls2002 <lsyincs@gmail.com >
2026-10-14 17:17:22 -07:00
Junnan Li and hnyls2002
0f36c8015c
[Bug] Fix multi-tokenizer BatchEmbeddingOutput repack dropping fields ( #31892 )
...
Co-authored-by: hnyls2002 <lsyincs@gmail.com >
2026-10-14 16:56:56 -07:00
209f8220d6
[Fix] sglang-server: Honour extra_key on the Rust frontend's /generate ( #44207 )
...
Co-authored-by: fei-xx <fei-xx@users.noreply.github.com >
Co-authored-by: shodoco <shodoco@users.noreply.github.com >
2026-10-14 16:44:31 -07:00
e7d6a88bfc
[HiCache] Buffer-only L3 for an auxiliary SWA window; load back only missing window rows ( #43932 )
...
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com >
Co-authored-by: houseroad <houseroad@users.noreply.github.com >
Co-authored-by: xiezhq-hermann <xiezhq-hermann@users.noreply.github.com >
2026-10-14 16:43:24 -07:00
aa8793f33b
[Rust] Serve scheduler metrics on the Rust frontend's /metrics ( #43942 )
...
Co-authored-by: hanminglu <hanminglu@users.noreply.github.com >
Co-authored-by: shodoco <shodoco@users.noreply.github.com >
2026-10-14 16:43:06 -07:00
2a9f6a25a7
[Core] Add subclass hooks for the generation request class and optimistic prefill eligibility ( #44113 )
...
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com >
Co-authored-by: CuiCoco <CuiCoco@users.noreply.github.com >
2026-10-14 16:42:30 -07:00
5f2861bf59
[Perf] session: Cache multimodal prompt token counts across turns ( #44112 )
...
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com >
Co-authored-by: joshuadeng <joshuadeng@users.noreply.github.com >
2026-10-14 16:41:34 -07:00
9732188a58
[Rust] Serve /weights_checker from the Rust frontend ( #43926 )
...
Co-authored-by: hanminglu <hanminglu@users.noreply.github.com >
Co-authored-by: shodoco <shodoco@users.noreply.github.com >
2026-10-14 16:41:10 -07:00
FREETRUMP and hnyls2002
97d3150a6a
[Perf] Cache msgspec struct fields in the batch splitter ( #43520 )
...
Co-authored-by: hnyls2002 <lsyincs@gmail.com >
2026-10-14 14:51:26 -07:00
4273814771
[Perf] detokenizer: Group return IPC sends by tokenizer worker ( #44288 )
...
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com >
Co-authored-by: hnyls2002 <lsyincs@gmail.com >
Co-authored-by: Liangsheng Yin <hnyls2002@gmail.com >
2026-10-14 14:26:51 -07:00
Charles Chen and alphabetc1
d3dc14eb38
[mem_cache][PD] Fix decode radix cache ownership and allow EAGLE/EAGLE3 with decode radix cache ( #40857 )
...
Co-authored-by: alphabetc1 <2508695655@qq.com >
2026-10-14 14:19:51 -07:00
Rain Jiang
12dc623102
[Chore] rust: Remove the sglang-renderer crate ( #43451 )
2026-10-14 11:43:37 -07:00
Shuwen Wang
b67785153a
[Fix] glm5.2: Align HiSparse warmup backends and move test to nightly ( #44303 )
2026-10-15 01:28:03 +08:00
Yoray Zack
87ed1b9526
[4/N] elastic-ep: Scale at DP-attention replica granularity ( #33728 )
2026-10-15 01:17:01 +08:00
c964d9ba20
[Feat] sgl-router: Separate LoRA block IDs from prompt lookup keys ( #42818 )
...
Co-authored-by: Kan Wu <kan.wu@radixark.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Shangming Cai <csmthu@gmail.com >
2026-10-15 00:50:19 +08:00
amote-i
3a9cae27ed
[Doc] [NPU] docs: Update NPU CANN version to 9.1.0 ( #44177 )
2026-10-15 00:27:18 +08:00
Michael
1737251730
[Test] [AMD] diffusion: Run the IPC all-to-all transport tests only on CUDA ( #44096 )
2026-10-14 23:04:23 +08:00
010776e94f
[KV events] Publish opaque LoRA block IDs and adapter names ( #42817 )
...
Co-authored-by: Kan Wu <kan.wu@radixark.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Shangming Cai <csmthu@gmail.com >
2026-10-14 22:56:28 +08:00
Xinyuan Tong
ed40c7fc66
[Doc] cookbook: add Ling-3.1-flash ( #44301 )
2026-10-14 22:40:01 +08:00
Kan Wu and Claude Opus 5.5
d2a436446c
[Feat] sgl-router: Fix cache-aware routing for MTP/EAGLE on hybrid (Mamba/KDA) models ( #44252 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-14 21:21:01 +08:00
204644c2dc
[Fix] sglang-processor: Follow Python's OpenAI layer through #43027 , #37577 , #43006 , #43026 , #43005 ( #44243 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Shangming Cai <csmthu@gmail.com >
2026-10-14 21:06:13 +08:00
triple-mu
5e90312eb1
[diffusion] perf: support bit-exact norm/bias fusions and channels_last decode for Qwen-Image 2.1( #41340 )
2026-10-14 20:47:46 +08:00
Shuwen Wang
7b53a13401
[Fix] glm5.2: Align PD HiSparse prefill KV layout ( #44256 )
2026-10-14 20:40:28 +08:00
bab9c81fea
fix: FA4 paged-KV zero-size page-table registers on SM90 ( #35757 )
...
Co-authored-by: razorback16 <razorback16@users.noreply.github.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
Co-authored-by: BBuf <1182563586@qq.com >
2026-10-14 19:10:15 +08:00
Po-Han Huang (NVIDIA) and Claude Opus 5.5
3b72b70ef4
[CI] vlm: Pin baidu/Unlimited-OCR revision in test_unlimited_ocr_server ( #44235 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-14 17:50:18 +08:00
Mohammad Miadh Angkad and Mohammad Angkad
c82e8bd5fe
[Fix] DeepSeek-V4.1: Align paged indexer score rows for DeepSelect decode ( #44210 )
...
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai >
2026-10-14 17:31:19 +08:00
df1a089cd3
[Spec] Fix DFLASH verify mask policy for wrapped attention backends ( #43054 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com >
2026-10-14 17:15:36 +08:00
817781b828
[Feat] sgl-router: Render Qwen3.8 (Max, 27B, Flash-Next) through sglang-processor ( #43689 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Shangming Cai <csmthu@gmail.com >
2026-10-14 17:09:35 +08:00
Po-Han Huang (NVIDIA) and Claude Opus 5.5
72730d6360
[Doc] cookbook: Mirror SA cluster NCCL_CUMEM_ENABLE in AgentX launches, fix GPU counts ( #44228 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-14 16:07:40 +08:00
c54ba66cd4
[Perf] Enable native MXFP4 routing and CuTeDSL fusion for MiMo ( #43639 )
...
Co-authored-by: lianghougong <lianghouhong@xiaohongshu.com >
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com >
2026-10-14 15:39:44 +08:00
Mohammad Miadh Angkad and Mohammad Angkad
6daae9c98e
[Perf] Bind scheduler CPUs to the GPU's NUMA node when memory policy is not permitted ( #44023 )
...
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai >
2026-10-14 00:01:32 -07:00
58f2c923ed
[Feat] sgl-router: skip engines at an admission cap in power-of-two; forward x-sglext headers ( #44218 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Codex <codex@openai.com >
2026-10-14 14:59:06 +08:00
Kan Wu and Claude Opus 5.5
53a24c448e
[Feat] sgl-router: Render Kimi-K3 through sglang-processor ( #43608 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-13 23:40:18 -07:00
Michael
6f646563a2
[AMD] diffusion: expect the residual-gate fast path to be off on ROCm in test_modulate ( #40660 )
2026-10-13 23:34:31 -07:00
Ma Mingfei
68c18a5701
Revert "[XPU] Support Gemma-3 bidirectional image attention in XPUAttentionBackend" ( #44216 )
2026-10-14 14:27:46 +08:00
desmond-intel
b87d62e581
[XPU] Support Gemma-3 bidirectional image attention in XPUAttentionBackend ( #39805 )
2026-10-14 14:24:04 +08:00
Shunkangz
0d130f1915
[Qwen3.8 CP 4/4] Prefill context parallel E2E support of Qwen3.8 ( #39724 )
2026-10-13 23:19:46 -07:00
7ca4cbfcef
[Disagg] Return prompt routed experts from the PD prefill server ( #44002 )
...
Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: ishandhanani <82981111+ishandhanani@users.noreply.github.com >
2026-10-13 23:13:44 -07:00
AndyLi429 and AndyLi429
05bb5afb62
[NPU][Bugfix] use CANN libdevice for DSpark Markov sampling ( #43849 )
...
Co-authored-by: AndyLi429 <AndyLi429@noreply.gitcode.com >
2026-10-14 14:12:22 +08:00
Zhaoyi Li
e29eee18df
[AMD][DI][CI] nightly mi355x disagg: stage packed logs under RUNNER_TEMP, not /tmp ( #43945 )
2026-10-14 13:53:40 +08:00
Kan Wu and Claude Opus 5.5
4560da58ee
[sgl-router] Render GLM-5.3 and GLM-5.3-Flash through sglang-processor ( #43625 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-14 13:19:44 +08:00
Po-Han Huang (NVIDIA) and Claude Opus 5.5
d5f80aa3c3
[Test] dsv4: Fix Hopper indexer replay test after DeepSelect decode ( #41561 ) ( #44193 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-14 13:15:35 +08:00
ANSHUMAN TRIPATHY
30bdd09aed
DSPARK enable for XPU ( #33739 )
2026-10-14 13:10:24 +08:00
Po-Han Huang (NVIDIA) and Claude Opus 5.5
721953c6f6
[Doc] cookbook: Refresh SemiAnalysis AgentX launches to InferenceX main (GLM-5.2 NVFP4 B300 agg TP4, Dynamo frontend counts) ( #44065 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-14 13:03:51 +08:00
Khoa Pham
e00754bab1
[DSV4.1] Derive the late-layer DP geometry from the captured shape under the breakable prefill graph ( #44144 )
2026-10-13 21:54:35 -07:00
Mohammad Miadh Angkad and Mohammad Angkad
ff8d964cf5
[CI] glm5.2: Temporarily disable PD HiSparse+MTP B300 test ( #44192 )
...
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai >
2026-10-13 21:53:58 -07:00
chenxb002 and chenxiaobing
491d4ef178
[MLU][CI] Bootstrap MLU runner workflows ( #41005 )
...
Co-authored-by: chenxiaobing <chenxiaobing@cambricon.com >
2026-10-13 21:40:49 -07:00
Chetan Kumar Verma and Claude Opus 5.5
2c1e85964e
[diffusion] cache-dit: respect CFG parallel for dual-transformer models ( #43704 )
...
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
2026-10-14 12:25:07 +08:00
xinchen9 and Ma Mingfei
2428d0f531
[XPU] Use the shared fused decode-metadata kernel in the intel_xpu ba… ( #42122 )
...
Co-authored-by: Ma Mingfei <mingfei.ma@intel.com >
2026-10-14 12:10:12 +08:00
16176278d0
[diffusion] perf: replay small Triton wrappers' launches per operand signature ( #43859 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-14 11:18:19 +08:00
Mohammad Miadh Angkad and Mohammad Angkad
4f751da975
[Agents] Add a rule to prefer sgl-eval for accuracy evals ( #44040 )
...
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai >
2026-10-13 20:17:04 -07:00
53fe21cdfd
[diffusion] perf: replace padded rows without a host sync for Z-Image ( #44069 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-14 11:15:28 +08:00
xiaofei-zheng and Xiaoyu Zhang
7e226b1b71
[AMD] Use the preallocated EAGLE topk=1 draft chain on HIP ( #43640 )
...
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com >
2026-10-14 11:06:07 +08:00
5efb7384c1
[diffusion] comfyui: consolidate 6 unrelated plugin fixes ( #43493 )
...
Co-authored-by: Mick <mickjagger19@icloud.com >
Co-authored-by: niehen6174 <1639206518@qq.com >
2026-10-14 10:56:09 +08:00
Cheng Wan
c053fc71d3
[CI] pr-title: Validate pull request title format ( #44165 )
2026-10-13 19:53:52 -07:00
Ziyi Xu and Claude Opus 5.5
4f5fcbfef5
[JIT] Clean up kernel headers: move runtime helpers, add once.h ( #41147 )
...
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
2026-10-14 10:46:51 +08:00
yuyu5333 and Claude Code
b50fdeb0ef
[DeepSeek V4.1] Use DeepSelect for Hopper full top-k decode. ( #41561 )
...
Co-authored-by: Claude Code <noreply@anthropic.com >
2026-10-14 10:46:12 +08:00
Ma Mingfei and MingxuZh
87beb3a052
[CPU] Temporarily disable subblock sparse attention test on Xeon CI ( #43468 )
...
Co-authored-by: MingxuZh <109504044+MingxuZh@users.noreply.github.com >
2026-10-14 10:44:24 +08:00
13a6dee99b
[Fix] moe_sum_reduce: gate the bf16 vectorized path on hidden % 16 ( #42837 )
...
Signed-off-by: Monishver Chandrasekaran <monishverchandrasekaran@gmail.com >
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com >
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com >
2026-10-14 10:32:50 +08:00
Mohammad Miadh Angkad and Mohammad Angkad
7f0434e4af
[GLM-5.3-Flash] Drop the NextN layer from the KDA state pool ( #44098 )
...
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai >
2026-10-13 19:31:02 -07:00
Avaya Aggarwal
ac187c0cd5
[diffusion] Bound frame_interpolation_exp to RIFE's supported range ( #43014 )
...
Signed-off-by: onepunchmonk <aggarwal.avaya@yahoo.com >
2026-10-14 10:29:28 +08:00
Jin and Xinyuan Tong
44b44774c3
[Bugfix] Keep rope_theta when --json-model-override-args overrides rope_scaling ( #43067 )
...
Co-authored-by: Xinyuan Tong <xinyuantong.cs@gmail.com >
2026-10-14 10:16:35 +08:00
Baizhou Zhang
536d065370
[CP] Automatically enable BCG for supported prefill CP configurations ( #44022 )
2026-10-13 19:00:42 -07:00
shyeh25 and Claude Opus 5.5
684fea8af3
[DSV4.1] Skip Mega mHC fusion on 0-row (DP-idle) batches ( #43993 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-13 16:36:15 -07:00
Liangsheng Yin
2ebdf9360b
[Fix] Work around a CPython fork deadlock in the CPU preprocess pool ( #44143 )
2026-10-13 16:31:57 -07:00
b3991b53da
GLM5.2 DSA Indexer Fallback ( #32947 )
...
Co-authored-by: Shu Wang <shuw@login-hecate03.hecate.clusters.nvidia.com >
Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com >
2026-10-13 16:27:33 -07:00
Khoa Pham
5917604a95
[DSV4.1] Support decoder SWA bounded replay with CUDA DP attention ( #43812 )
2026-10-13 16:10:57 -07:00
HEJIAN SANG and Claude Opus 5.5
a832609b92
[Feature] Return output top logprobs as flat base64 arrays on request ( #42890 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-13 15:58:31 -07:00
baa632b049
[Score] Send stacked GPU embedding overrides through a CUDA IPC pool ( #39730 )
...
Co-authored-by: Sundara Raman Ramachandran <sundar24295@gmail.com >
Co-authored-by: hnyls2002 <lsyincs@gmail.com >
2026-10-13 15:46:50 -07:00
Andrew Briand
08550acc28
Mark FP32 redux asm volatile ( #44137 )
2026-10-13 15:08:52 -07:00
Jiaqi Li and Zijie Xia
5569b14d26
fix(benchmark): request token IDs when counting OpenAI prompts ( #34511 )
...
Co-authored-by: Zijie Xia <zijie.xia@radixark.ai >
2026-10-13 14:31:45 -07:00
Yizheng Jiao
2341e647d3
[DeepSeek-V4.1] Run decoder SWA bounded replay inside the breakable prefill CUDA graph ( #43736 )
2026-10-13 13:55:09 -07:00
79af3df346
[gRPC] Move runtime.v1 bindings to tonic 0.14, report OpenAI status codes, return text on request ( #42673 )
...
Co-authored-by: Kan Wu <kan.wu@radixark.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-13 13:46:15 -07:00
huangtingwei
18210ed849
[HiSparse] Add speculative decoding support with multi-step KV cache swapping ( #37771 )
2026-10-13 13:38:31 -07:00
Xinyuan Tong and Nihar Patel
7dc2930585
[Feature] Add Aaronson-Gumbel text watermarking ( #37577 )
...
Co-authored-by: Nihar Patel <nihpat95@gmail.com >
2026-10-13 12:47:50 -07:00
Xinyuan Tong
5eda6b7e67
docs: add MiniCPM-V 4.7 cookbook ( #43829 )
2026-10-13 12:04:47 -07:00
ce5352a814
[Rust] Serve any model name on the OpenAI chat and completions routes ( #43923 )
...
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com >
Co-authored-by: shodoco <shodoco@users.noreply.github.com >
2026-10-13 11:58:57 -07:00
01f3ed18a5
[unified memory pool] Prefer SWA reclaim quotas with a HiCache storage backend or on PD decode ( #43919 )
...
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com >
Co-authored-by: ZYHowell <ZYHowell@users.noreply.github.com >
2026-10-13 11:58:24 -07:00
7e3da1b784
[HiCache] Extract the buffer-mode prefetch and load-back lifecycle into pipeline operations ( #43870 )
...
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com >
Co-authored-by: xiezhq-hermann <xiezhq-hermann@users.noreply.github.com >
2026-10-13 11:51:44 -07:00
5bcaf0f8ed
[Core] Add request-level customized_info keys and mark Full prefill capture batches ( #43772 )
...
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com >
Co-authored-by: jiayisuse <jiayisuse@users.noreply.github.com >
2026-10-13 11:45:52 -07:00
149ef7d1b7
[Rust] Let an embedding package wrap the frontend HTTP app ( #43924 )
...
Co-authored-by: hanminglu <hanminglu@users.noreply.github.com >
Co-authored-by: shodoco <shodoco@users.noreply.github.com >
2026-10-13 11:44:43 -07:00
a8381d73be
[HiCache] Block-group pool hits, caller-owned load destinations, allocation-target FULL eviction ( #43868 )
...
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com >
Co-authored-by: xiezhq-hermann <xiezhq-hermann@users.noreply.github.com >
2026-10-13 11:39:48 -07:00
Khoa Pham and vincentzed
8bf145d119
[DCP] Decode context parallelism for DSA (GLM-5.2 / GLM-5.3) with MTP ( #43735 )
...
Co-authored-by: vincentzed <207368749+vincentzed@users.noreply.github.com >
2026-10-13 11:35:34 -07:00
6ade23696d
[Core] Keep request priority within the signed 64-bit range in radix caches ( #43920 )
...
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com >
Co-authored-by: qandrew <qandrew@users.noreply.github.com >
2026-10-13 11:33:26 -07:00
7f9f742535
[Rust] Refuse an unsupported multimodal model before the Rust listener binds ( #43925 )
...
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com >
Co-authored-by: shodoco <shodoco@users.noreply.github.com >
2026-10-13 11:30:43 -07:00
2a8e9db3b0
[lora] Support LoRA for the NemotronH_Nano_VL_V2 model family ( #39798 )
...
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
Co-authored-by: Po-Han Huang <pohanh@nvidia.com >
Co-authored-by: Yanbin Jiang <jybsuper@gmail.com >
2026-10-13 11:29:48 -07:00
ff3f8c348c
[Observability] Report per-request storage prefetch and staging waits in meta_info ( #43946 )
...
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com >
Co-authored-by: xiezhq-hermann <xiezhq-hermann@users.noreply.github.com >
2026-10-13 11:13:01 -07:00
Rain Jiang
e98c6d7b25
Rainj me/rust bench ( #43950 )
2026-10-13 11:07:42 -07:00
Kailash Ranganathan and Yanbin Jiang
c0584cf9af
[perf] [multi-lora] remove empty CTAs from fused LoRA expand and add split-K to shrink ( #41951 )
...
Co-authored-by: Yanbin Jiang <jybsuper@gmail.com >
2026-10-13 10:16:14 -07:00
Zhaoyi Li
e9400c5a8f
[AMD][DI][CI] kimik26 2p1d-ep16: capture CUDA graph on decode (eager decode is slow) ( #43985 )
2026-10-13 08:46:53 -07:00
Zhaoyi Li
96f519e4fa
[AMD][DI][CI] glm52 2p1d-ep16: capture CUDA graph on decode (eager decode is too slow) ( #43977 )
2026-10-13 08:45:46 -07:00
11cd2701a8
fix(openai): preserve zero cached tokens when cache reporting is enabled ( #43005 )
...
Co-authored-by: ai-jz <ai-jz@users.noreply.github.com >
Co-authored-by: Xinyuan Tong <115166877+JustinTong0323@users.noreply.github.com >
2026-10-13 23:29:19 +08:00
3f1e185f6e
fix(openai): keep chat stream created timestamp stable ( #43026 )
...
Co-authored-by: ai-jz <ai-jz@users.noreply.github.com >
Co-authored-by: Xinyuan Tong <115166877+JustinTong0323@users.noreply.github.com >
2026-10-13 22:43:33 +08:00
jiaqiang-dot-liu and Cursor
4c5ed0597e
Skip fully-masked KV tiles in sliding-window extend attention ( #39769 )
...
Co-authored-by: Cursor <cursoragent@cursor.com >
2026-10-13 22:40:53 +08:00
27b68303d0
fix(openai): omit absent tool_calls from chat responses ( #43027 )
...
Co-authored-by: ai-jz <ai-jz@users.noreply.github.com >
Co-authored-by: Xinyuan Tong <xinyuantong.cs@gmail.com >
2026-10-13 20:56:58 +08:00
Po-Han Huang (NVIDIA) and Claude Opus 5.5
8e37c6cf37
[CI] Run test_pi05_modelopt_fp8_cuda in the kernel-unit lane ( #44055 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-13 20:39:02 +08:00
Ho-Ren (Jack) Chuang and Xiaoyu Zhang
5100f8d11d
[DSA] Q8KV8 FP8 Sparse Prefill on GLM-5.3-Flash with BF16 KV Cache ( #41790 )
...
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com >
2026-10-13 20:30:13 +08:00
amote-i
29974d543e
[NPU] [DOC] update kimi k3 cookbook and supported features on NPU ( #44052 )
2026-10-13 20:27:26 +08:00
jlqibm and Claude Opus 5
74e0a47863
Fix/compressed tensors nvfp4 hopper ( #40022 )
...
Co-authored-by: Claude Opus 5 <noreply@anthropic.com >
2026-10-13 20:10:31 +08:00
LittleNyima and Mick Qian
5727f08932
[diffusion] feat: add configurable reference image resizing for MiniMax H3 Ref2VA ( #35839 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
2026-10-13 19:48:18 +08:00
WenhaoZhang
85254b5a8f
[diffusion] comfyui: run Qwen-Image 2.1 in SGLD integrated mode ( #43581 )
2026-10-13 19:47:02 +08:00
Yunhao Zhou and 周云昊
991a29d0ad
fix(multimodal): convert grayscale tensors to RGB in PIL backend ( #42127 )
...
Co-authored-by: 周云昊 <zhouyunhao@zhouyunhaodeMacBook-Pro.local >
2026-10-13 19:03:48 +08:00
4b5b727145
Fix EVS for NemotronH_Omni_Reasoning_V3 ( #42933 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com >
2026-10-13 10:59:00 +00:00
Michael and Claude Opus 5.5
aa15b06abe
[AMD][CI] Run test_tracing.py in the extra tier, matching CUDA ( #43928 )
...
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
2026-10-13 02:45:58 -07:00
Xiaoyu Zhang
4e0fa52530
[Perf] Compact GDN causal-conv grids for skewed prefill batches ( #43570 )
2026-10-13 17:45:06 +08:00
Ziang Li
01f6840693
[FlashInfer] Add startup autotuning for MegaMoE ( #43789 )
2026-10-13 02:30:31 -07:00
Mohammad Miadh Angkad and Mohammad Angkad
195cb7afb4
[CI] Use sgl-eval for GLM-5.3-Flash GSM8K tests ( #43975 )
...
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai >
2026-10-13 02:26:50 -07:00
Shu Wang and Anish Shanbhag
e0c957e222
[HiCache][PD] Allow DSA index-K elision under HiCache and disaggregation ( #41769 )
...
Co-authored-by: Anish Shanbhag <ashanbhag@nvidia.com >
2026-10-13 02:07:54 -07:00
Shu Wang and Po-Han Huang
27be99ae73
[NVIDIA][MegaMoE] Fix the default spec decoding moe backend when the target model uses flashinfer megamoe ( #40772 )
...
Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com >
2026-10-13 02:00:24 -07:00
Chunan Zeng and Claude Opus 5.5
7ff3a11284
[Perf] Enable fast-math for JIT activations on Blackwell ( #43236 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-13 15:18:05 +08:00
AMD-yanfeiwang
d8f8522348
[router] power_of_two: in-flight request load for --dp-aware decode ( #43483 )
2026-10-13 14:50:54 +08:00
Michal Shalev and Po-Han Huang
35a014c7e8
[PD][NIXL] Fix staging buffer startup by accepting slot_layer_ids ( #43723 )
...
Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com >
2026-10-12 23:48:13 -07:00
Po-Han Huang (NVIDIA) and Claude Opus 5.5
dbad07a06f
[Docs] Cookbook: refresh SemiAnalysis AgentX launches (DeepSeek-V4.1 GB200/GB300, GLM-5.2 FP8 B300) ( #43983 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-13 14:39:45 +08:00
50a144057c
[AMD][PD] enable mori transfer for PD decode DCP ( #37358 )
...
Signed-off-by: inkcherry <mingzhi.liu@amd.com >
Co-authored-by: Duyi-Wang <duyi.wang@amd.com >
Co-authored-by: HAI <hixiao@gmail.com >
Co-authored-by: Shangming Cai <csmthu@gmail.com >
2026-10-13 14:35:09 +08:00
Xiaoyu Zhang
46b7357779
[Diffusion] Speed up LTX-2 guidance and video output ( #43667 )
2026-10-13 14:23:33 +08:00
Jimmy Shong
3eebf987ed
[PD] Support heterogeneous TP for hybrid SWA KV transfer ( #40909 )
2026-10-12 23:21:54 -07:00
603a05e6c0
[ROCm][Spec] Route NGRAM tree reconstruction to the JIT kernel on HIP ( #43328 )
...
Co-authored-by: Cursor <cursoragent@cursor.com >
Co-authored-by: Thomas Wang <thomawan@amd.com >
2026-10-12 23:17:49 -07:00
AMD-yanfeiwang
17c3b259f5
[router] Fix power_of_two with --dp-aware ( #42027 )
2026-10-13 14:15:36 +08:00
Zhaoyi Li
fc404bbbb3
[AMD][DI][CI] dsv4pro-fp8 2p1d-ep16-mtp: raise prefill mem fraction for MORI EP v2 memory ( #43944 )
2026-10-12 23:03:57 -07:00
Xinyi Song
a69e893b1a
[AMD] Fuse prefill index-Q RoPE fake-quant and fp4 quantize kernels in DSV4.1 ( #42911 )
2026-10-12 22:33:53 -07:00
jacky.cheng and Mick Qian
42e43cf33f
[AMD] Fix Flux2 kv-gather abort on non-contiguous K/V ( #43793 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
2026-10-12 22:20:55 -07:00
kk and wunhuang
43356304b4
[AMD][Kimi-K3] Use public FlyDSL APIs in KDA decode kernels ( #43968 )
...
Co-authored-by: wunhuang <wunhuang@amd.com >
2026-10-12 22:09:55 -07:00
WenhaoZhang and Mick Qian
ea13359da9
[diffusion] perf: implement paint_use_remesh (decimate to 40k faces before UV unwrap) for Hunyuan3D ( #42770 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
2026-10-13 12:28:32 +08:00
6a89181614
tokenizer perf optimization ( #39771 )
...
Co-authored-by: BBuf <1182563586@qq.com >
Co-authored-by: Xinyuan Tong <xinyuantong.cs@gmail.com >
2026-10-13 12:27:21 +08:00
WenhaoZhang and Mick Qian
feba04fa49
[diffusion] fix: avoid redundant LoRA scaling while preserving exact numerics ( #43028 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
2026-10-13 12:27:04 +08:00
WenhaoZhang
cda267ab0f
[diffusion] fix: keep MiniMax-H3 VAE fused norms off ROCm ( #43350 )
2026-10-13 12:26:16 +08:00
WenhaoZhang
b4822bae03
[diffusion] comfyui: run MiniMax H3, FastH3 and VDN in SGLD integrated mode ( #43504 )
2026-10-13 12:01:48 +08:00
4d5692b6cd
fix(openai): honor tool_choice=none in DeepSeek encoders ( #43006 )
...
Co-authored-by: ai-jz <ai-jz@users.noreply.github.com >
Co-authored-by: Xinyuan Tong <115166877+JustinTong0323@users.noreply.github.com >
2026-10-13 11:41:50 +08:00
Piotr Mazurek and Codex
40f5d9411f
Reuse Hugging Face LFM2-MoE configuration ( #43939 )
...
Co-authored-by: Codex <codex@openai.com >
2026-10-13 11:29:18 +08:00
Mohammad Miadh Angkad and Mohammad Angkad
7bd41aec12
[CI] Remove Mistral Small 4 e2e test ( #43973 )
...
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai >
2026-10-12 20:14:22 -07:00
Thomas Wang
c2dfd5d3d6
[AMD] Update v4 amd cookbook 1012 ( #43962 )
2026-10-12 19:57:55 -07:00
Xiaoyu Zhang
03416e8c0f
[Fix] Account for captured inputs in DeepGEMM masked workspace budget ( #43563 )
2026-10-13 10:53:10 +08:00
Liangsheng Yin
43e4e84731
[PD] Recompute retracted DeepSeek-V4 requests on prefill instead of crashing decode ( #43936 )
2026-10-12 19:32:03 -07:00
Yuwei An
07566cbf0d
[DSv4.1] Temporarily disable B300 PD/CP accuracy test ( #43948 )
2026-10-12 19:31:10 -07:00
75a0d91c1f
[Fix] Preserve HTTP errors for non-streaming /v1/messages ( #43796 )
...
Co-authored-by: Xinyuan Tong <xinyuantong.cs@gmail.com >
Co-authored-by: Xinyuan Tong <115166877+JustinTong0323@users.noreply.github.com >
2026-10-13 10:10:45 +08:00
45418d7dc6
[HiCache] Avoid full-store scans for uncached file prefix lookups ( #42077 )
...
Signed-off-by: apex-mochen <2756823972@qq.com >
Co-authored-by: apex-mochen <265981728+apex-mochen@users.noreply.github.com >
Co-authored-by: Zhiqiang Xie <xiezhq@stanford.edu >
2026-10-12 18:52:50 -07:00
Yuwei An
23150f7ae0
[DSv4.1] Reduce sparse-prefill workspace padding ( #43921 )
2026-10-12 18:51:36 -07:00
Ziyi Xu and Claude Opus 5.5
107cff72d3
[DeepSeek-V4.1] mHC: drop seam_open, narrow precomputed stats in take_rows ( #43833 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-12 18:45:16 -07:00
e775fa7253
[Build][CI] Drop Python 3.10: require Python 3.11+ and run CUDA CI on Python 3.12 ( #43021 )
...
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai >
Co-authored-by: Mohammad Miadh Angkad <176301910+mmangkad@users.noreply.github.com >
2026-10-12 18:40:16 -07:00
YAMY
59445b3da1
[DSV4][CP] Write low-ratio page indices when CP prefill runs paged attention ( #43935 )
2026-10-12 18:36:11 -07:00
Michael
7224fce1b5
[AMD] Recalibrate test_quark_mxfp4 GSM8K thresholds to observed MI355X noise ( #41980 )
2026-10-12 16:46:03 -07:00
Yizheng Jiao
6a7abcfc9b
[DeepSeek-V4.1] Admit the NIXL transfer backend for DSpark PD ( #43734 )
2026-10-12 15:32:17 -07:00
Khoa Pham
93809202a3
[Kernel] Expose bundled FlashMLA MegaAttention kernels and APIs ( #43918 )
2026-10-12 14:50:04 -07:00
Duyi-Wang
6781da8e5b
[AMD] [Docker] Update MoRI to v1.2.4 ( #43834 )
2026-10-12 14:45:42 -07:00
Thomas Wang
64cffed139
[AMD] Fix test_compact_attention_hip.py CI ( #43836 )
2026-10-12 14:43:19 -07:00
Thomas Wang
d6853f5066
[AMD] Add flydsl mxfp8 dense gemm kernel for dpsk v4.1 prefill ( #43803 )
2026-10-12 14:42:02 -07:00
Michael
6c2a682b59
[AMD] Keep the FP4 varctx schedule test off AITER's 2048-row compile ( #41793 )
2026-10-12 14:39:08 -07:00
Zhang, Jiejing and JohnQinAMD
6966c5d027
[ROCm] Eliminate HiCache migration stalls and use exact-size host allocations ( #42553 )
...
Signed-off-by: JohnQinAMD <yanyuan.qin@amd.com >
Co-authored-by: JohnQinAMD <yanyuan.qin@amd.com >
2026-10-12 14:05:51 -07:00
ix-48
b60444c621
Fix autograd retention in CUDA graph capture tail hooks ( #43854 )
2026-10-12 12:31:18 -07:00
Liangsheng Yin
0efbb3d2b9
[DFlash] Decide the Domino candidate pool in one place and apply simulated acceptance inside the accept ( #43791 )
2026-10-12 11:35:57 -07:00
Liangsheng Yin
392039d752
[DSpark] Share draft params, logits dtype, confidence and the accept tail between graph and eager ( #43790 )
2026-10-12 11:35:41 -07:00
Shuwen Wang
f92199b569
[Unified Cache] Abort requests whose external linker load fails instead of crashing the scheduler ( #40896 )
2026-10-13 00:03:21 +08:00
Sam Shleifer and Claude Fable 5
d88d5c325a
Prebuild fixup_zero_kv JIT module at TRTLLM-MLA backend init ( #37525 )
...
Co-authored-by: Claude Fable 5 <noreply@anthropic.com >
2026-10-12 23:27:21 +08:00
Yuan Luo and luoyuan.luo
23a1f53fc3
[QSA] Bound the decode indexer's work by the context length ( #43463 )
...
Co-authored-by: luoyuan.luo <luoyuan.luo@antgroup.com >
2026-10-12 22:47:45 +08:00
Alexandr and Артем Савкин
1a553927b0
[CI][Diffusion] Add caching for npu GT images in CI ( #37426 )
...
Co-authored-by: Артем Савкин <58187114+OrangeRedeng@users.noreply.github.com >
2026-10-12 17:18:01 +03:00
279f1fad90
[diffusion] chore: bind worker CPUs to the GPU's NUMA node when memory policy is not permitted ( #43756 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-12 21:09:41 +08:00
Артем Савкин
d1357bf87b
[NPU] Diffusion CI fixes ( #43210 )
2026-10-12 14:53:51 +03:00
ming_wang
c6e822c29b
[Fix] Adapt NPU Qwen-VL processor patch to transformers 5.19 resize() signature change ( #43309 )
2026-10-12 19:24:05 +08:00
Sasha Sidorov and Shangming Cai
7953751bfa
[PD][Spec] Give a PD decode's first EAGLE draft step a proposal distribution ( #41783 )
...
Co-authored-by: Shangming Cai <csmthu@gmail.com >
2026-10-12 18:49:52 +08:00
8afe107c24
[XPU] Upgrade XPU backend to PyTorch 2.14 ( #42003 )
...
Co-authored-by: Ashwini Rathi <arathi@habana.ai >
Co-authored-by: Y <ashwini.rathi@intel.com >
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com >
2026-10-12 18:24:48 +08:00
C and Cheng Wan
28621739fc
[Fix] Make tokenizer request-state initialization atomic ( #41911 )
...
Co-authored-by: Cheng Wan <54331508+ch-wan@users.noreply.github.com >
2026-10-12 03:10:57 -07:00
180933518e
[sgl-router] Serve the Anthropic Messages API on /v1/messages ( #42316 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Xinyuan Tong <115166877+JustinTong0323@users.noreply.github.com >
2026-10-12 17:56:08 +08:00
Jinyan Yi
f006fb49ec
[NPU] Stabilize HiCache and registered CI tests ( #43308 )
2026-10-12 17:10:26 +08:00
Po-Han Huang (NVIDIA) and Claude Opus 5.5
b0c84d4518
[Docs] Cookbook: refresh SemiAnalysis AgentX launches (GLM-5.2 B300, DeepSeek-V4.1 GB300) ( #43799 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-12 17:08:11 +08:00
Baizhou Zhang
450837e770
[Docs] Update GLM-5.3-Flash playground: B300 default, CP8, EP and PD ( #43802 )
2026-10-12 01:38:12 -07:00
BingjiaWang
62ff466ac0
[DSV4.1][PD][1/N] Support prefill node use cp parallel strategy ( #40323 )
2026-10-12 01:05:21 -07:00
BingjiaWang
54409c2fa0
[BCG][DSV4-Flash][CP] Support DeepSeek V4 prefill with interleave CP and breakable CUDA graphs ( #38603 )
2026-10-12 01:03:10 -07:00
Anupa Sajikumar and Siju Samuel
8d491fbc3e
[XPU] Support decode context parallelism (DCP) on Intel XPU ( #34355 )
...
Co-authored-by: Siju Samuel <siju.samuel@intel.com >
2026-10-12 15:50:33 +08:00
Ke Bao
a70a83bcd2
Keep NIXL Mamba checkpoint keys separate by TP rank ( #43594 )
2026-10-12 15:40:59 +08:00
Ke Bao
b6ba148384
Capture KV host transfer buffers ( #43711 )
2026-10-12 15:40:06 +08:00
Mohammad Miadh Angkad and Mohammad Angkad
06887827c7
[CI] Move tests that do not need GB300 off the per-PR GB300 suite ( #43784 )
...
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai >
2026-10-12 00:38:10 -07:00
5ca0695aa6
[HiCache] Preserve all MTP KV and recurrent state ( #40223 )
...
Co-authored-by: alphabetc1 <2508695655@qq.com >
Co-authored-by: Shuwen Wang <47200617+alphabetc1@users.noreply.github.com >
2026-10-12 15:37:59 +08:00
Shuwen Wang
f5f5003323
[HiCache] Support Mamba in buffer-only host memory mode ( #36345 )
2026-10-12 00:35:16 -07:00
Shangming Cai and Claude Fable 5.1
4195980f5a
[PD] Poison the deferred abort ack when a Mooncake send fails ( #43719 )
...
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com >
2026-10-12 15:33:09 +08:00
da4bf1ef8e
[Perf] Route BF16 mixed-batch decode tails through decode attention ( #43238 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com >
2026-10-12 15:27:22 +08:00
Xinyuan Tong
a0e6effe50
[Fix] Link cuda_ipc JIT against CUDA driver stubs dir ( #43764 )
2026-10-12 15:26:32 +08:00
Baizhou Zhang
62c853b654
docs: clarify prefill CP terminology and simplify wording ( #43787 )
2026-10-12 00:24:57 -07:00
208df292e5
[diffusion][xpu] FLUX.2: enable the bit-exact Triton SwiGLU on XPU ( #41940 )
...
Co-authored-by: Claude Opus 5 <noreply@anthropic.com >
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
2026-10-12 15:16:58 +08:00
Mohammad Miadh Angkad and Mohammad Angkad
166b4ae7fc
[Fix] Adapt to transformers 5.18 / 5.19 changes ( #43688 )
...
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai >
2026-10-11 23:24:57 -07:00
ashwini rathi
8ce76b88df
[XPU CI] Optimize PR CI per-file timeouts and fail hanging tests fast ( #42048 )
2026-10-12 14:24:26 +08:00
Mohammad Miadh Angkad and Mohammad Angkad
130e11d50e
[Deps] Bump FlashInfer to 0.7.1 ( #43494 )
...
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai >
2026-10-11 23:23:11 -07:00
sglang-bot and sglang-bot
f9bd83e7a9
docs: sync LMSYS SGLang blog cards ( #43462 )
...
Co-authored-by: sglang-bot <sglang-bot@users.noreply.github.com >
2026-10-12 06:07:15 +00:00
Liangsheng Yin
9e0b001db4
[CI] Skip eval retries in the PD transfer-failure test ( #43769 )
2026-10-11 22:37:40 -07:00
Liangsheng Yin
2240049194
[RL] Return a failed weight update when the cache flush is refused ( #43758 )
2026-10-11 21:57:58 -07:00
a297cd3da1
[Fix] scheduler: stop latching batch_is_full when nothing can run ( #41872 )
...
Co-authored-by: hnyls2002 <lsyincs@gmail.com >
Co-authored-by: Liangsheng Yin <hnyls2002@gmail.com >
2026-10-11 21:56:19 -07:00
Mohammad Miadh Angkad and Mohammad Angkad
bf703013f2
[Test] Re-anchor the VLM flashinfer throughput floor ( #43762 )
...
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai >
2026-10-11 21:34:50 -07:00
d434ea444d
[DSV4.1] Reduce eager Engram extend preparation ( #42159 )
...
Co-authored-by: Shangming Cai <csmthu@gmail.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-11 21:31:55 -07:00
Liangsheng Yin
a1001bcf3a
[PD] Retract transferred decode requests on a retract pause ( #43757 )
2026-10-11 21:29:39 -07:00
ChengYM
0d0be8fc98
[AMD][MORI] Add MORI EP V2 support ( #40666 )
2026-10-11 20:54:13 -07:00
Shunkangz
20c49f12c9
Support kv cache sharding for DSA indexer ( #40925 )
2026-10-12 11:50:23 +08:00
37d2e5c34f
[AMD][V4.1][*/N] Fix DP attention + TP MoE decode NaN and support bounded replay ( #43077 )
...
Co-authored-by: wunhuang <wunhuang@amd.com >
Co-authored-by: HAI <hixiao@gmail.com >
2026-10-11 20:47:13 -07:00
weireweire
296774da37
[Tokenizer] Require fastokens>=0.3.2 and include it in the all extra ( #43104 )
2026-10-11 20:41:53 -07:00
adenzhou1350
cd8ed1bb55
[Bugfix] Preserve available host memory for integrated MUSA devices ( #43398 )
...
Signed-off-by: Xucheng Zhou <aden1350@outlook.com >
2026-10-11 20:34:58 -07:00
Po-Han Huang (NVIDIA) and Claude Opus 5.5
6ac8602e8d
[CI] Fix TestQwenImageModulationCache after Qwen-Image modulation sharding ( #43747 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-11 20:31:29 -07:00
77a15cf4e2
[MemCache] Add shared component size and path APIs to the tree core interface ( #43722 )
...
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com >
Co-authored-by: Jialin <Jialin@users.noreply.github.com >
2026-10-11 20:14:07 -07:00
95ce9caaf2
[Rust] Lock hyper 1.12.0 and want 0.3.2 in the rust workspace ( #43724 )
...
Co-authored-by: hanminglu <hanminglu@users.noreply.github.com >
Co-authored-by: shodoco <shodoco@users.noreply.github.com >
2026-10-11 20:12:55 -07:00
e4cfc0c42a
[MemCache] Split the SWA mapping move out of locked-full recovery as RecoverEvictedSWA ( #43721 )
...
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com >
Co-authored-by: Jialin <Jialin@users.noreply.github.com >
2026-10-11 20:11:52 -07:00
WangHemeng
0c52cb1519
[MoE] Skip block-padding rows in the TMA down activation and quant ( #42212 )
2026-10-12 11:10:30 +08:00
Alison Shao
1a7af3290d
Fix Mistral-native format detection when the Hub is unreachable ( #43446 )
2026-10-11 19:59:19 -07:00
Chunan Zeng
3c74f68a9c
[Fix] Initialize BF16 GEMM dispatch in the one-batch benchmark ( #43702 )
2026-10-12 02:53:02 +00:00
WenhaoZhang and Mick Qian
d0a098b2e0
[diffusion] perf: split and batch LTX-2.3 one-stage guidance passes under CFG parallel ( #43684 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
2026-10-12 09:36:01 +08:00
WenhaoZhang
a954b395e6
[diffusion] perf: layerwise offload for the LTX-2.5 Gemma-4 text encoder ( #43683 )
2026-10-12 09:34:35 +08:00
WenhaoZhang
a67228aa43
[diffusion] fix: shard Qwen-Image adaLN modulation and text FFN under TP ( #43691 )
2026-10-12 09:32:52 +08:00
Liangsheng Yin
e6072935bf
[DSpark] Apply SGLANG_SIMULATE_ACC_LEN inside the accept instead of switching paths ( #43623 )
2026-10-11 15:33:24 -07:00
Liangsheng Yin
a65b776963
[DSpark] Add SGLANG_DSPARK_USE_GREEDY_DRAFT, an argmax draft accepted as one-hot ( #43670 )
2026-10-11 15:18:59 -07:00
Liangsheng Yin
3cc23b3b86
[DSpark] Merge the draft TP sync sites into dspark-draft ( #43669 )
2026-10-11 15:17:47 -07:00
Liangsheng Yin
00ba90fd1a
[DSpark] Run the same markov block in the draft graph tail; drop SGLANG_DSPARK_FOLDED_SAMPLING ( #43668 )
2026-10-11 15:15:59 -07:00
Liangsheng Yin
48abbdb845
[DSpark] Draft the eager markov block through MarkovSampleStep ( #43622 )
2026-10-11 15:13:02 -07:00
Liangsheng Yin
467209e838
[DSpark] Add MarkovSampleStep, one op for a markov draft step's add and token pick ( #43621 )
2026-10-11 15:10:58 -07:00
Liangsheng Yin
9943cf8eef
[DSpark] Run every markov head through one step interface ( #43620 )
2026-10-11 15:07:51 -07:00
Banruo Liu and Shangming Cai
d62b9c2480
[KV Events] Publish LoRA adapter names and namespaced block hashes ( #42455 )
...
Signed-off-by: Banruo Liu <lichangxinlivingshade@gmail.com >
Co-authored-by: Shangming Cai <csmthu@gmail.com >
2026-10-11 23:23:30 +08:00
97bfa5cc53
[diffusion] fix: select VSA-H3 for FastH3 when no transformer backend is given ( #43690 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-11 23:05:25 +08:00
Ke Bao
46a97e5540
Preserve serialized page dtype in NIXL copy fallback ( #43593 )
2026-10-11 22:39:14 +08:00
Ke Bao
b0a75c8196
Share MLA host capacity initialization ( #43595 )
2026-10-11 22:36:29 +08:00
Ke Bao
c938b6e454
Build Mamba host pools from device buffer infos ( #43596 )
2026-10-11 22:35:52 +08:00
Shuwen Wang
ef3e5c83c5
[HiCache] Keep storage workers alive on backend exceptions ( #43550 )
2026-10-11 22:26:56 +08:00
Mick and Mick Qian
11bfd868ac
[diffusion] perf: encode LTX video/audio through the shared bounded MP4 path ( #43656 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
2026-10-11 21:15:49 +08:00
Shuwen Wang
0923bcd258
[Unified Radix] Invalidate lazy Mamba checkpoints before in-place tracking ( #43567 )
2026-10-11 21:12:51 +08:00
FeiHong Guo
cf88ed4352
[BugFix][DP] Prevent health checks from perturbing user request routing ( #35721 )
...
Signed-off-by: Phoenix3334 <Phoenix3334@users.noreply.github.com >
2026-10-11 21:08:55 +08:00
Shuwen Wang
d26b6a5a9c
fix: default Inkling to fa4 on Hopper ( #35292 )
2026-10-11 20:57:42 +08:00
trebladev and Mick Qian
586d98cc15
[Diffusion] Add ModelOpt FP8 checkpoint quantization for Pi0.5 ( #43062 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
2026-10-11 20:55:59 +08:00
Mick and Mick Qian
04f740580c
[VLM] fix: repair multimodal VMM lease cleanup and source-stream ordering ( #43475 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
2026-10-11 19:13:39 +08:00
efd14cfb66
[DeepSeek-V4.1] Use num_stages=1 for the hc_mix stats partial kernel ( #43070 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Yuan Luo <4908075+yuan-luo@users.noreply.github.com >
2026-10-11 19:03:19 +08:00
Sage
53295c6972
[kvevents] fix missing topics in replayed kv events ( #43386 )
...
Signed-off-by: Sage Ahrac <sagiahrak@gmail.com >
2026-10-11 17:25:43 +08:00
Liangsheng Yin
0a21a3b798
[Scheduler] Keep a DP-converted decode batch as the running batch ( #43283 )
2026-10-11 01:23:51 -07:00
jiaqiang-dot-liu
41652225d0
[Sampler] Sort only the max(top_k) leading columns in the torch sampler ( #43346 )
2026-10-11 01:00:59 -07:00
Liangsheng Yin
dab108b9f3
[Scheduler] Hold back decode rows on a mixed step when they do not fit the KV pool ( #43290 )
2026-10-11 00:18:27 -07:00
Liangsheng Yin
89e24e7a20
[KV canary] Expect sweep counters on the target runner only ( #43286 )
2026-10-10 23:52:17 -07:00
Liangsheng Yin
ad8b1e1f85
[Test] Do not hang scripted-runtime teardown when the server died ( #43285 )
2026-10-10 23:13:56 -07:00
Liangsheng Yin
448759b235
[Scheduler] Requeue preempted requests when admission admits nothing ( #43298 )
2026-10-10 22:46:55 -07:00
Qiaolin Yu
047405f534
[qwen 3.8 next] Fuse QSA prefill KV gathering and dtype conversion ( #41933 )
2026-10-10 22:40:23 -07:00
paulzhang-tm
5678b71b6c
[Spec] Allow Qwen3.5 backends to opt into packed target verification ( #43037 )
2026-10-10 22:39:17 -07:00
Nikolai
22d63f5e43
[AMD]Qwen add Sparse attention to capturing hip graphs ( #43383 )
2026-10-10 22:03:54 -07:00
Jzz1943
260551ebe9
[KDA] Support Helion MTP verification and harden prefill, decode ( #42857 )
2026-10-11 12:47:57 +08:00
SunskyXH and Xiaoyu Zhang
65e6bb6be7
[GDN] Fix CuTe DSL decode to read the [N, HV, V, K] state layout ( #43332 )
...
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com >
2026-10-11 12:43:16 +08:00
a1500b10e3
[diffusion] Don't report video jobs completed after their outputs are deleted ( #43084 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com >
2026-10-11 12:42:38 +08:00
Chunan Zeng and Claude Opus 5.5
aa5c969dc3
[Perf] Fuse BF16 writes to native HND KV cache storage ( #43237 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-11 12:41:18 +08:00
Shuwen Wang
58cb1a2be6
[Scheduler] Keep waiting-prefix LRU refresh consistent across TP ranks ( #43479 )
2026-10-10 21:18:41 -07:00
Avaya Aggarwal and niehen6174
3458c94fad
[diffusion] comfyui: consolidate executor conditioning-cache fixes ( #43492 )
...
Co-authored-by: niehen6174 <1639206518@qq.com >
2026-10-11 11:42:00 +08:00
Avaya Aggarwal and niehen6174
638330d975
[diffusion] comfyui: consolidate FluxAdapter correctness fixes ( #43490 )
...
Co-authored-by: niehen6174 <1639206518@qq.com >
2026-10-11 11:41:36 +08:00
Khoa Pham
6576e50cd5
[1/N][DSv41] Enable DP attention with MegaMoE ( #43228 )
2026-10-10 20:36:35 -07:00
b931b2c8bd
[diffusion] perf: run the attention section eager under torch.compile for MiniMax-H3 ( #43634 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-11 11:02:54 +08:00
b21826e609
[diffusion] fix: replay Qwen-Image-2.1 BCG graphs across prompt lengths ( #43633 )
...
Co-authored-by: Mick <mickqian@radixark.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-11 10:55:01 +08:00
b237afa547
[diffusion] fix: Qwen-Image BCG prompt padding misses every graph and attends the pad ( #43632 )
...
Co-authored-by: Mick <mickqian@radixark.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-11 10:54:18 +08:00
b3a65efe7e
[diffusion] perf: keep Cosmos3's fused kernels in the graph ( #43577 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-11 10:53:43 +08:00
d45ef0f2d2
[diffusion] perf: keep Qwen-Image-2.1 and Ulysses graphs whole ( #43586 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-11 10:52:43 +08:00
Shangming Cai and Kevin Flansburg
e1f43c9a61
[PD][HiCache] Fold the decode restore outcome into the MIN-reduced KV poll ( #43562 )
...
Co-authored-by: Kevin Flansburg <kflansburg@cloudflare.com >
2026-10-11 10:12:34 +08:00
Richard Wang
d60e0c8570
[diffusion] Support LLaDA-Image BF16 generation and editing ( #37907 )
2026-10-11 02:10:51 +00:00
e4b423256e
[diffusion] fix: serve whole graphs after offload-during-compile ( #43631 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-11 10:03:47 +08:00
Richard Wang
d7ed5b56f5
[diffusion] fix: keep the overlay probe from stalling cached startup ( #43590 )
2026-10-11 10:01:01 +08:00
156d54e56c
fix: wait for HiCache load-back before hybrid MLA KV reads ( #38217 )
...
Co-authored-by: zhouzechun <zhouzechun@kuaishou.com >
Co-authored-by: alphabetc1 <2508695655@qq.com >
2026-10-11 09:50:13 +08:00
Jzz1943 and Xiaoyu Zhang
9336681310
[DSA] Support TileLang FP8 KV cache on CUDA ( #42957 )
...
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com >
2026-10-11 09:12:18 +08:00
Xiaoyu Zhang
33d2ef1756
[Fix] Reserve remaining tokens for ignore_eos requests during prefill admission ( #43564 )
2026-10-11 09:05:48 +08:00
WenhaoZhang and OnePunchMonk
5730345717
[diffusion] comfyui: fail early instead of silently dropping features in SGLD integrated mode ( #43495 )
...
Co-authored-by: OnePunchMonk <aggarwal.avaya@yahoo.com >
2026-10-11 08:58:23 +08:00
d1313dd5fe
[diffusion] fix: skip inductor's saved multi-kernel pick for torch.compile (Z-Image warm starts crash or hang) ( #43601 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-11 08:56:55 +08:00
3022989bad
[diffusion] CI: skip the LTX-2 RMSNorm-modulation guard test off the CUDA platform ( #43574 )
...
Co-authored-by: Mick <mickqian@radixark.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-11 08:54:37 +08:00
65386cb95a
[diffusion] perf: add the dynamic LoRA delta inside the second GEMM ( #43501 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-11 08:50:53 +08:00
e328c7ecbd
[diffusion] fix: skip QKV packing while layers are offloaded ( #43576 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-11 08:47:35 +08:00
7373665085
[diffusion] fix: keep exported buffers out of expandable segments for IPC all-to-all( #43584 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-11 08:46:30 +08:00
5e872557cb
[diffusion] feat: keep the TP all-reduce inside the graph ( #43602 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-11 08:45:54 +08:00
664e4c6499
[diffusion] feat: warn instead of disabling on unvalidated models for BCG( #43568 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-11 08:40:28 +08:00
23764b1242
[diffusion] feat: bind each GPU worker to its GPU's NUMA node ( #43565 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-11 08:38:58 +08:00
9933cf5437
[diffusion] feat: declare regional compile regions for flux ( #43575 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-11 08:33:29 +08:00
68e1a3dbb1
[diffusion] fix: accept request-level profiling on the image API ( #43573 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-11 08:30:20 +08:00
d32d8fa0d1
[Dsv4.1] Mega-mhc integration ( #42808 )
...
Co-authored-by: Yuwei An <yuwei.an@MacBook-Pro-yuweian.local >
Co-authored-by: kpham-sgl <khoa.pham@radixark.ai >
2026-10-10 17:25:17 -07:00
Liangsheng Yin
7187bb593e
[Model] Keep gpt-oss router logits batch-invariant under deterministic inference ( #43624 )
2026-10-10 17:22:29 -07:00
Mohammad Miadh Angkad and Mohammad Angkad
c640dc97ce
[CI] Fix Kimi-Linear DCP PD test startup ( #43598 )
...
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai >
2026-10-10 16:53:54 -07:00
Cheng Wan
234ccba7f6
[Refactor] Declare every model's stages from a shared function and drop the meta-device neighbour build ( #43460 )
2026-10-10 12:43:09 -07:00
Cheng Wan
3b16eeedce
[Refactor] Declare pipeline neighbours from a shared declaration function ( #43459 )
2026-10-10 12:42:46 -07:00
Cheng Wan
3b717c0f13
[Refactor] Decide an exit's fixed facts when the stage is built ( #43458 )
2026-10-10 12:42:23 -07:00
Cheng Wan
a956aa192a
[Refactor] Bind a layer stack in one pass over each stage's data flow ( #43457 )
2026-10-10 12:42:00 -07:00
Cheng Wan
d2c597c8db
[Refactor] Decide input-scattered eligibility and stage rejections in one place ( #43456 )
2026-10-10 12:41:38 -07:00
Cheng Wan
b0dde87de2
[Refactor] Make residual read and update capabilities explicit and checked ( #43455 )
2026-10-10 12:41:16 -07:00
Cheng Wan
514feb1afc
[Fix] Keep residual rows and sums consistent across appends and pipeline handoffs ( #43454 )
2026-10-10 12:40:53 -07:00
Cheng Wan
8bd7f2d178
[Refactor] Record what a bound stage path does instead of reading it back ( #43453 )
2026-10-10 12:40:30 -07:00
0acd93c35c
[Scheduler] Use committed length in spec mamba track boundary check ( #43029 )
...
Co-authored-by: Ke Bao <ispobaoke@gmail.com >
Co-authored-by: Shuwen Wang <47200617+alphabetc1@users.noreply.github.com >
2026-10-10 12:29:40 -07:00
Shuwen Wang and Ke Bao
f9cee8d2b2
[HiCache] Keep protected cgroup page cache charged in host budgets ( #42788 )
...
Co-authored-by: Ke Bao <ispobaoke@gmail.com >
2026-10-10 17:03:40 +00:00
Xiaoyu Zhang
f61163b4b0
[Docs] Let the Qwen3 B300 recipe use the default attention backend ( #43554 )
2026-10-10 10:00:04 -07:00
Xiaoyu Zhang
587b6d622d
[Docs] Document Qwen3-4B/8B H200 mixed-chunk latency tradeoffs ( #43551 )
2026-10-10 09:59:46 -07:00
Shunkangz
de487f8039
[kv-shard 3/4] Enable Control Plane C ( #40102 )
2026-10-10 22:53:58 +08:00
huangtingwei
406540ab5f
[HiCache] Add staged write-back for the page-unified KV cache layout ( #39606 )
2026-10-10 22:49:45 +08:00
Shuwen Wang
99d6f4e158
[MemCache] refactor: require an anchored lock receipt for non-root releases ( #41452 )
2026-10-10 22:43:49 +08:00
Shuwen Wang
44bc478863
[HiSparse] Avoid host synchronization in eager backup ( #41446 )
2026-10-10 21:49:55 +08:00
hhy-seven
616178a596
[Responses] Fix max_output_tokens overestimate on multimodal requests ( #35524 )
2026-10-10 13:42:53 +00:00
a510340aeb
[diffusion] perf: run head_dim-128 QK-norm + RoPE two rows per warp for every RoPE layout ( #43508 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-10 21:34:54 +08:00
72657f7ca5
[diffusion] perf: pipeline every USPAttention Ulysses path's exchange over the copy engine ( #43521 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-10 21:34:18 +08:00
Shuwen Wang
416ebe6090
[CI] Fix EAGLE3 CUDA parity prefill chunk alignment ( #43538 )
2026-10-10 21:11:17 +08:00
49e629d239
[Rust] Upgrade sglang-radix-tree to PyO3 0.29 ( #42807 )
...
Co-authored-by: colleen-tml <296620403+colleen-tml@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-10 20:56:50 +08:00
Bingxu Chen
a916d91cd3
[AMD][CI] Cap pyarrow below 26 for lmms-eval ( #43547 )
2026-10-10 18:59:49 +08:00
Sage
6fc8d9da32
[rust-server] preserve PD routing in chat and completions ( #42989 )
...
Signed-off-by: Sage Ahrac <sagiahrak@gmail.com >
2026-10-10 17:29:46 +08:00
Liangsheng Yin
c55790d38b
[CI] Drop CPU tests with no bug-catching signal and run Rust workspace tests in their own workflow ( #43503 )
2026-10-10 02:13:53 -07:00
siliangchen-amd
ff19159e16
[AMD] Add an opt-in AITER CK blockscale GEMM for gfx942 block-FP8 linears ( #41071 )
...
Signed-off-by: siliangchen-amd <SiLiang.Chen@amd.com
2026-10-10 01:31:20 -07:00
chuyeh
e35f60b9bc
[AMD][CI] Drop removed tp_rank/tp_size kwargs from small-M FP8 proj test ( #43318 )
2026-10-10 00:52:28 -07:00
d7d22e8a30
[AMD] Resolving bug in per batch size graph capture due to multiple capture runs per batch size ( #39840 )
...
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com >
Co-authored-by: HAI <hixiao@gmail.com >
2026-10-10 00:47:13 -07:00
d8c5eca3ac
[diffusion] fix: keep a LoRA unmerged under merge_mode auto when merging would round its update away ( #43385 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-10 15:02:03 +08:00
Richard Wang
441de659f4
[CI] Fix Python 3.10 failure in prefill complete integration test ( #43505 )
2026-10-10 00:00:03 -07:00
tao
6e8f3e09dd
[Model Loader] Add InstantTensor loader support ( #40453 )
2026-10-09 23:01:59 -07:00
cctry
33f6b00f32
Clamp default decode CUDA graph max_bs to max_running_requests ( #43172 )
2026-10-09 22:54:32 -07:00
Mohammad Miadh Angkad and Mohammad Angkad
16be9df25f
[CI] Remove the GLM-5.3-Flash PTX KDA prefill nightly test ( #43470 )
...
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai >
2026-10-09 22:52:17 -07:00
Yuwei An
8b2aaf7060
[Refactor][TCPCG] Retire backend selection and configuration (3/9) ( #42287 )
2026-10-09 22:42:35 -07:00
Yuwei An
0aa580f904
[Dsv4.1] Add DeepSeek V4.1 DSPARK TP and DP EP accuracy coverage ( #43478 )
2026-10-09 22:41:30 -07:00
YAMY
17852cdfe8
[DSv4.1] Initialize the full-to-SWA mapping for request-window pools ( #43165 )
2026-10-09 22:39:26 -07:00
Frederik Gossen
3831e7e091
[diffusion] Z-Image: autotune the shared transformer block at both token scales ( #35755 )
2026-10-10 13:04:43 +08:00
Raymond Huang and Claude Opus 5
aa79087017
[diffusion] fix: fail MiniMax-H3 quality="high" closed on substituted DiT weights ( #34978 )
...
Co-authored-by: Claude Opus 5 <noreply@anthropic.com >
2026-10-10 13:03:44 +08:00
ccae3b3639
[diffusion] perf: pipeline MiniMax-H3's Ulysses exchange against cuDNN SDPA on Blackwell ( #43373 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-10 12:58:30 +08:00
Shangming Cai
1814ece57f
[PP+PD] Harden PPConsensusStore: msgspec wire format, drop malformed messages ( #43329 )
2026-10-10 12:38:38 +08:00
cctry
20948cdcc6
[PD] Reclaim parked optimistic prefills when request slots are exhausted ( #43442 )
2026-10-09 20:31:24 -07:00
Liangsheng Yin
7d44269907
[MemCache] Fix the strict busy check on prefill-aware SWA and tidy small leftovers ( #43474 )
2026-10-09 20:27:27 -07:00
f2e9a2bed1
[unified memory pool] Allocate without transfer waits, pin floating SWA reservations, drop the HiCache startup asserts ( #43206 )
...
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com >
Co-authored-by: ZYHowell <ZYHowell@users.noreply.github.com >
2026-10-09 20:19:30 -07:00
Mick and Mick Qian
a65f4ffba5
[diffusion] docs: standardize diffusion cookbook command builders ( #43472 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
2026-10-10 11:14:31 +08:00
436d73ddda
[diffusion] model: support HunyuanImage-3.0-Instruct ( #36311 )
...
Co-authored-by: Elizaveta Martirosian <you@example.com >
Co-authored-by: wes-lyu <294033513+wes-lyu@users.noreply.github.com >
Co-authored-by: Wes <its.wesley.lyu@gmail.com >
Co-authored-by: AuFlow <73925903+AuFlow@users.noreply.github.com >
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
Co-authored-by: Mick <mickjagger19@icloud.com >
2026-10-10 11:11:28 +08:00
506f140166
[sgl-router] Read finished /generate streams to the end; expose reorg admission caps as flags ( #43233 )
...
Co-authored-by: Kan Wu <kan.wu@radixark.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-10 10:59:09 +08:00
Benjamin Truong and Xiaoyu Zhang
37678784c6
[Gemma4][Attention] Enable Gemma4 FA4 on SM90 with head-dim 512 support ( #42019 )
...
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com >
2026-10-10 10:40:19 +08:00
55a8dffc87
Skip the eager activation reserve in KV sizing on a capped Full prefill role ( #43435 )
...
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com >
Co-authored-by: 842974287 <842974287@users.noreply.github.com >
2026-10-09 19:36:48 -07:00
a338b9858a
[PD] Receive decode KV into separate paged SWA host pools; let host-capable receivers own abort drain tracking ( #43410 )
...
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com >
Co-authored-by: cctry <cctry@users.noreply.github.com >
2026-10-09 19:33:06 -07:00
Vladislav Nosivskoy and BBuf
2ff3e6dce1
[SM90] Use native block32 DeepGEMM for DeepSeek-V4.1 ( #42604 )
...
Co-authored-by: BBuf <1182563586@qq.com >
2026-10-10 10:29:42 +08:00
Liangsheng Yin
465253ef52
[Scheduler] Fix the deterministic-inference hang when a prefill chunk is shorter than its alignment ( #43444 )
2026-10-09 19:06:18 -07:00
78218194ad
perf(kda): rebalance MTP verify precompute and state synchronization ( #42683 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com >
2026-10-10 10:03:19 +08:00
Michael
37114afe20
[AMD][Fix] Saturate FP8 KV commits instead of reproducing e4m3fnuz NaN overflow ( #42982 )
2026-10-10 09:49:56 +08:00
Mick and Mick Qian
77ba54f7df
[diffusion] docs: add Wan-Animate-2 cookbook command picker ( #43380 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
2026-10-10 09:45:16 +08:00
16423cb6d2
[sgl-router] Serve DeepSeek-V4.1 Flash through sglang-processor ( #43000 )
...
Co-authored-by: Kan Wu <kan.wu@radixark.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-10 09:23:09 +08:00
5f94a45896
[srt] Allow mixed chunk prefill on the Full prefill CUDA graph backend ( #43461 )
...
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com >
Co-authored-by: joshuadeng <joshuadeng@users.noreply.github.com >
2026-10-09 18:21:22 -07:00
Mick and Mick Qian
cde16aa64d
[diffusion] model: support Qwen-Image 2.1 Turbo and checkpoint sampling grids ( #43391 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
2026-10-10 09:12:56 +08:00
e8fb52c526
[Fix] Keep request-pool row generations monotonic across clear() ( #43439 )
...
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com >
Co-authored-by: colinchan15 <colinchan15@users.noreply.github.com >
2026-10-09 18:08:54 -07:00
ashwini rathi
8cf4d73c84
[XPU] Fix XPU CI breaks from #41105 (KDA chunk_offsets) and #42588 (int4 test tp_group) ( #43334 )
2026-10-10 09:06:24 +08:00
Avaya Aggarwal
66e6b68d82
[diffusion] comfyui: report the real runtime import error ( #43178 )
2026-10-10 09:05:12 +08:00
ff3aa58f11
[diffusion] perf: run MiniMax-H3's QK-norm + RoPE two rows per war while bit-identical ( #43337 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-10 09:03:36 +08:00
bookinstock and Mick Qian
32f34e898b
[diffusion] fix: honor requested inference step count for MiniMax H3( #38671 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
2026-10-10 09:02:57 +08:00
bd7c7971ba
[diffusion] perf: move MiniMax-H3's pipelined Ulysses copies off the SMs ( #43361 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-10 08:58:07 +08:00
Alison Shao
9b37e28b83
Disable DualGemm fusion for Inkling dense MLP ( #43447 )
2026-10-09 17:56:20 -07:00
6acaf3374a
[kernel] SwiGLU-OAI MXFP8 producer; move swizzled MXFP8 producers to ops/quantization ( #42810 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Denji-kk <51020465+Denji-kk@users.noreply.github.com >
2026-10-10 08:51:57 +08:00
Rain Jiang
961bcf481d
support DeepEP v2 on GLM-5.3 ( #43432 )
2026-10-09 16:57:12 -07:00
Yuwei An and Yuwei An
2d38934493
[DSv4.1] mxfp8 dispatch cache ( #42156 )
...
Co-authored-by: Yuwei An <yuwei.an@MacBook-Pro-yuweian.local >
2026-10-09 16:36:28 -07:00
Liangsheng Yin
46498bc8c5
[MLX] Sync decode KV to the pool once at release, clamped to the owned prefix ( #43292 )
2026-10-09 16:20:45 -07:00
Liangsheng Yin
114cb44920
[MLX] Read radix prefix slots from the model runner's req_to_token pool ( #43291 )
2026-10-09 16:18:58 -07:00
Liangsheng Yin
8020ea2ee9
[MLX] Drop a retracted request's runner state at its re-prefill ( #43284 )
2026-10-09 16:16:59 -07:00
Liangsheng Yin
f7e601b936
[MLX] Drain in-flight jobs before a retract-mode pause_generation ( #43282 )
2026-10-09 16:14:23 -07:00
f6c848c882
[PD] Use 1 to disable decode host receive and 0 for always-on ( #43407 )
...
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com >
Co-authored-by: cctry <cctry@users.noreply.github.com >
2026-10-09 15:50:52 -07:00
Alison Shao
f6e3444b2e
ci: diffusion-only option for the runner utilization report ( #41620 )
2026-10-09 15:47:47 -07:00
Cheng Wan
58f6976ce7
[Fix] Unblock CUDA graph startup for hybrid parallelism ( #43211 )
2026-10-09 15:39:59 -07:00
cb9bd1d4d4
[Perf] Reduce multi-detokenizer router IPC sends ( #42238 )
...
Co-authored-by: hnyls2002 <lsyincs@gmail.com >
Co-authored-by: Liangsheng Yin <hnyls2002@gmail.com >
2026-10-09 15:32:14 -07:00
66d840a8ef
[cuda_graph] Size dp-local decode graphs by the runner's own gather requirement ( #43303 )
...
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com >
Co-authored-by: charlotte12l <charlotte12l@users.noreply.github.com >
2026-10-09 15:21:23 -07:00
Kevin Mi and Claude Opus 5.5
6d8ddaf1d7
[AMD] MiniMax-M3 indexer CP: packed scoring for EAGLE verify rows ( #42614 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-09 15:05:50 -07:00
fbf8d730fd
[Fix] MiniMax-M3: default to the breakable prefill CUDA graph ( #41845 )
...
Co-authored-by: Hao Phan <htphan@nvidia.com >
Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com >
Co-authored-by: Kevin Mi <45493463+kevin-mii@users.noreply.github.com >
Co-authored-by: Po-Han Huang <pohanh@nvidia.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-09 15:05:05 -07:00
Kevin Mi and Claude Opus 5.5
70e82f7b73
[AMD] Run the MiniMax indexer CP test on MI35X, not MI300 ( #43259 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-09 15:04:25 -07:00
weireweire and weireweire
46c2585faa
[DSpark] Fold draft sampling for DeepSeek-V4.1 heads under AUTO ( #42987 )
...
Co-authored-by: weireweire <20922698+weireweire@users.noreply.github.com >
2026-10-09 14:40:28 -07:00
Huabin
2f1acef3d8
[Model] Add LLaDA2.2 Block Routing MoE support ( #31768 )
2026-10-09 12:14:44 -07:00
Mohammad Miadh Angkad and Mohammad Angkad
72eab59bf6
[GLM-5.3-Flash] Enable breakable prefill CUDA graph by default ( #42845 )
...
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai >
2026-10-10 02:41:02 +08:00
87845cda2b
[Multimodal] Chunk and cache audio items whose offsets span several placeholder runs ( #43301 )
...
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com >
Co-authored-by: YavorGIvanov <YavorGIvanov@users.noreply.github.com >
2026-10-09 10:57:01 -07:00
3bffe69b74
[moe] Fill num_token_non_padded for the customized A2A backend at EP 1 ( #43245 )
...
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com >
Co-authored-by: cctry <cctry@users.noreply.github.com >
2026-10-09 10:55:02 -07:00
3c612132fd
Overlap scheduler startup with the data parallel controller ( #43177 )
...
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com >
Co-authored-by: Yongji Wu <30348494+libertyeagle@users.noreply.github.com >
2026-10-09 10:54:22 -07:00
Mohammad Miadh Angkad and Mohammad Angkad
5cbf949839
Fix torch.compile crash in fused gate-sigmoid-mul launcher ( #43384 )
...
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai >
2026-10-09 23:51:30 +08:00
76ccf0850c
[diffusion] fix: fix TeaCache CFG state lifecycle and skip boundaries ( #42422 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
Co-authored-by: Mick <mickjagger19@icloud.com >
2026-10-09 23:20:13 +08:00
121aa8dcc7
[diffusion] perf: fuse MiniMax-H3's RMSNorm and indexed AdaLN under quality lossless ( #43327 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-09 22:39:11 +08:00
Jimmy Shong
408f5d057a
docs: add Clef accuracy results to the deployment wizard ( #43349 )
2026-10-09 22:19:27 +08:00
1203eb9181
[diffusion] perf: pipeline MiniMax-H3's Ulysses exchange against attention over the copy engine ( #43164 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-09 22:04:29 +08:00
xiaofei-zheng
fba3ad4b1f
[Frontend] Parallelize long chat prompt encoding ( #41259 )
2026-10-09 21:30:22 +08:00
xiaofei-zheng
db32673b78
[AMD] Let the GLM DSA NextN draft declare its own shared-expert fusion architecture ( #41258 )
2026-10-09 21:28:45 +08:00
xiaofei-zheng
d5c3ae9752
[AMD] Eliminate remaining gfx95 FP8 scale relayout copies ( #41030 )
2026-10-09 21:27:38 +08:00
78996e3f02
[Performance] Batch Mooncake hybrid-pool metadata and puts ( #39696 )
...
Co-authored-by: weireweire <20922698+weireweire@users.noreply.github.com >
Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com >
Co-authored-by: Po-Han Huang <pohanh@nvidia.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-09 21:12:42 +08:00
d511e6bd1d
[sgl-router] Parse tool calls when serving chat through /generate ( #42999 )
...
Co-authored-by: Kan Wu <kan.wu@radixark.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-09 21:07:30 +08:00
67d063f63b
[diffusion] feat: allow per-role attention backend override ( #35310 )
...
Co-authored-by: Mick <mickjagger19@icloud.com >
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
2026-10-09 20:57:29 +08:00
348689da40
[PP+PD] Fix #38206 Reduce bootstrap consensus latency ( #38959 )
...
Co-authored-by: inkcherry <mingzhi.liu@amd.com >
Co-authored-by: Shangming Cai <csmthu@gmail.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-09 20:53:48 +08:00
f1e0472fcc
[diffusion] model: support Wan-Animate-2 ( #42941 )
...
Co-authored-by: Ankit Dhall <8938083+ankitdhall@users.noreply.github.com >
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
2026-10-09 20:46:23 +08:00
Shuwen Wang
0955169ed2
[mem_cache] Bound FlexKV checkpoint stores and restore DSPARK cache checks ( #35694 )
2026-10-09 20:25:33 +08:00
59eb71a831
[diffusion] fix: align SGLD CFG combine formula with diffusers Z-Image ( #23772 )
...
Co-authored-by: SGLang CI <ci@sglang.ai >
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
Co-authored-by: Mick <mickjagger19@icloud.com >
2026-10-09 19:47:03 +08:00
Shuwen Wang
f48ed2127e
test: allow bounded checkpoint lag in Mamba HiCache tests ( #42523 )
2026-10-09 19:24:39 +08:00
3805d78796
[sgl-router] Serve /v1/chat/completions through /generate, with reasoning parsing ( #42998 )
...
Co-authored-by: Kan Wu <kan.wu@radixark.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Shangming Cai <csmthu@gmail.com >
2026-10-09 19:17:52 +08:00
Xiaoyu Zhang
0eb24e7d5b
[Performance] Optimize Qwen3.8-Flash-Next BF16 decode on Blackwell ( #42175 )
2026-10-09 18:46:12 +08:00
516fd1c1b5
[diffusion] perf: tile MiniMax-H3's indexed modulation kernels by 2048 columns ( #43267 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-09 18:23:52 +08:00
Xinyuan Tong
91b3692a4c
docs: add thinking budget guide ( #41601 )
2026-10-09 17:50:34 +08:00
BingjiaWang
888de24cc8
Add BJWang-ant to CI_Permission ( #43321 )
2026-10-09 02:31:38 -07:00
黄孝君
e1c6ddca1d
[NPU] Bump SGLANG_KERNEL_NPU_TAG to 2026.9.0.post9 ( #43340 )
2026-10-09 17:30:38 +08:00
vOv and Xiaoyu Zhang
7cfecbef56
[diffusion][model] Add native SANA-Video 2.0 support ( #41492 )
...
Signed-off-by: cr-gao <gaochenrui@sjtu.edu.cn >
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com >
2026-10-09 17:28:03 +08:00
DevashishLal-CB
4debae7534
Add blackwell dual gemm fusion support for MLP ( #41887 )
...
Signed-off-by: Devashish Lal <devcode@meta.com >
2026-10-09 17:26:33 +08:00
Thomas Wang
5eca4d8caf
[AMD] Set maxseq length limit for DSV4.1 BCG metadata buffers ( #42570 )
2026-10-09 02:15:43 -07:00
Xinyi Song
23f890613e
[AMD] Patch aiter's group32 MXFP8 GEMM masked-scale fix ( #43319 )
2026-10-09 02:08:44 -07:00
Jimmy Shong
14e3414513
[Docs] Add Clef cookbook with selectable Clef Flash variant ( #43235 )
2026-10-09 01:56:45 -07:00
amote-i
ff146fa20c
[NPU] [DOC] remove --enable-torch-compile from Ascend support matrix ( #43336 )
2026-10-09 16:55:39 +08:00
Mohammad Miadh Angkad and Mohammad Angkad
7578432c21
[LoRA][Test] Use the 0.9 ROUGE-L tolerance for the CUDA multi-LoRA batch test ( #43297 )
...
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai >
2026-10-09 01:51:01 -07:00
Avaya Aggarwal and Mick Qian
d677848968
[diffusion] fix: fix CFG normalization to use per-sample norm instead of batch-flattened norm ( #41706 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
2026-10-09 16:43:18 +08:00
Murthy L and Claude Opus 5
3ab142adc4
[diffusion] feat: enable breakable CUDA graphs for FLUX.2 Klein ( #36470 )
...
Co-authored-by: Claude Opus 5 <noreply@anthropic.com >
2026-10-09 16:40:09 +08:00
b922f5ca20
[diffusion] fix: share the IPC all-to-all transport between AllToAll4D and USP ( #43269 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-09 16:31:55 +08:00
Mick and Mick Qian
17b2f35ae7
[diffusion] refactor: remove unused helpers and write-only state ( #43064 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
2026-10-09 16:30:55 +08:00
James Xu and Mick Qian
ee4e2f7818
[diffusion] optimization: enable channels_last_3d for Cosmos3 VAE ( #28021 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
2026-10-09 16:29:46 +08:00
Cheng Wan
dc0b394471
[Refactor] Remove Kimi K3's own layer communication and let the consumer set an FFN's exit ( #42936 )
2026-10-09 01:27:46 -07:00
Cheng Wan
305ad5503d
[Refactor] Run Kimi K3's fused collectives through stage boundaries ( #42898 )
2026-10-09 01:27:23 -07:00
Cheng Wan
2b469040d2
[Refactor] Support attention-TP dense FFNs and unpadded batches on stage boundaries ( #42897 )
2026-10-09 01:27:00 -07:00
Cheng Wan
b9ae3d1a65
[Fix] Run attention on a padded extend batch's real rows ( #42896 )
2026-10-09 01:26:38 -07:00
Cheng Wan
7203747cc8
[Fix] Shard Kimi K3's shared experts over the TP group they sum over ( #42895 )
2026-10-09 01:26:15 -07:00
Mick and Mick Qian
45ad463c87
[diffusion] refactor: deduplicate UniPC solvers and Gemma3 shard loading ( #42899 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
2026-10-09 16:26:04 +08:00
Cheng Wan
125f88cb75
[Refactor] Run Kimi K3 through stage boundaries ( #42894 )
2026-10-09 01:25:51 -07:00
Cheng Wan
666da4f56b
[Fix] Kimi K3 KDA value head size and SiTU input alignment ( #42893 )
2026-10-09 01:25:28 -07:00
Cheng Wan
48a6c0a5d0
[Fix] Return an FFN's output to the attention rows at a pipeline handoff ( #42892 )
2026-10-09 01:25:05 -07:00
Cheng Wan
b2402ccad6
[Fix] Merge DeepSeek-V4 two-batch-overlap microbatches without a residual ( #42891 )
2026-10-09 01:24:42 -07:00
b130f8af02
[sgl-router] Serve /v1/completions through /generate, with the shared OpenAI layer ( #42997 )
...
Co-authored-by: Kan Wu <kan.wu@radixark.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-09 16:04:04 +08:00
f620d733d2
[rust-processor] Share the reasoning splitter and tool constraint across hosts ( #42996 )
...
Co-authored-by: Kan Wu <kan.wu@radixark.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Shangming Cai <csmthu@gmail.com >
2026-10-09 15:19:41 +08:00
37ae292e6f
[diffusion] fix: skip LTX-2 SP masks only for globally unpadded sequences ( #43134 )
...
Co-authored-by: yatholam <yatholam@amd.com >
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
2026-10-09 15:08:41 +08:00
Lianmin Zheng
c44a42e6d3
docs: reorder README workloads, hardware, and ecosystem ( #43307 )
2026-10-08 23:46:24 -07:00
e0608d890e
fix: use dp_size instead of attn_dp_size for msprobe rank id ( #43041 )
...
Co-authored-by: l30044004 <louyujing@huawei.com >
Co-authored-by: Claude Code <noreply@anthropic.com >
2026-10-09 09:40:43 +03:00
Gabriel Wu
217980566a
fix(fa4): isolate sheared-bias workspaces by CUDA stream ( #33000 )
2026-10-08 23:37:02 -07:00
areneau-cf and Jimmy Shong
bd2d73daa5
[model] support Clef and Clef-Flash decision models on /v1/systemone ( #42721 )
...
Co-authored-by: Jimmy Shong <69131491+Jiminator@users.noreply.github.com >
2026-10-09 14:25:52 +08:00
Michael
51f7ea84c7
[AMD] Keep MiniMax decode-score tile at 128 tokens off gfx950 ( #42772 )
2026-10-08 23:22:19 -07:00
08fc290f48
[AMD] gfx950 small-batch MoE: expert-count gate for the small sort and FP8 block-scale small-M kernel ( #41982 )
...
Co-authored-by: Cursor <cursoragent@cursor.com >
Co-authored-by: jacky.cheng <yichiche@amd.com >
2026-10-08 23:05:50 -07:00
cba1d07a94
[Memory] Share one full prefill graph output buffer across shapes for tuple and list outputs ( #43176 )
...
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com >
Co-authored-by: 842974287 <842974287@users.noreply.github.com >
2026-10-08 23:05:29 -07:00
e14ef4b658
[Scheduler] Skip the decode for a request whose last token is already in flight ( #42720 )
...
Co-authored-by: hnyls2002 <lsyincs@gmail.com >
Co-authored-by: Liangsheng Yin <hnyls2002@gmail.com >
2026-10-08 23:00:46 -07:00
Po-Han Huang (NVIDIA) and Claude Opus 5.5
1fc6b8962c
[Docs] Cookbook: AgentX launches for Qwen3.5 / Qwen3.8-Flash-Next; refresh GLM-5.2, DeepSeek-V4, DeepSeek-V4.1 ( #43083 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-09 13:55:35 +08:00
e46aaaa789
[diffusion] feat: accept strided row views in the packed QKV all-to-all ( #40588 )
...
Co-authored-by: Mick <mickjagger19@icloud.com >
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
2026-10-09 13:55:31 +08:00
Colin Z
3c1703d2ff
[AMD] Support Quark DeepSeek-V4-MXFP4 checkpoints on the DSV4 stack, enabling shared-expert fusion and MTP ( #31054 )
2026-10-08 22:54:49 -07:00
Yuan Luo and luoyuan.luo
9a1aeacc3b
[GLM-5.3-Flash] Capture the KDA extend in breakable prefill CUDA graphs ( #41105 )
...
Co-authored-by: luoyuan.luo <luoyuan.luo@antgroup.com >
2026-10-09 13:37:20 +08:00
33e7b4cc4b
[diffusion] perf: decode plain base64 MiniMax-H3 materials without a per-character loop ( #43117 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-09 13:25:23 +08:00
Chetan Kumar Verma and Claude Fable 5.1
5d8e98b7fd
[XPU][Diffusion] Plain-torch scale-shift for inductor ( #41885 )
...
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com >
2026-10-09 13:07:14 +08:00
saatwiknagpal
6e7beace14
[diffusion] fix: fix int32 offset overflow in the Ulysses packed-QKV pack kernel ( #43140 )
2026-10-09 11:51:46 +08:00
3ba90f1b52
[rust] Update Dynamo frontend crates to latest compatible releases ( #42666 )
...
Co-authored-by: Kan Wu <kan.wu@radixark.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-09 11:46:01 +08:00
9172a73725
[sgl-router] Keep charset=utf-8 on streamed responses ( #43232 )
...
Co-authored-by: Kan Wu <kan.wu@radixark.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-09 11:44:52 +08:00
Baizhou Zhang
33bffcd490
[CP] GLM-5.3-Flash Prefill CP implemented on layer boundaries ( #42044 )
2026-10-08 20:42:11 -07:00
metamergebot and joshuadeng
7859eff6ca
[srt] Inject "__req__" into custom_params only for custom logit processors ( #43181 )
...
Co-authored-by: joshuadeng <joshuadeng@users.noreply.github.com >
2026-10-08 20:40:50 -07:00
4ee3f0f370
[Session] Skip the KV free when page alignment empties a streaming-session rewind ( #43183 )
...
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com >
Co-authored-by: Joshua Deng <91448271+joshuadeng@users.noreply.github.com >
2026-10-08 20:40:25 -07:00
4e3eaba619
[HiCache] Re-admit a split node's queued buffer-mode write instead of dropping it ( #43174 )
...
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com >
Co-authored-by: xiezhq-hermann <xiezhq-hermann@users.noreply.github.com >
2026-10-08 20:39:41 -07:00
HEJIAN SANG and Claude Opus 5.5
3e13f1b2c7
[Perf] Serialize OpenAI chat/completion responses in one pass ( #42889 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-09 11:17:25 +08:00
ormandj and kpham-sgl
438df9a2a4
[Spec] Fix wrong tokens from compact-verify CUDA graphs when a token tier is captured more than once ( #42709 )
...
Co-authored-by: kpham-sgl <khoa.pham@radixark.ai >
2026-10-08 19:49:23 -07:00
Khoa Pham
8e19db19b0
[KDA] Default safe-gate Blackwell prefill to FlashInfer ( #43246 )
2026-10-08 19:46:28 -07:00
Chetan Kumar Verma and Claude Opus 5.5
fedab5537b
[XPU] Enable ring attention on XPUAttentionBackend ( #41406 )
...
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
2026-10-09 10:41:11 +08:00
Mick and Mick Qian
df1e4f3f2b
[diffusion] refactor: share SANA-WM refiner block computation ( #42980 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
2026-10-09 10:38:36 +08:00
Liangsheng Yin
f6569bce4d
[MemCache] Skip KV events and eviction config on a disabled radix cache ( #43253 )
2026-10-08 19:27:39 -07:00
Polisetty V R K Jyothendra Varma
e54f08c860
[XPU] Misc changes to support XPU ( #39817 )
...
Signed-off-by: P V R K Jyothendra Varma <polisetty.v.r.k.jyothendra.varma@intel.com >
2026-10-09 10:26:46 +08:00
ashwini rathi
51686cf8e7
[XPU] split Intel nightly into kernel-main and kernel-wheel jobs ( #41857 )
2026-10-09 09:58:39 +08:00
shodoco
86624f14eb
[Rust] Pass the media item fps hint to multimodal processors ( #43143 )
2026-10-08 17:51:06 -07:00
Liangsheng Yin
e0ab91e3ca
[MemCache] Rename SWAChunkCapPoolConfigurator to SWARequestCapPoolConfigurator ( #43202 )
2026-10-08 17:50:51 -07:00
b2786a670e
[MemCache] Count reclaimable tokens in the unified tree ledgers ( #43185 )
...
Co-authored-by: merrymercy <merrymercy@users.noreply.github.com >
Co-authored-by: xiezhq-hermann <xiezhq-hermann@users.noreply.github.com >
2026-10-08 17:48:11 -07:00
Liangsheng Yin
b4fe282836
[MemCache] Delete ChunkCache ( #43201 )
2026-10-08 17:31:16 -07:00
Liangsheng Yin
508ba79688
[MemCache] Serve disabled hybrid-SWA models with UnifiedRadixCache ( #43200 )
2026-10-08 17:30:01 -07:00
Liangsheng Yin
265efe5e01
[MemCache] Serve disabled full-attention models with UnifiedRadixCache ( #43199 )
2026-10-08 17:27:24 -07:00
Liangsheng Yin
977e2719c7
[MemCache] Serve disabled pure-SWA models with PureSWARadixCache ( #43198 )
2026-10-08 17:25:54 -07:00
Liangsheng Yin
57ce47862d
[MemCache] Report no prefix sharing from a disabled RadixCache ( #43197 )
2026-10-08 17:23:11 -07:00
Khoa Pham
fc3385d530
feat(kda): integrate FlashInfer prefill checkpoints ( #41400 )
2026-10-08 17:16:33 -07:00
Khoa Pham
1e77db3215
[Perf] Avoid gate copies in gated RMSNorm ( #43227 )
2026-10-08 17:11:58 -07:00
20d1f3f56a
[Scheduler] Refresh waiting requests' cached prefixes under FCFS-like policies with LRU eviction ( #41593 )
...
Signed-off-by: Pei Li <peili@nvidia.com >
Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com >
Co-authored-by: Po-Han Huang <pohanh@nvidia.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: hnyls2002 <lsyincs@gmail.com >
2026-10-08 17:05:04 -07:00
Ankur Singh
2b94cb4eb5
docs(cookbook): add Qwen3.8 GB300 long-context recipe ( #43159 )
...
Signed-off-by: Ankur-singh <ankusingh@nvidia.com >
2026-10-08 16:56:30 -07:00
bb43927404
[PD] Charge decode admission the page-rounded allocation when the radix cache is disabled ( #43173 )
...
Co-authored-by: hanming-lu <69857889+hanming-lu@users.noreply.github.com >
Co-authored-by: xiezhq-hermann <xiezhq-hermann@users.noreply.github.com >
2026-10-08 16:48:31 -07:00
Liangsheng Yin
2e64740442
[CI] Avoid Hugging Face Hub API calls when the local model cache is complete ( #43229 )
2026-10-08 16:27:31 -07:00
Liangsheng Yin
baee5f3b62
[CI] Drop the labeled trigger from the extra CI workflows ( #43234 )
2026-10-08 16:25:50 -07:00
Liangsheng Yin
ae4da8cd78
[MemCache] Skip SWA request-cap pool sizing when chunked prefill is off ( #43203 )
2026-10-08 16:19:35 -07:00
Liangsheng Yin
71584b62c0
[CI] Fix stale fakes in the NVFP4 MoE dispatch and CP strategy unit tests ( #43226 )
2026-10-08 15:51:39 -07:00
20bf93b6bf
[metrics] Call DPBalanceStats.create by keyword in the ratio ladder test ( #43218 )
...
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com >
Co-authored-by: hanming-lu <hanming-lu@users.noreply.github.com >
2026-10-08 15:24:38 -07:00
2abb761639
[metrics] Count per-rank DP attention pairs and their imbalance ( #43180 )
...
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com >
Co-authored-by: hanming-lu <hanming-lu@users.noreply.github.com >
2026-10-08 15:01:37 -07:00
metamergebot and Hanming Lu
0b658c12ff
[Bench][MoE] Deal the simulated round-robin experts evenly across EP ranks ( #43188 )
...
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com >
2026-10-08 15:01:16 -07:00
Siyuan Chen
b0e06359ac
[DSV4.1][PD][5/N] Decode node support DP attention ( #40177 )
2026-10-08 14:42:16 -07:00
Xuanteng Huang
d20bdc1443
Add per-token NVFP4 MoE support for ReLU2 activation ( #39199 )
...
Signed-off-by: Xuanteng Huang <xuanteng.huang@outlook.com >
2026-10-08 14:28:22 -07:00
Shunkangz and Claude Opus 5
3d8d44a4a9
[Qwen3.8 CP 3/4] Collocated prefill CP integration ( #39723 )
...
Co-authored-by: Claude Opus 5 <noreply@anthropic.com >
2026-10-08 14:20:27 -07:00
Sage
5a5ea81036
[rust-server] warm up each rust listener in-process ( #43004 )
...
Signed-off-by: Sage Ahrac <sagiahrak@gmail.com >
2026-10-08 14:09:46 -07:00
e902207a4e
[Speculative] Support block verification for the DFlash family ( #43012 )
...
Co-authored-by: maocheng <mao.cheng@MacBook-Pro-MD663G60VV.local >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-08 13:59:52 -07:00
d1ba943016
[Model] Add support for EmbeddingGemma 2 ( #42782 )
...
Signed-off-by: Luciano Martins <lucianommartins@users.noreply.github.com >
Co-authored-by: Luciano Martins <lucianommartins@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-08 13:54:45 -07:00
Baizhou Zhang
b193ef382a
[CI] Clean some unnecessary DSV4 tests ( #43195 )
2026-10-08 13:53:41 -07:00
Mohammad Miadh Angkad and Mohammad Angkad
d8448d2e62
[CI] Re-enable GB300 ( #43058 )
...
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai >
2026-10-08 13:23:26 -07:00
metamergebot and hanming-lu
93f2acf614
[Metrics] Use widely supported bucket boundaries for the DP attention imbalance ratio ( #43187 )
...
Co-authored-by: hanming-lu <hanming-lu@users.noreply.github.com >
2026-10-08 12:58:58 -07:00
Ming Yang
3fef129ff8
[PD] Allow state-only KV checksum inputs ( #42904 )
2026-10-08 12:54:03 -07:00
075fd8fa0b
[Mamba] Describe the track snapshot dtype by its kernel instead of fp32 ( #43192 )
...
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com >
Co-authored-by: jiayisuse <jiayisuse@users.noreply.github.com >
2026-10-08 12:47:46 -07:00
e3c5719713
[DFlash] Build the attention QKV projection through an overridable hook ( #43179 )
...
Co-authored-by: Hanming Lu <69857889+hanming-lu@users.noreply.github.com >
Co-authored-by: 842974287 <842974287@users.noreply.github.com >
2026-10-08 12:46:28 -07:00
kunchengit
00b3a48753
[dLLM] Add JointThresholdInDel algorithm for insertion & deletion decoding ( #31773 )
2026-10-08 12:44:14 -07:00
Aurick Qiao and Qiaolin-Yu
b2cb24995d
[Debug] Add opt-in NaN detection before sampling ( #42690 )
...
Co-authored-by: Qiaolin-Yu <liin1211@outlook.com >
2026-10-08 12:20:57 -07:00
Shenxiu Liu and Qiaolin-Yu
393eaed509
[Fix] overlap scheduler: record_stream mix_running_indices for the forward-stream relay gather ( #34076 )
...
Co-authored-by: Qiaolin-Yu <liin1211@outlook.com >
2026-10-08 12:19:35 -07:00
shodoco
1638c5d699
[http_server] Let custom /generate routes skip the second contract check ( #43114 )
2026-10-08 11:56:06 -07:00
214347891a
[Bugfix] Skip post-experts EP all-reduce on the FlashInfer cutlass FP4 all-gather MoE path ( #41072 )
...
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com >
2026-10-08 18:01:27 +00:00
cbf49695c3
[CI] Add dedicated sglang-processor workflow ( #43022 )
...
Co-authored-by: Kan Wu <kan.wu@radixark.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-09 01:57:05 +08:00
shodoco
e2fbc6a550
Stream a chunk once stream_interval tokens are unsent ( #42643 )
2026-10-08 10:12:37 -07:00
Eric Zhang
344b8a863b
[Perf] Fuse short-convolution checkpoint tracking metadata ( #43013 )
2026-10-09 00:26:15 +08:00
Ke Bao
642e81fbb8
Fix DSA index host buffer cleanup ( #43100 )
2026-10-09 00:21:38 +08:00
Ke Bao
46ad9eb439
Fix K-only host buffer cleanup ( #43101 )
2026-10-09 00:21:27 +08:00
billishyahao
4ab4d90d78
[AMD][DCP 3/N] add aiter asm for kimi k3 target verify ( #34537 )
2026-10-08 09:13:03 -07:00
Mick and Mick Qian
3030e31ff8
[diffusion] refactor: deduplicate request extraction and ComfyUI test helpers ( #42903 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
2026-10-08 23:44:35 +08:00
黄孝君
9f5d871251
[NPU] Quote the arm64 Triton-ascend wheel URL in npu.Dockerfile ( #43035 )
2026-10-08 21:44:49 +08:00
943621c336
[sgl-router] Render DeepSeek-V4 through sglang-processor ( #42665 )
...
Co-authored-by: Kan Wu <kan.wu@radixark.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Shangming Cai <csmthu@gmail.com >
2026-10-08 20:16:20 +08:00
11b1fb21de
[MiniMax-M3] Decode: paged K/V tile loads and a tiny GEMM for the router projection ( #38841 )
...
Co-authored-by: Hao Phan <htphan@nvidia.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-08 19:39:31 +08:00
6e6ccb26f2
[MiniMax-M3] Instantiate the decode block top-k in register buckets ( #38615 )
...
Co-authored-by: Hao Phan <htphan@nvidia.com >
Co-authored-by: Claude Opus 5 <noreply@anthropic.com >
Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com >
2026-10-08 19:33:17 +08:00
45abea0269
[rust-processor] DeepSeek-V4 parity with SGLang, plus a reusable parity harness and skill ( #42664 )
...
Co-authored-by: Kan Wu <kan.wu@radixark.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-08 18:35:30 +08:00
Shangming Cai
9578cb1c63
[CI] Don't run the full GPU suite for renderer-only Cargo.lock changes ( #43025 )
2026-10-08 18:25:06 +08:00
Mick and Mick Qian
7dd512b0e8
[diffusion] refactor: deduplicate FLUX positions and RoPE application ( #43002 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
2026-10-08 18:18:39 +08:00
Horace He
8ad24fb560
[Perf] Cache compiled ModelOpt exclusion patterns ( #42927 )
2026-10-08 17:10:25 +08:00
Liangsheng Yin
4b384df0c9
[CI] Keep the KL tests' LongBench cache outside the checkout and bound its download ( #43068 )
2026-10-08 01:17:31 -07:00
Liangsheng Yin
7b9c42cf49
[CI] Remove the unregistered chunked-prefill tests under test/manual ( #43081 )
2026-10-08 01:16:39 -07:00
Liangsheng Yin
89b5935793
[MemCache] Return only the loaded length from init_load_back; drop empty_device_indices ( #43023 )
2026-10-08 01:14:33 -07:00
Liangsheng Yin
5a4d01a3ae
[MemCache] Return only the matched length from match_prefix; read KV indices off the node path ( #42923 )
2026-10-08 01:10:14 -07:00
Spandan Tiwari
2d26823f94
[AMD] Drop static input_scale on aiter per-token FP8 path (use dynamic per-token) ( #42152 )
2026-10-08 01:06:34 -07:00
Cheng Wan
b3e6c87be3
[Refactor] Translate each KV loc once per iteration ( #42753 )
2026-10-08 00:58:28 -07:00
508ddc859a
[unified-memory] Admit DFLASH and DSPARK with fused draft KV ( #40492 )
...
Co-authored-by: Caihua Li <caihua.li@bytedance.com >
Co-authored-by: Cheng Wan <cheng.wan@radixark.ai >
2026-10-08 00:56:21 -07:00
c2d491e38b
[unified-memory] Admit EAGLE/EAGLE3 onto fused draft KV, on mamba-MHA and MLA hosts ( #40491 )
...
Co-authored-by: Caihua Li <caihua.li@bytedance.com >
Co-authored-by: Cheng Wan <cheng.wan@radixark.ai >
2026-10-08 00:55:54 -07:00
Michael
5afe4d6334
[AMD] Fix DSpark draft bucket tests for the raw in-graph metadata default ( #42230 )
2026-10-08 00:54:26 -07:00
8b80808366
[unified-memory] Fuse the draft model's KV into the target's page envelope (mechanism) ( #37628 )
...
Co-authored-by: Caihua Li <caihua.li@bytedance.com >
Co-authored-by: Cheng Wan <cheng.wan@radixark.ai >
2026-10-08 00:54:19 -07:00
08d30f2300
[unified-memory] Speculative decoding on the unified memory pool: translate every read and write path ( #37627 )
...
Co-authored-by: Caihua Li <caihua.li@bytedance.com >
Co-authored-by: Cheng Wan <cheng.wan@radixark.ai >
2026-10-08 00:52:21 -07:00
33883a5fa3
[AMD] Small-M W8A8 FP8 projection GEMM for Qwen3.5 AttnFP8 on gfx950 ( #41134 )
...
Co-authored-by: Cursor <cursoragent@cursor.com >
Co-authored-by: jacky.cheng <yichiche@amd.com >
2026-10-08 00:50:16 -07:00
b2cd65b8b0
[AMD] One-launch small-M MoE router for Qwen3.5 on gfx950 ( #41133 )
...
Co-authored-by: Cursor <cursoragent@cursor.com >
Co-authored-by: jacky.cheng <yichiche@amd.com >
2026-10-08 00:47:52 -07:00
f795fe352d
[Test] Add a docs drift check for server_arguments.mdx ( #38520 )
...
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com >
Co-authored-by: Cheng Wan <54331508+ch-wan@users.noreply.github.com >
Co-authored-by: Cheng Wan <cheng.wan@radixark.ai >
2026-10-08 00:37:22 -07:00
Chunan Zeng
785005d21b
[DSpark] Mask uncommitted slots before commit-inject cache lookups ( #42624 )
2026-10-08 00:16:54 -07:00
Yuwei An
54b551df8e
[Refactor][TCPCG] Bind live forward batches in BCG eager calls (2/9) ( #42286 )
2026-10-07 23:55:41 -07:00
Mick and Mick Qian
5c5f593110
[diffusion] refactor: share KL VAE encode/decode and tiling operations ( #42841 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
2026-10-08 14:43:19 +08:00
Mick and Mick Qian
d1b751d958
[diffusion] refactor: inherit existing defaults instead of repeating overrides ( #42963 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
2026-10-08 14:42:21 +08:00
74308cf1d6
[diffusion] fix: wait for multimodal process teardown ( #40765 )
...
Signed-off-by: Xucheng Zhou <aden1350@outlook.com >
Co-authored-by: Mick <mickjagger19@icloud.com >
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
2026-10-08 14:36:17 +08:00
4f320fb85a
[diffusion] perf: FastH3 8-Step V2 native support, native Blackwell VSA and batched H3 VAE decode ( #37662 )
...
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com >
Co-authored-by: mickqian <mickqian@users.noreply.github.com >
Co-authored-by: Mick <mickjagger19@icloud.com >
2026-10-08 14:35:27 +08:00
Mick and Mick Qian
57e9ff5403
[diffusion] refactor: share timestep retrieval across pipeline stages ( #42967 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
2026-10-08 14:33:39 +08:00
Liangsheng Yin
0f5bc4af4b
[CI] Fix stale fakes in the NPU CUDA-graph backend unit test ( #43056 )
2026-10-07 23:04:48 -07:00
Mohammad Miadh Angkad
ac656ee79a
[CI] Re-enable B300 ( #42959 )
2026-10-07 22:03:16 -07:00
Jzz1943
b7b2975b57
fix(dsa): align FA3 metadata with unpadded queries ( #42793 )
2026-10-07 21:37:21 -07:00
hmalgewatta and Alex Nails
7327fb225f
[DSV4] Guard V4.1 indexer SM100 check on non-CUDA devices ( #42114 )
...
Co-authored-by: Alex Nails <alex.nails@radixark.ai >
2026-10-08 12:06:28 +08:00
Thomas Wang
b72da1b817
[AMD] Update v4 amd cookbook 1008 ( #43033 )
2026-10-08 11:18:41 +08:00
AndyLi429 and AndyLi429
89bea053ac
[NPU] Add NPU cudagraph runner unit tests ( #40404 )
...
Co-authored-by: AndyLi429 <AndyLi429@noreply.gitcode.com >
2026-10-08 11:18:38 +08:00
chilltongx and chilltongx
e7530a3a26
[diffusion] fix: fix forward multimodal token type IDs in image encoding ( #34494 )
...
Co-authored-by: chilltongx <284668524+chilltongx@users.noreply.github.com >
2026-10-08 10:55:55 +08:00
WenhaoZhang
b0d7254fbc
[diffusion] CI: rebase LTX H100 timing references on recent CI runs ( #42986 )
2026-10-08 10:42:12 +08:00
Jiajun Li and Claude Opus 5.5
99463390f8
[Fix] MiMo-V2: accept checkpoints with the split attention layout ( #40978 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-07 19:18:32 -07:00
c892301ff7
[Platform] Thread vendor KV pool classes through the in-tree builders; add PlatformCapabilities ( #35594 )
...
Co-authored-by: Claude Fable 5 <noreply@anthropic.com >
Co-authored-by: Yihao Wang <42559837+AgainstEntropy@users.noreply.github.com >
2026-10-07 19:16:10 -07:00
Po-Han Huang (NVIDIA) and Claude Opus 5.5
7b6b6ee25e
[Docs] Cookbook: interactive SemiAnalysis InferenceX AgentX launches for GLM-5.2, DeepSeek-V4 and DeepSeek-V4.1 ( #42726 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-08 09:33:40 +08:00
Mick and Mick Qian
21fa2251a0
[diffusion] refactor: share sigma schedules and causal cross-attention ( #42848 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
2026-10-08 09:29:33 +08:00
Mick and Mick Qian
28c08b02c3
[diffusion] refactor: deduplicate MOVA initialization ( #42983 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
2026-10-08 09:28:06 +08:00
Mick and Mick Qian
243f3dc35f
[diffusion] refactor: deduplicate Qwen-VL inputs and component accuracy suites ( #42860 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
2026-10-08 09:27:22 +08:00
Mick and Mick Qian
2489bd1d2e
[diffusion] refactor: share ModelOpt checkpoint conversion I/O ( #42856 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
2026-10-08 09:26:39 +08:00
Mick and Mick Qian
35608e48d1
[diffusion] refactor: reuse the single-process parallel test fixture ( #42964 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
2026-10-08 09:24:53 +08:00
ashwini rathi
b91919ffe2
[XPU][diffusion] Re-baseline Wan2.1-T2V-1.3B TextEncodingStage on B60 ( #42794 )
2026-10-08 09:22:41 +08:00
Mick and Mick Qian
287b8b793f
[diffusion] refactor: deduplicate VAE resampling and LTX tiling ( #42866 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
2026-10-08 09:22:11 +08:00
Mick and Mick Qian
0c00bffc6f
[diffusion] refactor: deduplicate Qwen Image RoPE and SenseNova backbones ( #42843 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
2026-10-08 09:20:55 +08:00
Mick and Mick Qian
1e2f524726
[diffusion] refactor: share stacked encoder weight loading ( #42988 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
2026-10-08 09:16:03 +08:00
Liangsheng Yin
0427ea8032
[CI] Split the shared Cargo target dir per glibc ( #43024 )
2026-10-07 18:14:26 -07:00
Mick and Mick Qian
d52ea5ee90
[diffusion] refactor: deduplicate realtime transport and LingBot tests ( #42990 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
2026-10-08 09:10:10 +08:00
Richard Wang
5cb7fb886c
[Feature] support pplx-decider-v1.1 and update the pplx-decider cookbooks ( #42645 )
2026-10-08 09:03:50 +08:00
Mick and Claude Opus 5.5
9101e895ea
[diffusion] feat: encode the MP4 while the VAE decodes (MiniMax-H3, Wan) ( #41819 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-08 09:01:19 +08:00
Cheng Wan
7760d5264a
[Refactor] Use destination layouts for model checkpoint mappings ( #42590 )
2026-10-07 16:24:22 -07:00
Cheng Wan
5e12831cec
[Refactor] Load auxiliary and expert checkpoints using owner partitions ( #42589 )
2026-10-07 16:24:03 -07:00
Cheng Wan
64c7f65eac
[Refactor] Retain process groups in parallel linear layers ( #42588 )
2026-10-07 16:23:40 -07:00
Cheng Wan
267e34516c
[Refactor] Select parallel groups for remaining model projections ( #42587 )
2026-10-07 16:22:57 -07:00
Cheng Wan
dec5d9d6fc
[Refactor] Let vision encoders select their parallel group ( #42586 )
2026-10-07 16:22:32 -07:00
Cheng Wan
b83827ea02
[Fix] Quantize reloaded FP8 weights after native sharding ( #42585 )
2026-10-07 16:20:47 -07:00
Cheng Wan
59740a728f
[Refactor] Let MLA projections select their parallel group ( #42484 )
2026-10-07 16:19:33 -07:00
Cheng Wan
961d7290fe
[Refactor] Let model MLPs select their parallel group ( #42468 )
2026-10-07 16:18:14 -07:00
Cheng Wan
4c5130ac5a
[Refactor] Let linear layers select their parallel group ( #42426 )
2026-10-07 16:15:26 -07:00
Liangsheng Yin
1066decce6
[Scheduler] Replace extend_range with extend_end derived from prefix_len ( #42822 )
2026-10-07 16:07:16 -07:00
Liangsheng Yin
5349d3e9d2
[Scheduler] Derive prefix KV indices from last_node at allocation; drop Req.prefix_indices ( #42825 )
2026-10-07 16:03:09 -07:00
Liangsheng Yin
e9240b3243
[Scheduler] Track prefill progress as prefix_len ( #42824 )
2026-10-07 15:59:06 -07:00
Kevin Mi and Claude Opus 5.5
79ef49cb6b
[Docs] GLM-5.3-Flash: measured MI355X MXFP4 agentic recipe ( #43015 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-07 15:49:49 -07:00
Liangsheng Yin
4bf8f28d15
[Scheduler] Fold match write-back into match_kv_cache ( #42823 )
2026-10-07 15:43:05 -07:00
Liangsheng Yin
b1ffe5c16a
[dLLM] Keep the request row across FDFO blocks ( #42846 )
2026-10-07 15:39:33 -07:00
Ping Qiu and Zhiqiang Xie
685d4d453d
[HiCache] Add SeaweedFS L3 storage backend ( #42399 )
...
Co-authored-by: Zhiqiang Xie <xiezhq@stanford.edu >
2026-10-07 15:05:37 -07:00
WenhaoZhang
0b635266d4
[diffusion] CI: warm up CI cases at the request's quality level ( #42958 )
2026-10-07 23:26:36 +08:00
00b21d6415
[diffusion] fix: reset DiT cache state at the start of each request ( #42437 )
...
Co-authored-by: Mick <mickjagger19@icloud.com >
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
2026-10-07 23:25:49 +08:00
Mohammad Miadh Angkad and Mohammad Angkad
44d10eab5e
[GLM-V] Shard the vision tower over the attention-TP group under DP attention ( #42931 )
...
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai >
2026-10-07 23:17:06 +08:00
d9a0f1cb34
[diffusion] fix: fix hybrid SP+TP for I2V models ( #41917 )
...
Co-authored-by: Mick <mickjagger19@icloud.com >
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
2026-10-07 22:33:03 +08:00
Po-Han Huang (NVIDIA) and Claude Opus 5.5
1e86143bf0
[Bugfix] Return None for empty reasoning_content outside K2 Horizon ( #42962 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-07 21:02:40 +08:00
WenhaoZhang
492dbd0807
[CI] keep diffusion-only kernels and tests out of the srt stages ( #42706 )
2026-10-07 20:28:53 +08:00
Tianyang Gu and Claude Fable 5.1
aa5551d9b6
[diffusion] fix: do not interleave comfy-kitchen MXFP8 scales twice ( #38028 )
...
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com >
2026-10-07 17:56:39 +08:00
YC Yen-Ching Tseng
c389cbaf99
[AMD][diffusion] Route bf16 fuse_scale_shift to the native fallback on gfx1250 ( #42859 )
2026-10-07 02:37:43 -07:00
YC Yen-Ching Tseng
49d7d60dfa
[AMD][diffusion] VAE attention to the math SDPA backend on gfx1250 ( #42858 )
2026-10-07 02:32:02 -07:00
Yihao Wang and Claude Opus 5.5
11972e5207
[deps] bump transformers to 5.19.0 ( #42784 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-07 17:04:08 +08:00
Rain Jiang
c52dcc5a01
The sgalng api specs ( #41585 )
2026-10-07 01:55:01 -07:00
Po-Han Huang (NVIDIA) and Claude Opus 5.5
ba77d30256
[FA4] Keep stream last in the SM100 MLA kernel so disk-cached objects load ( #42926 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-07 15:58:55 +08:00
Khoa Pham
e23093077e
[DSv4] Fix shared index cache accessor page size ( #42832 )
2026-10-07 00:53:17 -07:00
jiaryang
1c42ad3679
[ROCm] Skip the tilelang act_quant in the DSA indexer on gfx1250 ( #42747 )
2026-10-07 00:16:46 -07:00
Shuwen Wang and Claude Opus 5
2889158792
[MemCache] test: require every ComponentData field to be classified by each split hook ( #42518 )
...
Co-authored-by: Claude Opus 5 <noreply@anthropic.com >
2026-10-07 14:53:34 +08:00
paulzhang-tm
c721602ad2
Cover SWA admission after declined HiCache restores ( #39839 )
2026-10-07 14:51:45 +08:00
Hyogeun Oh (오효근) and mickqian
1d019ce1c8
[diffusion] fix: honor the lazy transformer loader contract ( #15215 )
...
Signed-off-by: Zerohertz <ohg3417@gmail.com >
Co-authored-by: mickqian <mickqian@users.noreply.github.com >
2026-10-07 14:23:00 +08:00
Mark
b0dff4009f
[diffusion] cli: restore gated Diffusers model detection ( #34903 )
2026-10-07 14:21:37 +08:00
Mohammad Miadh Angkad and Mohammad Angkad
f47d89564d
[KDA] Default GLM-5-Next spec verify to the CuTe DSL KDA kernel ( #42905 )
...
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai >
2026-10-06 22:31:41 -07:00
AMD-yanfeiwang and Thomas Wang
9ddbba50dd
[AMD][DSV4] feat: enable PCP with fp8 unified_kv on gfx950 ( #39923 )
...
Co-authored-by: Thomas Wang <thomawan@amd.com >
2026-10-06 22:09:18 -07:00
Seas0 and Mick Qian
43b4abd857
[diffusion] fix: validate SageAttention 2 head sizes without overriding component policy ( #41529 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
2026-10-07 12:39:54 +08:00
Xinyi Song
168458fd90
[AMD][V4.1] Fix wo_b input flatten for packed fp8 activations on gfx950 ( #42796 )
2026-10-06 21:35:51 -07:00
Khoa Pham
9e26600f3a
[DSA] Clarify pooled-page naming and ownership ( #42671 )
2026-10-06 21:05:29 -07:00
CuzMi
bde01f8dc8
[Diffusion] Fuse Anima AdaLN modulation and cache the FFI callable ( #42608 )
2026-10-07 12:01:59 +08:00
Tuan Nguyen Gia and Cursor
c0df52be2d
[AMD] Fix Quark W4A4 MXFP4 MoE clamped-SwiGLU on the AITER path ( #34540 )
...
Co-authored-by: Cursor <cursoragent@cursor.com >
2026-10-07 03:20:40 +00:00
Yi (Vincent) Zhong and Claude Opus 5.5
012cd3858b
[Fix] Make the fused KDA beta sigmoid bit-identical to torch.sigmoid ( #42611 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-06 20:04:59 -07:00
Thomas Wang
9a21d82958
[AMD] Update v4.1 amd cookbook 1006 ( #42847 )
2026-10-06 19:58:58 -07:00
Jay Zenith
35f18848db
[diffusion] fix: validate output_quality and fix return type annotation ( #19530 )
2026-10-07 10:53:29 +08:00
Yash Akhauri and BBuf
9c8fc6a8e4
[Bugfix] Preserve empty K2 Horizon reasoning in chat responses ( #42623 )
...
Co-authored-by: BBuf <1182563586@qq.com >
2026-10-07 10:44:55 +08:00
d7190b0122
[Perf] Accelerate DiffusionGemma with FA4 and CUDA graphs on Blackwell ( #42239 )
...
Co-authored-by: lianghougong <lianghouhong@xiaohongshu.com >
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com >
2026-10-07 10:31:00 +08:00
2f6e0ec45d
[DSv4.1] Trim prefill model-entry overhead ( #41704 )
...
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
Co-authored-by: BBuf <1182563586@qq.com >
2026-10-07 09:50:34 +08:00
7a4d6dfd2f
[diffusion] feat: name the quality tiers after what they guarantee ( #42370 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
Co-authored-by: Claude Opus 5 <noreply@anthropic.com >
2026-10-07 09:49:30 +08:00
Lianmin Zheng
8f1790f595
Add Rust code owners and grant shodoco CI permissions ( #42826 )
2026-10-06 18:37:54 -07:00
660b17e8dc
[AMD] Let AITER unified verify run without host seq_lens ( #42635 )
...
Co-authored-by: merrymercy <merrymercy@users.noreply.github.com >
Co-authored-by: zx3xyy <zx3xyy@users.noreply.github.com >
2026-10-06 18:27:27 -07:00
Eddie A. Tsai
e13933c4fd
[diffusion] docs: add gfx1151 Strix Halo serving guidance ( #41670 )
2026-10-07 09:14:07 +08:00
Michael
3e914e69d5
[AMD] ci: nightly-test the miles ROCm 10 MI30X image on MI300 ( #42771 )
2026-10-06 18:05:09 -07:00
WenhaoZhang
6da08fd7cf
[diffusion] CI: fix qwen cache-dit step 10/49 perf baselines ( #42495 )
2026-10-07 08:51:12 +08:00
Mick and Mick Qian
1351f32bb4
[diffusion] refactor: simplify Kandinsky 6 models, SR sampling and tests ( #42768 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
2026-10-07 08:48:16 +08:00
f1065423d3
[PD] Stop reserving unallocated mamba spec-verify scratch in prefill KV sizing ( #42649 )
...
Co-authored-by: merrymercy <merrymercy@users.noreply.github.com >
Co-authored-by: 842974287 <842974287@users.noreply.github.com >
2026-10-06 17:16:22 -07:00
Baizhou Zhang
f497b626b6
build: use one DeepEP ref for implementation and packaging ( #42827 )
2026-10-06 17:04:28 -07:00
Duyi-Wang
66dd4e15b9
[AMD][Docs] Add MI355X DeepSeek-V4 Pro PD recipes ( #42757 )
2026-10-06 16:58:40 -07:00
08aad0c196
[HiCache] Manage buffer-mode backups per storage pool ( #42633 )
...
Co-authored-by: merrymercy <merrymercy@users.noreply.github.com >
Co-authored-by: xiezhq-hermann <xiezhq-hermann@users.noreply.github.com >
2026-10-06 16:45:05 -07:00
jain-ria and ishandhanani
fd62ee84f7
feat(grpc): expose follower metadata and node-local KV sources ( #39659 )
...
Signed-off-by: jain-ria <riajain@NVIDIA.com >
Co-authored-by: ishandhanani <82981111+ishandhanani@users.noreply.github.com >
2026-10-06 16:39:01 -07:00
Chunan Zeng and Cheng Wan
90aa1cb0fb
[Scheduler] Gather prefill-delayer queue timeout across ranks ( #42625 )
...
Co-authored-by: Cheng Wan <54331508+ch-wan@users.noreply.github.com >
2026-10-06 16:15:34 -07:00
Jan Bernlöhr and Po-Han Huang
0a0fe001f1
fix: route Thor SM110 FP8 and ModelOpt NVFP4 auto backends ( #41765 )
...
Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com >
2026-10-06 16:02:45 -07:00
6b737fd4c6
Bind the MessageQueue remote socket atomically ( #42695 )
...
Co-authored-by: merrymercy <merrymercy@users.noreply.github.com >
Co-authored-by: charlotte12l <charlotte12l@users.noreply.github.com >
2026-10-06 15:50:59 -07:00
Trevor Morris
44a2558d40
[NVIDIA] Fix nightly-cu134 build and update ( #42661 )
2026-10-06 14:52:42 -07:00
264d1c2015
Back the DFlash-family draft KV pool to the post-capture token count ( #42651 )
...
Co-authored-by: merrymercy <merrymercy@users.noreply.github.com >
Co-authored-by: 842974287 <842974287@users.noreply.github.com >
2026-10-06 14:23:18 -07:00
4506927635
[Metrics] Add per-rank DP attention imbalance metrics ( #42755 )
...
Co-authored-by: merrymercy <merrymercy@users.noreply.github.com >
Co-authored-by: hanming-lu <hanming-lu@users.noreply.github.com >
2026-10-06 14:20:41 -07:00
Mohammad Miadh Angkad and Mohammad Angkad
38bdf25f8d
[Docker] Use uv-managed Python 3.12 and uv for all installs in the CUDA image ( #42612 )
...
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai >
2026-10-06 14:10:09 -07:00
Po-Han Huang (NVIDIA) and Claude Opus 5.5
3eec178e48
[Spec][MegaMoE] Let the speculative draft choose its own W4A4 MXFP4 MegaMoE MMA type ( #42022 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-06 14:09:57 -07:00
Liangsheng Yin
12004043b8
[Metrics] Read scheduled prefill KV tokens from batch prefix_lens ( #42797 )
2026-10-06 13:11:29 -07:00
a34cba3c62
[HiCache] Resolve write_back to write_through under buffer_only host memory ( #42754 )
...
Co-authored-by: merrymercy <merrymercy@users.noreply.github.com >
Co-authored-by: xiezhq-hermann <xiezhq-hermann@users.noreply.github.com >
2026-10-06 11:02:23 -07:00
0f57a39152
[rust-processor] Gate render, tokenizer and parser behind cargo features ( #42663 )
...
Co-authored-by: Kan Wu <kan.wu@radixark.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-06 10:58:38 -07:00
Byron Hsu
abed685de1
[PD] Fix crash for allocation when mamba + pd + decode radix cache ( #41927 )
2026-10-06 09:36:31 -07:00
Michael and Claude
209e190ee5
[AMD] ci: build the miles ROCm 10 MI30X image nightly ( #41605 )
...
Co-authored-by: Claude <noreply@anthropic.com >
2026-10-07 00:00:11 +08:00
Emil Bogomolov and Emil Bogomolov
e17935820a
[diffusion] fix: fix width/height silently dropped on multipart /v1/videos requests ( #35108 )
...
Co-authored-by: Emil Bogomolov <zetyquickly@googlemail.com >
2026-10-06 23:31:31 +08:00
WenhaoZhang and Mick
2579875b5d
[diffusion] feat: support loading community ComfyUI-GGUF Qwen-Image-2.1 transformers ( #42556 )
...
Co-authored-by: Mick <mickjagger19@icloud.com >
2026-10-06 23:30:48 +08:00
49bafb58cd
perf(mla): use native SATFINITE FP8 conversion in MLA KV/Q preparation ( #42659 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com >
2026-10-06 22:47:47 +08:00
Xiaoshuai Zhang and jet
679d2851ba
fix(mimo-vl): keep multimodal features when capturing aux hidden states and correct vision preprocessing ( #40888 )
...
Co-authored-by: jet <dev@jetd.one >
2026-10-06 22:15:14 +08:00
16a23a672b
[diffusion] model: support Kandinsky 6 ( #42743 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
Co-authored-by: Lev Novitskiy <57654885+leffff@users.noreply.github.com >
Co-authored-by: Kirill <kykozlov@gmail.com >
Co-authored-by: Artem Borisov <divotionchannel@gmail.com >
2026-10-06 22:13:29 +08:00
730f1f3e5b
[DeepSeek-V4.1] Unify mHC into one state machine and drop medium-batch fusion variants ( #42245 )
...
Co-authored-by: Claude Fable 5 <noreply@anthropic.com >
Co-authored-by: BBuf <1182563586@qq.com >
2026-10-06 21:01:42 +08:00
WenhaoZhang
2a5f45d3d4
[diffusion] fix: skip zero blocks when hashing large conditioning inputs ( #42748 )
2026-10-06 20:43:08 +08:00
WMC and jacky.cheng
ba94488b42
[diffusion] feat: default Qwen-Image 2.1 to VAE tiling on gfx1151 ( #41655 )
...
Co-authored-by: jacky.cheng <yichiche@amd.com >
2026-10-06 20:20:58 +08:00
Shangming Cai
1291f3175d
[rust-processor] Fix chat-template precedence wording in the processor README ( #42756 )
2026-10-06 19:10:26 +08:00
Shangming Cai and Claude Fable 5.1
28525b43c4
[sgl-router] Document and reject a zero request timeout ( #42751 )
...
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com >
2026-10-06 19:07:37 +08:00
cf19ae6d04
[rust-processor] Split sglang-processor into tokenizer, render and parser ( #42662 )
...
Co-authored-by: Kan Wu <kan.wu@radixark.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-06 18:59:02 +08:00
Mohammad Miadh Angkad and Mohammad Angkad
45db430530
[DSA] Fix pooled page table refresh for target verify CUDA graph replay ( #42750 )
...
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai >
2026-10-06 18:57:52 +08:00
2167cf90c3
[sgl-router] Bound streaming time-to-headers by --request-timeout-secs (retry 5/5) ( #42632 )
...
Co-authored-by: Kan Wu <kan.wu@MacBook-Pro.local >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-06 18:35:11 +08:00
YC Yen-Ching Tseng
f3b5a28f43
[AMD] M3-mxfp8 Nightly Test Update ( #42605 )
2026-10-06 02:22:53 -07:00
YC Yen-Ching Tseng
542addad32
[AMD] M3-MXFP4 Nightly Test ( #42602 )
2026-10-06 02:22:01 -07:00
662879e495
[sgl-router] Back off between retry attempts (retry 4/5) ( #42631 )
...
Co-authored-by: Kan Wu <kan.wu@MacBook-Pro.local >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Shangming Cai <csmthu@gmail.com >
2026-10-06 17:06:45 +08:00
Gawr Gura 0721
b563bbd597
[Fix] Don't let a failed deep_ep import-time check kill servers that never use DeepEP ( #40671 )
2026-10-06 01:58:02 -07:00
e49e971131
[mem_cache] Keep prefix_indices current for chunked requests kept out of the tree ( #42467 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: hnyls2002 <lsyincs@gmail.com >
Co-authored-by: Liangsheng Yin <hnyls2002@gmail.com >
2026-10-06 01:40:35 -07:00
bb4d0a13c1
[sgl-router] Retry failed dispatches on another worker (retry 3/5) ( #42630 )
...
Co-authored-by: Kan Wu <kan.wu@MacBook-Pro.local >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Kan Wu <kan.wu@radixark.ai >
2026-10-06 16:28:52 +08:00
Jzz1943
1be3013118
[diffusion] feat: support Ideogram TurboTime LoRA ( #30487 )
2026-10-06 16:15:21 +08:00
Max Kong
01345197af
[diffusion] chore: harden OpenAI upload filename handling ( #17270 )
...
=
2026-10-06 16:14:39 +08:00
Jinbao Chen and Mick Qian
d0494af08b
[diffusion] security: pin multimodal-gen ZMQ scheduler ingress to loopback on wildcard --host ( #36854 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
2026-10-06 16:05:17 +08:00
Liangsheng Yin
c7f3ae1dd5
[mem_cache] Hold a request's tree lock as one TreeLock ( #42686 )
2026-10-06 00:59:38 -07:00
05340e9ecb
[sgl-router] Make one request dispatchable more than once (retry 2/5) ( #42629 )
...
Co-authored-by: Kan Wu <kan.wu@MacBook-Pro.local >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-06 15:24:02 +08:00
c683190b40
[PD] Wait for prefill completion before Mooncake early KV transfer ( #41395 )
...
Co-authored-by: Shangming Cai <csmthu@gmail.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-10-06 15:17:12 +08:00
8c273d6813
[sgl-router] Let selection skip workers a request already failed on (retry 1/5) ( #42628 )
...
Co-authored-by: Kan Wu <kan.wu@MacBook-Pro.local >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-06 15:10:51 +08:00
Rumit Desai
33ce0943e9
[Bugfix] Guard merge_state sgl_kernel import on non-CUDA platforms ( #42682 )
2026-10-06 14:19:20 +08:00
Joe
40fb5935c0
Allow platforms to declare Mamba extra-buffer support ( #36410 )
2026-10-06 14:18:49 +08:00
Liangsheng Yin
9a09b48da7
[Fix] Add maybe_hand_to_session to the one-batch benchmark TreeCacheNamespace ( #42723 )
2026-10-05 23:09:41 -07:00
Mohammad Miadh Angkad and Mohammad Angkad
60878c3fba
[CI] Temporarily disable B300 ( #42704 )
...
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai >
2026-10-05 22:27:54 -07:00
b524de2de6
[diffusion] optimization: release the allocator cache when the server idles not per request ( #41718 )
...
Co-authored-by: Yihao Wang <42559837+AgainstEntropy@users.noreply.github.com >
Co-authored-by: Claude Fable 5 <noreply@anthropic.com >
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
2026-10-06 12:27:08 +08:00
e446698cdd
[SM120&90] Add CUDA fused Triton sparse-MLA prefill backend for DSA ( #32779 )
...
Co-authored-by: Claude Fable 5 <noreply@anthropic.com >
Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com >
Co-authored-by: Po-Han Huang <pohanh@nvidia.com >
2026-10-05 21:08:05 -07:00
1e1bdeedbb
[Perf] Improve GLM-5.3-Flash DFlash decoding on B300 ( #42489 )
...
Co-authored-by: lianghougong <lianghouhong@xiaohongshu.com >
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com >
2026-10-06 12:05:37 +08:00
Mohammad Miadh Angkad and Mohammad Angkad
fcbbefa82a
[GLM-5.3-Flash] Autotune MoE kernels at prefill sizes ( #42693 )
...
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai >
2026-10-05 21:04:54 -07:00
Yuwei An
620c76f25d
[DSV4.1] Add opt-in TRT-LLM sparse attention support ( #41603 )
2026-10-06 12:02:10 +08:00
Zhang, Jiejing
c76faa1ff2
docs(cookbook): enable Top-K v2 for GLM-5.2 MI355X MTP ( #42613 )
2026-10-06 11:33:24 +08:00
Ke Bao
ed75fe67f1
Declare hybrid Mamba indexer host pool ( #40915 )
2026-10-06 10:18:36 +08:00
CuzMi
67959f80ab
[diffusion] optimization: fuse Anima FP32 split-half Q/K RoPE ( #42609 )
2026-10-06 10:18:09 +08:00
Thomas Wang
1abd356595
[AMD] Add engram host/device address mapping ( #42552 )
2026-10-05 19:12:12 -07:00
Po-Han Huang (NVIDIA) and Claude Opus 5.5
f2f0be731c
[Docs] GLM-5.2: move GB300 AgentX recipe to the agentic HiCache section and drop no-op settings ( #42167 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-06 09:40:30 +08:00
Mick and Mick Qian
21c9bbdf2c
[diffusion] fix: fix concurrent server perf dump writes across ranks ( #41541 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
2026-10-06 09:29:51 +08:00
Yash Akhauri and Xiaoyu Zhang
36b216e2ee
[Docs] Refresh K2 Horizon checkpoint revisions ( #42130 )
...
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com >
2026-10-06 09:19:04 +08:00
5d385007b3
[diffusion] kernels: take the Wan VAE decoder's remaining first-decode work off the request path ( #41830 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-06 09:05:29 +08:00
3120bb87a2
[diffusion] fix: fix the OOM hint's placement advice and add a 24 GB Wan2.1 recipe ( #41723 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-06 09:01:02 +08:00
Liangsheng Yin
ad7a9b53cc
[mem_cache] Make release_kv_cache's final checkpoint an explicit checkpoint flag ( #42270 )
2026-10-05 17:32:07 -07:00
Po-Han Huang (NVIDIA)
62af5f56a5
[CI] Drop the redundant and broken tc_piecewise tests ( #42571 )
2026-10-05 17:29:32 -07:00
Liangsheng Yin
c01d59eb90
[Session] Let a streaming session own its KV record and tree lock from the first turn ( #42537 )
2026-10-05 17:28:09 -07:00
Liangsheng Yin
f3fe753475
[misc] Rename SGLANG_OPT_SWA_RELEASE_LEAF_LOCK_AFTER_WINDOW to SGLANG_OPT_RELEASE_PREFILL_SWA ( #42657 )
2026-10-05 17:24:15 -07:00
Sage
4409d2fd22
[ci] add renderer automatic labeler ( #40923 )
...
Signed-off-by: Sage Ahrac <sagiahrak@gmail.com >
2026-10-05 17:12:51 -07:00
Yuwei An
18ec2ecce3
[Refactor][TCPCG] Relocate shared graph tensor and DSA head-gate helpers (1/9) ( #42285 )
2026-10-05 17:06:52 -07:00
ADiL
968726f3ce
[HiCache] Route --file-storage-path to the file storage backend ( #33883 )
...
Signed-off-by: adlashab <adlashab@amd.com >
2026-10-05 15:33:49 -07:00
61ba98443d
[DSA] k-pool: 256-token logical page so index page id = logical page id ( #42178 )
...
Co-authored-by: Cursor <cursoragent@cursor.com >
Co-authored-by: Ke Bao <ispobaoke@gmail.com >
2026-10-05 15:02:44 -07:00
c765f8818a
[DFlash2] Keep the greedy selector walk in range for all-NaN score rows ( #41737 )
...
Co-authored-by: Rahul Chalamala <22563365+rchalamala@users.noreply.github.com >
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-05 14:29:13 -07:00
maocheng23
57826a62ea
[Speculative] Add opt-in block verification ( #42297 )
2026-10-05 11:49:23 -07:00
70f0b7351e
[sgl-router] Improved policy balanced mode ( #42545 )
...
Co-authored-by: Kan Wu <kan.wu@MacBook-Pro.local >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Kan Wu <kan.wu@radixark.ai >
2026-10-05 09:48:37 -07:00
debanshd
efb62ce269
[diffusion] fix: resolve QKV layout inference for MiniMax-H3 checkpoints ( #40158 )
...
Signed-off-by: Debanshu Das <debanshu@google.com >
2026-10-05 23:01:37 +08:00
150f568900
[diffusion] kernels: stop specializing on per-request sequence lengths ( #41710 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-05 22:21:01 +08:00
a977e3b9d5
[diffusion] optimization: stream an oversized DiT in auto mode instead of OOMing ( #41721 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-05 21:58:27 +08:00
ad3e80d4de
[diffusion] chore: warm the image writer and the HTTP route models at startup ( #41833 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-05 21:58:00 +08:00
c83c84b422
[diffusion] chore: keep the DiT off component offload under explicit multi-GPU fsdp ( #41722 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-05 21:56:52 +08:00
wxy
23bf107469
[diffusion] optimization: cache LTX-2 RoPE coords to avoid per-step recompute ( #22441 )
2026-10-05 21:21:36 +08:00
Shiwen Cheng
91f9bf8d17
[diffusion] fix: fix scheduler host not working when ipv6 in multi modal gen ( #22813 )
2026-10-05 21:14:09 +08:00
liuke and Mick
f20f8e120d
[diffusion] model: restore folded H3 GGUF patch embedding ( #39523 )
...
Co-authored-by: Mick <mickjagger19@icloud.com >
2026-10-05 21:10:06 +08:00
Mick and Mick Qian
83a6e1d39b
[diffusion] docs: keep model compatibility details in cookbook recipes ( #42227 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
2026-10-05 20:02:43 +08:00
Akshat Anand and cipheraxat
860ebe720e
[diffusion] fix: skip warmup preferred preload when it would OOM ( #40761 )
...
Co-authored-by: cipheraxat <cipheraxat@users.noreply.github.com >
2026-10-05 19:59:35 +08:00
aiyueqi and Mick
8663e3b670
[diffusion] optimization: avoid full-video clone in VAE post-processing for MiniMax-H3 ( #40095 )
...
Co-authored-by: Mick <mickjagger19@icloud.com >
2026-10-05 19:56:11 +08:00
d349e672bc
[diffusion] feat: recycle warmup-only entries of conditioning-cache at the first served store ( #41832 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-05 19:55:30 +08:00
7352073ec6
[diffusion] feat: support --async-output-save to overlap output saving with the next request ( #37549 )
...
Co-authored-by: Claude Fable 5 <noreply@anthropic.com >
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
2026-10-05 19:53:38 +08:00
WenhaoZhang and Mick Qian
f048d5aa4b
[diffusion] feat: add community LoRA recipes and Kohya mapping ( #35857 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
2026-10-05 18:22:56 +08:00
08fa37e48d
[sgl-router] Add reorg bucket config with per-group admission ( #42429 )
...
Co-authored-by: Kan Wu <kan.wu@radixark.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Kan Wu <kan.wu@MacBook-Pro.local >
2026-10-05 18:17:32 +08:00
Shangming Cai and Claude Opus 5.5
ab4f8a44f4
[sgl-router] Point the design doc at the shared engine ranking ( #42572 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-05 18:16:42 +08:00
a96b146491
[sgl-router] Move pressure and prefix signals out of legacy policies ( #42428 )
...
Co-authored-by: Kan Wu <kan.wu@radixark.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-05 18:10:29 +08:00
mrusanovsky and Po-Han Huang
26a1377c50
[Spec] LiLiCorr: named head MLPs and quantized head linears ( #42057 )
...
Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com >
2026-10-05 02:43:59 -07:00
WenhaoZhang and Mick Qian
284cda01fb
[diffusion] optimization: fuse the video VAE decoder's RMSNorm and QK RoPE for MiniMax-H3 ( #41906 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
2026-10-05 17:30:39 +08:00
0263fdacb2
[diffusion] fix: keep generic warmup valid for step-floored models and stop reporting a failed warmup as warm ( #41641 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-05 17:23:06 +08:00
elvischenv and Claude Opus 5.5
55dcf3254f
Fix XQA draft extend with FP8 KV cache in trtllm_mha ( #41656 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-05 01:37:54 -07:00
Xiaoyu Zhang
abd6c374c7
[Perf] Optimize MiMo local audio attention on Hopper ( #42101 )
2026-10-05 16:14:56 +08:00
Michael
b3f967e809
Revert "Revert "[AMD][ROCm] Keep cos_sin_cache fp32 on HIP for fused QSA indexer kernel ( #41282 )"" ( #42558 )
2026-10-05 16:12:21 +08:00
Aleksi Vesanto and jacky.cheng
9f7d38ffd3
[diffusion] attention: Allow disabling sequence masking in SP ( #33855 )
...
Co-authored-by: jacky.cheng <yichiche@amd.com >
2026-10-05 15:22:53 +08:00
Aleksi Vesanto and jacky.cheng
c41b2d26e5
[diffusion][ROCm][Perf]: Set gfx942 AITER FMHA rounding mode to rtz instead of rtna ( #28650 )
...
Co-authored-by: jacky.cheng <yichiche@amd.com >
2026-10-05 15:21:04 +08:00
Mick and Mick Qian
dd2ffdb680
[diffusion] nightly: keep nightly regression baselines on a comparable methodology ( #42226 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
2026-10-05 15:19:21 +08:00
438d9d2074
[Feature] DeepEP v2: expanded (do_expand=True) prefill dispatch ( #37261 )
...
Co-authored-by: Cheng Wan <cheng.wan@radixark.ai >
Co-authored-by: Cheng Wan <54331508+ch-wan@users.noreply.github.com >
2026-10-05 00:02:10 -07:00
cafb37a10d
[JIT] Fuse FP8 KV-cache quantization into the prefix-valid commit kernel ( #40785 )
...
Co-authored-by: Jayden Yu <jaydenyu@fb.com >
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com >
2026-10-05 14:47:54 +08:00
WenhaoZhang and Mick Qian
f218798006
[diffusion] CI: relax the load latency check and extend the 2-GPU retry deadline ( #42505 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
2026-10-05 14:43:27 +08:00
520d6d0229
[DSv4.1] Prefill indexer: raw selection, row-pair schedule ids, no host syncs ( #41659 )
...
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
Co-authored-by: BBuf <1182563586@qq.com >
2026-10-05 14:33:37 +08:00
Liangsheng Yin
503b898540
[Session] Count streaming session KV by owner in pool accounting ( #42539 )
2026-10-04 22:15:57 -07:00
Michael
456d533e61
Revert "[AMD][ROCm] Keep cos_sin_cache fp32 on HIP for fused QSA indexer kernel ( #41282 )" ( #42541 )
2026-10-04 21:59:12 -07:00
Liangsheng Yin
1818db4e54
[CI] Drop the echoed command from /rerun-test result replies ( #42543 )
2026-10-04 21:50:53 -07:00
ashwini rathi
b1bbd74f28
[XPU][CI] Disable test_xpu_graph until tc_piecewise is removed ( #41634 ) ( #42524 )
2026-10-05 11:09:26 +08:00
Xiaoyu Zhang
f385390be5
[Refactor] Simplify GDN, vision RoPE and MLA kernel dispatch ( #42506 )
2026-10-05 09:37:53 +08:00
734cf3cf3b
[PD] Reload NIXL peer metadata after the peer is invalidated ( #41691 )
...
Co-authored-by: agent <agent@local>
Co-authored-by: Jimmy Shong <69131491+Jiminator@users.noreply.github.com >
2026-10-05 00:49:26 +00:00
Yuwei An
f70e8c68fc
[Dsv4.1] Bounded replay with sparse mla path ( #42273 )
2026-10-04 17:19:14 -07:00
Eric.Chin.AMD and Bingxu Chen
b792228b35
[ROCm] topk v2: split one long row across blocks, the CDNA cluster-path equivalent ( #39931 )
...
Co-authored-by: Bingxu Chen <bingxche@amd.com >
2026-10-04 16:54:37 -07:00
Cheng Wan
1f299e9f2f
[Refactor] Build layer stacks in order with append_stages ( #42481 )
2026-10-04 16:40:24 -07:00
Cheng Wan
a99567ab74
[Fix] GigaChat 3.5: build the stage boundaries once ( #42480 )
2026-10-04 16:39:34 -07:00
Cheng Wan
5c8fb9e938
[Fix] Offer the fused MoE finalize all-reduce only to a block that owes the sum ( #42479 )
2026-10-04 16:39:06 -07:00
Cheng Wan
f07c1a398e
[Refactor] Build the Qwen4 experimental decoders from stage boundaries ( #42478 )
2026-10-04 16:38:36 -07:00
Cheng Wan
92d60351e2
[Refactor] Build the Hunyuan V4 decoder from stage boundaries ( #42477 )
2026-10-04 16:38:03 -07:00
Lianmin Zheng and jiayisuse
f5ccc4d7d9
[Kernel] Use explicit reciprocal scaling for recurrent Q/K normalization ( #42486 )
...
Co-authored-by: jiayisuse <jiayisuse@users.noreply.github.com >
2026-10-04 15:30:03 -07:00
Tarang Khanna
d475b5a25c
[weight_cache] Coordinate daemon readiness across ranks ( #38828 )
2026-10-04 15:20:13 -07:00
Aditya Sharma
caf96324fc
[Apple Silicon] Add an Apple Silicon (MPS) platform to the standard Torch model runner ( #36780 )
2026-10-04 15:03:40 -07:00
SuperSong and Cheng Wan
bab04cd791
fix(unified-memory): propagate unified-memory lazy checkpoint policy and handle unallocated session slots ( #42072 )
...
Co-authored-by: Cheng Wan <cheng.wan@radixark.ai >
2026-10-04 12:51:22 -07:00
Mohammad Miadh Angkad and Mohammad Angkad
fea3c088a1
[CI] Temporarily disable GB300 ( #42463 )
...
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai >
2026-10-04 12:24:04 -07:00
Zhang, Jiejing and kyle-256
aa0ce20201
[ROCm] GLM-5.2 decode path: decode-shaped MoE/MLA tiles, split speculative softmax, and bf16 GEMM routing ( #41725 )
...
Co-authored-by: kyle-256 <Kyle.Zhao@amd.com >
2026-10-04 11:19:37 -07:00
Xiaoyu Zhang
7a719e9a65
[AMD] Fix RoPE cache dtype and diffusion CI failures ( #42494 )
2026-10-05 00:36:26 +08:00
billishyahao
0343bc1483
[AMD][Docs] fix mori io and umbp playground ( #42512 )
2026-10-04 09:28:11 -07:00
Shuwen Wang
137c084000
[HiCache] refactor: retire in-flight storage prefetches through one helper ( #41453 )
2026-10-05 00:13:07 +08:00
WenhaoZhang
affa261e3d
[diffusion] feat: enable cuda graphs for observation encoding and denoising steps for action models ( #42171 )
2026-10-04 22:22:08 +08:00
WenhaoZhang
279d3f3888
[diffusion] feat: load ComfyUI/ai-toolkit fused gate_up LoRAs and diffusers metadata alpha for qwen-image-2.1 ( #42174 )
2026-10-04 22:21:13 +08:00
WenhaoZhang
44fac689f5
[diffusion] optimization: decode json pixel lists to numpy at the entrypoint ( #41895 )
2026-10-04 22:20:20 +08:00
Michael
ab9d8f029c
[AMD][DI][CI] mi355x spur: skip RDMA ports that are not PORT_ACTIVE; dump driver log on failure ( #41405 )
2026-10-04 21:23:54 +08:00
Shuwen Wang
35f3c96ff4
[CI] Fix bootstrap sender registration test on main ( #42487 )
2026-10-04 18:45:31 +08:00
Shangming Cai and Claude Opus 5
148da8e77d
[PD] Validate decode state layout once at registration ( #42051 )
...
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-10-04 18:32:30 +08:00
Xiaoyu Zhang
bc5a055dd7
[diffusion] move kernel validation into launchers and simplify dispatch ( #42391 )
2026-10-04 18:23:29 +08:00
Michael
29bcb74a89
[AMD][CI] Fix ROCm kernel wheel sources: drop eagle_utils, add DSV4 kernels ( #42461 )
2026-10-04 17:54:28 +08:00
SuperSong and Cheng Wan
6cec8f98ce
[Bugfix] Fix unified-memory compaction gates and pending page reuse ( #39982 )
...
Co-authored-by: Cheng Wan <cheng.wan@radixark.ai >
2026-10-04 02:48:08 -07:00
SuperSong and Cheng Wan
1d02a36bb7
fix(inkling): translate unified-memory checkpoint destinations to physical slots ( #38229 )
...
Co-authored-by: Cheng Wan <cheng.wan@radixark.ai >
2026-10-04 02:45:45 -07:00
WenhaoZhang and Claude Opus 5.5
1e490772e5
[diffusion] fix: pin the JoyAI-Echo overlay's source revision ( #42471 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-04 17:19:09 +08:00
ba68306256
[diffusion] feat: let requests choose the video libx264 preset ( #41797 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-04 16:22:36 +08:00
Tokha233 and WenhaoZhang
fbce0d9478
[diffusion] fix: load serialized h3 INT8 in ComfyUI integrated mode ( #42121 )
...
Signed-off-by: Tokha233 <61346912+Tokha233@users.noreply.github.com >
Co-authored-by: WenhaoZhang <42087078+niehen6174@users.noreply.github.com >
2026-10-04 16:08:56 +08:00
Mohammad Miadh Angkad and Mohammad Angkad
0bdd0cc8dc
[Deps] Upgrade the CUDA PyTorch stack to 2.14 ( #38641 )
...
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai >
2026-10-04 00:48:02 -07:00
Kotthagattu Meher Sai
9631f6f2d0
[XPU] Enable fused QK-norm + RoPE for Qwen3-MoE ( #40190 )
2026-10-04 15:45:18 +08:00
Mohammad Miadh Angkad and Mohammad Angkad
7eb628612c
[Fix] Publish the parallel config the CLIP attention test reads ( #42460 )
...
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai >
2026-10-04 00:44:51 -07:00
Baizhou Zhang
55c5415c17
[CP] Support scattered interleave CP inputs with expert parallelism ( #42041 )
2026-10-04 00:43:31 -07:00
937c0a6cfe
Fix unified HiCache physical transfers ( #39479 )
...
Co-authored-by: yhzhuang <yhzhuang@fb.com >
Co-authored-by: Lianmin Zheng <lianminzheng@gmail.com >
Co-authored-by: Yonghao Zhuang <yhzhuang@users.noreply.github.com >
Co-authored-by: Cheng Wan <cheng.wan@radixark.ai >
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com >
2026-10-04 00:38:56 -07:00
billishyahao
6cc661f223
[AMD][DCP 2/N] Enable aiter asm ps for kimi k3 prefill ( #34438 )
2026-10-03 23:34:45 -07:00
Liangsheng Yin
048b2a07f3
[Session] Release aborted streaming turns through the normal release path ( #42330 )
2026-10-03 22:30:25 -07:00
Mohammad Miadh Angkad and Mohammad Angkad
1315d171a0
[Diffusion][CI] Add missing B200 Cirrascale runner E2E baselines ( #42329 )
...
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai >
2026-10-03 21:10:32 -07:00
Brayden Zhong and Mohammad Angkad
a3275295c4
[MoE] Use FlashInfer A2A for prefill instead of AG+RS ( #39818 )
...
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai >
2026-10-03 21:09:00 -07:00
Mohammad Miadh Angkad and Mohammad Angkad
c12bc635ca
[CI] Install only userspace libgdrapi for GDRCopy ( #42233 )
...
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai >
2026-10-03 21:05:35 -07:00
Mohammad Miadh Angkad and Mohammad Angkad
0e6d7eba5f
[Fix] Count attention-DP ranks in one-batch bench running cap ( #42331 )
...
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai >
2026-10-03 21:04:27 -07:00
Baizhou Zhang
fccc61c1c5
[CP] Support DP x TP x CP for attention with interleave CP strategy ( #42038 )
2026-10-03 20:57:36 -07:00
Aurick Qiao
f24db66729
[PD] Publish prefill DP rank whenever forced lookup is enabled ( #41786 )
2026-10-04 11:46:20 +08:00
metamergebot and metamergebot
169e6223ef
[PD] Drop the removed MooncakeKVSender args from test_prefill_complete_integration ( #42454 )
...
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com >
2026-10-03 20:29:06 -07:00
4611ea365e
[HiCache]: Fix host lock ownership across radix splits ( #38480 )
...
Signed-off-by: Lukas Humbel <lhumbel@nvidia.com >
Co-authored-by: Jialin Ouyang <jialino@meta.com >
Co-authored-by: Shuwen Wang <47200617+alphabetc1@users.noreply.github.com >
2026-10-04 11:16:04 +08:00
ff219e159e
[AMD] MiniMax-M3: fuse sparse QK norm, RoPE and cache writes with AITER ( #35357 )
...
Co-authored-by: zijiec <300606707+zijiecode@users.noreply.github.com >
Co-authored-by: Thomas Wang <thomawan@amd.com >
2026-10-03 19:49:53 -07:00
sglang-bot and sglang-bot
8afef58af4
docs: sync LMSYS SGLang blog cards ( #41296 )
...
Co-authored-by: sglang-bot <sglang-bot@users.noreply.github.com >
2026-10-04 02:26:05 +00:00
5606fb8592
[Fix] HiCache: carry registered Mamba slot side states (Qwen4-Exp PLE) through the host tier ( #39862 )
...
Co-authored-by: Shuwen Wang <47200617+alphabetc1@users.noreply.github.com >
Co-authored-by: alphabetc1 <2508695655@qq.com >
Co-authored-by: ispobock <ispobaoke@gmail.com >
2026-10-04 10:15:59 +08:00
Michael
f2385314b2
[AMD] Add DeepSeek-V4.1-Flash MI35x nightly accuracy test ( #41476 )
2026-10-03 18:16:44 -07:00
Xinyi Song and kk
663984b5a1
[AMD] Keep aiter's tuned GEMM for the DeepSeek-V4 compressors ( #41931 )
...
Co-authored-by: kk <43161300+kkHuang-amd@users.noreply.github.com >
2026-10-03 18:11:37 -07:00
trebladev
295a53c90e
[diffusion] optimization: fuse Flux3 rowwise FP8 quantization with triton ( #41671 )
2026-10-04 09:01:17 +08:00
71dc292b97
[AMD] Use AITER ASM prefill for MiniMax-M3 HD128 attention ( #41707 )
...
Co-authored-by: Zijie Chen <300606707+zijiecode@users.noreply.github.com >
Co-authored-by: Thomas Wang <thomawan@amd.com >
2026-10-03 17:54:49 -07:00
4ab720e655
[HiCache] Fix cgroup page-cache accounting and the sizing fallback when cgroup discovery fails ( #42420 )
...
Co-authored-by: merrymercy <merrymercy@users.noreply.github.com >
Co-authored-by: xiezhq-hermann <xiezhq-hermann@users.noreply.github.com >
Co-authored-by: nvjullin <nvjullin@users.noreply.github.com >
2026-10-03 16:01:32 -07:00
metamergebot and merrymercy
4301be7bdb
Pick the default nccl_port below the kernel ephemeral range ( #42405 )
...
Co-authored-by: merrymercy <merrymercy@users.noreply.github.com >
2026-10-03 16:00:53 -07:00
Cheng Wan
10edb05e72
[Refactor] Let the remaining placement consumers read the parallel context ( #42348 )
2026-10-03 14:59:32 -07:00
07064fd4a4
[sgl-router] Credit routed prompts before their KV events arrive ( #42276 )
...
Co-authored-by: Kan Wu <kan.wu@radixark.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Kan Wu <kan.wu@MacBook-Pro.local >
2026-10-03 14:57:49 -07:00
Cheng Wan
b2aa993cb8
[Fix] Skip a draft's decode recapture when it owns no decode graph ( #42347 )
2026-10-03 14:54:12 -07:00
Cheng Wan
1a18de4b29
[Fix] Load a draft's tensor weight update from its deployment TP rank ( #42346 )
2026-10-03 14:52:25 -07:00
sglang-bot and sglang-bot
061e712bab
chore: bump sgl-kernel version to 0.4.9 ( #42427 )
...
Co-authored-by: sglang-bot <sglang-bot@users.noreply.github.com >
2026-10-03 14:29:04 -07:00
Mohammad Miadh Angkad and Mohammad Angkad
3122b9c995
Update AOT kernels for Torch 2.14 ( #42366 )
...
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai >
2026-10-03 14:26:34 -07:00
Liangsheng Yin
df53c978f1
[mem_cache] Run mamba models on UnifiedRadixCache when the radix cache is disabled ( #42354 )
2026-10-03 14:11:13 -07:00
Liangsheng Yin
3e9a120add
[Fix] Make Hf3fsMockClient reads and writes thread-safe with pread/pwrite ( #42425 )
2026-10-03 14:02:24 -07:00
8bf780dda5
feat(metrics): expose deferred decode KV release metrics ( #41128 )
...
Co-authored-by: Shangming Cai <csmthu@gmail.com >
Co-authored-by: Claude Fable 5 <noreply@anthropic.com >
2026-10-04 04:52:21 +08:00
Yanbin Jiang
749a89abd6
[LoRA] Reorganize kernels and add CODEOWNERS ( #42299 )
2026-10-03 13:43:40 -07:00
fd5e68f99e
[sgl-router] Repair KV-event sequence gaps from the engine's replay socket ( #42275 )
...
Co-authored-by: Kan Wu <kan.wu@radixark.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Kan Wu <kan.wu@MacBook-Pro.local >
2026-10-04 04:43:17 +08:00
aaf7ebe077
Advertise the KV-event replay endpoint in /server_info ( #42274 )
...
Co-authored-by: Kan Wu <kan.wu@radixark.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-04 04:42:33 +08:00
AMD-yanfeiwang
3dc7f4b816
fix(cuda-graph): remove unnecessary CP batch alignment ( #40543 )
2026-10-03 12:14:28 -07:00
AMD-yanfeiwang
e83c95b650
[AMD] Fix AITER weight slicing for CP decode TP ( #39950 )
2026-10-03 12:09:08 -07:00
AMD-yanfeiwang
f327f92424
[AMD] Port CP V2 to the DeepSeek-V4 HIP backend ( #34200 )
2026-10-03 12:08:17 -07:00
6acf2f4736
[PD] Keep ingesting requests while a prefill forward result is pending ( #42035 )
...
Co-authored-by: merrymercy <merrymercy@users.noreply.github.com >
Co-authored-by: Jialin <Jialin@users.noreply.github.com >
2026-10-03 12:02:21 -07:00
b016ca406f
Support post-capture KV sizing for the unified hybrid-SWA pool ( #41961 )
...
Co-authored-by: merrymercy <merrymercy@users.noreply.github.com >
Co-authored-by: ZYHowell <ZYHowell@users.noreply.github.com >
2026-10-03 09:54:20 -07:00
2bae12b9b3
[HiCache] Fix write-back SWA insert backups tripping the write-through pending-ack assert ( #42264 )
...
Co-authored-by: merrymercy <merrymercy@users.noreply.github.com >
Co-authored-by: 842974287 <842974287@users.noreply.github.com >
2026-10-03 09:45:56 -07:00
5b5d721239
[PD] Add opt-in prefill-complete decode KV allocation ( #40703 )
...
Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com >
Co-authored-by: Po-Han Huang <pohanh@nvidia.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Shangming Cai <csmthu@gmail.com >
2026-10-03 21:03:09 +08:00
5916999afb
[DSv4.1] Fused c1/c2 compress for eager extend, faster c2 decode ( #41660 )
...
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
Co-authored-by: BBuf <1182563586@qq.com >
2026-10-03 19:31:05 +08:00
d75d5b33f7
[DSv4.1] Faster fp4 index-K gather and combine_topk_swa_indices ( #41658 )
...
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
Co-authored-by: BBuf <1182563586@qq.com >
2026-10-03 18:25:22 +08:00
DarkSharpness and Claude Opus 5
379ec90fb2
[DSv4.1] Fold q_rope_store into fused_q_norm_rope ( #41657 )
...
Co-authored-by: Claude Opus 5 <noreply@anthropic.com >
2026-10-03 18:24:41 +08:00
Cheng Wan
7be5e3473c
[Refactor] Drop the reduction-skip mechanisms stage boundaries no longer use ( #42312 )
2026-10-03 02:21:55 -07:00
Cheng Wan
71b04e02cf
[Refactor] Build the ZAYA1, IQuest-Q1 and Gemma 4 decoders from stage boundaries ( #42311 )
2026-10-03 02:21:34 -07:00
Cheng Wan
f85c2c4923
[Fix] GigaChat 3.5: apply the sandwich norms to the complete sums ( #42310 )
2026-10-03 02:21:10 -07:00
Cheng Wan
6faccf3e84
[Refactor] Build the GLM-4, GLM-Image and Granite MoE hybrid decoders from stage boundaries ( #42309 )
2026-10-03 02:20:40 -07:00
Cheng Wan
e4554fd5e5
[Refactor] Build the Llama and Nemotron-NAS decoders from stage boundaries ( #42308 )
2026-10-03 02:20:18 -07:00
Cheng Wan
740a4d5955
[Refactor] Build the Qwen2 decoders from stage boundaries ( #42307 )
2026-10-03 02:19:56 -07:00
Cheng Wan
2c739b225b
[Fix] Jet-Nemotron build and Granite MoE hybrid final norm ( #42306 )
2026-10-03 02:19:33 -07:00
Cheng Wan
b7fc516986
[Fix] EXAONE MoE under DP attention and DeepEP ( #42305 )
2026-10-03 02:19:19 -07:00
Cheng Wan
2fa17a6467
[Refactor] Build the ERNIE 4.5 VL MoE and EXAONE MoE decoders from stage boundaries ( #42304 )
2026-10-03 02:18:56 -07:00
Cheng Wan
f89ead192f
[Fix] EXAONE and ERNIE 4.5 VL MoE architecture, backend and PP issues ( #42303 )
2026-10-03 02:18:33 -07:00
Cheng Wan
be20e7453c
[Fix] Step-3.5 DeepEP routed scaling and Sarvam shared expert under dense TP1 ( #42302 )
2026-10-03 02:18:11 -07:00
Cheng Wan
b157d548bb
[Refactor] Let stage boundaries complete every stage-output sum ( #42301 )
2026-10-03 02:17:48 -07:00
Cheng Wan
d5a8d76bf5
[Refactor] Drop TBO op methods that no strategy ever schedules ( #42300 )
2026-10-03 02:17:26 -07:00
Liangsheng Yin
122b59152c
[mem_cache] Replace is_chunk_cache / is_tree_cache with supports_prefix_sharing ( #42362 )
2026-10-03 02:07:55 -07:00
Banruo Liu
8293f9af54
[PD] Centralize drain-aware abort acknowledgements ( #37077 )
2026-10-03 16:51:11 +08:00
saatwiknagpal
1093c501df
[diffusion] optimization: route subBlock sparse attention in head chunks ( #41985 )
2026-10-03 15:54:54 +08:00
Yanbin Jiang
254aa41f78
[Fix] Guard DeepSeek NVFP4 shared-expert fusion for LoRA and FP4 backends ( #42203 )
2026-10-03 00:37:57 -07:00
Tokha233
ce617d410b
[diffusion] fix: fix native FP8 format handling for FLUX 3 row-wise linears ( #42255 )
2026-10-03 15:23:41 +08:00
Mohammad Miadh Angkad and Mohammad Angkad
00bcc25f6c
[HiCache] Fix DeepSeek-V4 storage backend crash from missing storage_format_tag ( #42215 )
...
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai >
2026-10-02 22:27:57 -07:00
Jimmy Shong
0072e2cc08
[Fix] Validate per-layer rope_parameters on transformers 5.17 (Laguna-S/XS-2.1 nightly) ( #42214 )
2026-10-02 21:49:45 -07:00
Jacob0226 and Thomas Wang
af1bef3eaf
[AMD] GLM-5.3-Flash: fuse shared expert and KDA projections on Quark MXFP4 ( #41870 )
...
Co-authored-by: Thomas Wang <thomawan@amd.com >
2026-10-02 20:47:47 -07:00
Richard Wang and Richard Wang
482b062d28
[Feature] serve pplx-decider decision checkpoints on /v1/systemone ( #42183 )
...
Co-authored-by: Richard Wang <11150595+rwang5203@users.noreply.github.com >
2026-10-03 10:18:53 +08:00
WenhaoZhang and Xiaoyu Zhang
41c6ffea89
[diffusion] fix: keep residual_gate_add on the JIT CUDA path for cont… ( #42240 )
...
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com >
2026-10-03 10:12:11 +08:00
Xiaoyu Zhang
391e665dab
[Perf] Optimize DeepSeek V4.1 Flash Hopper paths and Blackwell prefill selection ( #41251 )
2026-10-03 10:04:48 +08:00
Mick and Mick Qian
65c7425a56
[diffusion] fix: respect explicit residency over pipeline preload hints ( #41622 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
2026-10-03 09:54:21 +08:00
Liangsheng Yin
a8b54f138e
[Session] Run streaming sessions only on UnifiedRadixCache; reject unverified tree caches ( #42295 )
2026-10-02 18:15:59 -07:00
8ff02f5820
[QSA] Enable breakable prefill CUDA graphs for text-only Qwen3.8 Flash-Next (capture-safe metadata, MTP side-channel padding) ( #41729 )
...
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com >
Co-authored-by: Qiaolin Yu <liin1211@outlook.com >
2026-10-02 17:57:42 -07:00
Kevin Mi and Claude Opus 5
65f759144d
[HiCache] Let the MiniMax-M3 K-only host pool join a host pool group ( #42166 )
...
Co-authored-by: Claude Opus 5 <noreply@anthropic.com >
2026-10-02 17:00:13 -07:00
Baizhou Zhang
3bde8eb2d8
[CP 1/5] Remove the CP adapter and separate interleave transport from boundary reduction ( #42036 )
2026-10-02 16:50:20 -07:00
Zhang, Jiejing
cbe070b6bf
[Docs][AMD] Update GLM-5.2 MI355X daily image to 20260930 ( #42278 )
2026-10-02 15:39:07 -07:00
b69a5f296a
[AMD] Add opt-in MiniMax-M3 TP4 indexer context partitioning ( #41488 )
...
Co-authored-by: ThomasNing <thomas.ning@amd.com >
Co-authored-by: Kevin Mi <mikevin920@yahoo.com >
Co-authored-by: Claude Opus 5 <noreply@anthropic.com >
Co-authored-by: Kevin Mi <45493463+kevin-mii@users.noreply.github.com >
2026-10-02 15:34:29 -07:00
fa7784d140
[unified-memory] Copy page envelopes in place during compaction ( #42252 )
...
Co-authored-by: merrymercy <merrymercy@users.noreply.github.com >
Co-authored-by: ZYHowell <ZYHowell@users.noreply.github.com >
2026-10-02 15:12:14 -07:00
e8fab85a02
[KDA] Add an opt-in decode-parity mode to the Triton multi-token recurrence ( #42251 )
...
Co-authored-by: merrymercy <merrymercy@users.noreply.github.com >
Co-authored-by: jiayisuse <jiayisuse@users.noreply.github.com >
2026-10-02 14:59:56 -07:00
19794f5743
[sgl-router] Add SGLang's /v1/classify ( #42258 )
...
Co-authored-by: Kan Wu <kan.wu@radixark.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-03 05:56:16 +08:00
Liangsheng Yin
8a328e867b
[mem_cache] Rename the tree's request-level insert_req to checkpoint ( #42202 )
2026-10-02 14:42:35 -07:00
2ec9e3b11d
[eplb] Warm default-group NCCL P2P transports before KV-cache sizing ( #42026 )
...
Co-authored-by: merrymercy <merrymercy@users.noreply.github.com >
Co-authored-by: luccafong <luccafong@users.noreply.github.com >
2026-10-02 14:34:55 -07:00
ebf161cea4
[sgl-router] Add SGLang's /v1/rerank ( #42205 )
...
Co-authored-by: Kan Wu <kan.wu@radixark.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-03 05:33:40 +08:00
Kan Wu and Kan Wu
8c9030ce3f
[sgl-router] Unify reorg session and cache affinity modes ( #41990 )
...
Co-authored-by: Kan Wu <kan.wu@radixark.ai >
2026-10-03 05:21:22 +08:00
Yuhao Wang
0098b9d29d
[Spec] Fix Qwen3.5 text model EAGLE3/DFLASH aux-layer capture ( #42002 )
2026-10-02 14:05:01 -07:00
raghotham
a64725d407
[HiCache] Fall back to host memory when no cgroup fs is mounted ( #42120 )
2026-10-02 13:59:40 -07:00
7dd93b01a5
[PD] Make the prebuilt last-token H2D copy non-blocking under overlap ( #42113 )
...
Co-authored-by: merrymercy <merrymercy@users.noreply.github.com >
Co-authored-by: bbus <bbus@users.noreply.github.com >
2026-10-02 13:48:39 -07:00
Liangsheng Yin
07e1a4f082
[mem_cache] Skip the release-time insert on optimistic prefill requeue; make refresh_fill_ids public ( #42194 )
2026-10-02 13:46:32 -07:00
354ed2a7b6
Size the VMM graph-input exchange by the widest input across ranks ( #42125 )
...
Co-authored-by: merrymercy <merrymercy@users.noreply.github.com >
Co-authored-by: oulgen <oulgen@users.noreply.github.com >
2026-10-02 13:45:32 -07:00
4a4dad1cf1
[Metrics] Label scheduler stage wall time by sampled forward overlap ( #42025 )
...
Co-authored-by: merrymercy <merrymercy@users.noreply.github.com >
Co-authored-by: Jialin <Jialin@users.noreply.github.com >
2026-10-02 13:43:53 -07:00
ChangLiu0709 and Nikolai Protasov
f6fcda8827
[AMD][ROCm] Keep cos_sin_cache fp32 on HIP for fused QSA indexer kernel ( #41282 )
...
Co-authored-by: Nikolai Protasov <289648307+amd-nprotaso@users.noreply.github.com >
2026-10-02 11:57:03 -07:00
Shuwen Wang
89f21671bb
fix: preserve QSA indexer state through HiCache ( #39893 )
2026-10-02 23:49:38 +08:00
kk and wunhuang
a74f259f38
[AMD][V4.1][*/N] Fuse MXFP8 activation quant into producer kernels on gfx950 ( #42055 )
...
Co-authored-by: wunhuang <wunhuang@amd.com >
2026-10-02 08:34:44 -07:00
Alex O. P. and Po-Han Huang
e667ab10d5
[diffusion] quantization: ConvRot INT8 online W8A8 for DiTs (sgl-kernel + convrot_int8_customkernel) ( #38040 )
...
Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com >
2026-10-02 23:26:28 +08:00
Xiaoyu Zhang
1a2eb6c426
[Kernel] Use Cake softmax for qualified SM103 sampling workloads ( #42004 )
2026-10-02 23:06:43 +08:00
bb9a820f09
[diffusion] chore: make perf dump one writer per replica with no first-request NCCL setup ( #41711 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-02 21:44:10 +08:00
Michael
2cd92d10b9
[AMD] Skip the AITER #6042/#5967 patches on gfx1250 ( #42219 )
2026-10-02 21:10:01 +08:00
edee4308bc
[sgl-router] Add OpenAI /v1/embeddings ( #42191 )
...
Co-authored-by: Kan Wu <kan.wu@radixark.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-02 20:18:24 +08:00
1330e437e1
[sgl-router] Add native /generate endpoint ( #42147 )
...
Co-authored-by: Kan Wu <kan.wu@radixark.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-02 20:09:04 +08:00
Michael
e020b389b6
[AMD][CI] Partition stage-b-test-1-gpu-small-amd-mi35x to stop the 30-min timeout ( #41973 )
2026-10-02 19:29:20 +08:00
Mick and Mick Qian
0f71ec8656
[diffusion] UX: deduplicate SM120 FP8 fallback warnings across layers ( #41542 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
2026-10-02 19:25:15 +08:00
Michael
544e46c080
[AMD][CI] Fix VLM MMMU nightly max_tokens for CoT prompt ( #41979 )
2026-10-02 18:59:05 +08:00
YAMY and Po-Han Huang
895525f062
[Mamba] Warm cache COW kernel before serving ( #36266 )
...
Signed-off-by: Yangmin Li <yangminl@nvidia.com >
Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com >
2026-10-02 18:52:37 +08:00
Alex
1db33bc367
[PD][NIXL] Fix prepared dlists for KV entries of different lengths (DeepSeek-V4) ( #42005 )
2026-10-02 17:59:31 +08:00
Po-Han Huang (NVIDIA) and Claude Opus 5.5
41bd213c5f
[Test] Add get_swa_key_page_size to the Q8KV8 sparse-prefill fake KV pool ( #42211 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-02 16:00:15 +08:00
Mohammad Miadh Angkad and Mohammad Angkad
d954c88a4f
[Fix] Update stale source-patch match in dumper comparator e2e test ( #42179 )
...
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai >
2026-10-02 00:37:37 -07:00
Mohammad Miadh Angkad and Mohammad Angkad
e7d9c0299b
[Docs] Keep the last CUDA 12 image tag fixed during install version bumps ( #42201 )
...
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai >
2026-10-02 00:17:13 -07:00
sglang-bot and sglang-bot
d61a9c28ba
chore: bump docs install version to 0.5.21 ( #42013 )
...
Co-authored-by: sglang-bot <sglang-bot@users.noreply.github.com >
2026-10-01 23:16:26 -07:00
Liangsheng Yin
50be533d09
[mem_cache] Rename cache_unfinished_req to checkpoint_req and count cache hits only when a request finishes ( #41520 )
2026-10-01 21:41:45 -07:00
Mingyi and Mick Qian
b906cd3f45
doc: refresh README ( #41967 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
2026-10-02 12:11:33 +08:00
saatwiknagpal
bf8adf9602
[diffusion] fix: fix int32 offset overflow in SubBlock router kernels ( #41965 )
2026-10-02 11:22:53 +08:00
Khoa Pham
6be84e1b72
[Fix] Select the Marlin MoE runner for MiMo-V2 packed MXFP4 experts on SM90 ( #42172 )
2026-10-01 20:01:03 -07:00
Yuwei An
ef867fa40d
[Fix][DSV4.1] SWA page size with bounded replay ( #42128 )
2026-10-01 18:50:43 -07:00
jiaryang and Thomas Wang
58f0d250ec
[ROCm] Use the fused MLA absorb + RoPE + KV-write kernel for decode-sized forward modes only ( #41533 )
...
Co-authored-by: Thomas Wang <thomawan@amd.com >
2026-10-01 18:45:29 -07:00
kk and wunhuang
29f6d408c0
[AMD][V4.1][*/N] Fix shared-expert fusion accuracy and speed up MoE routing on ROCm ( #42011 )
...
Co-authored-by: wunhuang <wunhuang@amd.com >
2026-10-01 18:19:06 -07:00
Kedar Potdar and Claude Opus 5.5
8acd36b56b
[Docs] GLM-5.2 GB300 NVFP4: add env vars from InferenceX AgentX recipe ( #42109 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-02 09:11:29 +08:00
Qiaolin Yu
67eab57057
[qwen 3.8 next] Fuse NEXTN verify and draft graph input preparation ( #41175 )
2026-10-01 17:17:34 -07:00
caihuali95 and Cheng Wan
2a3655fea1
[unified-memory] Name the fused KV translate for what it computes (7/7) ( #40331 )
...
Co-authored-by: Cheng Wan <cheng.wan@radixark.ai >
2026-10-01 17:14:03 -07:00
df530479c6
[unified-memory] Stride the KV translate kernel and route every translate through it (6/7) ( #40330 )
...
Co-authored-by: Caihua Li <caihua.li@bytedance.com >
Co-authored-by: Cheng Wan <cheng.wan@radixark.ai >
2026-10-01 17:12:14 -07:00
9855e9c6f8
[unified-memory] Remove the kernel-page multiplier plumbing (5/7) ( #40329 )
...
Co-authored-by: Caihua Li <caihua.li@bytedance.com >
Co-authored-by: Cheng Wan <cheng.wan@radixark.ai >
2026-10-01 17:11:25 -07:00
4c4e352a4b
[unified-memory] Mark the write loc physical and check it at every write door (4/7) ( #40328 )
...
Co-authored-by: Caihua Li <caihua.li@bytedance.com >
Co-authored-by: Cheng Wan <cheng.wan@radixark.ai >
2026-10-01 17:10:37 -07:00
cdadb1c79a
[unified-memory] Token-major dense views for the unified memory pool (3/7) ( #38592 )
...
Co-authored-by: Caihua Li <caihua.li@bytedance.com >
Co-authored-by: Cheng Wan <cheng.wan@radixark.ai >
2026-10-01 17:09:38 -07:00
caihuali95 and Cheng Wan
5df52e6382
[unified-memory] Build paged KV views through one helper (2/7) ( #40327 )
...
Co-authored-by: Cheng Wan <cheng.wan@radixark.ai >
2026-10-01 17:08:42 -07:00
caihuali95 and Cheng Wan
d3dbb71e5a
[unified-memory] Derive KV row addresses from strides, not shapes (1/7) ( #40326 )
...
Co-authored-by: Cheng Wan <cheng.wan@radixark.ai >
2026-10-01 17:07:47 -07:00
shyeh25 and Brayden Zhong
5423a4d885
[MegaMoE] Preserve W13 layout for ModelOpt NVFP4 experts ( #39388 )
...
Co-authored-by: Brayden Zhong <b8zhong@uwaterloo.ca >
2026-10-01 16:12:34 -07:00
f17f7705a5
[AMD] [GLM-5.3-Flash] Enable FP8 and MXFP4 serving on gfx950 ( #39273 )
...
Co-authored-by: Byron Hsu <byron+per@periodiclabs.ai >
Co-authored-by: Cheng Wan <54331508+ch-wan@users.noreply.github.com >
Co-authored-by: long10024070 <long10024070@users.noreply.github.com >
Co-authored-by: Kevin Mi <45493463+kevin-mii@users.noreply.github.com >
Co-authored-by: Kevin Mi <mikevin920@yahoo.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: sunxxuns <126995791+sunxxuns@users.noreply.github.com >
Co-authored-by: Long Luong <long.luong2+personal@onemount.com >
2026-10-01 15:59:44 -07:00
Jan Bernlöhr and jbernloehr
80bb3fb651
[Fix] Select the MXFP4 MoE runner for MiMo-V2 packed experts on SM100 ( #41668 )
...
Co-authored-by: jbernloehr <janbernloehr@users.noreply.github.com >
2026-10-01 15:33:30 -07:00
Kan Wu and Kan Wu
a21dcbd320
[sgl-router] Support engines started with --api-key (--worker-api-key) ( #41996 )
...
Co-authored-by: Kan Wu <kan.wu@radixark.ai >
2026-10-01 14:54:59 -07:00
Kan Wu and Kan Wu
d7565215ec
[sgl-router] Route to a specific DP rank inside multi-rank workers (--dp-aware) ( #41974 )
...
Co-authored-by: Kan Wu <kan.wu@radixark.ai >
2026-10-01 14:50:59 -07:00
Martin Hua
52d5faed5b
[Bugfix] Llama4 local attention: read page ids from the graph tables in CUDA-graph capture/replay (page_size > 1) ( #38187 )
2026-10-01 14:42:22 -07:00
bf2d686a82
Add triton autotune on the Mamba2 SSD kernels ( #39130 )
...
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com >
2026-10-01 14:25:37 -07:00
Jiajun Li and Claude Fable 5.1
dcf4e18283
fix(nccl): disable graph buffer registration for pausable graph pools ( #40648 )
...
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com >
2026-10-01 12:34:23 -07:00
Shangming Cai and Claude Fable 5
a71c7d19e7
[Router] Document peer bootstrap: flags, RBAC, downward API, probe implications ( #42049 )
...
Co-authored-by: Claude Fable 5 <noreply@anthropic.com >
2026-10-02 02:20:36 +08:00
3f4bfda13c
[Bugfix] Fix DeepSeek V4 Pro TP24 vocabulary-padding failure on Hopper ( #31801 )
...
Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com >
Co-authored-by: Po-Han Huang <pohanh@nvidia.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-01 11:11:18 -07:00
a0fcebb3bb
NIXL: Use stride desc API ( #37827 )
...
Signed-off-by: Ilia Yastrebov <iyastrebov@nvidia.com >
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com >
Co-authored-by: ishandhanani <82981111+ishandhanani@users.noreply.github.com >
2026-10-01 10:53:00 -07:00
jain-ria
fb92ba1127
[Rust] Wire the gRPC server into the frontend lifecycle ( #41964 )
...
Signed-off-by: jain-ria <riajain@NVIDIA.com >
2026-10-01 10:44:56 -07:00
Joe and Alex Nails
f03a183719
Fix Kimi-K3 MLA output gate dispatch on non-CUDA devices ( #36406 )
...
Co-authored-by: Alex Nails <alex.nails@radixark.ai >
2026-10-01 08:53:31 -07:00
Joe and Alex Nails
3031091c36
[Kimi-K3] Guard optimized paths by platform ( #40269 )
...
Co-authored-by: Alex Nails <alex.nails@radixark.ai >
2026-10-01 08:53:05 -07:00
Артем Савкин
1bbce57d9e
[NPU] [Diffusion] Fix NPU multimodal-gen CI ( #41528 )
2026-10-01 18:02:37 +03:00
41cbe65de0
[sgl-router] Enforce compatible PD version groups ( #41613 )
...
Co-authored-by: Kan Wu <kan.wu@radixark.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Kan Wu <kan.wu@MacBook-Pro.local >
Co-authored-by: Shangming Cai <csmthu@gmail.com >
2026-10-01 20:24:20 +08:00
3b2ad1c6ae
[sgl-router] Fail fast and cancel decode when prefill fails ( #41612 )
...
Co-authored-by: Kan Wu <kan.wu@radixark.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Shangming Cai <csmthu@gmail.com >
2026-10-01 19:35:56 +08:00
Aurick Qiao and Aurick Qiao
1a0250c5b6
[Disagg] Validate state strides before Mooncake transfers ( #41607 )
...
Co-authored-by: Aurick Qiao <6137920+aurickq@users.noreply.github.com >
2026-10-01 18:47:41 +08:00
d669efba84
[AMD][V4.1][*/N] Greedy dspark draft/accept under SGLANG_SIMULATE_ACC_LEN ( #41981 )
...
Co-authored-by: wunhuang <wunhuang@amd.com >
Co-authored-by: Cursor <cursoragent@cursor.com >
2026-10-01 03:35:28 -07:00
46637965ab
[sgl-router] Allow load-only routing without a tokenizer ( #41611 )
...
Co-authored-by: Kan Wu <kan.wu@radixark.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Shangming Cai <csmthu@gmail.com >
2026-10-01 18:30:35 +08:00
3c4e218779
[AMD][V4.1][*/N] OPUS sparse prefill on gfx950 through layout conversion ( #42017 )
...
Co-authored-by: HAI <hixiao@gmail.com >
Co-authored-by: thomawan <thomawan@amd.com >
2026-10-01 03:21:25 -07:00
kk and wunhuang
f45c004e18
[AMD][V4.1][*/N] Build DSpark draft metadata inside the CUDA graph on ROCm ( #42014 )
...
Co-authored-by: wunhuang <wunhuang@amd.com >
2026-10-01 02:17:13 -07:00
黄孝君
7be0e85658
[CI][NPU][Diffusion] Bump ascend consistency GT to the CANN 9.1.0 baseline ( #41500 )
2026-10-01 17:13:28 +08:00
Xinyi Song
73ba6513f4
[AMD][V4.1][*/N] Switch the fp8 dense GEMMs on gfx950 to aiter's MXFP8 GEMM ( #41970 )
2026-10-01 02:04:39 -07:00
Xinyi Song
0224fcbd1c
[AMD][V4.1][*/N] Route low-ratio indexer and candidate-block top-k through top-k v2 ( #41947 )
2026-10-01 01:56:31 -07:00
Cheng Wan
5fab1f0f51
[Fix] Let the aiter DCP ASM decode test run without aiter ( #42018 )
2026-10-01 16:55:15 +08:00
Thomas Wang
69240ba7d6
[AMD][V4.1][*/N] Pick the gfx950 wo_a batched gemm tile by row count ( #41971 )
2026-10-01 01:52:39 -07:00
Thomas Wang
ffb134188b
[AMD] Update aiter version for v41 ( #42023 )
2026-10-01 01:49:45 -07:00
Shunkangz
266d9d1fa7
[Qwen3.8 CP 1/4] Context parallelism for QSA attention and the sparse indexer ( #39721 )
2026-10-01 01:28:55 -07:00
b51d4a04f0
[DFlash] Keep grouped-conv taps inside each request's block so NaN/Inf cannot leak across requests ( #41736 )
...
Co-authored-by: Rahul Chalamala <22563365+rchalamala@users.noreply.github.com >
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-01 00:18:37 -07:00
Mohammad Miadh Angkad and Mohammad Angkad
42f2e4fcb7
[DSA] Stop forcing the per-step CPU seq_lens sync for the k-pool indexer ( #41987 )
...
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai >
2026-10-01 00:16:28 -07:00
DarkSharpness
3ed6367d3f
[DSV4/DSA] Name the FlashMLA KV format and drop the V4.1 support probe ( #41337 )
2026-10-01 00:02:18 -07:00
d010e50ad6
[PPU][1/N] CI: Add backend registration and runner preflight ( #39788 )
...
Co-authored-by: Yihao Wang <42559837+AgainstEntropy@users.noreply.github.com >
Co-authored-by: Alex Nails <alex.nails@radixark.ai >
2026-10-01 00:01:17 -07:00
amd-danli103
9c3262d855
[AMD][DSV4] feat: enable PD-disagg with fp8 unified_kv on gfx950 ( #39166 )
2026-09-30 23:42:31 -07:00
315080a0de
[sgl-router] Exclude portless prefills and retry bootstrap discovery ( #41610 )
...
Co-authored-by: Kan Wu <kan.wu@radixark.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Kan Wu <kan.wu@MacBook-Pro.local >
Co-authored-by: Shangming Cai <csmthu@gmail.com >
2026-10-01 14:32:14 +08:00
Shangming Cai and Claude Fable 5
c78fa09290
[Router] Drop the snapshot client's redirect-following fallback ( #42010 )
...
Co-authored-by: Claude Fable 5 <noreply@anthropic.com >
2026-10-01 14:31:53 +08:00
Eric Zhang
947fee9384
[PD] Fix block scale registration for mixed full/SWA KV dtypes ( #41924 )
2026-10-01 14:25:04 +08:00
Xiaoyu Zhang and kernel-design-agents
77b64e353f
[KDA+Kimi K3] Speed up LTX-2.3 QK norm and split RoPE on H200 ( #41459 )
...
Co-authored-by: kernel-design-agents <334018530+kernel-design-agents@users.noreply.github.com >
2026-10-01 14:21:31 +08:00
Peng Wu
11500701f1
[DSV4] Reserve the FULL logical page in the c4 indexer pool like its KV pool ( #41163 )
2026-09-30 23:19:27 -07:00
Shangming Cai and Claude Fable 5
047be72a8c
[Router] Harden peer-bootstrap edges: refuse redirects and metric docs ( #41999 )
...
Co-authored-by: Claude Fable 5 <noreply@anthropic.com >
2026-10-01 14:15:42 +08:00
chien-an-chen
99c9d65b32
[AMD] AITER MLA DCP decode ASM path (opt-in) ( #40750 )
2026-09-30 22:47:55 -07:00
billishyahao
dc7defd294
[Kimi-K3][DCP][DSpark] fix eager mode nonetype crash due to no extend_prefix_lens ( #38991 )
2026-09-30 22:35:25 -07:00
ashwini rathi
86807b87dd
[XPU] Fix test_qsa_indexer_xpu after defer_expansion change ( #41915 )
2026-09-30 22:34:10 -07:00
Ke Bao
5733bcaad3
Declare HiCache host pools for DSA indexer ( #40913 )
2026-10-01 13:08:26 +08:00
f02828ea39
fix(dsv4): ship the fp8 unified_kv rope pool through the direct external linker ( #40471 )
...
Co-authored-by: Claude Opus 5 <noreply@anthropic.com >
Co-authored-by: HaiShaw <hixiao@gmail.com >
Co-authored-by: billishyahao <bill.he@amd.com >
2026-09-30 21:46:56 -07:00
Mohammad Miadh Angkad and Mohammad Angkad
40e1bb0f36
[Docs] Add NVIDIA NVFP4 checkpoint to GLM-5.3-Flash cookbook ( #41986 )
...
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai >
2026-10-01 12:15:30 +08:00
3b6d37a56e
[Router] Gate readiness on peer bootstrap, and make it observable (13/13) ( #40699 )
...
Co-authored-by: Kangyan Zhou <kangyan.zhou@radixark.ai >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
Co-authored-by: Shangming Cai <csmthu@gmail.com >
2026-10-01 11:55:46 +08:00
ebdeba2fee
[Fix] Emit Responses reasoning content-part events ( #41789 )
...
Co-authored-by: ai-jz <ai-jz@users.noreply.github.com >
Co-authored-by: Yuwei An <ayw.sirius19@gmail.com >
Co-authored-by: Xinyuan Tong <xinyuantong.cs@gmail.com >
Co-authored-by: Xinyuan Tong <115166877+JustinTong0323@users.noreply.github.com >
2026-10-01 11:15:24 +08:00
Mohammad Miadh Angkad and Mohammad Angkad
99c59a2ea6
Drop the GLM-5.3-Flash breakable prefill chunk-size pin ( #41837 )
...
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai >
2026-09-30 19:26:58 -07:00
Shuwen Wang
bb62f125aa
[HiSparse] fix: return HiSparse slots on PD decode waiting abort and DeepSeek V4 free ( #41346 )
2026-10-01 10:24:51 +08:00
Shuwen Wang
c9d6c966d0
[HiSparse] fix: reject speculative decoding and mixed chunk at startup ( #41347 )
2026-10-01 10:24:15 +08:00
stanleys12 and Claude Opus 5
3fb721917d
[Fix] Require admin auth for /load_lora_adapter_from_tensors and /set_trace_level ( #41322 )
...
Co-authored-by: Claude Opus 5 <noreply@anthropic.com >
2026-10-01 10:23:12 +08:00
Shuwen Wang
b62b21e7f4
[HiCache][PD] fix: keep decode load-back polling out of rank-local collectives ( #41393 )
2026-10-01 10:22:30 +08:00
zijie and Zijie Chen
0148d9a2fd
[AMD] Fix MiniMax-M3 EAGLE3 verification on ROCm ( #41497 )
...
Co-authored-by: Zijie Chen <300606707+zijiecode@users.noreply.github.com >
2026-09-30 19:05:49 -07:00
Mick and Mick Qian
2ab1b537f8
[diffusion] docs: recommend docker for gpu deployments ( #41966 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
2026-10-01 09:52:31 +08:00
36330b8c08
[Router] k8s e2e for cache-aware peer bootstrap (12/13) ( #40698 )
...
Co-authored-by: Kangyan Zhou <kangyan.zhou@radixark.ai >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
Co-authored-by: Shangming Cai <csmthu@gmail.com >
2026-10-01 09:42:59 +08:00
Shafeeq Ibraheem and Yanbin Jiang
b4df214948
Fix LoRA usage-counter accounting across request lifecycle paths ( #31808 )
...
Co-authored-by: Yanbin Jiang <jybsuper@gmail.com >
2026-09-30 18:37:53 -07:00
Sage and ishandhanani
baae0195d9
[ci] fix renderer image publish ( #41936 )
...
Signed-off-by: Sage Ahrac <sagiahrak@gmail.com >
Co-authored-by: ishandhanani <82981111+ishandhanani@users.noreply.github.com >
2026-09-30 18:27:11 -07:00
jain-ria
ebccddce19
[Rust] Implement the Rust gRPC adapter for native generation ( #41766 )
...
Signed-off-by: jain-ria <riajain@NVIDIA.com >
2026-09-30 18:17:42 -07:00
Yu-Yun Chang and kk
6af651ea0c
[AMD][Quark] Serve the Kimi-K3 MXFP4 checkpoint on ROCm ( #40811 )
...
Co-authored-by: kk <43161300+kkHuang-amd@users.noreply.github.com >
2026-09-30 18:14:15 -07:00
leonardHONG and lianghougong
a702970ecb
[Perf] Use FA4 by default for MiMo on SM100 ( #41886 )
...
Co-authored-by: lianghougong <lianghouhong@xiaohongshu.com >
2026-10-01 09:05:12 +08:00
Sage
b37e6f79fc
[ci] grant ci permissions to sagearc ( #41935 )
...
Signed-off-by: Sage Ahrac <sagiahrak@gmail.com >
2026-09-30 17:21:41 -07:00
Liangsheng Yin
63321e3693
[Scheduler] Move internal-state readback and updates into a collaborator ( #41950 )
2026-09-30 16:40:57 -07:00
Liangsheng Yin
517cfa218c
[Test] Prune redundant scheduler unit tests ( #41952 )
2026-09-30 16:30:18 -07:00
Jan Bernlöhr and jbernloehr
488869c2d0
[Fix] Keep MiMo-V2 processor available without TorchCodec ( #41667 )
...
Co-authored-by: jbernloehr <janbernloehr@users.noreply.github.com >
2026-09-30 15:28:16 -07:00
826f15bf9c
[Fix] Preserve inline instructions for Responses and Kimi K3 Messages ( #40077 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: hnyls2002 <lsyincs@gmail.com >
Co-authored-by: Liangsheng Yin <hnyls2002@gmail.com >
2026-09-30 15:20:52 -07:00
Sage
8e4bcc5ca9
[rust-renderer] sglang-processor lib ( #41896 )
...
Signed-off-by: Sage Ahrac <sagiahrak@gmail.com >
2026-09-30 15:09:35 -07:00
f050ab194a
fix(spec): enable Qwen3.5 EAGLE3 capture and streaming overlap ( #41154 )
...
Co-authored-by: Yuhao Wang <59914901+jin-chuan@users.noreply.github.com >
Co-authored-by: Liangsheng Yin <95566987+hnyls2002@users.noreply.github.com >
2026-09-30 14:41:46 -07:00
Cheng Wan
a9c97c9f69
[Feature] Add --attn-dp-size and deprecate --enable-dp-attention ( #41818 )
2026-09-30 13:34:18 -07:00
Jason Mancuso and hnyls2002
201c4b9b2a
[RL, Spec] Introduce top-p mask capture for spec and add DFlash/DSpark impl ( #34201 )
...
Co-authored-by: hnyls2002 <lsyincs@gmail.com >
2026-09-30 13:32:26 -07:00
Cheng Wan
986cb8268b
[Refactor] Enter draft TP scopes by attention ownership and drop ModelRunner.tp_group ( #41817 )
2026-09-30 13:28:50 -07:00
Cheng Wan
e7f6a99313
[Refactor] Add make_pp_layers so models stop handing their PP position down ( #41816 )
2026-09-30 13:28:34 -07:00
Cheng Wan
00659ac099
[Refactor] Stop passing models' layers the placement they already read ( #41815 )
2026-09-30 13:28:17 -07:00
Cheng Wan
c7fa37a5ca
[Refactor] Let FusedMoE's weight-loading helpers read the layer's MoE-TP rank ( #41814 )
2026-09-30 13:27:58 -07:00
Cheng Wan
499e86db98
[Refactor] Keep only the TP and PP groups on the model runner ( #41813 )
2026-09-30 13:27:31 -07:00
Cheng Wan
a09aedf1ca
[Refactor] Drop model placement attributes and parameters nothing reads ( #41812 )
2026-09-30 13:27:11 -07:00
Cheng Wan
b0cc246ff8
[Refactor] Drop placement values nothing reads ( #41811 )
2026-09-30 13:26:51 -07:00
Cheng Wan
4f3ee94d25
[Fix] Make the Solar model constructible and runnable ( #41810 )
2026-09-30 13:26:32 -07:00
Cheng Wan
a659e6461f
[Fix] Size the prefill delayer's gather buffer to its TP group ( #41809 )
2026-09-30 13:26:12 -07:00
Cheng Wan
ecc0f0dec2
[Fix] Pass the draft's attention ownership to DFLASH's eager LiLiCorr scope ( #41808 )
2026-09-30 13:25:54 -07:00
Cheng Wan
e289989933
[Fix] Shard MoE WNA16 and Quark INT4-FP8 weights by the MoE placement ( #41807 )
2026-09-30 13:25:38 -07:00
Cheng Wan
3f20738846
[Fix] State the configured MoE-DP width in the weight-cache fingerprint ( #41806 )
2026-09-30 13:25:19 -07:00
Cheng Wan
e27b6e0971
[Fix] Read TensorCast's WORLD placement under its current names ( #41805 )
2026-09-30 13:25:03 -07:00
YAMY
4407a21f3c
[Spec][PP] Fix hybrid recurrent-state commit and micro-batch pairing under PP x speculative decoding ( #40001 )
2026-09-30 13:05:39 -07:00
547286bb5a
[Deps] Bump transformers to 5.17.0 ( #39012 )
...
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai >
Co-authored-by: Xinyuan Tong <xinyuantong.cs@gmail.com >
2026-09-30 12:52:02 -07:00
Mohammad Miadh Angkad and Mohammad Angkad
036a3b3e14
[Test] Lower the SM120 NVFP4 KV GSM8K threshold to 0.60 ( #41799 )
...
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai >
2026-09-30 12:50:42 -07:00
Mohammad Miadh Angkad and Mohammad Angkad
eae808903f
[Deps] Bump FlashInfer to 0.7.0.post1 ( #40709 )
...
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai >
2026-09-30 12:42:19 -07:00
4dfc0ac131
[BugFix] Pass token-major Q/K tensors from Gemma-3 to RadixAttention ( #37984 )
...
Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com >
Co-authored-by: Po-Han Huang <pohanh@nvidia.com >
2026-09-30 12:05:41 -07:00
Kevin Mi and Kevin Mi
3a398442bf
dsv4.1-amd: serve DeepSeek-V4.1 on gfx950 ( #41308 )
...
Co-authored-by: Kevin Mi <kevin.mi@radixark.ai >
2026-09-30 11:47:14 -07:00
9f9f9e702a
[HiCache] Attribute buffer-mode storage hits against the joint device match ( #41758 )
...
Co-authored-by: merrymercy <merrymercy@users.noreply.github.com >
Co-authored-by: xiezhq-hermann <xiezhq-hermann@users.noreply.github.com >
2026-09-30 11:46:30 -07:00
zijiexia
d26ec4690d
docs: remove unreliable DeepWiki badge ( #41851 )
2026-09-30 10:51:36 -07:00
Shuwen Wang and Shangming Cai
aec4b799e6
[PD][Mamba] fix: free the COW mamba slot of decode requests dropped before preallocation ( #41451 )
...
Co-authored-by: Shangming Cai <csmthu@gmail.com >
2026-09-30 17:50:44 +00:00
jain-ria
21ed9fbf44
[Rust] Extract shared runtime.v1 protobuf bindings ( #41726 )
...
Signed-off-by: jain-ria <riajain@NVIDIA.com >
2026-09-30 10:13:08 -07:00
Sohom chakraborty and Shuwen Wang
28c5e7f5cb
[Fix] Stop PD-decode queue_time from counting decode time before a retraction ( #41380 )
...
Signed-off-by: Sohom Chakraborty <sohomchakraborty.iitkgp@gmail.com >
Co-authored-by: Shuwen Wang <47200617+alphabetc1@users.noreply.github.com >
2026-09-30 23:17:03 +08:00
bd66ce343e
[diffusion] nightly: measure every framework with one client end-to-end methodology ( #41689 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-09-30 19:52:20 +08:00
4b3f63ec70
[Router] Prove a bootstrapped replica answers like the one it copied (11/13) ( #40697 )
...
Co-authored-by: Kangyan Zhou <kangyan.zhou@radixark.ai >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-09-30 19:19:26 +08:00
4c930125c7
[Router] Ask the fleet when a graft's splice goes unwitnessed (10/13) ( #40696 )
...
Co-authored-by: Kangyan Zhou <kangyan.zhou@radixark.ai >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-09-30 18:59:21 +08:00
Cheng Wan
53225fadf4
[Fix] Stop the namespace census at the leaf a read names ( #41838 )
2026-09-30 03:27:51 -07:00
f8500474c9
[Router] Share one fleet-wide fetch across a discovery burst (9/13) ( #40695 )
...
Co-authored-by: Kangyan Zhou <kangyan.zhou@radixark.ai >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-09-30 18:07:51 +08:00
jacky.cheng
47dcde70f4
[Docs] Add MI355X FP8 agentic recipe to the Qwen3.5 cookbook ( #41849 )
2026-09-30 17:44:19 +08:00
Yuan Luo and luoyuan.luo
57c81258d4
[KDA] Enable the ptx_kda prefill backend on SM100 (B200 / GB200) ( #41445 )
...
Co-authored-by: luoyuan.luo <luoyuan.luo@antgroup.com >
2026-09-30 17:24:51 +08:00
Yuan Luo and luoyuan.luo
c0296186b7
[KDA] Fix ptx_kda prefill NaN without a gate lower bound and workspace growth ( #41572 )
...
Co-authored-by: luoyuan.luo <luoyuan.luo@antgroup.com >
2026-09-30 17:23:54 +08:00
paulzhang-tm and Ke Bao
8055ccd2cd
[Linear Attention] Expose GDN/KDA prefill hooks and auxiliary cache accounting ( #40227 )
...
Co-authored-by: Ke Bao <ispobaoke@gmail.com >
2026-09-30 17:19:02 +08:00
EdwardXuy and EdwardXuy
9b7464c8dd
ci: temporarily disable A5 (950) nightly job during power maintenance ( #41856 )
...
Co-authored-by: EdwardXuy <EdwardXuy@users.noreply.github.com >
2026-09-30 17:17:39 +08:00
jiaryang
e9b0d0c0a3
[AMD] Gate flashinfer and TRT-LLM DSA paths on CUDA ( #40911 )
2026-09-30 02:10:02 -07:00
Michael
d7cce54885
[AMD] Resolve QSA packed-varlen decode to aiter on HIP ( #41513 )
2026-09-30 02:01:15 -07:00
amote-i
964c45cf31
[NPU] [DOC]: add Kimi-K3 NPU PD disaggregation recipes ( #41717 )
2026-09-30 16:42:46 +08:00
4d4d3f2e5c
[Router] Sweep the fleet for a peer snapshot and hand it to the pump (8/13) ( #40694 )
...
Co-authored-by: Kangyan Zhou <kangyan.zhou@radixark.ai >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-09-30 16:34:12 +08:00
45c8ddddd3
[diffusion] feat: check the final MP4 in-process instead of spawning ffprobe ( #41825 )
...
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-09-30 16:26:39 +08:00
WenhaoZhang
0931a72eb2
[diffusion] feat: support MiniMax H3 RL ( #34365 )
2026-09-30 16:23:56 +08:00
Michael
b87a241a6f
[AMD] Add Qwen3.8-Flash-Next-FP8 nightly validation ( #36901 )
2026-09-30 00:54:36 -07:00
682f4ef32f
[ROCm] Select DSA indexer top-k wave size by arch (wave32 on gfx1250) ( #41687 )
...
Co-authored-by: Cursor <cursoragent@cursor.com >
Co-authored-by: Thomas Wang <thomawan@amd.com >
2026-09-30 00:47:04 -07:00
Shuwen Wang
51cae5f303
[HiCache][PD] fix: clamp the decode restore to the prefix promised to prefill ( #41450 )
2026-09-30 15:26:15 +08:00
heziiop and sglang-npu-bot
ec0006673a
[NPU] Add decode context parallel support for dsa models ( #37787 )
...
Co-authored-by: sglang-npu-bot <sglangnpu@163.com >
2026-09-30 15:22:16 +08:00
Shangming Cai
8c75ad9e62
chore: bump mooncake version to 0.3.13.post1 ( #41678 )
2026-09-30 15:16:54 +08:00
Zhaoyi Li
3b537e96e2
[AMD][DI][CI] Leave the GLM-5.2 MTP decode room to load its Triton kernels ( #41135 )
2026-09-30 00:02:42 -07:00
Zhaoyi Li
25de118ae3
[AMD][DI][CI] Give the dsv4pro-fp4 MTP decode more memory headroom ( #41238 )
2026-09-29 23:57:25 -07:00
Zhaoyi Li
4207f1ae23
[AMD][DI][CI] Record the commit the MI355X nightly tested ( #41099 )
2026-09-29 23:55:38 -07:00
Zhaoyi Li
f85ca901e2
[AMD][DI][CI] Fix four failures in the MI355X disaggregation nightly ( #41025 )
2026-09-29 23:54:40 -07:00
24ee67d2fa
Readme refresh ( #41787 )
...
Co-authored-by: zijiexia <37504505+zijiexia@users.noreply.github.com >
Co-authored-by: Zijie Xia <zijie.xia@radixark.ai >
2026-09-29 23:48:38 -07:00
eb9c9ee99d
[Qwen4-Exp] Optional fp8 (e4m3) storage for the compressed QSA indexer cache ( #39614 )
...
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com >
Co-authored-by: YAMY1234 <74099316+YAMY1234@users.noreply.github.com >
2026-09-29 22:11:46 -07:00
Qiaolin Yu
88f95f4b87
[qwen next] Fuse QSA KV preparation and sparse block expansion ( #40972 )
2026-09-29 21:54:34 -07:00
Qiaolin Yu
b660100c00
[qwen 3.8 next] Fuse QSA graph replay metadata across draft steps ( #41173 )
2026-09-29 21:52:28 -07:00
JohnQinAMD and Claude Opus 5.5
964fd6af92
[ROCm] GLM-5.2: gfx950 four-kernel fused DSA indexer decode path ( #38583 )
...
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
2026-09-29 21:52:00 -07:00
hhhh1252023
be66c1b460
[NPU][CI] Fix stale multimodal-gen job name in fast-fail health check ( #41824 )
2026-09-30 12:39:23 +08:00
Cheng Wan
d41904f05e
[Refactor] Build seven more decoders from stage boundaries ( #41779 )
2026-09-29 21:16:45 -07:00
Cheng Wan
113e6a1496
[Refactor] Simplify layer boundary internals ( #41778 )
2026-09-29 21:15:58 -07:00
Wenfei (Charles) Qi
7fb85a9d88
[MLX] Fix startup after logical token capacity change ( #41314 )
2026-09-29 21:15:56 -07:00
Cheng Wan
9ee065e49a
[Fix] Step-3.5: build the shared expert without TP under all-to-all MoE backends ( #41754 )
2026-09-29 21:11:37 -07:00
Cheng Wan
faa5064f6e
[Refactor] Send producer-written residual streams through to_pp ( #41753 )
2026-09-29 21:11:02 -07:00
Cheng Wan
843f50f5c1
[Fix] Reduce the Sarvam dense MLP through its row-parallel projection ( #41752 )
2026-09-29 21:10:17 -07:00
Cheng Wan
ea855f0daf
[Refactor] Leave decoder stacks through final_norm(skip_empty=True) ( #41751 )
2026-09-29 21:08:19 -07:00
Cheng Wan
01b5f572b7
[Refactor] Rename layer boundary protocols, contracts and helpers ( #41750 )
2026-09-29 21:05:48 -07:00
cb3ba2f2b7
[XPU] Add dynamic_expert_bias and track_state support ( #39618 )
...
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Singh <rohitsi2@iil-login.iind.intel.com >
Co-authored-by: Singh <rohitsi2@iil-gnrap01.iind.intel.com >
Co-authored-by: Ma Mingfei <mingfei.ma@intel.com >
2026-09-30 12:05:35 +08:00
Alison Shao
8854857a99
[CI] Bound maintenance API requests to 30 seconds ( #41800 )
2026-09-29 20:02:02 -07:00
Mohammad Miadh Angkad and Mohammad Angkad
7b30ff26ee
[CI] Evaluate the maintenance gate with github-script instead of jq ( #41625 )
...
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai >
2026-09-29 18:47:51 -07:00
fc9bdc8a3e
[MoE] Keep the moe_align_block_size pad fill inside sorted_token_ids ( #41759 )
...
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com >
Co-authored-by: oulgen <oulgen@users.noreply.github.com >
2026-09-29 17:43:17 -07:00
Baizhou Zhang
e5cec303ae
[Fix] Restore deferred layer dumps and pin SentencePiece for InternVL ( #41777 )
2026-09-29 17:28:30 -07:00
sglang-bot
cae69be584
chore: bump sgl-kernel version to 0.4.8 ( #41784 )
2026-09-29 17:19:21 -07:00
DarkSharpness
57d4adeb83
[Kernel] Expose FlashMLA kv_format in sgl-kernel sparse decode ( #41336 )
2026-09-29 17:16:52 -07:00
Jason Mancuso
ba3e04c063
[CI] Grant CI permissions to four contributors ( #41775 )
2026-09-29 16:42:01 -07:00
Cheng Wan
8d2d874758
[Fix] Release consumed residual contributions ( #41749 )
2026-09-29 16:38:05 -07:00
Jiajun Li and Shi Dong
3d4953839c
[Fix] MoE: require TopK layer_id to ensure routed expert captures ( #40980 )
...
Co-authored-by: Shi Dong <hello@dongshi.me >
2026-09-29 16:29:27 -07:00
Jiajun Li
f0e4001930
[Fix] Allocate SWA KV pools inside the memory-saver region ( #40979 )
2026-09-29 16:29:02 -07:00
Yuwei An
d3f8a8f4f5
[Spec/Prefill Coordination] Dspark support ( #41623 )
2026-09-29 16:08:01 -07:00
Liangsheng Yin
211b1d9784
[Refactor] Share PD routing fields across request models ( #41756 )
2026-09-29 15:11:18 -07:00
37a47737c8
[Perf] Tune SM90 GDN recurrent verify launch for small batches ( #41486 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: hnyls2002 <lsyincs@gmail.com >
Co-authored-by: Liangsheng Yin <hnyls2002@gmail.com >
2026-09-29 15:03:33 -07:00
Mohammad Miadh Angkad and hnyls2002
4885b563c3
[Test] Demote PD test RDMA openability check to a warning ( #41681 )
...
Co-authored-by: hnyls2002 <lsyincs@gmail.com >
2026-09-29 14:52:31 -07:00
84523d6785
[Spec] Reuse K3 auxiliary outputs across decode CUDA graph sizes ( #40159 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: hnyls2002 <lsyincs@gmail.com >
2026-09-29 14:24:51 -07:00
jain-ria
46beacf13d
[grpc rust] Define semantic frontend contract ( #41709 )
...
Signed-off-by: jain-ria <riajain@NVIDIA.com >
2026-09-29 13:40:04 -07:00
James Liu and Claude Opus 5.5
77a81996ce
[PD] Preserve bootstrap metadata in native Messages requests ( #39711 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-09-29 13:32:54 -07:00
3c4653b6a3
[Metrics] Fix PD latency histogram accounting ( #39706 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: hnyls2002 <lsyincs@gmail.com >
Co-authored-by: Liangsheng Yin <hnyls2002@gmail.com >
2026-09-29 13:12:09 -07:00
zijiexia and Claude Sonnet 5.5
f884231f5a
[Docs] Add IQuestLab card logo to cookbook landing page ( #41746 )
...
Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com >
2026-09-29 12:47:21 -07:00
51ace48387
[Metrics] Add request-level TPOT histogram (sglang:request_time_per_output_token_seconds) ( #40275 )
...
Co-authored-by: Gilford Ting <gilford@modal.com >
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com >
2026-09-29 12:20:22 -07:00
ishandhanani
9a16e47437
chore: grant jain-ria CI permissions ( #41728 )
2026-09-29 11:56:51 -07:00
ashwini rathi
bf80732b92
Fix pip install: exclude multimodal_gen/.claude symlink from package data ( #41703 )
2026-09-29 11:35:11 -07:00
8f2a9638f7
[metrics] Fix non-streaming TTFT by flushing the first output through detokenization ( #38600 )
...
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: hnyls2002 <lsyincs@gmail.com >
2026-09-29 11:24:52 -07:00
Colin Z
875dd41e6f
[AMD] Fix Load and Inference of MLA models with Quark PTPC FP8 attention on ROCm ( #28734 )
2026-09-29 11:20:10 -07:00
Chunan Zeng
affc602f51
[CI] Extend DeepGEMM GB300 validation timeout to three hours ( #41720 )
2026-09-29 10:57:20 -07:00
ced5e9fe49
[Kimi] Enable GB300 TP4 and GB200/GB300 TP16 SP collectives ( #35330 )
...
Co-authored-by: Ajit Mistry <ajmistry@amistry.nvidia.com >
Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com >
2026-09-29 10:18:11 -07:00
CYJiang
fa090f7755
[MUSA] Fix fused MoE GEMV registration and torchada pin ( #41444 )
2026-09-29 09:51:08 -07:00
ashwini rathi
98fce73d5b
[XPU] publish nightly docker image with sgl-kernel-xpu built from main ( #41650 )
2026-09-29 20:09:53 +08:00
4a4e1ce014
[Router] Hold a booting rank's batches and graft a snapshot on the pump (7/13) ( #40693 )
...
Co-authored-by: Kangyan Zhou <kangyan.zhou@radixark.ai >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-09-29 20:07:37 +08:00
8d1643b21e
[Router] Vet a peer's snapshot before it may touch the tree (6/13) ( #40692 )
...
Co-authored-by: Kangyan Zhou <kangyan.zhou@radixark.ai >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-09-29 19:58:11 +08:00
8b3a4cad8b
[Router] Track a booting rank's bootstrap and fetch a peer's snapshot (5/13) ( #40691 )
...
Co-authored-by: Kangyan Zhou <kangyan.zhou@radixark.ai >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-09-29 19:06:13 +08:00
7bcfcf5aa0
[Router] Watch EndpointSlices for sibling router replicas (4/13) ( #40690 )
...
Co-authored-by: Kangyan Zhou <kangyan.zhou@radixark.ai >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-09-29 18:56:32 +08:00
c5a38d3998
[sgl-router] Add --tokenizer-backend fast and --tokenizer-l1-cache-mb ( #41268 )
...
Co-authored-by: Kan Wu <kan.wu@MacBook-Pro.local >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-09-29 18:55:58 +08:00
Michael and Claude
5828cd4927
[AMD] Add GLM-5.3 MI30x and MI35x nightly accuracy tests ( #41602 )
...
Co-authored-by: Claude <noreply@anthropic.com >
2026-09-29 02:49:01 -07:00
Rumit Desai and Xiaoyu Zhang
f731e82f09
Fix MUSA detection under torch.compile fullgraph ( #41065 )
...
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com >
2026-09-29 17:26:27 +08:00
Carrie Chen and Brayden Zhong
0e586fd12d
fuse shared experts with routed experts in MegaMoE's DeepGEMM ( #39313 )
...
Co-authored-by: Brayden Zhong <b8zhong@uwaterloo.ca >
2026-09-29 17:25:10 +08:00
c9cb32e310
[AMD] fix kda decode flydsl import ( #40546 )
...
Co-authored-by: Bingxu Chen <bingxche@amd.com >
Co-authored-by: Michael <13900043+michaelzhang-ai@users.noreply.github.com >
2026-09-29 02:20:42 -07:00
Liang Gong
34a1234d21
chore: expose agent skills via .agents directories ( #40515 )
2026-09-29 17:12:28 +08:00
Michael
05817a40c9
[AMD] Fix GLM-5.3 quark MoE MI35x test runner config ( #41464 )
2026-09-29 16:46:52 +08:00