Compare commits

...
588 Commits
Author SHA1 Message Date
minke.yu 8b67636010 Merge b300/integration into main: 2026-11-17 c580 baseline
CI Model Inventory / build-inventory (push) Failing after 41s
PR Benchmark (SMG Components) / Benchmark - Manual Policy (push) Skipped
PR Benchmark (SMG Components) / Benchmark - Request Processing (push) Skipped
PR Test (Arm64) / check-changes (push) Successful in 14s
PR Test (Xeon) / pr-gate (push) Successful in 2s
PR Test (SMG) / k8s-integration (push) Failing after 12m21s
PR Test (Rust extensions) / sglang-mm-unit (push) Failing after 12s
PR Benchmark (SMG Components) / Benchmark Compilation Check (push) Failing after 1m45s
PR Test (Arm64) / pr-gate (push) Successful in 2s
PR Test (sglang-processor) / ${{ github.event.action == 'labeled' && github.event.label.name != 'run-ci' && 'ignored-label-sgl-processor-gate' || 'sgl-processor-gate' }} (push) Successful in 2s
Seed Rust Ext Cache / seed (push) Failing after 1m53s
PR Benchmark (SMG Components) / Benchmark Summary (push) Skipped
PR Test (SMG) / docker-build-test (push) Failing after 18m37s
pr-test-xpu.yml / pr-gate (push) Successful in 2s
PR Test (sglang-processor) / ${{ github.event.action == 'labeled' && github.event.label.name != 'run-ci' && 'ignored-label-sgl-processor-lint' || 'sgl-processor-lint' }} (push) Failing after 10s
PR Test (SMG) / finish (push) Skipped
Lint / lint (push) Failing after 2m40s
pr-test-arm64.yml / pr-gate (push) Successful in 2s
PR Test (XPU) / multimodal-gen-test-1-gpu-xpu (push) Canceled after 0s
PR Test (XPU) / stage-a-test-1-gpu-xpu (push) Canceled after 0s
Release SGLang Kernels / build-rocm-matrix (3.10, ${{ fromJson(inputs.target == 'rocm720' && '["720"]' || inputs.target == 'rocm1000' && '["1000"]' || '["720", "1000"]') }}) (push) Canceled after 0s
Release SGLang Kernels / build-musa43 (43, 3.10) (push) Canceled after 0s
PR Test (Xeon) / build-test (spr1, 1, 0, 4, spr, xeon-spr, stage-a-test-cpu-intel,stage-b-test-cpu-intel) (push) Canceled after 0s
PR Test (Xeon) / build-test (spr2, 3, 2, 4, spr, xeon-spr, stage-a-test-cpu-intel,stage-b-test-cpu-intel) (push) Canceled after 0s
Seed Rust Ext Cache / seed-aarch64 (push) Canceled after 27h16m58s
PR Test (Arm64) / build-test (push) Canceled after 0s
Release SGLang Kernels / release-musa43 (push) Canceled after 0s
PR Test (XPU) / finish (push) Canceled after 0s
Release SGLang Kernels / release-cu130 (push) Canceled after 0s
Release SGLang Kernels / release-rocm720 (push) Canceled after 0s
Release SGLang Kernels / release-rocm1000 (push) Canceled after 0s
PR Test - Build Rust Extensions / Collect Rust Ext (push) Canceled after 0s
PR Test (Rust workspace) / cargo-workspace (push) Failing after 2m27s
PR Test (Rust workspace) / sglang-radix-tree (push) Successful in 2m45s
PR Test (SMG) / unit-tests (push) Failing after 1m47s
PR Test (sglang-processor) / ${{ github.event.action == 'labeled' && github.event.label.name != 'run-ci' && 'ignored-label-sgl-processor-changes' || 'sgl-processor-changes' }} (push) Successful in 14s
PR Test (sgl-router) / changes (push) Successful in 13s
PR Test (sgl-router) / gate (push) Successful in 2s
PR Test (XPU) / pr-gate (push) Successful in 2s
PR Test (Xeon) / build-test (gnr, gnr, xeon-gnr, stage-a-tp-test-cpu-intel) (push) Canceled after 0s
Release SGLang Kernels / build-cu130-matrix (aarch64, 13.0, 3.11, arm-kernel-build-node) (push) Canceled after 0s
Release SGLang Kernels / build-cu130-matrix (x86_64, 13.0, 3.11, x64-kernel-build-node) (push) Canceled after 0s
PR Test - Build Rust Extensions / Restore Rust Ext (push) Successful in 5s
PR Test - Build Rust Extensions / Build Rust Ext (3.12) (push) Canceled after 0s
PR Test (SMG) / build-wheel (push) Failing after 1m38s
pr-test-sgl-processor.yml / pr-gate (push) Successful in 2s
pr-test-sgl-router.yml / pr-gate (push) Successful in 2s
pr-test-xeon.yml / pr-gate (push) Successful in 2s
PR Test (sglang-processor) / ${{ github.event.action == 'labeled' && github.event.label.name != 'run-ci' && 'ignored-label-sgl-processor-test' || 'sgl-processor-test' }} (push) Skipped
PR Test (sgl-router) / tier-1 — lint (push) Failing after 55s
PR Test (sgl-router) / tier-3 — e2e (push) Skipped
PR Test (sglang-processor) / ${{ github.event.action == 'labeled' && github.event.label.name != 'run-ci' && 'ignored-label-sgl-processor-finish' || 'sgl-processor-finish' }} (push) Failing after 1s
PR Test (SMG) / python-unit-tests (push) Skipped
PR Test (SMG) / benchmarks (push) Skipped
PR Test (SMG) / chat-completions (push) Skipped
PR Test (SMG) / chat-completions-4gpu (push) Skipped
PR Test (SMG) / e2e (push) Skipped
PR Test (SMG) / summarize-benchmarks (push) Skipped
PR Test (Xeon) / check-changes (push) Successful in 14s
PR Test (XPU) / check-changes (push) Successful in 20s
PR Test (sgl-router) / tier-2 — build + test (push) Skipped
PR Test (sgl-router) / tier-3 — docker (placeholder) (push) Skipped
PR Test (sgl-router) / tier-3 — k8s integration (push) Skipped
PR Test (sgl-router) / finish (push) Failing after 2s
Promote the validated ds41-gist-exp baseline:
- image recipe-ns-l3ssd-433257c7b-baked-20261115 (SGLang + Mooncake 0.3.14 + L3 SSD)
- P: cache / no-optimistic / fastokens / tokenizer-worker 16 / L3 SSD 4T
- D: fastokens / tokenizer-worker 16
- gateway: normal health/circuit, no P admission caps
- c580 baseline 24.045 rps; c620 and buffer_only results recorded
- Feishu baseline doc and deploy manifests included

This is the base for the next Dynamo layer.
2026-11-17 20:08:08 +08:00
minke.yu 310ad35430 deploy: record 2026-11-17 c580 baseline (D16+fastokens) and experiments 2026-11-17 20:05:11 +08:00
minke.yu bdbd8dffb8 deploy(ds41-gist-exp): P buffer_only + optimistic, D tokenizer 16 2026-11-16 22:03:20 +08:00
minke.yu f363a6ab7b deploy: record 11/16 pre-optimistic stable config 2026-11-16 22:03:11 +08:00
minke.yu e4c7e540be deploy(g9i): T26 qps20 + P queue32 production candidate 2026-11-16 00:42:44 +08:00
minke.yu b8b205b367 deploy(g9i): merged ns+td gateway, 32-request P queue cap (p95 5.70s) 2026-11-15 22:25:33 +08:00
minke.yu b3d6d19c7d deploy(g9i): set regular P queue reject 32s (experiment) 2026-11-15 21:45:51 +08:00
minke.yu 8f53c1fb73 deploy(l3ssd): offload_on_evict=false, 4TB cap, L3 reset script 2026-11-15 20:33:38 +08:00
minke.yu 42f228bfb8 docs(deploy): fresh-instance multi-round tokenizer A/B (B2 TTFT p95 < 5s) 2026-11-15 18:20:53 +08:00
minke.yu 242a7ad23a docs(deploy): record L3 SSD tokenizer A/B (fastokens vs huggingface) 2026-11-15 15:49:56 +08:00
minke.yu 2eb163ad73 fix(deploy): point sglang cache at the shared warm cache 2026-11-15 14:23:52 +08:00
minke.yu d3240c745e deploy(b300): L3 Mooncake SSD baseline with mooncake 0.3.14 2026-11-15 14:10:30 +08:00
minke.yu 433257c7b0 fix(recipe): drop the pre-48928 _fast_prompt_tokenizer fallback stub 2026-11-15 10:53:15 +08:00
leon 7980a09c0a [Perf] DeepSeek-V4.1-Flash: Use segmented fastokens for prompt encoding (#48928)
(cherry picked from commit 219b7e7a688363bac9ef09eacf80e44eef9efe4b)
2026-11-15 10:51:40 +08:00
minke.yu a7c668a936 fix(chat): move namespace-tool flattening validator into ChatCompletionRequest 2026-11-14 14:10:07 +08:00
minke.yu 0c6849e378 chat: flatten namespace tools to ns__function for Chat Completions 2026-11-14 14:09:36 +08:00
minke.yu 85605b8c72 fix(protocol): ResponseTool accepts extra fields (tool_search execution) 2026-11-13 23:25:50 +08:00
minke.yu b6a5d70cba fix(responses): tolerate tools without model_extra in native validation 2026-11-13 23:24:00 +08:00
minke.yu cac60f7c2e fix(protocol): effective_tool_choice preserves namespace/type 2026-11-13 23:22:21 +08:00
minke.yu 68c2af7957 fix(protocol): Function.strict is Optional[bool] for recipe tool schemas 2026-11-13 23:21:04 +08:00
minke.yu 497aeff5f6 fix(recipe): define _fast_prompt_tokenizer fallback on this base 2026-11-13 22:44:35 +08:00
minke.yu 5fa8c23227 vendor: DeepSeek-recipe namespace tools (#46370 + #49625)
Backport the upstream namespace-tool stack onto the B300 recipe base:
- #46370: Function.namespace / ChatMessage.action protocol fields.
- #49625: Responses/Messages native recipe rendering (convert_native),
  tool_search/defer_loading, namespace identity in/out/history replay.

Adapted to this base: skips serving_kimi_k3.py (not present), drops the
Minimax/Beam/wire-only dependencies added between 10-23 and the PR base,
and keeps the existing tool-parsing fallback where the newer
`parses_without_tools()` helper is unavailable.

Also includes the PR's Anthropic tool-reference plumbing and unit tests.
2026-11-13 22:43:37 +08:00
minke.yu f4de9f2141 observability: ignore sub-tolerance CPU>wall skew in accounting anomaly
A 2 ms phase whose CPU clock reads 0.1 us over its wall clock is clock
granularity, not an accounting bug; the anomaly flag fired on nearly every
short phase in the live verification run.

Only flag (and record non_cpu_raw_ms) when the deficit exceeds both 100 us and
1% of the phase wall. Larger real anomalies keep the previous behaviour.
2026-11-12 12:30:58 +08:00
minke.yu 7f42a0806a observability: per-rank request counter for P/D RPS
The frontend sglang:num_requests_total has no dp_rank label (verified from the
series registry), so per-rank RPS was not derivable from any existing metric.

Add sglang:batch_requests_total{phase,dp_rank}, incremented by the scheduler
process that actually completed the request:

  phase=prefill  prefill compute finished and the request entered the PD
                 transfer queue (not a guarantee that the transfer succeeded)
  phase=decode   request finished decoding

Only the canonical scheduler rank reports (attn_tp_rank==0, attn_cp_rank==0,
last PP stage) — every TP/CP rank runs a scheduler and sees the same request,
so an ungated counter would be multiplied by TP*CP in the multiprocess
registry. Both hooks count each request exactly once: the decode path counts
after _handle_finish_state_updated_req and on the sampling-mask abort branch,
skipping rows that finished on an earlier step; the prefill path counts after
the request enters the disagg inflight queue (aborted/retried requests never
reach it). The per-request finished() check is skipped entirely when
SGLANG_BATCH_TELEMETRY is off.
2026-11-12 10:30:56 +08:00
minke.yu 17b922c84f test: prefill_batch_init detail and eviction semantics
Covers the design's unit-level acceptance points:
* disabled/unsampled loops never read the thread clock;
* wall/CPU/non-CPU semantics, unmeasured != zero, partial measurement omits
  non-CPU, busy vs sleeping windows are distinguishable;
* CPU > wall flags an accounting anomaly with a raw negative delta;
* child spans carry the CPU split;
* repeated phases (prefix indices) stay attribute-only, inclusive phases are
  not summed, and eviction demand/shortfall/plan plus Mamba plan/zero results
  are recorded as separate counters.
2026-11-12 10:30:47 +08:00
minke.yu 5ad9ff6a5f observability: prefill_batch_init wall+CPU, sub-phases and eviction coverage
Parent span scheduler.loop.prefill_batch_init records wall + the scheduler
thread's CPU at both existing close points (empty admission and normal
return); the CPU clock is only read when loop detail is enabled and sampled,
through the safe batch_telemetry.thread_cpu_ns() wrapper so a missing clock
degrades to 'not measured' instead of failing the scheduling path.

Fixed children, in call order:
  schedule_batch_init, ready_to_load_host_cache, prepare_for_extend,
  alloc_for_extend (+ prefix_device_indices attribute-only, sampling_batch_info),
  evict_to_free_tokens, evict_for_alloc, prefill_stats (+ pending_tokens),
  mixed_chunk_tail.

* prefill_stats wraps the whole assignment so _get_num_pending_tokens() is
  included; mixed_chunk_tail only exists when that branch is taken.
* prefill-only shape counters (n_reqs/n_new/n_cached) are snapshotted before
  mixed-chunk folding; a mixed batch additionally publishes decode_reqs and
  merged totals under explicit *_merged_* names.
* eviction is timed at the call site (outside the allocator wrapper) so
  rank-consensus waits are included, and covers base, SWA (x2), UnifiedSWA
  (x3) and the extend Mamba reclaim through evict_for_alloc_observed().
  Demand, shortfall and plan are separate counters; Mamba plan and realized
  counts are both recorded (including zero).

All of it is gated by detail on a sampled loop.
2026-11-12 10:30:38 +08:00
minke.yu fec10023e0 observability: optional thread-CPU recording for detail phases
Adds an optional thread CPU clock next to the existing wall clock:

* detail_begin/end record time.thread_time_ns() only when detail is on and the
  loop is sampled; detail_add() accepts an explicit thread_cpu_ns for callers
  that already hold absolute start/end points (the prefill_batch_init parent).
* phase records carry [wall, count, start_wall, end_wall, cpu_sum, cpu_count].
  thread_cpu_ms and thread_cpu_measured_count are emitted whenever any
  occurrence was measured; non_cpu_ms only when every occurrence was, so a
  partial measurement never attributes unmeasured wall to non-CPU time. A
  phase whose CPU exceeds its wall keeps the clamped value, adds
  non_cpu_raw_ms and accounting_anomaly, and bumps a per-loop anomaly count.
* new detail_attr_sum()/detail_attr_setdefault() accumulate low-cardinality
  numeric loop attributes (eviction tokens, free-token snapshots).
* prefix_device_indices joins the attribute-only prefixes so per-request
  repeats never create spans.
* child spans carry phase_thread_cpu_ms / phase_cpu_measured_count /
  phase_non_cpu_ms, so the CPU split is visible without the loop attributes.

Default-off path is unchanged: no thread clock, no new dicts, no spans.
2026-11-12 10:30:29 +08:00
minke.yu 9f9dce1ff6 test: restore _emit_pass_span residual default
The detail-phase refactor made residual a required positional argument while
test_loop_span_accounting.py still calls _emit_pass_span with four arguments,
so that test failed on the base commit. Default it to 0.0.

Split out of the prefill_batch_init observability series to keep that review
scope focused.
2026-11-12 10:30:17 +08:00
minke.yu c401c614fa fix(dp): rank uncached candidates by published backlog only
Deriving each candidate's uncached estimate from its own recent hit rate is
self-reinforcing: a rank that falls behind sees its hit rate drop (fewer
chances to hit), gets charged more for every new request, and falls further
behind. Measured on c580: one rank ended 16% below the others, 30 s cross-rank
dispersion max 952x, and 9.6% lower rps than the total_tokens arm.

Charge the fleet-average uncached size instead and rank by the published
num_waiting_uncached_tokens, which already carries each rank's exact prefix
match.
2026-11-11 02:06:59 +08:00
minke.yu 24b284d4e7 feat(dp): uncached_tokens load-balance method
DP balancing on total_tokens charges each request's whole prompt, but with
prefix caching only the unmatched part costs prefill compute. On a workload
with a 95% hit rate the prompt is ~95% noise: per request the uncached part
is a few percent of the prompt yet varies by more than an order of magnitude
(p50 1.9k / p99 59k tokens against a 95k prompt), so one cold request
outweighs many hot ones and per-rank prefill work drifts apart.

Add --load-balance-method uncached_tokens: DPBudget tracks the published
LoadSnapshot.num_waiting_uncached_tokens (prefix match minus accepted storage
hits) plus the rank's recent cache_hit_rate for a newly dispatched request.
2026-11-11 00:03:55 +08:00
QNan0513 fef023b64f [Fix] scheduler: Count all pending work in DP load snapshots
total_tokens DP balancing replaces its per-rank estimate with the rank's
latest load snapshot, so a snapshot that misses work already sent to the
rank lets the controller keep sending to it. Close three gaps:

- count grammar-queue requests in num_waiting_reqs and num_total_tokens
- count the in-progress chunked request's unprefilled tokens
- publish a snapshot once newly received requests are queued, since the
  one published after a forward predates the requests sent during it
2026-11-10 23:58:23 +08:00
ymk 15218b2f65 deploy: bake the live hostPath overrides into the tree
Copies the three files that the live cw mounts over the image:
- managers/load_snapshot.py: queued_total_input_tokens + drain_rate_tokens_per_s
- managers/scheduler_components/load_inquirer.py: fills the two new fields
- models/deepseek_v4.py: env-gated MM trace (SGLANG_MM_TRACE, off by default)

After this commit every runtime file the P/D pods need is in the tree, so the
hostPath file mounts can be dropped once the image is rebuilt.
2026-11-09 06:31:48 +00:00
minke.yu 883f7d08e1 fix(mooncake): use enable_client_http_server (not enable_http_server) 2026-11-09 11:04:01 +08:00
minke.yu e40c6dfe83 observability(mooncake): opt-in client metrics HTTP server per rank
SGLANG_MOONCAKE_CLIENT_METRICS_PORT_BASE=<base> makes MooncakeStore.setup()
pass enable_http_server=True and client_http_port=base+dp_rank, so each DP
rank's embedded Mooncake client exposes /metrics (needs MC_STORE_CLIENT_METRIC=1).
Off by default.
2026-11-09 10:55:17 +08:00
minke.yu 32e5e0f30f observability: GPU-window probe + forward/prefill sub-phase timing
Phase 1/3 of the async-backup handoff instrumentation plan:

* new fwd_timeline probe (SGLANG_FWD_TIMELINE=1): one CUDA event pair around
  the model execute call per forward, resolved without blocking and written as
  wall-clock JSONL per DP rank. Joined with loop spans it separates host time
  spent while the GPU is busy (overlapped) from GPU-idle host stalls.
* detail phases: fwd_init_new, fwd_prepare, fwd_execute, fwd_graph_load,
  fwd_graph_replay, fwd_post, fwd_runner_tail,
  prefill_policy, prefill_adder_init, prefill_scan, prefill_batch_init.

All spans ride the existing SGLANG_TRACE_LOOP_DETAIL path; the GPU probe is
off by default.
2026-11-08 01:19:09 +08:00
minke.yu a5fbe834e8 observability: split get_next residual into its four phases
Adds the Phase A instrumentation from the async-backup handoff (§7.1):

* PassTimings: pending_abort / chunk / new_batch / set_sched (+ chunk_count)
* prefill.get_next_disagg_prefill_batch_to_run: time
  process_pending_chunked_abort, process_prefill_chunk,
  get_new_batch_prefill and set_schedule_time_batch
* scheduler.prefill.pass gains sglang.{pending_abort,chunk,new_batch,set_sched}_ms,
  sglang.get_next_residual_ms (= other - these four) and sglang.chunk_count;
  the existing sglang.other_ms is unchanged for dashboard compatibility.

~5 monotonic calls per pass; enabled by SGLANG_BATCH_STAGE_TIMING (already on).
2026-11-08 00:20:55 +08:00
minke.yu d6f1e1efbd rebase(hicache): per-op async backup on top of #46884 + #48782
Net async change (from exp-async-backup-perop-20261107) applied onto
b300-pick-46884-20261107. Runtime files applied cleanly; the new tests and
the timeline probe are added as-is.

Interaction to validate on hardware: #46884 publishes prefill read-done before
the replay, so the scheduler-stream fence that the index copy waits on is
shorter; the async worker's wait should shrink and the copy may overlap more of
the in-flight forward. Re-baseline the 4-arm A/B on this branch.
2026-11-07 22:04:33 +08:00
alphabetc1 520ca548ea [Fix] hicache: Wait for overlapping write-through acknowledgements
(cherry picked from commit 032725770c33c61c236f2884ce7e7ff1fa99cef9)
2026-11-07 14:37:24 +08:00
metamergebot 5f7be7fab4 [Perf] hicache: Publish prefill read-done when offload copies are fenced on the forward stream
The prefill shared-read-done fastpath lets the schedule stream run ahead of
the forward that writes a stashed chunk's KV instead of waiting for the whole
forward. It was disabled whenever hierarchical cache or a storage backend was
enabled, because host and storage offload copies used to be ordered only
behind the schedule stream and could have read unfinished KV.

The cache controller now orders its device-to-host copies behind the forward
stream through write_fence_stream, and its load-back H2D copies through
load_fence_stream, so that reason no longer holds when every KV offload copy
carries that fence. The scheduler records on the model runner whether that is
the case: hierarchical cache with a cache controller and no decode offload
manager, which has no such fence. The publish helper keeps the coarse
whole-forward barrier for LMCache and for unfenced offload, and publishes
read-done otherwise.

With the coarse barrier, a synchronous page-index copy issued while preparing
the next batch inherited the wait on the entire in-flight forward, leaving the
GPU idle between batches. With read-done published, that copy only waits for
the shared-table reads to finish; KV ordering is unchanged because the backup
copies still wait on the forward stream.

Tests: the CPU runner test covers the fenced hierarchical-cache and storage
cases publishing, and LMCache keeping the barrier even when hierarchical cache
is fenced. The GPU ordering test checks that a write-through backup of a chunk
stashed before its forward finished still copies the final KV.

Co-authored-by: ZYHowell <ZYHowell@users.noreply.github.com>
(cherry picked from commit bb1cb2dded6fb9352d98076280342b10684e061b)
2026-11-07 14:36:38 +08:00
minke.yu 97c087778b observability: harden flush span attributes
* Normalize the host-pool layout to a string (a None attribute would make
  the OTel SDK drop the whole flush span).
* Drop any remaining None-valued flush attributes for the same reason.
2026-11-07 11:35:03 +08:00
minke.yu 53267ee18f observability: time loop-other inflight poll and release/insert
Adds loop-detail phases outside run_batch/process_result/get_next:

* inflight_poll: process_disagg_prefill_inflight_queue() (overlap loop)
* inflight_release: release_kv_cache(checkpoint=True) + tree_cache.finish()
  inside it, so the checkpoint/insert cost is separated from the poll
* launch_sample: launch_batch_sample_if_needed()

The get_next side already publishes match/adder/hicache/bootstrap/mlp_sync
and a pass residual (sglang.other_ms) on scheduler.prefill.pass.
2026-11-07 09:26:46 +08:00
minke.yu aa53670fcd observability: split flush_pending_backups host chain (off by default)
Adds the phase-3/phase-4 instrumentation from
handoff/prefill-flush-backups-validation-20261107.md:

* process_batch_result outer phases: flush_trace, load_snapshot,
  flush_backups, step_counters, metrics_report. process_residual now
  subtracts them, so the 37.5% residual is attributable.
* SGLANG_TRACE_BACKUP_DETAIL=1 (default 0) traces one flush:
  merge_ops / move_write (anchor + per-sidecar indices) / indices_cpu /
  index_sort / index_select / write_fence / build_l2 / submit_d2h
  (start-event wait, prepare, per-pool launch, ack events) / ack_enqueue,
  plus token/op/pool/device dimensions and ack queue depth.
* Per-flush resolution: one scheduler.cache_flush_backup span per
  non-empty flush, child of the loop span. Sub-phases never enter the
  loop aggregate, so the residual identity cannot double count.
* Empty vs non-empty flushes are counted on the loop.

All timings are scheduler host wall, not device transfer time.
2026-11-07 09:23:44 +08:00
ymk 7b3f34cbc0 test: cover async SWA state-payload call chain; clarify CP gate
Review follow-up for the v2 async state payload:

- P2: new tests use CustomTestCase (with a local fallback when sglang is not
  importable), matching the disaggregation/Mooncake test convention.
- P3: CP gate docstring now says all-CP-rank transfer *without DSA cache
  layer split* keeps the synchronous path.
- Add real call-chain coverage:
  * _send_kv_chunk -> sender.send -> add_transfer_request -> transfer_worker
    -> materialize_state_indices -> maybe_send_extra, with DSV4's four
    components (DSV4_REQUEST_STATE, REQUEST_ROW, SWA, SWA);
  * the worker materializes pinned tensors only after indices_event and
    fails loudly when a tensor has no event;
  * a second materialization without an event (staging retry) is a no-op;
  * real CUDA ordering: the pinned helper returns before a long kernel and
    the event-protected values match kv_to_page_indices.
- Verified on b300 (P container, overlay source): 9 passed in
  test_async_state_payload.py, 2 passed in the new
  TestMooncakeAsyncStateMaterialization class; the old batching suite has
  11 pre-existing failures that also fail on the image source.

The test-only batching suite requires the detail envs to be off when run in
the live pod (the inherited SGLANG_TRACE_MOONCAKE_DETAIL=1 exposes an existing
test fixture omission); noted, not changed here.
2026-11-07 00:22:57 +08:00
ymk 3b0baa3455 prefill: async SWA state-payload D2H (first version)
Implements the v2 plan in handoff/prefill-state-payload-async-review-20261106.md.

- mem_cache/common.py: add kv_to_page_indices_pinned() (pinned CPU buffer,
  non-blocking copy; no synchronous .cpu()).
- prefill.py: gate via should_async_state_payload() (last chunk, independent
  supports_async_state_indices, kv_shard_size==1, CP guard); only the
  _swa_payload request_window-is-None hot branch uses the pinned helper.
  Final segment asserts async state has an indices_event. Other state
  builders (mamba/full/DSA) stay synchronous.
- mooncake/conn.py: MooncakeKVSender advertises
  supports_async_state_indices; the worker materializes pinned state tensors
  to numpy only after indices_event.synchronize(), and fails loudly if a
  tensor arrives without an event.
- tests: guard/event/materialize/pinned-body unit tests + CUDA ordering test
  (skipped unless torch+CUDA+sglang are available).

No scheduler/transfer semantics change; rollback is the helper switch or
reverting this commit. 49 tests pass, 1 skipped locally.
2026-11-06 23:00:22 +08:00
ymk fe9c6574fe observability: trace KV transfer chunks end-to-end (M2)
Implements M2 of handoff/prefill-cpu-kv-overlap-design-20261106.md, behind
SGLANG_TRACE_MOONCAKE_DETAIL=1 (default off; reuses the existing trace stack).

- TransferKVChunk gains optional observation fields: chunk_seq, loop_seq,
  send_kind, enqueue/last_enqueue monotonic+wall stamps, queue_wait_accum_ns,
  queue_id, requeue_count, num_pages, approx_bytes, observe
- chunk_seq and enqueue timestamps are assigned at the real queue.put()
  point (after every early return), so skipped paths cannot leave holes
- Mooncake transfer worker measures queue wait, source-event wait, indices
  wait, KV/state/aux transfer, worker busy and residual; emits one
  mooncake.kv_chunk span per attempt (+ queue/source/indices wait children)
  and updates a per-room join summary
- request completion emits kv.request_join with chunk_count,
  last_chunk_queue_wait/worker, final_exposed_ms, completion_poll_ms,
  request_join_ms and first-enqueue-to-completion. First enqueue is stamped
  by the Mooncake manager at the real queue.put(); no ReqTimeStats field is
  added and the legacy queue_entry semantics are unchanged.
- nixl/mori add_transfer_request accept the observation kwarg (no-op there)

No scheduling/transfer semantics change; queue wait is excluded from worker
busy, and worker_busy excludes source/indices event waits. Unit tests: test_mooncake_chunk_detail.py (15 cases, including the
abort/worker interleaving and attempt-vs-total queue wait windows).
2026-11-06 20:01:56 +08:00
ymk 72e556ec81 observability: split prefill run_batch/process_result into host phases (M1)
Implements M1 of handoff/prefill-cpu-kv-overlap-design-20261106.md:
per-loop detail phases behind SGLANG_TRACE_LOOP_DETAIL=1 (requires
SGLANG_BATCH_STAGE_TIMING=1 + SGLANG_TRACE_LOOP_SPAN=1, only sampled loops).

- run_batch: early_send_host, resolve_inputs, model_forward_host,
  copy_result_submit (+ run_residual_ms)
- process_result: copy_done_wait, auxiliary_and_logprob, per_request_result,
  kv_state_payload/page_indices/sender_send by send_kind, report_stats
  (+ process_residual_ms)
- war_barrier is measured as its own loop phase (outside run_batch's span)
- repeated/per-request phases are summed attributes; single phases are child
  spans, so span count never grows with n_reqs/n_segments
- DP-skew join fields: pass_end_wall_ns, arrival_est_wall_ns = pass_end -
  mlp_sync, previous-loop phase summary (sglang.prev_*, published only for
  consecutive loop_seq; prev_detail_missing marks a gap)

No CUDA syncs added; when disabled every hook is a module-level bool check.
Unit tests: test_loop_detail_phases.py (10 cases).
2026-11-06 20:01:56 +08:00
minke.yu 23053730fc observability: count SWA gate rejections per prefill admission
Review follow-up. match_kv_cache also runs on waiting-queue probes
(calc_priority), PD decode _match_prefix_and_lock and HiCache rematches.
Emitting the counter there counted the same gap once per scheduling round
and attributed "re-prefilled" tokens to decode engines.

- match_kv_cache now only records req.full_kv_hit_length (plus the match-time
  gap); it no longer emits.
- the metric is emitted once in PrefillAdder._commit_prefill_admission, where
  the accepted prefix is final after any host load-back.
- rename the request counter to swa_gate_reject_admissions_total and document
  that a retracted and re-admitted request is counted again (it is re-prefilled).
- pre-create zero-valued children so gate-off deployments expose the series
  before the first rejection.
- tests: repeated match does not emit, admission emits once, a retry counts
  again, and host load-back closes the gap.

Not runtime verified (no GPU run on this build).
2026-11-06 13:18:07 +08:00
ymk bf8cf7f1dc hicache: pipeline prefetch/backup IO lanes + stage timing
- prefetch_io_aux_func: allow up to SGLANG_HICACHE_PREFETCH_IO_THREADS
  transfers in flight (default 1 = legacy behaviour); PrefetchAck is still
  emitted strictly FIFO so the per-rank ack stream (and the TP all_reduce in
  prefetch_sync_thread_func) is unchanged.
- backup_thread_func: same pipelining for L2->L3 writes via
  SGLANG_HICACHE_BACKUP_IO_THREADS.
- SGLANG_HICACHE_TIMING=1: emit [HC_TIMING] prefetch/backup queue, inflight,
  p50/p95 per 100 ops (the prefetch/backup lanes were previously single
  threaded; this is what caps L3<->L2 at ~1 op at a time).
2026-11-06 13:18:07 +08:00
minke.yu c911d1b572 observability: expose SWA prefix-match gate rejections
The unified-cache SWA component gates prefix matching on a live sliding
window: when the window is missing, the match stops before an otherwise
available Full-KV prefix. The existing hit counters do not show the tokens
lost to that gate, which makes the gate-on/gate-off trade-off invisible.

Record the gap between the longest Full-KV match and the accepted prefix
(full_kv_hit_length - (device_prefix_len + host_hit_length)) on the Req and
export it as:
  sglang:swa_gate_reject_tokens_total
  sglang:swa_gate_reject_reqs_total

This is the token volume that will be re-prefilled because a component
validator (SWA window; Mamba where present) rejected the match. With the
gate enabled it measures the hit length lost to a missing window; with the
gate disabled it only captures the host-only / load-back-gap fallback.

Includes unit tests for the gap and the host-hit case. Not runtime verified
yet (no GPU run on this build).
2026-11-06 10:33:52 +08:00
minke.yu 145e05c44d observability: fix scheduler no-batch accounting 2026-11-05 16:42:39 +08:00
minke.yu 5d5870f3ea observability: add scheduler loop span and no-batch accounting
Adds an opt-in scheduler.loop span (SGLANG_TRACE_LOOP_SPAN) with recv /
get_next / run_batch / process_result / kv_send children. The span is only
created for sampled loops that have pending work; fully idle loops stay in
the counters so the 714 pass/s idle spin never floods Jaeger.

No-batch accounting is always on with SGLANG_BATCH_STAGE_TIMING:
- sglang:scheduler_no_batch_{seconds,total}{dp_rank,reason,forward_overlap}
- sglang:scheduler_dp_lockstep_idle_seconds_total{dp_rank,forward_overlap}
with the design's reason priority (result_backpressure, deferred_input,
waiting_bootstrap, waiting_prefetch, admission_blocked, waiting_pd,
fully_idle, no_ready_other) and local_batch_kind/global_outcome from the DP
sync. DP lockstep idle is counted separately from pure no-batch wait.

loop gpu_ms reuses the existing DeviceTimer intervals (no new CUDA events);
the forward.gpu lane is optional and approximate. Pass/recv spans become
children of the loop span when it is active.

Tests: test_loop_span_accounting (5) and the existing stage-timing suite (8)
pass.
2026-11-05 15:00:23 +08:00
minke.yu 309af6e6e1 observability: record prefetch ready latency on the L3 hit success path
_handle_prefetch_result() retires the operation with a direct
del ongoing_prefetch[request] instead of _retire_ongoing_prefetch(), so
the ready-latency hook only ever saw miss/revoke paths: with a cache-hit
workload storage_prefetch_ready_seconds{result="hit"} stayed at 0 even
though storage_prefetch_hit_tokens_total was moving ~312k tok/s.

Observe right after the del, classifying by loaded_from_storage.
2026-11-04 19:14:02 +08:00
minke.yu 1a274df27e observability: fix prefetch ready latency to cover fast lookups
The first version stamped the request on its first in-flight
check_prefetch_progress() call. A fast L3 query can be drained by
_process_hicache_events before that check runs (the op is already gone
from ongoing_prefetch), so those requests produced no sample -- exactly
the common case.

Stamp at issue time in prefetch_from_storage() and observe once at
_retire_ongoing_prefetch(), classifying by operation.storage_hit_count.
Retire is the single exit for hit completion, below-threshold revoke,
anchor loss and host-capacity fallback, so hit and miss are both covered.

No behavior change; the hook is still behind SGLANG_BATCH_STAGE_TIMING.
2026-11-04 18:04:34 +08:00
minke.yu cb7988f849 observability: measure storage prefetch ready latency (hit/miss)
Under --hicache-storage-prefetch-policy wait_complete a request is not
admitted until its L3 lookup resolves, and a miss pays the same query round
trip as a hit. The scheduler only exposed a skip counter, so the actual wait
was unknown.

Add sglang:storage_prefetch_ready_seconds{dp_rank, result=hit|miss}:
  * stamp cache_request_handle on its first in-flight check,
  * observe at the moment check_prefetch_progress() turns usable, split by
    whether prefetch_loaded_tokens_by_reqid ended up > 0,
  * clear the stamp on revoke / abort / accounting discard.

Buckets cover 1 ms .. 30 s. Gated by SGLANG_BATCH_STAGE_TIMING (off by
default). The measurement is ~2 dict ops + one observe per request, not per
pass.

Tests: test/registered/unit/observability/test_batch_telemetry_stage_timing.py
adds the hit/miss split assertion (9 tests, all passing).
2026-11-04 17:26:41 +08:00
minke.yu 0866012b80 observability: split scheduler recv_requests into pull/broadcast/unwrap
Runtime accounting on the prefill service (64 x 14.6k-token requests,
8-way concurrency) showed the non-forward wall time is dominated by the
scheduler receive path, not by radix match:

  forward (extend+idle) 58.2% | get_next 6.9% | recv_requests 25.6%
  process_batch_result 1.8% | run_batch overhead 0.9% | other ~6%

The existing pass span only covers get_next, so it cannot explain that
25.6%.  Split it with the same opt-in gate:

  * sglang:recv_requests_seconds{dp_rank, part} (total/pull/broadcast/
    unwrap/mm), 5 new series per rank
  * sglang:recv_requests_msgs_total, sglang:recv_requests_prompt_tokens_total
  * OTLP span scheduler.recv_requests with the same breakdown, sampled
    deterministically by a per-process counter (SGLANG_RECV_SPAN_SAMPLE,
    default 0.05; SGLANG_RECV_SPAN=0 disables the span only)

Note on scope: tokenization itself does NOT run in the scheduler. The
tokenizer workers (--tokenizer-worker-num) produce TokenizedGenerateReqInput
and the scheduler receives the already-tokenized ids. What this measures is
ZMQ pull + decode, the cross-rank broadcast of the pickled objects, and
unwrap_pickle_fields() of the token-id buffers -- which is why large prompts
make recv_requests expensive.

Tests: test/registered/unit/observability/test_batch_telemetry_stage_timing.py
now also covers the recv histogram parts, counters, zero-message passes and
deterministic recv-span sampling (8 tests total, all passing).
2026-11-04 16:11:13 +08:00
minke.yu 85f4f9576c scheduler: add opt-in per-pass prefill stage timing (SGLANG_BATCH_STAGE_TIMING)
Motivation: on the PD prefill node the wall-clock budget is dominated by
get_next_batch_to_run (~50% of the hour on 2026-11-02 16:00-17:00), and
forward_overlap=partial is only 21-87 s/h, i.e. it does not overlap the GPU
forward. We know the total but not which segment inside it costs: radix
match, PrefillAdder admission, the HiCache event collective, the bootstrap
all-reduce, or the DP metadata gather.

Measurement design (one sample per scheduling pass, including passes that
produce no batch; see handoff/prefill-batch-stage-timing-20261103.md v2/v3):

  L1  7 per-pass histograms:
      sglang:batch_prefill_{get_next,match,adder,hicache_events,
      bootstrap,mlp_sync,other}_seconds
      other = total - (match + adder + hicache + bootstrap + mlp_sync),
      clamped to >= 0. Nested stages are not double counted.

  L3  5 per-pass counters:
      candidates / skipped_prefetch / admitted / resumed_chunk / new_chunk

  L2  a dedicated per-pass span scheduler.prefill.pass (not the batch span),
      emitted from end_pass() on every pass. Its sampling decision is a pure
      function of mlp_sync_seq, so a sampled gather round is sampled on all
      DP ranks. Idle ranks and ranks without a local batch still emit, which
      is exactly what the cross-rank skew analysis needs. No links.

Hooks are installed in both scheduling loops:
  * get_next_disagg_prefill_batch_to_run() for --disaggregation-mode prefill
    (hicache / bootstrap / mlp_sync segments), and
  * get_next_batch_to_run() for standalone prefill (hicache / mlp_sync),
so the instrumentation can be validated on ds41-prefill-dp4 without a decode
peer. match/adder/counters live in the shared get_new_batch_prefill path.

Semantics:
  * match times only init_next_round_input; check_prefetch_progress counts as
    skipped_prefetch, never as match.
  * bootstrap and hicache_events are cross-rank sync points measured at the
    outer call, not attributed to local CPU work.
  * dp_gather_ms comes from the existing MLPSyncBatchInfo.sync_wait_seconds;
    sglang:dp_attention_sync_wait_seconds is reused, no duplicate metric.
    The value is cleared before each outer call and mlp_sync_seq only
    increments when a gather actually ran, so stale values cannot leak.
  * resumed vs new chunked requests are captured separately: had_chunked is
    recorded at pass start (before process_prefill_chunk/admission can
    consume the last chunk) and new_chunk from adder.new_chunked_req. This
    fixes both the first-chunk and last-chunk mislabels.

Opt-in:
  * SGLANG_BATCH_STAGE_TIMING=1 (default 0);
    SGLANG_BATCH_PASS_SPAN=1 / SGLANG_BATCH_PASS_SPAN_SAMPLE=0.05.
  * When off every hook is one module-level bool check; no logging, no CUDA
    calls, no new collectives.

Tests: test/registered/unit/observability/test_batch_telemetry_stage_timing.py
covers first/middle/last chunk labelling, all-prefetch-skip passes without a
batch, stale-gather clearing, rank-deterministic pass-span sampling, and the
disabled no-op path. py_compile passes on all touched files.
2026-11-04 14:53:29 +08:00
minke.yu 2434d09848 observability: commit the in-image per-batch telemetry baseline (btelem4-6)
These three files were already running in the gist production image
(gist-nvlink-ee9170e7bb-pzcfix) but were never committed: the image was
produced by docker-commit with the btelem4/5/6 files overlaid on top of
the built source tree.

Port them so the tree matches the image before layering new instrumentation:

  * observability/batch_telemetry.py: per-batch histograms, per-DP-step
    shape histograms and the sampled OTLP span scheduler.batch.{prefill,decode}.
  * managers/scheduler_components/metrics_reporter.py: observe_prefill/
    observe_decode call sites and the batch-trace log suffix.
  * managers/scheduler_components/dp_attn.py: observe_dp_step().

Content is identical to the image; only line endings were normalised to LF
(the image copy was mixed CRLF/LF). No behavior change when
SGLANG_BATCH_TELEMETRY is unset (default off).
2026-11-04 14:48:38 +08:00
Florian Wunderlich d47a99b2ad test: import SWA eviction helper directly
(cherry picked from commit 5729f61e175bde3ef2b67dc3a95abb1042f4bf1b)
2026-11-04 10:41:52 +08:00
Florian Wunderlich 2ba9b8431a Fix SWA eviction behind the replay-trimmed prefix frontier
(cherry picked from commit f399d1e9a2bf7c33c7244ecf405e756292ddaf2b)
2026-11-04 10:41:52 +08:00
zicai.pu ee9170e7bb fix(mooncake): bind IntraNode batches to the bootstrap CUDA device
Sender and executor threads can enter synchronous IntraNode transfers with
another current CUDA device. Keep the explicit bootstrap ordinal, check it
against the published model device, and scope the native batch call to that
device while restoring the calling thread on success or failure.

Ordinary transports, force-TCP and non-CUDA devices keep their existing
path. Missing or inconsistent CUDA identity is rejected before native IO.

Add 13 CPU regressions importing the production wrapper, including real
constructor calls with a mocked SDK, concurrent thread-local device states,
error restoration and transport/device compatibility. All 13 pass; two
selected tests fail on the unchanged wrapper at behavior assertions. Native
and CUDA boundaries are mocked, so this does not certify physical CUDA IPC
or attribute the separate container-PID collision to this change.
2026-10-30 11:42:19 +08:00
zicai.pu 17505e6cf6 hicache: align prefetch and host restore boundaries
Use the request radix-key cap for L3 prefetch so bounded replay cannot
split a queried trailing-state checkpoint at the later rematch. Reject
Host-only SWA-gap candidates before they mask a complete ancestor, while
preserving device matching and the final load-back completeness guard.

Add CPU regressions that import the real scheduler, cache and TreeCore
methods. All 13 test methods pass; the same tests against the unmodified
scheduler/SWA code fail six methods (ten assertions including subtests).
The checks use synthetic cache state and controller boundaries, not GPU
or live Store execution.
2026-10-30 11:42:19 +08:00
zicai.pu 81f7398855 fix(hicache): reject unproven trailing restore endpoints
Do not invent shorter checkpoints when a trailing pool has no restorable-prefix inventory. Reject partial allocation and vote no common endpoint while preserving the collective shape and plain-KV contiguous behavior.

Add negative controls for missing inventory and retain pure-KV coverage. Validation: 33 cache/PP CPU tests passed with an external Mac native-hash import adapter using the repository Python hash oracle; repository pre-commit checks passed.
2026-10-30 11:42:19 +08:00
zicai.pu 0a56a5f6e7 fix(hicache): preserve sparse restore checkpoints under host pressure
Choose a common validated trailing-state endpoint before partial host allocation, retain that endpoint set across query synchronization, and size auxiliary staging from the selected length. Assumed trailing hits prove only their original endpoint.

Add controller/cache CPU regressions and a PP checkpoint-vote control. Validation: 14 targeted CPU tests and repository pre-commit checks passed.
2026-10-30 11:35:29 +08:00
minke.yu 7f6fe22445 mem_cache: fail-soft on SWA load-back gaps instead of asserting
With long prompts and L3 under pressure, a matched prefix can contain a node
whose SWA component has neither a device nor a host value (host tombstone
evicted between match and load-back). build_hicache_transfers asserted on
this and killed the prefill scheduler.

Add SWAComponent.has_load_back_gap() and have build_load_back_spec return an
empty transfer spec when a gap is found, so the caller backs off and the
request is recomputed. Measured on peak15-v6 (1M, cap 64): 0 scheduler
exceptions, hit 59.4%->94.0%, TTFT p95 47.0s->3.4s with the SWA prefix-match
gate disabled.
2026-10-26 13:23:17 +00:00
minke.yu f6fc6ac3e7 mem_cache: allow disabling the DSv4.1 SWA prefix-match gate via env
The unified-cache SWA component gates prefix matching on a live SWA window,
so a long prefix only matches up to the last sliding window and the tail is
recomputed. Add SGLANG_DSV4_SWA_GATES_PREFIX_MATCH (default True) so the gate
can be turned off for measurement; the SWA window is then rebuilt/recovered
downstream as for the req-ring paths. Measured on 4xB300 DP4 at 160k prefix:
c32 hit tokens 40.6% -> ~100% (with HiCache L2/L3 enabled), needle test
(cold vs hit) identical.
2026-10-26 04:06:25 +00:00
minke.yu 0a3c370451 mooncake: probe pointers via cuda.bindings instead of ctypes libcudart
The ctypes-based probing loaded a second libcudart copy into the process,
which corrupted CUDA/runtime state and segfaulted the scheduler processes
shortly after registration (reproducible on prefill ranks, intermittent on
decode). cuda.bindings.runtime.cudaPointerGetAttributes is a properly
typed binding and boots cleanly on both roles.
2026-10-25 14:04:56 +00:00
minke.yu b1b303c9f2 mooncake: support host-memory payloads with intra-node NVLink transport
IntraNodeNvlinkTransport can only register/reach device memory. sglang's
register list includes host regions (aux buffers, some state components);
one host region failed the whole registerLocalMemoryBatch and the engine
rolled back every region, leaving the segment descriptor empty so all KV
transfers failed with 'Requested address ... not found'.

When the intra-node NVLink transport is active (MC_INTRANODE_NVLINK etc.):
- register only device-memory regions (probe via cudaPointerGetAttributes)
- route transfer blocks whose local source is host memory over the ordered
  per-endpoint zmq channel (new STATE_DATA message) instead of the engine;
  the receiver validates and writes the block into its buffer, relying on
  the same ordering guarantee as the aux TCP path
- force aux over the TCP socket path automatically
2026-10-25 14:04:52 +00:00
Liangsheng Yin 7d996300b9 [Perf] dsv41: Use the fused ratio-2 compressor in the prefill graph (#46230) 2026-10-23 21:33:42 -07:00
Liangsheng Yin 54df642fe6 [Perf] dsv41: Translate SWA locations once per prefill graph step (#46229) 2026-10-23 21:33:15 -07:00
Liangsheng Yin 564aec61c8 [Perf] dsv41: Feed wo_b the fused MXFP8 attention output in the prefill graph (#46231) 2026-10-23 21:31:33 -07:00
Yihao WangandClaude Opus 5.5 a7183a8246 [Feat] spec: Let out-of-tree platforms provide EAGLE draft graph runners (#45145)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-23 21:28:56 -07:00
Yihao WangandClaude Opus 5.5 af3f9e4c8f [Feat] spec: Let out-of-tree platforms provide their own speculative worker (#45144)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-23 21:24:58 -07:00
MickandMick Qian f09cec65e9 [Fix] diffusion/storage: Join uploads before request cleanup (#46068)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-24 12:19:10 +08:00
MickandMick Qian f4ecaae888 [Fix] diffusion/mesh: Guard asynchronous job publication and defaults (#46081)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-24 12:18:25 +08:00
MickandMick Qian cdab6c1bcc [Fix] diffusion/loader: Preserve checkpoints on read failures (#46095)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-24 12:17:46 +08:00
f2c634961b [Perf] diffusion: Project LTX-2 prompt K/V once per denoising loop (#46105)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-24 12:17:04 +08:00
Liangsheng Yin 3b95120f00 [Fix] dsv41: Revert MegaGate routing in prefill (#46242) 2026-10-23 21:15:38 -07:00
55f23a8b59 [Fix] [AMD] dsv4.1: Fix decode graph capture with quantized low-ratio activations (#46151)
Co-authored-by: Kevin Mi <kevin.mi@radixark.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-23 20:55:13 -07:00
Liangsheng Yin 3c6a6c8068 [Perf] dsv41: Run MegaGate routing in prefill (#46232) 2026-10-23 20:31:50 -07:00
Liangsheng Yin 17d92ea671 [Feat] openai: Parse DeepSeek text for the native endpoints through deepseek-recipe (#46219) 2026-10-23 20:17:37 -07:00
Richard WangandMick Qian 7cefa60765 [Perf] diffusion: Wait on the socket when the scheduler is idle (#46007)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-24 11:02:24 +08:00
25ba24e9a7 [Perf] diffusion: Build one LTX-2.3 AdaLN row per sample when every token shares the sigma (#46109)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-24 11:00:14 +08:00
42ac81c655 [Perf] diffusion: Route dense attention with a sub-tile side to FA4 on Blackwell (#46107)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-24 10:59:25 +08:00
jambow0320andShangming Cai de7055f596 [Refactor] pd: Consolidate sender and receiver implementations (#41140)
Co-authored-by: Shangming Cai <csmthu@gmail.com>
2026-10-24 10:50:10 +08:00
jacky.cheng 88ac7c02a6 [Fix] [AMD] diffusion: Fix the first-step denoise regression from compiling ENABLE_CK=0 BF16 attention (#45838) 2026-10-23 19:42:54 -07:00
Jackey Huaandhnyls2002 a77998e157 [Feat] openai: parse DeepSeek output through deepseek-recipe (#45907)
Co-authored-by: hnyls2002 <lsyincs@gmail.com>
2026-10-23 19:23:22 -07:00
Liangsheng Yinandkpham-sgl 79701a2549 [Perf] dsv41: Read Engram rows from peer device shards (#46180)
Co-authored-by: kpham-sgl <khoa.pham@radixark.ai>
2026-10-23 19:01:26 -07:00
Yihao WangandClaude Opus 5 327c906764 [Perf] [XPU] diffusion: Avoid QKV copies in flash attention (#41944)
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-10-24 09:33:26 +08:00
Yihao WangandClaude Opus 5 5bef70c0a2 [Perf] flux2: Build the single-block output projection without a cat (#41943)
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-10-24 09:33:26 +08:00
MickandMick Qian 1d4dcc81cc [Perf] diffusion/registry: Reduce model discovery work at startup (#46043)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-24 09:30:04 +08:00
47c0791ad1 [Perf] diffusion: Write uncompressed PNG responses without Pillow's row filters (#46099)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-24 09:28:55 +08:00
bbc77e7db4 [Perf] diffusion: Cut LTX-2.3 per-block elementwise work (contiguous AdaLN values, gate-folded masks, vectorized head gate) (#46110)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-24 09:27:39 +08:00
4e2d5b17ad [Perf] diffusion: Keep every component resident in auto mode when memory allows (#45978)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-24 09:26:13 +08:00
be8600d02d [Fix] diffusion: Step MOVA latents outside the serial-CFG branch (#45993)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-24 09:23:54 +08:00
a08bb0ec9e [Perf] diffusion: Run LTX-2 stage 2 on a premerged resident DiT (#46108)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-24 09:23:08 +08:00
e66c9e981d [Perf] diffusion: Turn on breakable CUDA graphs by default for LTX-2/LTX-2.3 two-stage serving (#45977)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-24 09:22:06 +08:00
2e6d87e2ca [Perf] diffusion: Serve FA4 head sizes on Blackwell instead of falling back to sm80 SDPA (#46102)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-24 09:20:32 +08:00
386fa9ecfd [Perf] diffusion: Drop LTX-2's all-valid prompt mask under lossless quality (#46106)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-24 09:19:12 +08:00
5b8955e70f [Perf] diffusion: Shard LTX-2 stage 2 over the CFG ranks as Ulysses (#45965)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-24 09:17:41 +08:00
738e4535ea [Perf] diffusion/qwen-image-2.1: Keep an image prefix's K/V in the packed buffer across steps (#46101)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-24 09:14:10 +08:00
3712b5944d [Perf] diffusion: Patchify Wan latents straight into tokens (#46148)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-24 09:12:49 +08:00
MickandMick Qian 2d5ad3243f [Fix] diffusion/http: Allow bounded graceful shutdown (#46169)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-24 09:11:15 +08:00
Jackey Huaandhnyls2002 e22a530859 [Feat] openai: Render DeepSeek-V4/V4.1 chat prompts with deepseek-recipe (#45315)
Co-authored-by: hnyls2002 <lsyincs@gmail.com>
2026-10-23 18:05:35 -07:00
weireweireandweireweire 49a2fbf817 [Fix] docker: Pin sgl-nvidia-mega to 3333e4c so the megakernel starts on main (#46030)
Co-authored-by: weireweire <20922698+weireweire@users.noreply.github.com>
2026-10-23 17:58:04 -07:00
Khoa PhamandClaude Fable 5.1 ee2dd727a3 [Perf] dsv41: Let Mega-mHC write the MegaMoE input under MegaGate (#46163)
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-23 17:54:03 -07:00
Liangsheng Yin 79e5c4bb9b [Perf] dsv41: Trim per-step DSpark decode overhead (#46176) 2026-10-23 17:24:06 -07:00
Qiaolin Yu 5994c5597d [Perf] spec-decoding: fuse simulated speculative acceptance tensor preparation (#41169) 2026-10-23 17:15:21 -07:00
Baizhou Zhangandsid-rp 6a319e7b34 [Fix] muse: Support Responses namespace tools and defer_loading (#44685)
Co-authored-by: sid-rp <sidrp@meta.com>
2026-10-23 16:48:08 -07:00
0d592c475b [Fix] rust-server: Keep a transfer backend's own bootstrap port on prefill (#46066)
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com>
Co-authored-by: shodoco <shodoco@users.noreply.github.com>
2026-10-23 16:47:14 -07:00
Cheng Wan 855c092deb [Fix] inkling: Keep BF16 LM-head operands with FP32 output (#45335) 2026-10-23 15:51:44 -07:00
Ziang Li 197be2203a [Doc] quantization: Link W4A16 design from environment reference (#46170) 2026-10-23 15:49:18 -07:00
Liangsheng Yin f05ad5ec01 [Perf] logits: Cast bf16 logits into the fp32 buffer with a Triton kernel (#46167) 2026-10-23 15:06:09 -07:00
Rohan VarmaandJun Wan 15dc838f9b [Fix] http_server: Apply SGLANG_TIMEOUT_KEEP_ALIVE to Granian HTTP/1.1 connections (#45760)
Signed-off-by: Rohan Varma <rohanv@nvidia.com>
Co-authored-by: Jun Wan <8323392+junwan01@users.noreply.github.com>
2026-10-23 14:43:54 -07:00
Baizhou Zhang ea226cf782 [Feat] glm5.3flash: Support padded CP BCG and prefix replay with PD/DFlash2 (#43774) 2026-10-23 14:21:31 -07:00
Yuhao YangandClaude Opus 5.5 5f7542d762 [Perf] dsv41: Build the MegaGate row masks once per forward (#46071)
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-23 14:15:01 -07:00
27ac614f05 [Feat] parser: Add checkpoint-driven response-template adapters (#40479)
Signed-off-by: Yoni Gozlan <yonigozlan@users.noreply.github.com>
Co-authored-by: Yoni Gozlan <yonigozlan@users.noreply.github.com>
Co-authored-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
Co-authored-by: Jimmy Shong <69131491+Jiminator@users.noreply.github.com>
2026-10-23 14:05:54 -07:00
em-dreyer 4542fc8f59 [Fix] mistral3: Subclass LlavaForConditionalGeneration instead of wrapping it (#45656) 2026-10-23 14:05:32 -07:00
Michael 194abb8e54 [CI] [AMD] diffusion: Drain the Cosmos Dreams decode stream after freeing its outputs (#45983) 2026-10-23 13:26:56 -07:00
1d605b7ece [CI] vlm: Harden fixture cleanup after setup and shutdown failures (#45461)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Alison Shao <a.shao@wustl.edu>
2026-10-23 13:20:06 -07:00
005333d029 [Refactor] radix-tree: Make each cache action a typed CacheAction implementation (#45444)
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com>
Co-authored-by: Jialin <Jialin@users.noreply.github.com>
2026-10-23 12:49:30 -07:00
Cheng Wan d0022e1df3 [Fix] lora: Preserve MoE projection identity and routing lifetimes (#45681) 2026-10-23 12:32:55 -07:00
Cheng Wan ac3db731b3 [Fix] moe: Preserve CUTLASS routing metadata and execution budgets (#45917) 2026-10-23 12:32:05 -07:00
Cheng Wan 8b9b992e41 [Perf] moe: Preserve Marlin workspace and route-count budgets (#45462) 2026-10-23 12:31:17 -07:00
0931dfedc9 [Fix] [AMD] dsa: Fix AITER prefill with an FP8 KV cache (#39083)
Co-authored-by: Pawel Kowalski <pawel.kowalski@silo.ai>
Co-authored-by: Thomas Wang <thomawan@amd.com>
2026-10-23 12:30:42 -07:00
Cheng Wan e6451c1a95 [Fix] gguf: Preserve MoE routing, weighting and allocation budgets (#45916) 2026-10-23 12:29:51 -07:00
Cheng Wan 1340044e97 [Fix] deepseek-v4: Preserve decoder-tail token-count ownership (#45915) 2026-10-23 12:28:43 -07:00
9d7ad308bd [Fix] [AMD] Quark: fix Quark block-FP8 loading in DeepSeek loader (#40776)
Co-authored-by: clintg6 <7388379+clintg6@users.noreply.github.com>
Co-authored-by: Thomas Wang <thomawan@amd.com>
Co-authored-by: Bowen Bao <bowenbao@amd.com>
Co-authored-by: Hubert Lu <Hubert.Lu@amd.com>
2026-10-23 12:11:21 -07:00
Alex NailsandClaude Opus 5.5 baf80c61b7 [Feat] platforms: Let out-of-tree platforms bind group devices and custom ops (#45897)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-23 12:02:42 -07:00
Shangming Cai 1e894eb92d [CI] disaggregation: Run the SWA decode radix cache test on Mooncake (#46126) 2026-10-24 01:41:39 +08:00
Peng WuandClaude Opus 5.5 16aa816060 [Fix] disaggregation: Restart a lost prefill's heartbeat miss count (#45922)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-24 01:26:10 +08:00
74c08f3dc5 [Feat] rust-server: Let a multimodal processor finalize item layout after caller hashes (#45914)
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com>
Co-authored-by: Lu Fang <30275821+houseroad@users.noreply.github.com>
2026-10-23 09:26:52 -07:00
30dc2f9430 [Feat] benchmark: Add --input-len-list to bench_one_batch_server for per-request input lengths (#45982)
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com>
Co-authored-by: wpc <wpc@users.noreply.github.com>
2026-10-23 09:04:59 -07:00
5a86d6ee17 [Fix] rust-server: Deliver scheduler-side aborts to Rust frontend clients (#45899)
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com>
Co-authored-by: shodoco <shodoco@users.noreply.github.com>
2026-10-23 09:04:54 -07:00
eb8ef80842 [Feat] rust-server: Serve /abort_request from the Rust frontend (#45888)
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com>
Co-authored-by: shodoco <shodoco@users.noreply.github.com>
2026-10-23 09:04:49 -07:00
Avaya Aggarwal efd2ce8a9d [Feat] comfyui: video-to-video input and prompt enhancement for server-mode nodes (#43673) 2026-10-23 23:09:30 +08:00
c677171c1d [Feat] hicache: Support MLA DCP with HiCache L3(file/mooncake) (#41187)
Co-authored-by: Codex <codex@openai.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: alphabetc1 <2508695655@qq.com>
Co-authored-by: Venkat Raman <venkat.raman@verda.com>
2026-10-23 23:09:04 +08:00
02b983603d [Fix] diffusion: Keep dynamic LoRA deltas on fused linear-GELU sites (#45943)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-23 21:19:53 +08:00
0e34962884 [Perf] diffusion: Cut LTX-2 denoising-step host overhead (#45966)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-23 21:18:57 +08:00
82ff16dad0 [Perf] diffusion: Skip the residency calibration probe in auto mode until a planner consumes it (#45979)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-23 21:18:06 +08:00
3e5f5505ed [Perf] diffusion: Serve image responses from in-memory encoded bytes (#45987)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-23 21:16:44 +08:00
2fbd094a8c [Fix] diffusion: Offset LTX-2.3 BCG video RoPE by the SP shard start (#45995)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-23 21:14:37 +08:00
pllimax 9b46f27090 [Fix] [NPU] tests: Restore evalscope GSM8K baseline via NPUGSM8KMixin backend (#45732) 2026-10-23 21:06:21 +08:00
shun8686 2a46e32fbc [CI] [NPU] precise-test: skip recommend-tests-from-coverage when no NPU paths changed (#46055) 2026-10-23 21:01:42 +08:00
86933b1f33 [Fix] deps: Pin python-dateutil<3 to keep botocore resolvable (#45857)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Po-Han Huang <pohanh@nvidia.com>
2026-10-23 20:57:44 +08:00
shun8686 0347b2bb4e [CI] [NPU] precise-test: gate precision on PR judgeability, not bucket count (#46042) 2026-10-23 20:41:22 +08:00
Jimmy Shong 4b1941e889 [CI] diffusion: Run the FLUX.2 XPU norm-fusion test on the XPU suite only (#46040) 2026-10-23 19:54:03 +08:00
chuyehandCursor f967418a35 [Doc] cookbook: Qwen3.5 MI355X FP8 agentic adds scheduler/tokenizer tuning from the InferenceX recipe (#46032)
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-23 19:12:05 +08:00
shun8686 535aeae2a7 [CI] [NPU] precise-test: run only the recommended multimodal_gen cases (#46014) 2026-10-23 18:06:58 +08:00
chuyehandCursor 69f830756e [Doc] cookbook: Qwen3.5 MI355X FP8 agentic uses HiCache write_through on the v0.5.22 image (#46017)
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-23 17:47:52 +08:00
Sage 611a703061 [Chore] api: add fields to generate request (#45164)
Signed-off-by: Sage Ahrac <sagiahrak@gmail.com>
2026-10-23 17:10:23 +08:00
494eeecb6c [Fix] diffusion: Keep the linear-GELU verdict cache out of torch.compile traces (#45944)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-23 17:01:51 +08:00
f9129915a3 [Fix] diffusion: Give every SP rank the same Wan TI2V timestep (#45994)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-23 17:00:40 +08:00
Xinyuan Tong 8b9f72417f [Feat] Inkling: render and parse chat via tml-renderers (#45340) 2026-10-23 16:43:38 +08:00
Michael bb72b47daf [Test] [AMD] deepseek-v4.1: Give the encoder-replay test's request window a zero_row (#45728) 2026-10-23 01:38:54 -07:00
Shuwen Wang 77bc8cab18 [Fix] hicache/pd: Promise prefill only the L3 span the decode prefetch takes (#43548) 2026-10-23 16:32:41 +08:00
Michael 7d1a16d3eb [CI] [AMD] diffusion: Skip the LTX-2.5 diffusion decoder case on the 2-GPU lane (#45861) 2026-10-23 01:28:12 -07:00
2b8e4cde95 [Doc] [CPU] Cookbook: Update to the cookbook for models supported by Intel Xeon CPUs (#40092)
Co-authored-by: Niu Ziru <niuziru@a4bf018d3341.jf.intel.com>
Co-authored-by: Niu Ziru <niuziru@gnr799217.jf.intel.com>
2026-10-23 16:22:50 +08:00
Avaya Aggarwal 4b273537de [Fix] diffusion: runtime utils bug fixes (#44887) 2026-10-23 16:08:13 +08:00
Po-Han Huang (NVIDIA)andClaude Opus 5.5 02d2445138 [Doc] cookbook: Refresh SemiAnalysis AgentX launches with seven open InferenceX PRs (#46005)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-23 16:04:01 +08:00
shun8686 472208fec0 [CI] [NPU] precise-test: recommend multimodal_gen cases at test-case granularity (#45955) 2026-10-23 15:57:31 +08:00
Yao MatrixandAlex Nails 6d6c9cd4ae [Fix] offloader: Preserve cached weights during CPU offload (#44139)
Co-authored-by: Alex Nails <alex.nails@radixark.ai>
2026-10-23 00:50:24 -07:00
Yihao WangandClaude Opus 5 6f466f655a [Perf] [XPU] diffusion/flux2: Fuse bit-exact LayerNorm and gated residual norm (#41942)
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-10-23 15:38:12 +08:00
8708d2ee99 [Fix] diffusion/ltx: Keep tiled VAE decode off the sharded convs under parallel decode (#45942)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-23 15:32:03 +08:00
4f47c48b33 [Fix] diffusion: Keep the N-rank IPC all-to-all's flags and staging out of expandable segments (#45948)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-23 15:31:19 +08:00
1cd0ebb3d3 [Perf] diffusion: Remove per-step and per-request host syncs in Cosmos3 and Qwen-Image-2.1 (#45967)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-23 15:30:09 +08:00
Avaya Aggarwal 4da6f585f3 [Fix] diffusion: OpenAI API hardening (#43585) 2026-10-23 15:22:00 +08:00
ShunkangzandZhangheng 79ab8f6d7c [Feat] KVShard: Support kv cache sharding for MTP (#40929)
Co-authored-by: Zhangheng <hzh0425@apache.org>
2026-10-23 15:20:19 +08:00
Banruo LiuandClaude Opus 5.5 6e667c4310 [Fix] disaggregation: Poison the deferred abort ack when a NIXL or Mori write fails (#45973)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-23 14:47:59 +08:00
Liangsheng Yin 6c844eb12e [Perf] dsv4: Replay the V4.1 prefill graph past 16K contexts on SM90 (#45941) 2026-10-22 23:47:17 -07:00
ashwini rathi 6dd0dfabf0 [CI] [Intel] docker/xpu: Tag nightly images with main-whl and main-main kernel variant (#45854) 2026-10-23 14:44:55 +08:00
Jimmy Shong 2be6577ba9 [Doc] beam: Offer Python and Docker commands in the cookbook (#45991) 2026-10-22 23:39:10 -07:00
Chetan Kumar VermaandClaude Opus 5.5 76c382e80f [Perf] [XPU] diffusion: Use the AOT SYCL fused residual-norm-scale-shift kernel (#41401)
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-23 14:32:25 +08:00
ZiruandNiu Ziru 904691e0b5 [Feat] [CPU] model: Support Cosmos3 series models on CPU (#45478)
Co-authored-by: Niu Ziru <niuziru@gnr799217.jf.intel.com>
2026-10-23 14:24:28 +08:00
vikram singh shekhawat 3f36fc0833 [Fix] [XPU] runtime: Fix XPU rotary embeddings and tokenizer padding (#41213) 2026-10-23 14:09:59 +08:00
Liangsheng Yin 0e338777b0 [Perf] dsv4: Let the fused MXFP8 producers run inside the breakable prefill graph (#45980) 2026-10-22 23:00:10 -07:00
Pinjie Xu 7c680a0d45 [Perf] diffusion/cosmos3: optimize v2v preprocess performance (#45398)
Signed-off-by: pinjie <pinjiex@nvidia.com>
2026-10-23 13:57:38 +08:00
Michael 7baacd1d1b [CI] [AMD] docker: Add a build-only dispatch option to the ROCm nightly image workflows (#45882) 2026-10-23 13:41:02 +08:00
Khoa Pham c7994e7076 [Fix] dsv41: Keep MegaGate out of prefill graphs by gating on the forward mode (#45969) 2026-10-22 22:11:18 -07:00
Khoa Phamandhnyls2002 5bdbec0d48 [Perf] dsv41: Drop the MegaGate padded-row copy on sgl-deep-gemm 0.2.3.post2 (#45933)
Co-authored-by: hnyls2002 <lsyincs@gmail.com>
2026-10-22 22:10:57 -07:00
e83f585ae6 [Perf] dsv41: Serve token-interleaved V4.1 KV caches with sgl-flash-mla (#45893)
Co-authored-by: kpham-sgl <khoa.pham@radixark.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-22 22:02:49 -07:00
Liangsheng Yin 49933325cc [Perf] dspark: Combine the draft's mHC streams from bf16 without an fp32 copy (#45964) 2026-10-22 21:44:01 -07:00
Mick f4a2df3dac [CI] diffusion: Calibrate single-GPU shard scheduling from CI (#45831) 2026-10-23 12:03:24 +08:00
Avaya Aggarwalandniehen6174 fa628c5359 [Fix] comfyui: Fix GGUF detection, import guard and VRAM accounting in SGLDiffusion plugin (#43536)
Co-authored-by: niehen6174 <1639206518@qq.com>
2026-10-23 11:58:23 +08:00
em-dreyerandClaude Opus 5.5 79eb10e182 [CI] [NVIDIA] gb300: Restore the aarch64 rust-ext cache on ARM runners (#45920)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-22 20:01:42 -07:00
Wookje Han f2873f1613 [Perf] [NVIDIA] muse-spark-open: Fuse pre-expert RMSNorm with static-scale NVFP4 quantization (#44554) 2026-10-22 20:00:37 -07:00
Baizhou Zhang 698651acd0 [Feat] HiCache: Support fused DFLASH KV in unified hybrid SWA pools (#45700) 2026-10-22 20:00:28 -07:00
chenyang08056032 ce2261863f [CI] [NPU] precise-test: add empty-recommendation skip-all rule, slim baseline artifacts, and move setup-covstub off the NPU pool (#45571) 2026-10-23 10:52:14 +08:00
Liangsheng Yin a4838160b3 [Perf] dsv4: Write V4.1 K rows with one warp per token in the fused k-norm-rope store (#45934) 2026-10-22 19:50:50 -07:00
Liangsheng Yin c4fa35b829 [Perf] dsv4: Map V4.1 prefill index selections to pool slots in one kernel (#45938) 2026-10-22 19:50:31 -07:00
Liangsheng Yin 2da6c0c237 [Perf] dsv4: Combine the final mHC streams from bf16 without an fp32 copy (#45940) 2026-10-22 19:43:46 -07:00
MickandMick Qian 363e6c6b4d [Feat] diffusion/disaggregation: split FLUX.3 prompt and image encoders (#45309)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-23 10:35:41 +08:00
Lianmin Zheng 5daa1352ba [CI] tests: Retire test/srt and reject legacy test paths (#42567) 2026-10-22 19:15:55 -07:00
Khoa PhamandClaude Opus 5.5 e5dc0c363a [Perf] dsv41: Integrate DeepGEMM MegaGate routing (#43262)
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-22 19:08:39 -07:00
YAMYandOasis-Git c82f7141ad [Perf] [NVIDIA] dspark: Move the host work between DSv4.1 decode graphs into the graphs for the megakernel path (#45450)
Co-authored-by: Oasis-Git <ayw.sirius19@gmail.com>
2026-10-22 19:08:06 -07:00
metamergebotandJialin 94e4611eac [Refactor] unified-cache: Factor match validators and match completion into helpers (#45714)
Co-authored-by: Jialin <Jialin@users.noreply.github.com>
2026-10-22 18:58:15 -07:00
Xinyuan Tong 92e57d5de7 [Perf] request_logger: skip per-token watermark redaction walk (#44692) 2026-10-22 18:57:19 -07:00
Liangsheng Yin b588aa39e9 [Perf] dsv4: Optionally stage V4.1 engram rows from the shared host table on the scheduler (#45921) 2026-10-22 18:35:55 -07:00
zijiexiaandClaude Opus 5.5 f36ee5de93 [Doc] Cookbook: add MiniMax-M3.1 (#45924)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-23 09:07:35 +08:00
26396bf566 [Fix] [NVIDIA] deepseek-v4.1: Fix the breakable prefill CUDA graph in the late-layer tail on SM90 (#45407)
Co-authored-by: JohnZhao01 <325926370+JohnZhao01@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: hnyls2002 <lsyincs@gmail.com>
2026-10-22 17:57:16 -07:00
Michael a95ae5f1c5 [CI] [AMD] docker: Start the ROCm 10 nightly at 10:00 UTC and ROCm 7.2 at 11:00 UTC (#45848) 2026-10-23 08:52:06 +08:00
Liangsheng Yin 97b79f0796 [Fix] prefill-graph: Keep every DP rank eager when the replay tail captures behind rank barriers (#45881) 2026-10-22 17:36:00 -07:00
Liangsheng Yin 9c7dd3c45f [Perf] dsv4: Dequantize each shared sparse-prefill KV source once per forward (#45909) 2026-10-22 17:08:40 -07:00
Liangsheng Yin ae4258715c [Perf] dsv4: Keep V4.1 image rows inside the Engram gate kernel (#45904) 2026-10-22 17:05:01 -07:00
HEJIAN SANGandClaude Opus 5.5 77f90ade97 [Fix] weight_checker: Make the checksum device-agnostic (#45918)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-22 16:57:49 -07:00
bb9e9d5632 [Perf] mlp: Fuse dense SwiGLU with block-FP8 quantization on decode (#45219)
Co-authored-by: Jayden Yu <jaydenyu@fb.com>
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com>
2026-10-23 07:49:39 +08:00
Chunan ZengandClaude Opus 5.5 5849ff5337 [Fix] [NVIDIA] jit: Build sm_100f on CC 10.7 until the bundled CUTLASS supports SM107 (#45911)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-22 16:42:00 -07:00
Baizhou Zhang 8d47d515e4 [CI] deepgemm: Remove obsolete ref_mhc preflight check (#45910) 2026-10-22 16:27:40 -07:00
Michael 035ed73b9d [Fix] [AMD] kernels: Import the HIP launch helpers from jit.utils.hipcc in smallm_fp8_blockscale_gfx950 (#45676) 2026-10-22 16:19:34 -07:00
Alex Nails c394a90c2b [Perf] gemma4: Capture single-step frozen-KV drafts (#45771) 2026-10-22 15:37:19 -07:00
Liangsheng Yin f9b7225f2e [Perf] dsv4: Use the fused MXFP8 producers in eager prefill (#45903) 2026-10-22 15:29:09 -07:00
Liangsheng Yin 9c0f6d6a61 [Perf] dsv4: Write the MegaAttention output straight into the prefill graph buffer (#45901) 2026-10-22 14:56:08 -07:00
Liangsheng Yin 818d7932d7 [Perf] dsv4: Drop local references that keep residual-stream tensors alive through the prefill step (#45900) 2026-10-22 14:55:49 -07:00
3b0003dac1 [Fix] HiCache: stale host KV from write-through copies racing the in-flight forward (#42701)
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Zhiqiang Xie <xiezhq@stanford.edu>
2026-10-22 14:46:51 -07:00
Alison Shao a904e4d31a [Fix] ci: Reduce runner utilization scan overhead and bound API retries (#45873) 2026-10-22 14:34:28 -07:00
Liangsheng Yin 840a6313c9 [Perf] dsv4: Score the V4.1 index logits in bf16 on SM100 (#45880) 2026-10-22 14:29:08 -07:00
25ad569ff9 [Feat] speculative: Seed rejection-sampling draft proposals for deterministic inference (#33395)
Co-authored-by: Ajit Mistry <ajmistry@nvidia.com>
Co-authored-by: Ajit Mistry <ajmistry@amistry.nvidia.com>
Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com>
Co-authored-by: Po-Han Huang <pohanh@nvidia.com>
2026-10-22 14:04:32 -07:00
AlexandJimmy Shong e5db41b329 [Fix] nemotron_vl: Pad visual spans using processor offsets (#44581)
Co-authored-by: Jimmy Shong <69131491+Jiminator@users.noreply.github.com>
2026-10-22 13:58:47 -07:00
786cdfc040 [Fix] nccl: Disable graph buffer registration whenever a captured graph can hold a collective (#43216)
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Jimmy Shong <69131491+Jiminator@users.noreply.github.com>
2026-10-22 13:56:19 -07:00
Yanbin Jiang 13999673ef [Feat] lora: Add plan-driven dense and MoE LoRA engines (#45892) 2026-10-22 13:40:56 -07:00
Shunkangz 154e29c1ec [Perf] CP: QSA Context Parallel Optimization (#43792) 2026-10-22 13:33:04 -07:00
Liangsheng Yin ba83415eea [Perf] engram: Read host-table rows in address order on GPUs without host page tables (#45879) 2026-10-22 13:13:43 -07:00
bf0e193bfd [Refactor] radix-tree: Move insert overlap handling into a resumable OVERLAP phase (#45702)
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com>
Co-authored-by: Jialin <Jialin@users.noreply.github.com>
2026-10-22 13:12:37 -07:00
aae2145eb2 [Feat] radix-tree: Report the last visited node and a shorter FULL device match in match results (#45703)
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com>
Co-authored-by: Jialin <Jialin@users.noreply.github.com>
2026-10-22 13:11:33 -07:00
357323e912 [Refactor] hicache: Route buffer-mode backup admission keys and FULL transfers through pipeline hooks (#45701)
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com>
Co-authored-by: Jialin <Jialin@users.noreply.github.com>
2026-10-22 13:10:18 -07:00
Liangsheng Yin 4f96807ea8 [CI] radix-cache: Build unified radix cache test states through production paths (#45334) 2026-10-22 13:04:23 -07:00
YAMY d67d1ae02c [Fix] logits: Preserve lm_head precision with shared buffers (#45855) 2026-10-22 12:46:27 -07:00
Liangsheng Yin 0d6b283d7e [Refactor] dsv4: Keep only the two-level V4.1 indexer in the prefill graph (#45783) 2026-10-22 12:26:43 -07:00
WenhaoZhang 8d8916815b [Fix] ci: Add pytest entry point for Muse Spark router tests (#45843) 2026-10-22 11:39:34 -07:00
Zhangheng 913e91ed51 [Fix] CI: add missing entry point to Muse Spark router test (#45845) 2026-10-23 00:45:41 +08:00
Aarne Talman c80285ca9a [Fix] [AMD] deterministic: Refuse RCCL all-reduce in deterministic mode above TP2 (#43809)
Signed-off-by: Aarne Talman <aarne.talman@amd.com>
2026-10-22 09:31:25 -07:00
ShunkangzandClaude Opus 5.5 cc811ff0c3 [Feat] KVShard: Add attention and indexer support for page-sharded DSA caches (#44280)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-23 00:16:35 +08:00
Shuwen Wang 8cd3c494a3 [Fix] mem_cache: Skip linker L3 lookup on PD decode (#45538) 2026-10-22 22:36:32 +08:00
Wenbo Ji 嵇文博 afd0d7b47b [Fix] diffusion/flux: use true_cfg_scale as the FLUX CFG weight (#44260) 2026-10-22 21:30:45 +08:00
Mick 0f7d3a0451 [CI] diffusion: target thirty-minute CI shards (#45761) 2026-10-22 21:19:42 +08:00
SunandMick dcf05724f7 [Perf] diffusion: accumulate chunked LoRA merges with addmm (#37620)
Co-authored-by: Mick <mickjagger19@icloud.com>
2026-10-22 21:18:39 +08:00
trebladev 698fa949ec [Perf] diffusion: fuse FLUX3 SwiGLU with rowwise FP8 quantization (#42758) 2026-10-22 21:17:56 +08:00
Song Bianandkernel-design-agents 635836129b [Perf] [KDA] muse_spark: use BLOCK_M 240 for BF16 MegaMoE prefill shards (#45637)
Co-authored-by: kernel-design-agents <kernel-design-agents@users.noreply.github.com>
2026-10-22 20:57:30 +08:00
gjsheu a9902f47e3 [Fix] [NPU] qwen: adapt pcp-v2 (#41564) 2026-10-22 20:54:11 +08:00
Song Bianandkernel-design-agents 851419717a [Perf] [KDA] muse_spark: Run the one-token router GEMV on CuTe DSL (#45508)
Co-authored-by: kernel-design-agents <kernel-design-agents@users.noreply.github.com>
2026-10-22 20:49:29 +08:00
Trevor Morris 91fc46a017 [Fix] [NVIDIA] docker: Disable torch 2.15 nccl2 in CUDA 13.4 container to fix EPLB rebalance crash (#45456) 2026-10-22 20:46:32 +08:00
Song Bianandkernel-design-agents 0ec3fee429 [Perf] [KDA] cookbook: enable custom all-reduce for GB300 BF16 Muse Spark (#45429)
Co-authored-by: kernel-design-agents <kernel-design-agents@users.noreply.github.com>
2026-10-22 20:46:28 +08:00
Song Bianandkernel-design-agents e43f6128a0 [Fix] [KDA] cookbook: add MoE runner backend to Muse Spark BF16 recipes (#45422)
Co-authored-by: kernel-design-agents <kernel-design-agents@users.noreply.github.com>
2026-10-22 20:38:35 +08:00
Jzz1943 f4ccfd4183 [Perf] moe: Add H200 block-FP8 Triton configurations (#45591) 2026-10-22 20:37:54 +08:00
MonishverandXiaoyu Zhang 178febef5d [Perf] qknorm: run head_dim-128 rows two per warp with 16B lanes, bit-identical (#45214)
Signed-off-by: Monishver Chandrasekaran <monishverchandrasekaran@gmail.com>
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com>
2026-10-22 20:34:16 +08:00
81841d6bdc [Fix] hicache: Fail fast on a null pointer from the Mooncake host allocator (#40565)
Co-authored-by: huangtingwei <141888744+huangtingwei9988@users.noreply.github.com>
Co-authored-by: Teng Ma <sima.mt@alibaba-inc.com>
2026-10-22 20:12:25 +08:00
Jzz1943 8a6770d566 [Fix] NIXL: Preserve explicit UCX network device selection (#45722) 2026-10-22 20:07:27 +08:00
weireweireandweireweire 96f013b452 [Doc] Cookbook: update GB300 DSV4.1 disagg AgentX launches from InferenceX #4303 (#45495)
Co-authored-by: weireweire <20922698+weireweire@users.noreply.github.com>
2026-10-22 19:18:55 +08:00
Alex NailsandClaude Opus 5.5 d0a7565436 [Fix] diffusion: Drop stale mismatch_msg from FLUX 3 video VAE gate call (#45778)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-22 01:31:54 -07:00
Liangsheng Yin 3f6c313389 [Fix] diffusion: Restore the kernel registry import after #41894 (#45775) 2026-10-22 01:25:13 -07:00
ashwini rathi 5c2cb60790 [Fix] [Intel] diffusion: Restore copy offload for FLUX.2-dev nightly (#45725) 2026-10-22 16:24:41 +08:00
Michael 0a87f163dc [Test] [AMD] deepseek-v4.1: Fix the HIP breakable-graph unit tests for the default prefill context limit and the DSpark draft ring (#45507) 2026-10-22 01:14:14 -07:00
kangwangamd 1c967f5043 [Fix] [AMD] test: Recalibrate MiniMax-M2.7 GSM8K threshold 0.93->0.91 for MI300 runners (#34241) 2026-10-22 01:08:54 -07:00
jacky.cheng b5137693a5 [Fix] [AMD] diffusion: Enable --quantization fp8 on gfx1250 (torch.compile lowering + per-channel weight scales) (#45304) 2026-10-22 01:05:36 -07:00
WenhaoZhangandMick Qian 24e45961c6 [Perf] diffusion: fuse QKV norm + RoPE, cached RoPE tables for FLUX 3 video VAE (#41894)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-22 15:44:54 +08:00
James XuandCursor c64886f9e3 [Perf] diffusion: fuse Wan VAE DupUp3D shortcut add (#32860)
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-22 15:38:51 +08:00
Ankit DhallandAnkit Dhall f727076ed0 [Feat] diffusion: support wan-animate-2 distilled model (#44085)
Co-authored-by: Ankit Dhall <8938083+ankitdhall@users.noreply.github.com>
2026-10-22 15:37:08 +08:00
Shuwen Wang 9d5386fd13 [Perf] dsv41: Fuse request-window layout construction (#45726) 2026-10-22 00:13:33 -07:00
shun8686 33a0734ca1 [CI] [NPU] precise-test: collect multimodal_gen coverage per test case (#45553) 2026-10-22 15:11:47 +08:00
黄孝君 569dda1e61 [CI] [NPU] npu: Update SGLang kernel tag to 2026.9.0.post10 (#45288) 2026-10-22 14:55:56 +08:00
Baizhou Zhang 779adb4b82 [Perf] cuda-graph: Raise MLA prefill capture default to chunked size capped at 8192 (#45565) 2026-10-21 23:49:25 -07:00
f68e82e7bc [Fix] epd: Accept tokens-in requests on scheduler-side transfer backends (#41376)
Signed-off-by: Krishnan Prashanth <kprashanth@nvidia.com>
Co-authored-by: Shangming Cai <csmthu@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-22 14:45:38 +08:00
Liangsheng Yin 82f7249ce6 [Perf] dsv4: Gather V4.1 index-K slots once per prefill graph replay (#45741) 2026-10-21 23:34:17 -07:00
Ziyi XuandClaude Opus 5.5 a56024c267 [CI] flash-mla: Add the sgl-flash-mla release workflow (#45651)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-21 23:34:14 -07:00
Shangming Cai 9569a36561 [Feat] scheduler: Gate --prefill-decode-interval on running batch size (#44001) 2026-10-22 14:30:12 +08:00
Richard Wang ce66b3ab59 [Feat] systemone: serve Kev pointer head decision models (#45413) 2026-10-22 14:16:52 +08:00
Po-Han Huang (NVIDIA)andClaude Opus 5.5 12370da115 [Doc] cookbook: Refresh SemiAnalysis AgentX launches with five open InferenceX PRs (#45749)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-22 14:15:19 +08:00
Shangming Cai a6a6beffac [Refactor] disaggregation: Share the flag all-reduce and host-destination predicate in the decode transfer queue (#45641) 2026-10-22 13:42:55 +08:00
WenhaoZhangandMick d9ce511646 [Feat] diffusion: support FastH3 Omni (Ref2VA 8-step PDD) (#45530)
Co-authored-by: Mick <mickjagger19@icloud.com>
2026-10-22 13:07:13 +08:00
MickandMick Qian 3472e16f96 [Refactor] diffusion: remove the deprecated kitchen_int8 alias (#45706)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-22 13:06:07 +08:00
Shuwen Wang 80263b5580 [Fix] sgl-router: Do not abort forwarded requests on reqwest's default 30 s TCP_USER_TIMEOUT (#45696) 2026-10-22 12:43:54 +08:00
zijieandZijie Chen f91b81b82a [Doc] [AMD] cookbook: Add Muse Spark Open MI355X AgentX recipe (#45689)
Co-authored-by: Zijie Chen <300606707+zijiecode@users.noreply.github.com>
2026-10-21 21:12:52 -07:00
ApeiriaandRain Jiang 8d7df07a56 [Feat] rust-server: Add HiCache storage management endpoints (#44726)
Signed-off-by: 0130w <1157374677@qq.com>
Co-authored-by: Rain Jiang <96632942+rainj-me@users.noreply.github.com>
2026-10-22 04:01:35 +00:00
Liangsheng Yin 17babd67b4 [Fix] dsv4: Never read attention metadata for idle SWA stores (#45699) 2026-10-21 20:56:40 -07:00
WenhaoZhang 45de4adde9 [Feat] diffusion: Add worker-side ComfyUI compatibility patches (#44841) 2026-10-21 20:41:51 -07:00
Thomas Wang 9ee9858805 [Doc] [AMD] dsv41: Update amd cookbook 1021 (#45693) 2026-10-21 20:33:34 -07:00
Khoa Pham f1e26dc658 [Perf] logits: Write lm_head logits straight into the fp32 logits buffer (#45631) 2026-10-21 20:32:16 -07:00
WenhaoZhang 1962cbeebe [Perf] diffusion: convrot_int8 comfy_kitchen issues one GEMM per layer by default (#44179) 2026-10-22 11:30:32 +08:00
Viraat ChandraandClaude Opus 5.5 5f62e23d30 [Doc] cookbook: Add Muse Spark Open GB300 Dynamo + SGLang deployments measured with AA AgentPerf (#45629)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-22 11:24:58 +08:00
Shuwen Wang 6929afb528 [Fix] hisparse: Keep PD destination host pages until the decode KV release (#45108) 2026-10-22 11:19:21 +08:00
8dc7f98005 [Perf] pd: Send prefill KV chunks without a host sync on the scheduler thread (#45492)
Co-authored-by: alphabetc1 <2508695655@qq.com>
Co-authored-by: Shuwen Wang <47200617+alphabetc1@users.noreply.github.com>
2026-10-21 20:11:16 -07:00
3468a65b51 [CI] labeler: Add a mooncake PR label (#45578)
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Teng Ma <stmatengss@users.noreply.github.com>
2026-10-22 11:09:48 +08:00
MickandMick Qian 780a5b1acc [Doc] diffusion: document prompt enhancer tasks and link Cosmos recipes (#45705)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-22 10:28:58 +08:00
Tamir BaydasovandClaude Opus 5.5 b057d5b3f1 [Fix] [NPU] mla-pool: DCP allocator page and index-K bound, short-pool warning (#42976)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-22 05:24:14 +03:00
b453996b7d [Doc] diffusion/minimax-h3: Refresh VDN-H3 B200 benchmarks, fix 8-step bench presets and popular-models snippet (#45267)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: zijiexia <37504505+zijiexia@users.noreply.github.com>
2026-10-22 10:09:13 +08:00
em-dreyerandClaude Opus 5.5 3d508d614f [CI] [NVIDIA] gb300: Run DSV4.1-Flash tests on 2 GPUs with engram host tables (#45294)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-21 18:53:24 -07:00
wutao 67bbfcee3c [Perf] [NPU] scheduler: Remove blocking seq_lens D2H from graph replay (#45170) 2026-10-22 09:50:04 +08:00
Liangsheng Yin d61f618276 [Perf] spec: Let DSPARK prefill publish read-done by placing its draft KV inject before the target forward (#45682) 2026-10-21 18:40:44 -07:00
4049941a82 [Fix] [NPU] scope: fix dtype error in sampling backend ascend (#39609)
Co-authored-by: ronnie_zheng <zl19940307@163.com>
Co-authored-by: Артем Савкин <58187114+OrangeRedeng@users.noreply.github.com>
2026-10-22 04:34:29 +03:00
Khoa PhamandClaude Opus 5.5 aa10927271 [Perf] dsv41: Mega-mHC and the wo_a einsum write MXFP8 for wqkv_a and wo_b (#45632)
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-21 18:27:21 -07:00
Baizhou Zhang 59149553b9 [Refactor] attention: centralize linear and draft TP group selection (#45684) 2026-10-21 18:25:40 -07:00
Rohan VarmaandYangmin Li 9ec74a53cc [Fix] [NVIDIA] docker: Build sgl-deep-gemm 0.2.3 for cu134 to match the sglang pin (#45521)
Co-authored-by: Yangmin Li <yangminl@nvidia.com>
2026-10-22 09:01:05 +08:00
Jimmy Shong 9301ca37a0 [Fix] modelopt: Correct NVFP4 TRT-LLM clamps for untied gate/up scales (#45386) 2026-10-21 17:37:07 -07:00
Michael 5fff58a169 [CI] [AMD] miles: Fix the ROCm 10 nightly suites that never finish (#45469) 2026-10-21 17:24:51 -07:00
eacd0a07d7 [Feat] bench: support --tokenize-prompt for generated-shared-prefix (#40900)
Co-authored-by: Yihao Wang <42559837+AgainstEntropy@users.noreply.github.com>
Co-authored-by: chaoyujiang <chaoyujiang@hygon.com>
Co-authored-by: Shangming Cai <csmthu@gmail.com>
2026-10-22 08:17:15 +08:00
Khoa Pham ed89693c4a [Perf] dsv41: Fuse the q_norm MXFP8 quantize for wq_b at every row count (#45633) 2026-10-21 16:56:45 -07:00
Liangsheng Yin da7b97a737 [Perf] dsv4: Publish prefill read-done under the request window and on V4.1 (#45582) 2026-10-21 16:29:28 -07:00
Liangsheng Yin 3438ed660d [Test] cpu: prune mirror and tautology cases from the per-commit CPU suite (#45672) 2026-10-21 16:02:47 -07:00
ed1f56fead [Fix] [Apple] mlx: Give the stub model a shared-expert layout (#45419)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: R0CKSTAR <yeahdongcn@gmail.com>
2026-10-21 15:54:49 -07:00
Baizhou Zhang f4035cab0b [Feat] cp: Enable GLM-5.3 Flash interleave BCG with batched prefill and DFlash2 (#43182) 2026-10-21 15:51:28 -07:00
42db2db4a8 [Fix] rust-server: Check only a client-sent max_new_tokens against the context window (#45460)
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com>
Co-authored-by: shodoco <shodoco@users.noreply.github.com>
2026-10-21 15:50:13 -07:00
991b846227 [Perf] speculative: Parallelize EAGLE spec-decode KV index building over token blocks (flashinfer draft decode; verify/draft-extend for flashinfer, flashinfer_mla, aiter) (#42803)
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: YAMY1234 <74099316+YAMY1234@users.noreply.github.com>
Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com>
Co-authored-by: Po-Han Huang <pohanh@nvidia.com>
2026-10-21 15:26:35 -07:00
maocheng23 4b9936b5e3 [Fix] deepseek_v4: Fix mHC prewarm when chunked prefill is disabled (#43448) 2026-10-21 15:21:31 -07:00
Liangsheng Yin 99cf403c9e [Fix] dsv4: Write idle SWA stores to an idle request-window layout (#45586) 2026-10-21 15:18:34 -07:00
em-dreyer aa5880e2b8 [Fix] bailing: Renormalize top-k routing weights by default for Ling-2.5/2.6 (#45662) 2026-10-21 14:53:58 -07:00
Michael 51baa1f6e9 [Test] [AMD] debug-utils: Opt TestCrashDump into expect_server_exit (#45408) 2026-10-21 14:49:53 -07:00
Liangsheng Yin f3f0f7b68c [Perf] dsv4: Let the decoder tail leave the prefill graph under DP attention with MegaMoE (#45581) 2026-10-21 14:44:21 -07:00
Liangsheng Yin f5eca05b05 [Fix] [SM100] quantization: Honor deep_gemm for 32x32 block-FP8 linears (#45585) 2026-10-21 14:42:50 -07:00
29e78045fb [Fix] dsv4: Skip DP-attention pad rows in the engram hasher and low-ratio sources (#40424)
Co-authored-by: jiashuai <jiashuai@baidu.com>
Co-authored-by: hnyls2002 <lsyincs@gmail.com>
2026-10-21 14:37:55 -07:00
Liangsheng Yin 68623cae92 [Fix] test: Gather Mamba track indices from track_position in the mixed-batch track test (#45670) 2026-10-21 14:33:48 -07:00
Liangsheng Yin f7d4f06242 [Perf] dsv4: Compute request-window history rows once per batch (#45579) 2026-10-21 14:01:19 -07:00
Liangsheng Yin bc412213c1 [Perf] dsv4: Gather V4.1 index-K slots once per forward and cache the prefill index plan (#45577) 2026-10-21 14:00:35 -07:00
Liangsheng Yin 620718c8ea [Fix] moe: Keep the DeepSeek V4.1 vision router off the packed-ids path under routing simulation (#45575) 2026-10-21 13:58:44 -07:00
Liangsheng Yin 97945831f9 [Perf] dsv4: Restore sparse prefill for request-window rows (#45573) 2026-10-21 13:54:32 -07:00
Liangsheng Yinandlixiufei-leo 2c7a9e3b50 [Fix] radix-cache: Order linker offloads behind the forward that writes the pages (#45559)
Co-authored-by: lixiufei-leo <294029388+lixiufei-leo@users.noreply.github.com>
2026-10-21 13:53:37 -07:00
Khoa Pham 028de2c5d4 [Perf] dsv41: Use tuned MXFP8 tactics at decode sizes (#45630) 2026-10-21 13:35:56 -07:00
Yoni GozlanandJimmy Shong a2e4ffbcea [Fix] parser: Accept whole-number floats for integer tool arguments (#43941)
Co-authored-by: Jimmy Shong <69131491+Jiminator@users.noreply.github.com>
2026-10-21 13:04:15 -07:00
Michael 7e0e259435 [CI] [AMD] disaggregation: Register Mooncake KV buffers without HIP dmabuf (#45519) 2026-10-21 12:47:42 -07:00
a3d00eff58 [Perf] srt: Record on the host whether an extend batch completes a prompt (#45487)
Co-authored-by: Lu Fang <116399278+luccafong@users.noreply.github.com>
Co-authored-by: shodoco <shodoco@users.noreply.github.com>
2026-10-21 11:58:25 -07:00
7e5256d1ba [Feat] srt: Balance uvicorn tokenizer workers with per-worker SO_REUSEPORT listeners (#45482)
Co-authored-by: luccafong <116399278+luccafong@users.noreply.github.com>
Co-authored-by: joshuadeng <joshuadeng@users.noreply.github.com>
2026-10-21 11:57:26 -07:00
Baizhou Zhang f06d1bb1f4 [Perf] cuda-graph: Capture FlashInfer KDA prefill (#45473) 2026-10-21 11:57:23 -07:00
740d5f9f9e [Perf] scheduler: Plan hybrid Mamba extend batches without host-device syncs (#45472)
Co-authored-by: Lu Fang <116399278+luccafong@users.noreply.github.com>
Co-authored-by: Shiyan Deng <20773414+842974287@users.noreply.github.com>
2026-10-21 11:56:50 -07:00
5e79333c95 [Refactor] radix-tree: Reuse the insert walk, component hooks and bindings for custom tree cores (#45439)
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com>
Co-authored-by: Jialin <Jialin@users.noreply.github.com>
2026-10-21 10:53:26 -07:00
paulzhang-tm b57b5e2286 [Fix] disaggregation: Synchronize deferred decode KV release across TP ranks (#45453) 2026-10-22 01:37:28 +08:00
Khoa Pham ed268dd320 [Fix] moe: Apply SGLANG_SIMULATE_* routing on DeepSeek-V4.1 vision-biased gates (#45628) 2026-10-21 10:00:15 -07:00
vorapolsiloai 6e98c84c60 [Perf] [AMD] quark: Add opt-in AITER ASM FP4 GEMM for Quark dense linear (#39598) 2026-10-21 09:37:47 -07:00
Michael 4b1a44bc77 [Test] [AMD] diffusion: Avoid Python 3.11-only enterContext in test_usp_ring_replicated (#45409) 2026-10-22 00:11:08 +08:00
zijiexia 75878f27c8 [Doc] cookbook: Add Recursive Flint cookbook (#45625) 2026-10-21 08:58:16 -07:00
Shiyang Lyu cdbd478422 [Doc] [NPU] cookbook: add tested A2 deployment and A3 reference for MiniMax-M2.5 (#40120) 2026-10-21 22:59:37 +08:00
Huang LongandShangming Cai af49ada16d [Fix] pd: Allow runtime role switching with DP and DP attention (#44215)
Co-authored-by: Shangming Cai <csmthu@gmail.com>
2026-10-21 22:58:29 +08:00
Peng Xingchen 7163e97462 [Doc] [NPU] GLM: Add Ascend cookbook deployment recipes (#45597) 2026-10-21 22:44:08 +08:00
8e3e2980db [Feat] [Intel] disaggregation: Support XPU for staging-buffer KV transfer (#40861)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Shangming Cai <csmthu@gmail.com>
2026-10-21 22:09:04 +08:00
pllimax 7e5d3780ef [Fix] [NPU] tests: Run the HiCache Mamba case with bfloat16 SSM kernels (#45593) 2026-10-21 21:48:12 +08:00
Shawn Yang dd9ea06c00 [Fix] rust: Format health probe handle setup (#45606) 2026-10-21 21:40:08 +08:00
unclezhou e109bc1832 [Doc] [NPU] deepseek-v4: Add A3/950 single-node cookbook recipes (#45602) 2026-10-21 21:16:49 +08:00
amote-i 2815eed49c [Fix] [NPU] pre_warm_nccl: init npu backend before the parallel runtime touches a device (#43039) 2026-10-21 19:44:30 +08:00
amote-i fda84f80b7 [Doc] [NPU] docs: reference released Qwen3.8-Max W4A8 checkpoint in tutorial (#45574) 2026-10-21 19:38:17 +08:00
Shangming Cai a175e39b84 [CI] cuda: Rebuild workspace crates when the shared Cargo target dir came from another rust/ tree (#45557) 2026-10-21 19:32:22 +08:00
Avaya Aggarwalandniehen6174 62b4127668 [Feat] comfyui: Apply ControlNet residuals in Flux integrated mode (#43698)
Co-authored-by: niehen6174 <1639206518@qq.com>
2026-10-21 19:29:43 +08:00
Shangming Cai 43a3f956cb [Fix] sgl-router: Wait for the detached PD prefill in routing tests (#45502) 2026-10-21 19:04:26 +08:00
pllimax 43728298fe [Fix] [NPU] qwen3_6: Set accuracy cases to greedy decoding and tune retries (#40889) 2026-10-21 18:43:30 +08:00
ca72dad953 [Fix] rust-server: Answer health probes a busy scheduler skipped on the response ring (#45475)
Co-authored-by: Lu Fang <116399278+luccafong@users.noreply.github.com>
Co-authored-by: shodoco <shodoco@users.noreply.github.com>
2026-10-21 03:41:36 -07:00
Jimmy Shong 5bf10d4c79 [Fix] nemotron: Correct weight-export failures and GEMV dispatch (#45082) 2026-10-21 18:38:35 +08:00
Jzz1943 2e7c9d851c [Feat] NIXL: Support per-GPU UCX device selection (#43482) 2026-10-21 17:54:11 +08:00
bz f 9c10e0dd61 [Doc] [NPU] cookbook: add NPU (A3 series) deployment scenarios for Qwen3.6 (#38678) 2026-10-21 17:37:24 +08:00
Shangming Cai 7554be5a69 [CI] codeowners: Add owners for the top-level sgl-router (#45564) 2026-10-21 17:28:21 +08:00
kangwangamd 131c42ba33 [Perf] [AMD] aiter: Update ROCm AITER pin to 285579b62 (gfx942/gfx950 only) (#45498) 2026-10-21 02:25:02 -07:00
Shangming Cai be65391abd [Feat] mooncake: Enable the failed-session probe by default with backoff and idle pruning (#45338) 2026-10-21 17:16:19 +08:00
Thomas Wang 1278087336 [Fix] [AMD] mono-kernel: Update sgl-amd-mono version (#45546) 2026-10-21 02:02:03 -07:00
2c9d919071 [Fix] srt: Pin the source address of TCP ZMQ connects for the peer allowlist (#45437)
Co-authored-by: Lu Fang <116399278+luccafong@users.noreply.github.com>
Co-authored-by: fei-xx <fei-xx@users.noreply.github.com>
2026-10-21 01:59:03 -07:00
21f775cd97 [Feat] rust-server: Let the launcher set the default max_new_tokens for /generate (#45431)
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com>
Co-authored-by: shodoco <shodoco@users.noreply.github.com>
2026-10-21 01:58:34 -07:00
Aurick QiaoandXiaoyu Zhang 7a632cf998 [Fix] quantization: Preserve BF16 NaNs in per-token FP8 quantization (#42689)
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com>
2026-10-21 16:52:28 +08:00
8bd6aa9a6c [Feat] parser: Enrich chat_parsing streaming events (#40478)
Co-authored-by: Yoni Gozlan <yonigozlan@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
Co-authored-by: Jimmy Shong <69131491+Jiminator@users.noreply.github.com>
2026-10-21 01:46:26 -07:00
AlexandJimmy Shong dbe0a9a7c9 [Fix] nemotron_vl: Preserve tokenized prompts during media expansion (#44184)
Co-authored-by: Jimmy Shong <69131491+Jiminator@users.noreply.github.com>
2026-10-21 01:40:51 -07:00
Liangsheng Yin 6efdd33d07 [Refactor] spec: Move platform dispatch and Triton kernels into kernels/ops (#45504) 2026-10-21 01:22:16 -07:00
6cac4dc3fe [Refactor] diffusion: Rollout controller as a scheduler-step plugin instead of mixins (#44603)
Signed-off-by: rockdu <kangrdu@gmail.com>
Co-authored-by: WenhaoZhang <42087078+niehen6174@users.noreply.github.com>
Co-authored-by: JingwenGu0829 <75733630+JingwenGu0829@users.noreply.github.com>
2026-10-21 16:04:33 +08:00
Avaya AggarwalandWenhaoZhang fee32a5c1f [Feat] ComfyUI: Add online INT8 quantization for BF16 Qwen-Image-2.1 single files (#44865)
Co-authored-by: WenhaoZhang <42087078+niehen6174@users.noreply.github.com>
2026-10-21 16:01:18 +08:00
Jzz1943 a361f0c410 [Fix] kda: Align graph tracking branch dtype with the recurrent state pool (#44886) 2026-10-21 15:58:34 +08:00
f7b8c6fe40 [Fix] glm-5.2: Record side-stream work right before its join to keep CUDA-graph replay on one stream (#45349)
Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com>
2026-10-21 15:36:54 +08:00
Hongkuan Zhouandishandhanani 8acfabbd67 [Fix] observability: Correct speculative FPM timing and decode lengths (#38711)
Signed-off-by: hongkuanz <hongkuanz@nvidia.com>
Co-authored-by: ishandhanani <82981111+ishandhanani@users.noreply.github.com>
2026-10-21 15:31:18 +08:00
Shangming Cai cd61db49c7 [Fix] sgl-router: Bump rustls-webpki and webpki-roots for emSign advisories (#45526) 2026-10-21 15:24:11 +08:00
YC Yen-Ching Tseng 132836b369 [Fix] [AMD] docker: Build Triton from source for ROCm 10.1 gfx1250 (#45329) 2026-10-21 15:14:24 +08:00
Peng Xingchenandpengxingchen0810 4449d7ecb0 [Doc] [NPU] GLM: Add GLM-5.3 and GLM-5.3-Flash deployment tutorials (#43372)
Co-authored-by: pengxingchen0810 <pengxingchen0810@noreply.gitcode.com>
2026-10-21 15:02:58 +08:00
Junlin Wu 844d2b5dec [Fix] [NPU] qwen3.5: Fix dense serving on Ascend 950 (#32745) 2026-10-21 09:50:38 +03:00
Jimmy Shong 8599b7251f [Chore] deps: Bump the OpenAI SDK to 2.8.0 for Responses shell tools (2/8) (#45291) 2026-10-20 23:49:30 -07:00
wutao fb6e6a09ce [Fix] [NPU] disaggregation: Fix host-pool backup for decode retraction (#43814) 2026-10-21 14:48:01 +08:00
chenyang08056032 c3b2b55bbb [CI] [NPU] precise-test: add recommendation summary & run artifacts, fix NPU-only coverage view, and quiet benign teardown noise (#45374) 2026-10-21 14:40:50 +08:00
Khoa PhamandAMD-yanfeiwang da7e3f1c12 [Fix] cache: Accept packed draft layer mappings in the UMBP linker (#44929)
Co-authored-by: AMD-yanfeiwang <yanfei.wang@amd.com>
2026-10-20 23:27:22 -07:00
Thomas Wang bd5a445bf7 [Fix] [AMD] dsv41: Set moe act to fp8 by default (#45480) 2026-10-20 23:09:27 -07:00
Shangming Cai 574f9cdb2e [CI] slash-command: Add /run-cpu-ci to run only the CPU unit suite (#45501) 2026-10-21 13:59:16 +08:00
956b9a24ee [Perf] [AMD] qwen3.5: Fuse activation quant into small-M FP8 block-scale GEMM on gfx950 (#42229)
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: jacky.cheng <yichiche@amd.com>
2026-10-20 22:32:30 -07:00
Po-Han Huang (NVIDIA)andClaude Opus 5.5 1689cb2d52 [Doc] cookbook: Add GLM-5.3 AgentX section and refresh with eight open InferenceX PRs (#45513)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-21 13:22:38 +08:00
Aarne Talman 4b5c9c4b5e [Perf] [AMD] attention: Restore the gfx950 extend tile for head_dim <= 128 (#45099) 2026-10-20 22:02:50 -07:00
32c26788f3 [Fix] mamba: Keep prefill rows' checkpoint tracking through mixed chunked prefill (#45435)
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com>
Co-authored-by: YavorGIvanov <YavorGIvanov@users.noreply.github.com>
2026-10-20 21:56:50 -07:00
Liangsheng Yin 573b5716f5 [Fix] spec: Take post-verify KV move targets from the verify window (#45491) 2026-10-20 21:12:41 -07:00
1c2f42470c [Fix] disaggregation: Keep a requeued optimistic prefill request's multimodal inputs (#45440)
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com>
Co-authored-by: libertyeagle <30348494+libertyeagle@users.noreply.github.com>
Co-authored-by: fei-xx <135589532+fei-xx@users.noreply.github.com>
2026-10-20 20:27:34 -07:00
metamergebotandshodoco 6bd1c76735 [Fix] rust-server: Drop the fixed 64-item multimodal cap in favor of --limit-mm-data-per-request (#45432)
Co-authored-by: shodoco <shodoco@users.noreply.github.com>
2026-10-20 20:25:22 -07:00
Kan WuandClaude Opus 5.5 aba892fbdf [Refactor] sgl-router: Move experimental/sgl-router to top-level sgl-router (#45414)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-21 11:09:13 +08:00
eb91a41f38 [Feat] unified_cache: Support FULL + MAMBA MLA models (Kimi-K3) in the direct external linker (#44977)
Co-authored-by: Yutong Wu <yutong.wu@amd.com>
Co-authored-by: Zhangheng <hzh0425@apache.org>
2026-10-21 10:47:17 +08:00
metamergebot f1ba723c64 [Test] [AMD] moe: Expect gfx1250 W4A8 layers to build with expert parallelism (#45476) 2026-10-21 10:45:31 +08:00
Chunan Zeng 93088901fb [Chore] attention: Remove the tokenspeed_mla backend and dependency (#44803) 2026-10-20 18:50:05 -07:00
unclezhou c0d18c0254 [Fix] [NPU] dspark: Fix attn_tp > 1 draft wo_b all_reduce group (#40399) 2026-10-21 09:20:15 +08:00
c1c287a429 [Doc] cookbook: add TPU7x recipe for GLM-5.3-Flash (#45326)
Co-authored-by: Qinghan Chen <qinghan.chen@radixark.net>
Co-authored-by: zijiexia <37504505+zijiexia@users.noreply.github.com>
2026-10-20 17:10:18 -07:00
b326e8349d [Perf] mamba: Copy tracked decode states at HBM bandwidth in one persistent launch (#45213)
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com>
Co-authored-by: YavorGIvanov <YavorGIvanov@users.noreply.github.com>
2026-10-20 16:52:15 -07:00
Baizhou Zhang 5b1dbc2c80 [Fix] [NVIDIA] modelopt-nvfp4: Keep MoE global scales in gate/up order (#45289) 2026-10-20 16:49:13 -07:00
kkandwunhuang 9686edcf80 [Perf] [AMD] dsv41: faster mHC boundary Triton tile for 96+ rows, bitwise unchanged (#45124)
Co-authored-by: wunhuang <wunhuang@amd.com>
2026-10-20 16:39:34 -07:00
Cheng Wan 027208ae02 [Doc] layer-boundary/moe: Correct guides and comments that drifted from the code (#44896) 2026-10-20 16:26:40 -07:00
Cheng WanandAditya Suresh 299a75ac5d [Refactor] moe: Derive routing inputs from the layer and active forward (#44994)
Co-authored-by: Aditya Suresh <152752311+adi-suresh01@users.noreply.github.com>
2026-10-20 16:26:15 -07:00
kkandwunhuang 533020b500 [Perf] [AMD] dsv41: run gfx950 MXFP8 dense GEMMs on aiter's preshuffled FlyDSL kernel (#45123)
Co-authored-by: wunhuang <wunhuang@amd.com>
2026-10-20 16:19:32 -07:00
Cheng WanandAditya Suresh 4f8f06faf3 [Refactor] moe: Derive layer contracts from the selected quant method (#44993)
Co-authored-by: Aditya Suresh <152752311+adi-suresh01@users.noreply.github.com>
2026-10-20 16:18:45 -07:00
zydoandCheng Wan 7b1d5cd02d [Doc] agent-skills: Fix stale paths and deleted guard tests in skills and rules (#43666)
Co-authored-by: Cheng Wan <cheng.wan@radixark.ai>
2026-10-20 16:15:10 -07:00
bf145750a6 [Perf] prefill-cuda-graph: Make prefill shared-read completion explicit and default-on (#37958)
Co-authored-by: weireweire <20922698+weireweire@users.noreply.github.com>
Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com>
Co-authored-by: Po-Han Huang <pohanh@nvidia.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: hnyls2002 <lsyincs@gmail.com>
2026-10-20 16:12:55 -07:00
Aarne Talman d5e83ea4f6 [Perf] [AMD] deterministic: Faster extend attention launch on gfx950 (#45098) 2026-10-20 16:11:01 -07:00
averyjones4andaveryjones4 c1e43b95f4 [Doc] cookbook: Add DeepReinforce Ornith-1.5 (#45455)
Co-authored-by: averyjones4 <averyjones4@users.noreply.github.com>
2026-10-20 16:08:53 -07:00
Cheng WanandAditya Suresh f1eec1232d [Fix] config: Preserve ServerArgs positional constructor slots (#44991)
Co-authored-by: Aditya Suresh <152752311+adi-suresh01@users.noreply.github.com>
2026-10-20 16:08:28 -07:00
Aditya SureshandCheng Wan e2e1a97fc3 [Fix] moe: Map expert ids for layers whose runner needs local ids under EP (#44480)
Co-authored-by: Cheng Wan <cheng.wan@radixark.ai>
2026-10-20 16:07:48 -07:00
Baizhou Zhang b008ec3de6 [CI] muse_spark: Move unified memory test to B300 (#45454) 2026-10-20 15:56:49 -07:00
Liangsheng Yin 1dd0f361b8 [Fix] moe: Run the shared-slot routing test without a published server config (#45448) 2026-10-20 15:03:47 -07:00
Baizhou Zhang 583cfc4d55 [CI] muse_spark: Use complete local checkpoints in model tests (#45428) 2026-10-20 14:57:01 -07:00
Yuwei An f0b6590722 [Fix] DSpark: Disable simulated acceptance shortcuts by default (#45433) 2026-10-20 14:10:59 -07:00
YAMY 7d9bc10493 [Perf] [NVIDIA] DSpark: Enable the simulated-acceptance fast paths on CUDA (#45421) 2026-10-20 13:36:33 -07:00
cc7ae372ac [Refactor] radix-cache: Route pending-backup split bookkeeping through one component action (#45242)
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com>
Co-authored-by: Jialin <Jialin@users.noreply.github.com>
2026-10-20 13:24:49 -07:00
0b536ca112 [Fix] radix-cache: Separate logical and reclaimable value lengths in tree components (#45238)
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com>
Co-authored-by: Jialin <Jialin@users.noreply.github.com>
2026-10-20 13:24:20 -07:00
metamergebotandjoshuadeng 468e26f253 [Perf] detokenizer: Add SGLANG_BATCH_TOKENIZER_OUTPUTS to opt out of grouped return sends (#45280)
Co-authored-by: joshuadeng <joshuadeng@users.noreply.github.com>
2026-10-20 13:24:14 -07:00
bab4a808d8 [Perf] unified-memory: Allocate free pages from each sub-pool's own end first (#45222)
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com>
Co-authored-by: ZYHowell <ZYHowell@users.noreply.github.com>
2026-10-20 13:24:09 -07:00
0d7638b035 [Fix] scheduler: Exclude accepted storage-tier hits from the waiting prefill load (#45158)
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com>
Co-authored-by: Jialin <Jialin@users.noreply.github.com>
2026-10-20 13:24:04 -07:00
dc62c3effd [Feat] radix-tree: Report each path node's child count in collect_path_info (#45241)
Co-authored-by: metamergebot <metamergebot@users.noreply.github.com>
Co-authored-by: Jialin <Jialin@users.noreply.github.com>
2026-10-20 13:23:59 -07:00
metamergebotanddaniel-salib 194876bea0 [Fix] srt: Admit only allowlisted peers on TCP-bound ZMQ IPC sockets (#45250)
Co-authored-by: daniel-salib <daniel-salib@users.noreply.github.com>
2026-10-20 13:23:53 -07:00
Yuwei An afe3e1df31 [Fix] DSpark: Restore weight-loading CPU tests (#45427) 2026-10-20 13:13:39 -07:00
Jimmy Shong d69966f9c0 [Chore] docker: Add Beam CUDA image (1/8) (#45253) 2026-10-20 13:00:46 -07:00
Baizhou Zhang cfec1b91bb [Fix] muse_spark: support unified KV memory with TRT-LLM (#45356) 2026-10-20 12:54:53 -07:00
Aarne Talman 29274b41c6 [Fix] [AMD] activation: Run ReLU2 natively on ROCm (#44198)
Signed-off-by: Aarne Talman <aarne.talman@amd.com>
2026-10-20 12:05:14 -07:00
Thomas Wang 82ac83a8d0 [Perf] [AMD] dsv41: Skip unneeded layer 20 computation under decoder-bounded-replay (#45339) 2026-10-20 11:49:05 -07:00
Michaelandjacky.cheng f82465718b [Test] [AMD] diffusion: Skip MOSS-Image mismatched-operand test on ROCm (#44596)
Co-authored-by: jacky.cheng <yichiche@amd.com>
2026-10-20 11:37:32 -07:00
Khoa Pham c2e5742464 [Perf] deepseek-v4: DSpark draft stages take the TF32 mHC statistics path from 32 rows on SM100 (#45260) 2026-10-20 10:12:06 -07:00
sogalin_codegen 89860fbd2e [Fix] [AMD] gpt-oss: support EP for gfx1250 W4A8 MXFP4 MoE (#45365) 2026-10-20 10:08:39 -07:00
Michael 7591118794 [Fix] [AMD] grok: accept allow_patterns_overrides in presharded _prepare_weights patch (#45217) 2026-10-20 16:12:06 +00:00
eb4b167d27 [Feat] KVShard: PD transfer and e2e test (#38473)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Zhangheng <hzh0425@apache.org>
2026-10-20 22:48:43 +08:00
7656071309 [Feat] platform: Add initial T-HEAD SAIL platform support [2/N] (#43826)
Co-authored-by: Zhangheng <hzh0425@apache.org>
Co-authored-by: yzshi <163985193+Yunzshi310@users.noreply.github.com>
2026-10-20 22:19:07 +08:00
pllimax e262b44eef [CI] [NPU] tests: Replace DeepSeek-V2-Lite-W8A8 cases with Moonlight (#43063) 2026-10-20 21:35:49 +08:00
Shangming Cai 952c63b1bb [CI] sgl-router: Gate on live run-ci label so /tag-and-rerun-ci can start the tiers (#45383) 2026-10-20 21:33:42 +08:00
Jensenandronnie_zheng 10d5293546 [Perf] [NPU] glm5.2: Support dynamic INT8 MLAProlog for attention (#43469)
Co-authored-by: ronnie_zheng <zl19940307@163.com>
2026-10-20 16:33:08 +03:00
Jensen 3bf4cbc6cf [Perf] [NPU] mem_cache: Increase GLM-5.2-W4A8C8 KV cache capacity with compact DSA indexer allocation (#44436) 2026-10-20 16:30:01 +03:00
gjsheu aa91108f64 [Fix] [NPU] glm5.2: support pcp v2 with interleave & zigzag (#40165) 2026-10-20 21:26:30 +08:00
gjsheu ee2f95a522 [Fix] [NPU] glm5.2-mtp: L2 cache adaptation (#41642) 2026-10-20 21:25:37 +08:00
df13d5dce3 [Feat] sgl-router: Route prefill buckets by uncached tokens (#45103)
Co-authored-by: Xiufei Li <294029388+lixiufei-leo@users.noreply.github.com>
Co-authored-by: Shangming Cai <csmthu@gmail.com>
2026-10-20 20:56:14 +08:00
Po-Han Huang (NVIDIA)andClaude Opus 5.5 eae134d61f [Test] kernels: Import silu_and_mul_contig_post_quant from its new home (#45368)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 19:48:59 +08:00
Kurkur 07eb671b6a [Perf] [NPU] attn: add not needs_cpu_seq_lens for dsa modules (#38461) 2026-10-20 19:19:34 +08:00
Shijin ZhangandXinyuan Tong ca473ef3e4 [Feat] glm-5.3: Render GLM-5.3 and GLM-5.3-Flash chat prompts with a Python chat template (#45154)
Signed-off-by: Shijin Zhang <75300765+Dovis01@users.noreply.github.com>
Co-authored-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
2026-10-20 19:11:40 +08:00
Ho-Ren (Jack) Chuang 800e60e345 [Fix] moe: Publish a server config in the CUTLASS runner dispatch-format and placement tests (#45376) 2026-10-20 19:05:17 +08:00
Yuxuan Zhou 306ccf6f57 [Perf] spec: Let the FlashAttention backend capture the draft-extend CUDA graph (#40762) 2026-10-20 03:05:42 -07:00
Sihan Chen 9db58ab547 [Fix] eplb: Fix Qwen3 MoE output corruption after EPLB rebalance (#39580)
Signed-off-by: Spycsh <sihan.chen@intel.com>
2026-10-20 03:00:39 -07:00
a70b1b73d3 [Fix] tracing: Fix OTel tracing context lost in multi-tokenizer router processes (#38211)
Co-authored-by: Ma <T9FOPWP@cn.vwgroup.com>
Co-authored-by: Shangming Cai <csmthu@gmail.com>
2026-10-20 17:57:16 +08:00
16eb87c06f [Fix] openai: distinguish s3 served-model-name from LoRA adapter syntax (#44450)
Signed-off-by: Muqi Li <muqi1029@gmail.com>
Signed-off-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
Co-authored-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
Co-authored-by: Xinyuan Tong <115166877+JustinTong0323@users.noreply.github.com>
2026-10-20 17:11:06 +08:00
Xiaoyu ZhangandClaude Opus 5.5 2229345dff [Doc] kda_kernels: Point the SM121 QSA kernel provenance at its SGLang PR (#45359)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 16:50:58 +08:00
Ho-Ren (Jack) Chuang f8fd41c36e [Perf] moe: Extend FlashInfer CUTLASS MoE autotune coverage to prefill chunks (#43363) 2026-10-20 01:41:16 -07:00
Ma MingfeiandCopilot 06314dc2e8 [CI] [Intel] tests: Remove CPU monitor and crash-dump tests from Xeon CI (#45281)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-10-20 16:40:13 +08:00
Xiaoyu ZhangandClaude Opus 5.5 0788f1fdd0 [Refactor] kernels: Register agent-generated kernels under their KDA / Cake backend (#45277)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 16:35:11 +08:00
Xinyuan Tong fdd89d182a [Fix] moe: add uses_expert_location to MegaMoE test fixtures (#45350)
Signed-off-by: Xinyuan Tong <xinyuantong.cs@gmail.com>
2026-10-20 16:20:46 +08:00
Mohammad Miadh AngkadandMohammad Angkad 78185f4b3f [Fix] [NVIDIA] modelopt-nvfp4: Read w13 global scales in the TRT-LLM [up, gate] order (#44858)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-10-20 07:38:45 +00:00
XXXXRT666andAlex Nails 1739d962a4 [Refactor] [Apple] MLX: Use Native Extension Builder for Metal Kernels (#44367)
Co-authored-by: Alex Nails <alex.nails@radixark.ai>
2026-10-20 00:35:29 -07:00
Shuwen Wang 15e8e687db [Chore] environ: Remove retired unified radix tree environment variable (#45090) 2026-10-20 07:02:43 +00:00
Po-Han Huang (NVIDIA)andClaude Opus 5.5 9e23107772 [Doc] cookbook: Refresh SemiAnalysis AgentX launches with five open InferenceX PRs (#45311)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 14:48:48 +08:00
1ec902e00f [Fix] cache: Support DeepSeek-V4.1 external linker pools (#44623)
Co-authored-by: Zhangheng <hzh0425@apache.org>
Co-authored-by: Shuwen Wang <47200617+alphabetc1@users.noreply.github.com>
2026-10-19 23:40:25 -07:00
Michael 407bfc94f9 [Test] [AMD] diffusion: Make multimodal_gen unit tests ROCm-aware (#44599) 2026-10-20 14:24:37 +08:00
Xinyuan Tong 4b83f215f4 [Feat] entrypoints: add a chat encoding adapter registry (#45300) 2026-10-20 14:10:01 +08:00
lpdinkandZhangheng 4a651146de [Feat] hicache: Notify storage backends when the initial host-pool registration burst completes (#44030)
Co-authored-by: Zhangheng <hzh0425@apache.org>
2026-10-20 14:09:13 +08:00
Michael 36978d6c4d [Test] [AMD] diffusion: Skip the Kandinsky6 FA dispatch test on ROCm (#44421) 2026-10-20 13:54:23 +08:00
Duyi-Wang 50fa3727eb [Doc] [AMD] cookbook: Fix DeepSeek-V4 PD image and c1 capacity (#45297) 2026-10-19 22:45:50 -07:00
Liangsheng Yin 472f0d540b [Fix] deepseek-v4.1: Keep the padded query buffer inside decoder tail graphs (#45301) 2026-10-19 22:41:53 -07:00
Khoa PhamandClaude Fable 5.1 476fe86a12 [Perf] deepseek-v4.1: Score a request's verify rows as one request in the full-scan indexer (#45251)
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-19 22:25:04 -07:00
35c873cff4 [Fix] observability: Log scheduler stats once per attention CP group (#45319)
Co-authored-by: Hun-ho Kim <hunho.kim@samsung.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 22:22:31 -07:00
weireweireandweireweire c7687a0e7c [Fix] deepseek-v4: Keep the draft MoE out of the target's expert placement (#45299)
Co-authored-by: weireweire <20922698+weireweire@users.noreply.github.com>
2026-10-19 22:20:26 -07:00
602c92ccd8 [Perf] DeepSeek-V4.1: Raise the SM100 swap-AB decode cap to 40 rows (#44632)
Co-authored-by: weireweire <20922698+weireweire@users.noreply.github.com>
Co-authored-by: nbommi19 <227267412+nbommi19@users.noreply.github.com>
2026-10-19 22:20:10 -07:00
553836c164 [Feat] simulator: Add baseline collection and ML latency predictor training tools (#44299)
Co-authored-by: shijunyu.sjy <shijunyu.sjy@alibaba-inc.com>
Co-authored-by: baoyunong.byn <baoyunong.byn@alibaba-inc.com>
Co-authored-by: wangyi.wy <aaronwangyi0201@gmail.com>
2026-10-20 13:02:33 +08:00
parkeychenandparkeychen 43a896933e [Fix] deepseek-v4: Refresh attn_sink after loading weights (#41630)
Co-authored-by: parkeychen <parkeychen@tencent.com>
2026-10-19 21:59:32 -07:00
ashwini rathiandClaude Opus 5.5 d1c3346f74 [Fix] [Intel] spec: Fix XPU nightly dflash stop_regex and DSpark capture check (#45033)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 12:27:36 +08:00
72d886efb8 [Perf] DeepSeek-V4.1: Run the small-batch decode fusions at TP2 (#43838)
Co-authored-by: weireweire <20922698+weireweire@users.noreply.github.com>
Co-authored-by: nbommi19 <227267412+nbommi19@users.noreply.github.com>
Co-authored-by: Po-Han Huang <pohanh@nvidia.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com>
2026-10-19 21:17:05 -07:00
Mohammad Miadh AngkadandMohammad Angkad fdd53418a0 [CI] [NVIDIA] gb300: Drop Kimi-K2.5 and Qwen3.5 nightly tests (#45308)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-10-19 21:13:00 -07:00
Shuwen Wang c53b98d1fe [Fix] test-utils: Allow expected server exits in canary tests (#45172) 2026-10-19 21:07:03 -07:00
Mohammad Miadh AngkadandMohammad Angkad 5b683d6b72 [Fix] test: Add logical_forward_mode to the KDA prefill-graph stand-in batch (#45218)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-10-19 21:06:10 -07:00
Mingyi 4a0c6f2fe0 docs: promote SGLang Summit 2026 in README (#45271) 2026-10-20 03:36:56 +00:00
wxzhoucsandXiaoyu Zhang e3205080ce [Fix] qwen3.8: Fix batch-invariant kernels for Qwen3.8 Flash-Next (#42087)
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com>
2026-10-20 11:32:40 +08:00
1140a8dfd9 [Fix] [AMD] aiter: Fix garbage decode output for non-sliding-window models on the vectorized_5d KV layout (#43335)
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com>
2026-10-20 11:31:20 +08:00
Cao EandMa Mingfei 3c07ff2654 [Perf] [XPU] fp8: Optimize FP8 scale layouts (#40052)
Co-authored-by: Ma Mingfei <mingfei.ma@intel.com>
2026-10-20 11:24:16 +08:00
huangtingwei 3eb4bd598f [Perf] hicache: Unify PP prefetch hit queries and buffer allocation (#43344) 2026-10-20 11:05:00 +08:00
Trevor Morris ec1fdc22a9 [Fix] [NVIDIA] docker: Fix CUDA 13.4 container by updating torch nightly (#45252) 2026-10-20 10:52:42 +08:00
kpjeeja 8683f8ad4a [Feat] [Intel] hicache: Enable L1<->L2 support on XPU (#32503) 2026-10-20 10:42:21 +08:00
Liangsheng Yin 5e630fa571 [Perf] deepseek-v4.1: Drop host syncs from encoder-replay folding and tail metadata (#45275) 2026-10-19 19:39:39 -07:00
Mohammad Miadh AngkadandMohammad Angkad af856717c2 [Fix] mamba: Skip zero-width conv pools in the MTP verify scatter (#45212)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-10-19 19:23:17 -07:00
Vladimir Serovandronnie_zheng 55b2aef7ad [Fix] [NPU] LoRA: Update for follow-up with changes in sgl-kernel-npu (#40908)
Co-authored-by: ronnie_zheng <zl19940307@163.com>
2026-10-20 05:16:05 +03:00
wutao 2c00b5d129 [Test] constrained: Cover JSON whitespace limits (#43981) 2026-10-20 10:12:22 +08:00
Gabriel Wuandnumber-eleven11 67d9ee156c [Fix] [SM120] quantization: Requantize block-FP8 MoE weights to UE8M0 for DeepGEMM (#43556)
Signed-off-by: Zihua Wu <13583761+lucifer1004@users.noreply.github.com>
Co-authored-by: number-eleven11 <1635219023@qq.com>
2026-10-19 19:06:21 -07:00
Liangsheng Yin ce329c85fa [Perf] deepseek-v4.1: Skip the request-window history check when invariant checks are off (#45276) 2026-10-19 18:38:44 -07:00
79ed13fc2c [Fix] glm47: Disable tool grammar for tool_choice="none" (#41233)
Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com>
Co-authored-by: Xinyuan Tong <115166877+JustinTong0323@users.noreply.github.com>
Co-authored-by: Po-Han Huang <pohanh@nvidia.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 09:36:06 +08:00
Lee Nau 1897575ce1 [Fix] runtime: Version FlashInfer caches for default-tactic tuning. (#44514) 2026-10-19 18:33:49 -07:00
Anihcandsunguozhong ebc71e650e [Perf] [NPU] mem_cache: Add optional ScatterPa KV cache writes (#40535)
Co-authored-by: sunguozhong <sun_gz2772@163.com>
2026-10-20 09:33:34 +08:00
71ee2dc950 [Fix] inkling: Return 400 instead of 500 for unfetchable/undecodable multimodal input (#40899)
Co-authored-by: apex-mochen <265981728+apex-mochen@users.noreply.github.com>
Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com>
Co-authored-by: apex-mochen <2756823972@qq.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Po-Han Huang <pohanh@nvidia.com>
2026-10-20 09:18:18 +08:00
Xiaoyu ZhangandClaude Opus 5.5 693d333306 [Refactor] kernels: Regroup operators by computation and dissolve ops/muse_spark (#45112)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-20 08:47:57 +08:00
Liangsheng Yin ab580b2e06 [CI] radix-cache: Split unified radix cache unit tests into topic files and a shared kit (#45236) 2026-10-19 17:00:42 -07:00
Cheng Wan 6b90e5716d [Refactor] moe: Scope shared-expert configuration to each model (#44913) 2026-10-19 16:59:37 -07:00
Cheng Wan 964ab2b280 [Refactor] moe: Split the two meanings of num_fused_shared_experts (#44912) 2026-10-19 16:57:15 -07:00
Cheng Wan 48ea486928 [Fix] moe: Renormalize per-rank shared-slot routing over routed experts only (#44904) 2026-10-19 16:56:21 -07:00
Cheng Wan 3a059de94c [Fix] moe: Count routed_scaling_factor once when top-k folds it (#44901) 2026-10-19 16:54:27 -07:00
Qiaolin Yuandzijiexia 28448a86d4 [Doc] cookbook: Default Qwen3.8-Flash-Next datacenter recipes to FP8 KV cache and QSA indexer (#45245)
Co-authored-by: zijiexia <37504505+zijiexia@users.noreply.github.com>
2026-10-19 15:57:22 -07:00
Liangsheng Yin 1238edd954 [Perf] deepseek-v4.1: Reset request-window tags without a host sync (#45247) 2026-10-19 15:45:17 -07:00
Mingyi 9a949403f1 [Doc] tpu: Introduce SGLang-TorchTPU in README and TPU guide (#45246) 2026-10-19 15:34:47 -07:00
ab258b6453 [Fix] symm-mem: Name the window registration in the scheduler watchdog dump (#44681)
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: hnyls2002 <lsyincs@gmail.com>
2026-10-19 15:29:35 -07:00
Gabriel Wu 05d7f7a166 [Fix] attention: Honor selected FlashAttention implementation (#33007) 2026-10-19 15:27:52 -07:00
Jiuqiang Tang 49766a74f9 [Fix] attn_residual: int64 program ids in the score/combine kernels (int32 overflow at 37,451 tokens) (#43438) 2026-10-19 15:12:12 -07:00
0z5aandRichard Wang 80ea5ee638 [Fix] dLLM: Preserve prompt tails and literal mask tokens across prefix reuse (#39592)
Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
Co-authored-by: Richard Wang <wangrichard08@gmail.com>
2026-10-19 14:48:37 -07:00
Liangsheng Yin fdc2dd0d05 [Perf] dspark: Make the prefill SWA valid-from update sync-free (#45234) 2026-10-19 14:44:01 -07:00
Liu Keandhnyls2002 81ac15391c [Fix] metrics: record HTTP request metrics in tokenizer workers (#44663)
Co-authored-by: hnyls2002 <lsyincs@gmail.com>
2026-10-19 14:36:16 -07:00
Mohammad Miadh AngkadandMohammad Angkad 7656ea0e74 [CI] python: Move the remaining CUDA build paths off Python 3.10 (#44393)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-10-19 14:29:00 -07:00
Liangsheng Yin b20cb3665d [CI] radix-cache: Drop redundant unified radix cache unit tests (#45232) 2026-10-19 14:20:32 -07:00
Baizhou Zhang 642a671c30 [CI] tests: Re-enable muse spark B300 PD coverage (#44975) 2026-10-19 13:40:02 -07:00
Liangsheng Yin 8b1d1e0d74 [Perf] deepseek-v4.1: Check request-window SWA history without per-layer host syncs (#45223) 2026-10-19 13:34:52 -07:00
Rain Jiang aaa9388c91 [Fix] kda: Keep FlashInfer prefill off its grouped kernel's bf16/flat-dt_bias contract (#45018) 2026-10-19 13:16:54 -07:00
Liangsheng Yin f1a2ee763d [Perf] deepseek-v4.1: Run decoder replay tail graphs in prefill-graph steps and under DP attention (#45107) 2026-10-19 13:02:07 -07:00
Qiaolin Yu 5385866108 [Fix] moe: gated NVFP4 MoE intermediate alignment for FlashInfer TRTLLM (#44138) 2026-10-19 12:52:36 -07:00
Qiaolin Yu 9ab74b8517 [Fix] Qwen3.8next: Stop retaining per-size PLE outputs in breakable prefill CUDA graphs (#44150) 2026-10-19 12:41:39 -07:00
0ce41c5a06 [Fix] PD: retraction restores racing the in-flight decode forward (#42702)
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Zhiqiang Xie <xiezhq@stanford.edu>
2026-10-19 11:55:04 -07:00
YAMY 722669e606 [Feat] [NVIDIA] megakernels: Add --enable-nvidia-megakernel and a layer for NVIDIA megakernel plugins (#44945) 2026-10-19 11:54:44 -07:00
AMD-yanfeiwang 678913a0f0 [Feat] sgl-model-gateway: Add --decode-dispatch-limit to pick PD decode workers near KV arrival (#44689) 2026-10-20 02:16:21 +08:00
jacky.cheng 80b86dbecb [Fix] [AMD] diffusion: Accept residual and skip_bias in the ROCm Conv3d-to-Conv2d VAE patch (#44672) 2026-10-19 11:07:37 -07:00
jacky.cheng 5ed306bd69 [Fix] [AMD] diffusion: Keep gfx1250 FlyDSL BF16 attention out of torch.compile (#44671) 2026-10-19 10:36:28 -07:00
lixiufei-leoandXiufei Li 4ef9765e76 [Feat] sgl-router: Add --pd-stream-after-prefill (#45102)
Co-authored-by: Xiufei Li <294029388+lixiufei-leo@users.noreply.github.com>
2026-10-20 01:13:25 +08:00
chenyang08056032 716c9151f2 [CI] [NPU] npu-coverage: run only coverage-recommended tests on PRs (precise test selection) (#44019) 2026-10-20 00:57:27 +08:00
Jan SchalkampandXiaoyu Zhang 8e7601ae4a [Feat] paged-experts: Serve MoE models whose weights do not fit VRAM (#42557)
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com>
2026-10-19 23:10:04 +08:00
Monishver 806e7423af [Perf] deepseek_v4: size the masked silu-mul-quant grid to the GPU, not the padded capacity (#44618)
Signed-off-by: Monishver Chandrasekaran <monishverchandrasekaran@gmail.com>
2026-10-19 23:07:12 +08:00
Avaya Aggarwal c04f9a61f0 [Fix] diffusion: Fix VMoBA chunk selection bugs (#44286) 2026-10-19 23:04:26 +08:00
Siliang ChenandCursor 7c4cc5a51c [Perf] [AMD] attention: Spread uneven Triton decode batches across XCDs (#41229)
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-19 23:03:27 +08:00
Sam Shleifer 96ba108072 [Fix] dsa-eagle-topk: Copy token keys and scales when relocating index cache (#44564) 2026-10-19 22:54:08 +08:00
Sam Shleifer 417606527d [Fix] nvfp4: Own streamed tensors while waiting for paired weights (#44500) 2026-10-19 22:41:53 +08:00
pllimax 8c4680cb65 [CI] npu: Move llada2-mini FDFO test to full test (#45153) 2026-10-19 22:29:30 +08:00
lixiufei-leoandXiufei Li 62cee1b15c [Fix] sgl-router: Keep the kv-indexer connection through bursts of cancelled prefix queries (#45100)
Co-authored-by: Xiufei Li <294029388+lixiufei-leo@users.noreply.github.com>
2026-10-19 21:41:03 +08:00
zhaozx-cn ed0b8f433f [Fix] [NPU] kimi-k3: skip fused gmm1 for situ activation in Ascend runner (#43837) 2026-10-19 19:30:41 +08:00
Shangming Cai 0025f93944 [Fix] sgl-router: Refuse prefill workers that cannot compute before bootstrap under --decode-dispatch-limit (#45120) 2026-10-19 18:24:17 +08:00
Quanli Liand全力 e45cbce599 [Fix] diffusion: Prevent int32 overflow in Qwen-Image-2.1-Turbo VAE RMSNorm (#45118)
Co-authored-by: 全力 <liquanli.lql@antgroup.com>
2026-10-19 18:01:02 +08:00
Niko Maandinkcherry a90e9c5dd8 [Fix] [AMD] mhc: Bound the Triton fused post+pre buffer cache (#44443)
Co-authored-by: inkcherry <mingzhi.liu@amd.com>
2026-10-19 01:37:50 -07:00
8111841103 [Perf] deepseek-v4.1: Replay the breakable prefill CUDA graph at any context length (#43733)
Co-authored-by: Oasis-Git <ayw.sirius19@gmail.com>
Co-authored-by: hnyls2002 <lsyincs@gmail.com>
2026-10-19 01:29:14 -07:00
33dd8f6a2e [Fix] hybrid-SSM: DP attention failures at low concurrency (#34535)
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Jimmy Shong <69131491+Jiminator@users.noreply.github.com>
Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com>
2026-10-19 16:24:41 +08:00
Thomas Wang 5d9c956f8e [Doc] [AMD] dsv4: Update amd cookbook 1016 (#45070) 2026-10-19 16:11:11 +08:00
Po-Han Huang (NVIDIA)andClaude Opus 5.5 af9bb2a32d [Doc] cookbook: Refresh SemiAnalysis AgentX launches with twelve open InferenceX PRs (#45111)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 15:24:08 +08:00
Bingxu Chen 3d588779e1 [CI] [AMD] test: Run UMBP HiCache E2E through standalone DRAM server (#40674) 2026-10-19 00:23:52 -07:00
AMD-yanfeiwang f86a04a331 [Feat] sgl-router: Add --decode-dispatch-limit to pick decode DP ranks near KV arrival (#44402) 2026-10-19 15:11:40 +08:00
a5117beb11 [Perf] mhc: Parallelize small-batch post across hidden tiles (#44906)
Co-authored-by: leonardHONG <178642616+leonardHONG@users.noreply.github.com>
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com>
2026-10-19 15:07:13 +08:00
ashwini rathiandClaude Opus 5.5 b6d3b7e163 [CI] [Intel] xpu: Disable L0 copy offload to fix first-H2D-copy hang (#45013)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 14:48:45 +08:00
97857f9f96 [Doc] cookbook: GB300 full-set accuracy and tool follow-up output for Ling-3.1-flash (#44307)
Co-authored-by: andyluo7 <andy.luo@amd.com>
Co-authored-by: zijiexia <37504505+zijiexia@users.noreply.github.com>
2026-10-18 23:29:09 -07:00
desmond-intel dc81f17b90 [Fix] [XPU] gemma3: Enable bidirectional image attention (#44225) 2026-10-19 14:08:24 +08:00
Liangsheng Yin a3330237db [Perf] deepseek-v4.1: Run decoder replay tail layers from CUDA graphs on eager prefill steps (#45059) 2026-10-18 23:07:50 -07:00
Trevor Morris 04c8c9703f [Fix] docker: Install numactl in runtime images - needed for SGLANG_NUMA_BIND_V2 (#43883) 2026-10-18 22:55:58 -07:00
7c9a78efbe [Feat] nemotron: Port Nemotron 3.5 Super serving changes (#44518)
Co-authored-by: BBuf <1182563586@qq.com>
Co-authored-by: elvischenv <219235043+elvischenv@users.noreply.github.com>
Co-authored-by: Jimmy Shong <jimmy.shong@mac.attlocal.net>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 13:27:22 +08:00
Ziru f726461287 [Feat] [CPU] model: Support LTX and MOVA models on CPU (#36632) 2026-10-19 13:25:13 +08:00
ZiruandNiu Ziru 9a93a17c08 [Feat] [CPU] model: Support Muse models on CPU (#44624)
Co-authored-by: Niu Ziru <niuziru@gnr799217.jf.intel.com>
2026-10-19 13:12:32 +08:00
MickandMick Qian 2b75048e2a [CI] diffusion: tighten H100 baselines and enable B200 performance checks (#44996)
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
2026-10-19 12:54:54 +08:00
Avaya Aggarwal 58ad0748a6 [Fix] diffusion: Unmerge LoRA layers the next adapter does not cover (#44770) 2026-10-19 12:33:40 +08:00
Xinyuan Tongand文贤 266550048a [Fix] bailing: MoE gate dtype follows config.router_dtype (#44715)
Co-authored-by: 文贤 <luoyuan.luo@antgroup.com>
2026-10-19 12:21:29 +08:00
Liangsheng Yin 82c9a8b037 [Perf] deepseek-v4.1: Default PD to MegaAttention, with the KV layout in the PD handshake (#45075) 2026-10-18 21:16:45 -07:00
Yuwei An 834bfe4ef0 [Fix] attention: Opt FlashMLA scheduler into dynamic shared memory (#44580) 2026-10-18 20:31:39 -07:00
Duyi-Wang 8d3eee46a7 [Perf] [AMD] deepseek-v4: Skip top-k refinement for proven positive-zero ties (#44927) 2026-10-19 11:27:47 +08:00
06342acad2 [Perf] deepseek-v4.1: Stage late-layer DP row counts through pinned memory (#45062)
Co-authored-by: weireweire <20922698+weireweire@users.noreply.github.com>
Co-authored-by: kpham-sgl <khoa.pham@radixark.ai>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-18 20:19:26 -07:00
f02458fff1 [Fix] [CPU] memory: Honor cgroup memory.max when sizing CPU memory (#40066)
Co-authored-by: Zhang, Mingxu <mingxu.zhang@intel.com>
Co-authored-by: MingxuZh <109504044+MingxuZh@users.noreply.github.com>
2026-10-19 11:11:44 +08:00
Alexandr 617c8b67b1 [Fix] Z-Image: Synchronize device after warmup request (#44306) 2026-10-19 05:36:16 +03:00
Ankur SinghandClaude Opus 5.5 7b1710324c [Doc] cookbook: Refresh SemiAnalysis AgentX launches for Qwen3.8-Flash-Next H200 (v0.5.22, TP2 points) (#45039)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

Force merge to unblock SA AgentX submission
2026-10-19 10:31:51 +08:00
Kotthagattu Meher Sai 7f96dc6874 [Perf] [XPU] server_args: Enable fused QK-norm + RoPE by default (#43345) 2026-10-19 10:19:52 +08:00
642f43ce59 [Fix] diffusion: size the in-context slot from both input grids for wan-animate-2(#44090)
Co-authored-by: Ankit Dhall <8938083+ankitdhall@users.noreply.github.com>
Co-authored-by: Mick <mickjagger19@icloud.com>
2026-10-19 10:11:58 +08:00
Khoa Pham a7fd2e1158 [Feat] dsv4.1: Return prompt logprobs under encoder SWA bounded replay (#45050) 2026-10-18 19:08:20 -07:00
Wuhen DuanandEven Zhou 69076b9f9e [CI] [NPU] tests: Reduce HiCache sizes and enable expandable pinned memory in focused tests (#41675)
Co-authored-by: Even Zhou <even.y.zhou@outlook.com>
2026-10-19 10:06:35 +08:00
fea48754de [Feat] rust/grpc: Add ListModels, Detokenize, and Abort to api.v1 (#42118)
Signed-off-by: jain-ria <riajain@NVIDIA.com>
Co-authored-by: ishandhanani <82981111+ishandhanani@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Ishan Dhanani <ishandhanani@gmail.com>
2026-10-19 09:53:56 +08:00
Mohammad Miadh AngkadandMohammad Angkad 377cd867fd [Perf] dsa: Route the k-pool indexer top-k through the top-k v2 kernel (#43775)
Co-authored-by: Mohammad Angkad <mohammad.angkad@radixark.ai>
2026-10-19 09:49:39 +08:00
Yuwei An df078cc729 [Fix] DSv4.1: Support CUDA image requests with encoder bounded replay and CI Test (#45048) 2026-10-18 18:47:17 -07:00
Emre AlbayrakandXiaoyu Zhang 10ec53ac2f [Perf] [NVIDIA] fp8_fa_sm120: Use the block-scaled MMA encoding (#43390)
Co-authored-by: Xiaoyu Zhang <1182563586@qq.com>
2026-10-19 09:36:21 +08:00
Jimmy Shong 0dc6de2e74 [Doc] nemotron-3.5-super: update H200 and Spark wizard recipes (#44587) 2026-10-18 18:22:05 -07:00
A&W cff88d6eac [Fix] diffusion: Invalidate merged LoRA cache after adapter rewrites (#44204) 2026-10-19 01:08:57 +00:00
Liangsheng Yin 299da60131 [CI] test-utils: Abort the test run when its server dies inside a CustomTestCase method (#45053) 2026-10-18 18:00:06 -07:00
Xiaoyu ZhangandClaude Opus 5.5 329072d99a [Refactor] kernels/diffusion/tests: Prune dead code and fix tests that never ran (#44690)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-19 08:55:47 +08:00
Liangsheng Yin 59fec011dd [Fix] deepseek-v4.1: Reject encoder replay on a chunk too small for a hit or on other models (#45051) 2026-10-18 17:55:35 -07:00
Liangsheng Yin 3c560089e1 [Refactor] scheduler: Bill encoder replay rows in _update_prefill_budget (#45047) 2026-10-18 17:17:38 -07:00
Liangsheng Yin 1637a4e305 [Fix] scheduler: Clear stale encoder_swa_reset in convert_decode_to_extend (#45042) 2026-10-18 17:09:19 -07:00
Liangsheng Yin 6708426728 [Fix] spec: Remove --speculative-skip-dp-mlp-sync (#45043) 2026-10-18 16:41:21 -07:00
Liangsheng Yin 3f13f5d966 [Fix] scheduler: Drop prefill embedding overrides in prepare_for_decode (#45041) 2026-10-18 15:54:20 -07:00
Liangsheng Yin 1461152cb0 [Fix] deepseek-v4.1: Leave room for encoder replay rows when sizing a prefill chunk (#45040) 2026-10-18 15:43:08 -07:00
Liangsheng Yin fe89a7c890 [Fix] deepseek-v4.1: Start the PD decode draft SWA window at the prefill draft valid-from (#45037) 2026-10-18 15:06:20 -07:00
Liangsheng Yin da414adda7 [Fix] engram: Clamp multimodal pad ids in host-built history rows (#45038) 2026-10-18 12:24:05 -07:00
Liangsheng Yin 0f4aedfecc [Perf] deepseek-v4.1: Default decoder SWA bounded replay on for both PD roles on CUDA (#45035) 2026-10-18 11:53:35 -07:00
Ke Bao 4ec19ca27f [Refactor] memory-pool: Build KV host pools from buffer info (#40917) 2026-10-18 23:21:16 +08:00
Ke Bao bb1d4cddd9 [Refactor] hicache: Route Mamba host transfers from pool info (#44968) 2026-10-18 23:02:46 +08:00
Xinyuan Tong d4e3c163c9 [CI] radix-cache: Register the Inkling cache linker KL test to extra-b (#45004) 2026-10-18 22:23:34 +08:00
e9b22a0818 [Perf] [AMD] monokernels: Let a plugin skip per-step attention metadata (#44601)
Co-authored-by: Kevin Mi <kevin.mi@radixark.ai>
Co-authored-by: kkHuang-amd <43161300+kkHuang-amd@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-18 14:04:57 +00:00
Cheng Wan 57db5089c2 [Perf] attention: Capture translated FA4 verify metadata for DSPARK (#44973) 2026-10-18 05:42:16 -07:00
Cheng Wan 2e9b2cacda [Perf] speculative: Fuse draft write-window gathering and KV translation (#44972) 2026-10-18 05:41:34 -07:00
Cheng Wan 2a83471676 [Perf] mamba: Fuse strided state updates and replay index translation (#44971) 2026-10-18 05:40:57 -07:00
Cheng Wan d318a546da [Perf] unified-memory: Reduce allocation and SWA retirement overhead (#44970) 2026-10-18 05:40:25 -07:00
Cheng Wan a5b21145e1 [Refactor] attention: Separate decode metadata from pool ownership (#44969) 2026-10-18 05:39:44 -07:00
3edb156dec [Feat] unified-memory: Fuse drafts with sliding-window and recurrent-state layers (#43326)
Co-authored-by: Caihua Li <caihua.li@bytedance.com>
Co-authored-by: Cheng Wan <cheng.wan@radixark.ai>
2026-10-18 05:39:09 -07:00
caihuali95 19cb24f1e1 [Feat] unified-memory: Support asymmetric KV rows (#43325) 2026-10-18 05:37:50 -07:00
caihuali95andCaihua Li 1d17066bad [Perf] gdn: Select FlashInfer prefill for page-major pools (#43171)
Co-authored-by: Caihua Li <caihua.li@bytedance.com>
2026-10-18 05:37:24 -07:00
caihuali95andCaihua Li ba50ea3d77 [Perf] unified-memory: Free SWA segments without a host sync (#43170)
Co-authored-by: Caihua Li <caihua.li@bytedance.com>
2026-10-18 05:36:56 -07:00
caihuali95andCaihua Li 47eeddcd18 [Fix] swa: Retain a whole window behind insertion pages (#43168)
Co-authored-by: Caihua Li <caihua.li@bytedance.com>
2026-10-18 05:36:02 -07:00
lixiufei-leoandXiufei Li fe1851c3c2 [Fix] sgl-router: Receive kv-indexer-bridge event batches outside tokio's coop budget (#44305)
Co-authored-by: Xiufei Li <294029388+lixiufei-leo@users.noreply.github.com>
2026-10-18 20:35:50 +08:00
WenhaoZhang e17fed6ea7 [Perf] diffusion: fused gate/up + SwiGLU MLP on comfy_kitchen INT8 for Qwen-Image 2.1 (#44196) 2026-10-18 20:33:31 +08:00
LindaandMick 680a2ff27e [Perf] [NVIDIA] diffusion/FLUX.3: Run dynamic LoRA through PrimsTS GEMMs (#44761)
Co-authored-by: Mick <mickjagger19@icloud.com>
2026-10-18 20:32:19 +08:00
WenhaoZhangandMick 1fd46ddd56 [Perf] diffusion: fused indexed modulation for FP32 AdaLN tables (#44180)
Co-authored-by: Mick <mickjagger19@icloud.com>
2026-10-18 20:31:36 +08:00
Shuwen Wang 8dee3f6bdb [Fix] [H200] ci: Split GLM-5.3-Flash serving recipes into separate tests (#44892) 2026-10-18 19:26:40 +08:00
Shuwen Wang f8e32706fc [CI] radix-cache: Re-enable Inkling unified cache linker KL test (#44938) 2026-10-18 19:26:22 +08:00
AMD-yanfeiwang a3abea3560 [Fix] sgl-router: Resend on a closed pooled connection; add --disable-circuit-breaker (#44075) 2026-10-18 18:23:18 +08:00
3b932cde26 [Fix] VLM: harden Kimi image cache identity across preprocessing and radix hashing (#43043)
Co-authored-by: Rahul Chalamala <22563365+rchalamala@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Mick Qian <mickqian@users.noreply.github.com>
Co-authored-by: Liangsheng Yin <hnyls2002@gmail.com>
2026-10-18 17:39:37 +08:00
5a930cfd88 [Fix] glm4v: Clamp request video budgets to the deployment limit (#43871)
Co-authored-by: cipheraxat <cipheraxat@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-18 17:31:33 +08:00
2993 changed files with 230369 additions and 94848 deletions
+1 -1
View File
@@ -742,7 +742,7 @@ cd test && python3 run_suite.py --hw cuda --suite base-b-kernel-benchmark-test-1
- `docs/docs/developer_guide/development_jit_kernel_guide.mdx`
- `test/run_suite.py` — suite names, discovery of `test/registered/`, execution entrypoint for CI
- `python/sglang/test/ci/ci_register.py` — `register_cuda_ci` and AST registration rules
- `python/sglang/kernels/jit/utils/compile.py` — `load_jit`, `make_cpp_args`
- `python/sglang/kernels/jit/utils/compile/` — `load_jit` (`loader.py`), `make_cpp_args` (`cpp_args.py`)
- `python/sglang/kernels/jit/utils/common.py` — `cache_once`, `should_run_full_tests`, `get_ci_test_range`
- `python/sglang/kernels/jit/include/sgl_kernel/tensor.h` — `TensorMatcher`, `SymbolicSize/DType/Device`, `is_type`
- `python/sglang/kernels/jit/include/sgl_kernel/ffi.h` — `ffi::empty`, `ffi::empty_like`, `ffi::from_blob`
@@ -398,6 +398,7 @@ group: pr-test-{event_name}-{branch}-{pr_sha}-{stage}
| `/run-extra-ci` | Adds both labels + reruns **only** `PR Test Extra`, leaving baseline runs alone. Alias: `/rerun-extra-ci` |
| `/rerun-test <test-file> [<test-file> ...]` | Reruns specific test file(s) via `rerun-test.yml`. A file arg containing a glob metacharacter (`*`, `?`, `[...]`) expands against `test/registered/` and the multimodal test dir to every matching `test_*.py` (e.g. `/rerun-test test_*backend*.py` — wrap in backticks so GitHub doesn't italicize the `*`); matches are deduped, grouped by dispatch shape, and can't carry a `::test` selector. No match → single ⛔ reply, nothing dispatched. |
| `/rerun-group <group> [<group> ...]` | Expands registered test groups, then reuses `/rerun-test` |
| `/run-cpu-ci` | Dispatches `run-cpu-ci.yml` on main: only the `base-a-test-cpu` suite, at the PR head SHA, no caches saved. Same gate as `/rerun-test`. Alias: `/rerun-cpu-ci` |
Handled by `scripts/ci/utils/slash_command_handler.py` → `.github/workflows/slash-command-handler.yml`.
@@ -16,7 +16,7 @@ These are investigation hints, not a list of changes to apply.
| `No platform detected. Using base SRTPlatform with defaults.` | Platform detection | Decide whether fallback is expected on this machine or indicates a missing plugin. |
| `Unexpected error during package walk` in CUTE_DSL | Installed cutlass package, its handler, root propagation, and `CUTE_DSL_LOG_LEVEL` | Inspect the actual exception and duplicate-handler path; do not assume the current log level or hide all warnings. |
| Repeated dtype, hybrid-model, or tokenizer fallback messages | Model-config and tokenizer construction in launcher, scheduler, and detokenizer | Multiple constructors may explain repetition. Retain meaningful per-process information and distinguish these from handler duplication. |
| Repeated template detection messages | `managers/template_detection.py` and `managers/template_manager.py` | Check whether a summary already contains the same information. |
| Repeated template detection messages | `parser/template_detection.py` and `parser/template_manager.py` | Check whether a summary already contains the same information. |
| KV cache dtype logged separately from allocation | Model runner and memory pool | Check whether the allocation summary already carries dtype; retain useful allocation and SWA details. |
| CUTLASS backend disabled during graph capture | Attention backend selection | Expected fallback may be useful information; retain the reason and selected behavior. |
| `Ignore import error when loading ...` | Model, multimodal, or algorithm registries | Distinguish an unused optional model dependency from failure to load the requested model. |
+1 -1
View File
@@ -205,7 +205,7 @@ than restating.
`python/sglang/srt/function_call/function_call_parser.py` (ToolCallParserEnum) —
prose naming a near-miss id (e.g. the reasoning id where the tool id differs) is a
factual bug. `--…-parser auto` is acceptable ONLY if the template-detection rules
(`python/sglang/srt/managers/template_detection.py`) actually resolve THIS model's
(`python/sglang/srt/parser/template_detection.py`) actually resolve THIS model's
chat template to the right parser — no rule match means auto silently disables the
parser; when in doubt require explicit ids (the DSv4 page pins explicit ids).
+36 -13
View File
@@ -17,25 +17,47 @@ logical grouping and remove the old `_jit_` filename prefix.
The `sgl_kernel` wheel retains its own public API and packaging tests.
- Group by computation, not model or GPU: GEMM and GEMV in `gemm`, expert
routing in `moe`, attention index selection in `attention`, sampling in
`sampling`, normalization in `layernorm`. A quantized GEMM is still a GEMM;
quantization alone belongs in `quantization`.
`sampling`, normalization in `layernorm`.
- Classify a fused operator by its complete contract; do not split a fused
kernel into separate launches just to separate its parts. Split unrelated
public entry points that happen to share a source file. Decided rules:
- Any cross-rank collective or peer-memory access (all-reduce, all-gather,
reduce-scatter, a2a, multicast/NVLS) puts the op in `communication`,
whatever compute it fuses.
- Producer + output quantization stays with the producer (norm + quant in
`layernorm`, activation + quant in `activation`, GEMM with a quant epilogue
in `gemm`, MoE dispatch/permute + quant in `moe`). `quantization` holds only
pure format conversion: quant, dequant, scale-layout conversion, repack.
- RoPE stays in `attention` (standalone RoPE, position ids, QK-norm + RoPE).
- `kvcache` owns ops that read/write KV-cache contents; `attention` owns the
index/shape/schedule tables (page tables, `cu_seqlens`, `kv_indices`, split
metadata) and index selection. A fused compute + store stays with its
compute group.
- Model-specific files/subpackages and tuning data are allowed **inside** a
logical group. Do not add a model bundle or an implementation file directly
under `ops/`. Propose a new logical group only for a distinct responsibility,
and update `ops._GROUPS`, documentation, and tests together.
- Classify a fused operator by its complete contract. Do not split a fused
kernel into separate launches just to separate norm, RoPE, or quantization.
Split unrelated public entry points that happen to share a source file.
- Keep shared CUDA build/runtime infrastructure in `kernels/jit`; operator
wrappers call it from their logical group. Keep process groups, communicator
state, model dispatch, and buffer ownership in `srt`. K3-specific adapters in
`srt/layers/communication/` need not pretend to be generic interfaces.
and update `ops._GROUPS`, the README group list, and tests together
(`test_kernels_namespace.py` checks the README against `_GROUPS` and the
README layout against the directories under `kernels/`).
- Agent-generated implementations live in `kernels/kda_kernels/` (Kernel Design
Agents, `KernelBackend.KDA`) and `kernels/cake_kernels/` (Cake adapters over
FlashInfer). They are backends, not groups: register their public ops in the
logical group with a `KernelSpec` whose target points into them, and do not
reorganize those directories.
- Keep shared build/runtime infrastructure (including the Helion runtime
helper) in `kernels/jit`; operator wrappers call it from their logical group.
Keep process groups, communicator state, workspace classes, model dispatch,
and buffer ownership in `srt` (`srt/layers/communication/` for the K3 and Muse
Spark adapters); kernels must not import srt runtime modules to inspect
model objects, so pass flags in. Adapters there need not pretend to be generic
interfaces.
## Register the public entry point
Add lazy `KernelSpec` metadata in the owning group's `__init__.py`, or use the
existing `BaseFusedOp` registration when the operation has interchangeable
backends. Group imports must not import GPU implementations or compile kernels.
backends (a `BaseFusedOp` built per layer registers `module:Class.forward_<backend>`
targets). Group imports must not import GPU implementations or compile kernels.
Use `<group>.<name>` for the op id and `module:callable` for the target. Preserve
existing backend dispatch and describe actual device/architecture restrictions
@@ -53,8 +75,9 @@ factories, reference helpers, and runtime classes are not separate kernel APIs.
runtime module. A CUDA allocation does not make a runtime test a kernel test.
- Non-CI smoke scripts and benchmarks: `test/manual/kernels/`; do not leave
standalone test/benchmark entry points in production operator modules.
- Shared test helpers: `sglang.test.kernels`. Preserve vendored upstream trees
and AOT wheel packaging boundaries instead of reorganizing them incidentally.
- Shared test helpers and reference oracles: `sglang.test.kernels`. Preserve
vendored upstream trees and AOT wheel packaging boundaries instead of
reorganizing them incidentally.
For a move/split, preserve assertions, parametrization, fixtures, platform skips,
execution entry points, and CI stages/runners. Apportion existing time estimates
@@ -49,15 +49,15 @@ in-flight row as shipped.
| FlashInfer unified `allreduce_fusion` | `cross_device_reduce_1stage*`<br>`all_reduce`<br>`FusedAddRMSNormKernel`<br>`rmsnorm*` | `python/sglang/srt/layers/flashinfer_comm_fusion.py`<br>`python/sglang/srt/layers/layernorm.py::forward_with_allreduce_fusion`<br>`python/sglang/srt/layers/layer_boundary/residual/add_norm.py::apply_flashinfer_allreduce_fusion` | FlashInfer workspace creation plus `allreduce_fusion(..., pattern=AllReduceFusionPattern.kARResidualRMSNorm, ...)` | First suspect missing / disabled / unsupported FlashInfer allreduce fusion, not a brand new TP fusion idea. |
| AITER allreduce fusion | ROCm all-reduce plus RMSNorm still split | `python/sglang/srt/layers/layernorm.py::forward_with_allreduce_fusion`<br>`python/sglang/srt/distributed/communication_op.py::tensor_model_parallel_fused_allreduce_rmsnorm`<br>`python/sglang/srt/layers/layer_boundary/residual/add_norm.py::apply_aiter_all_reduce_fusion` | ROCm-side fused TP all-reduce + RMSNorm with fallback to plain all-reduce plus norm | On AMD, rule out existing AITER fusion before proposing a new communication fusion. |
| Fused activation-and-mul (`SwiGLU` / `GeGLU`) | `silu_and_mul`<br>`gelu_and_mul`<br>`npu_swiglu` | `python/sglang/srt/layers/activation.py` | Single op covers activation plus elementwise multiply across CUDA / CPU / NPU / XPU backends | Treat separate activation + mul on packed MLP outputs as missing existing fusion. |
| Fused dual residual RMSNorm | residual add plus two RMSNorm-like kernels around Grok blocks | `python/sglang/srt/layers/elementwise.py::fused_dual_residual_rmsnorm`<br>`python/sglang/srt/models/grok.py` | One Triton kernel computes intermediate residual update and next RMSNorm output together | On Grok-like residual layouts, treat split residual + norm as missing existing fusion. |
| Fused dual residual RMSNorm | residual add plus two RMSNorm-like kernels around Grok blocks | `python/sglang/kernels/ops/layernorm/fused_rmsnorm.py::fused_dual_residual_rmsnorm`<br>`python/sglang/srt/models/grok.py` | One Triton kernel computes intermediate residual update and next RMSNorm output together | On Grok-like residual layouts, treat split residual + norm as missing existing fusion. |
| In-place QK RMSNorm | split `q_norm` / `k_norm` kernels | `python/sglang/srt/models/utils.py::apply_qk_norm`<br>`python/sglang/kernels/ops/layernorm/norm.py::fused_inplace_qknorm` | In-place JIT QK norm plus optional `alt_stream` overlap for K | Check shape, dtype, deterministic mode, and in-place legality before proposing a new QK fuse. |
| TorchInductor horizontal Q/K norm combo-kernels | `combo_kernels`<br>`benchmark_combo_kernel`<br>`q_norm`<br>`k_norm`<br>`split_with_sizes` | `torch._inductor.config.combo_kernels` | TorchInductor can horizontally fuse sibling Q-norm and K-norm kernels in compiled traces, often deleting `split_with_sizes` / `clone` ladders | Treat separate Q/K norm ladders in compile-heavy traces as an existing compiler-fusion family first. |
| MiniMax TP fused QK RMSNorm | `MiniMaxM2RMSNormTP`<br>`rms_sumsq_serial`<br>`rms_apply_serial`<br>`forward_qk` | `python/sglang/srt/models/minimax_m2.py` | Triton kernels compute Q / K sumsq together, TP all-reduces shared stats, then apply both RMSNorms together | On MiniMax traces, separate Q norm and K norm are usually a missed model-specific Triton fusion. |
| Fused QK RMSNorm + RoPE | `qknorm*` + `rope*` + `rotary*` as separate steps | `python/sglang/kernels/ops/attention/fused_qknorm_rope.py`<br>`python/sglang/srt/models/qwen3_moe.py` | One JIT kernel applies QK RMSNorm and RoPE in-place on packed QKV | For compatible LLMs, classify split QK norm + RoPE as a missing existing fusion. |
| Fused QK RoPE reshape + KV cache write | `fused_qk_rope_reshape_and_cache*`<br>RoPE followed by reshape / cache DtoD | `python/sglang/srt/layers/attention/utils.py::fused_qk_rope_reshape_and_cache` | One Triton kernel applies RoPE to Q / K, reshapes cache layout, and writes K / V directly to paged cache | Treat separate RoPE + reshape + cache-write ladders as an existing attention-prep fusion family. |
| Fused QK RoPE reshape + KV cache write | `fused_qk_rope_reshape_and_cache*`<br>RoPE followed by reshape / cache DtoD | `python/sglang/kernels/ops/attention/fused_qk_rope_reshape_and_cache.py::fused_qk_rope_reshape_and_cache` | One Triton kernel applies RoPE to Q / K, reshapes cache layout, and writes K / V directly to paged cache | Treat separate RoPE + reshape + cache-write ladders as an existing attention-prep fusion family. |
| Fused RoPE + KV cache store | `fused_set_kv_buffer`<br>RoPE followed by KV-store, DtoD, or cache-write kernels | `python/sglang/kernels/ops/attention/rope.py`<br>`python/sglang/srt/models/utils.py::enable_fused_set_kv_buffer` | Shared entrypoints can route to fused RoPE + KV-store or model-side `fused_set_kv_buffer` fast paths | Compare against the fused cache-store path before proposing a new KV rewrite. |
| Fused decode metadata setup | `normal_decode_set_metadata`<br>`cache_seqlens_int32`<br>`cu_seqlens_k`<br>`page_table`<br>`swa_page_table` | `python/sglang/srt/layers/attention/flashattention_backend.py::normal_decode_set_metadata` | Triton decode path fuses seq-len cast/add, prefix-sum, req-to-token gather, page-table divide, and optional SWA metadata build into 1-2 kernels | If decode exposes multiple tiny metadata kernels before attention, first compare against this existing fused metadata-prep path. |
| NSA fused metadata copy for graph replay | `fused_metadata_copy`<br>`fused_metadata_copy_multi`<br>`fused_nsa_cache_seqlens`<br>`fused_flashmla_metadata` | `python/sglang/kernels/ops/attention/fused_metadata_copy.py` | CUDA graph replay path fuses multiple metadata copies into one kernel or one multi-destination kernel | Treat bursts of tiny metadata-copy kernels around NSA replay as a missed existing replay fusion. |
| NSA fused metadata copy for graph replay | `fused_metadata_copy`<br>`fused_metadata_copy_multi`<br>`fused_nsa_cache_seqlens`<br>`fused_flashmla_metadata` | `python/sglang/kernels/ops/attention/dsa/fused_metadata_copy.py` | CUDA graph replay path fuses multiple metadata copies into one kernel or one multi-destination kernel | Treat bursts of tiny metadata-copy kernels around NSA replay as a missed existing replay fusion. |
| DeepSeek MLA fused projection + norm + RoPE | `qkv_proj_with_rope_fused_weight`<br>`fused_qkv_a_proj_with_mqa`<br>`forward_absorb_fused_mla_rope*` | `python/sglang/srt/models/deepseek_common/attention_forward_methods/forward_mla_fused_rope_cpu.py`<br>`python/sglang/srt/models/deepseek_common/attention_forward_methods/forward_mla_fused_rope_rocm.py`<br>`python/sglang/srt/models/deepseek_v2.py` | CPU / ROCm paths fuse DeepSeek MLA projection packing with q / k norm, RoPE, and cache-oriented MLA prep | For DeepSeek MLA, split proj / norm / rope prep is usually an existing backend-specific fuse that did not fire. |
| Fused QK RoPE concat + MLA cache write | `fused_qk_rope_cat_and_cache_mla`<br>`set_mla_kv_buffer` | `python/sglang/srt/layers/rocm_linear_utils.py`<br>`python/sglang/srt/models/deepseek_common/attention_forward_methods/forward_mla.py` | ROCm MLA path can fuse Q / K RoPE packing, concat, and MLA cache write in one backend-specific op | On DeepSeek / MLA traces, separate RoPE-cat-cache steps are not automatically novel. |
| Qwen3 decode fused QK norm + 3D mRoPE + KV cache write | `fused_qk_norm_mrope_3d_cache_pts_quant_shuffle`<br>`mrope`<br>decode cache write | `python/sglang/srt/models/qwen3.py` | ROCm / AITER decode path fuses QK norm, 3D mRoPE, and paged KV cache write | On Qwen3-style decode, separate norm + mRoPE + cache-store kernels are not a novel opportunity. |
@@ -69,18 +69,18 @@ in-flight row as shipped.
| Qwen-style shared-expert append into routed top-k output | `_append_shared_to_topk_output`<br>`fused_append_shared_experts_with_weights`<br>`num_fused_shared_experts` | `python/sglang/srt/models/qwen2_moe.py`<br>`python/sglang/srt/layers/moe/moe_runner/triton_utils/fused_moe_triton_kernels.py` | Qwen-style MoE paths can append shared-expert ids and sigmoid gate weights to routed top-k output in one Triton kernel so the shared experts execute inside the fused MoE path | Treat routed top-k plus shared-expert pad / concat ladders as an existing MoE-prep fusion family first. |
| Fused MoE dispatch / permute / combine | token permutation<br>dispatch / combine<br>grouped top-k<br>many small MoE support kernels | `python/sglang/srt/layers/moe/fused_moe_triton/layer.py`<br>`python/sglang/srt/layers/moe/fused_moe_triton/fused_moe.py` | `FusedMoE` plus DeepEP / FlashInfer / FuseEP / standard dispatch backends and `permute_fusion=True` | First ask whether the model is missing an existing `FusedMoE`-style path or backend-specific dispatcher path. |
| Fused MoE sum + all-reduce | routed MoE followed by explicit sum-reduce kernels | `python/sglang/srt/layers/moe/fused_moe_triton/fused_moe.py`<br>`python/sglang/srt/layers/moe/fused_moe_triton/fused_moe_triton_kernels.py` | `fuse_sum_all_reduce=True` path in the second MoE GEMM | Before inventing a new MoE reduction fuse, check whether `enable_fused_moe_sum_all_reduce` is simply off or the quant path is incompatible. |
| Fused MoE activation + quant / re-quant | `silu_and_mul_*quant*`<br>`npu_dequant_swiglu_quant`<br>`swiglu_quant` | `python/sglang/srt/layers/moe/ep_moe/kernels.py`<br>`python/sglang/kernels/ops/quantization/nvfp4_gemm_swiglu_nvfp4_quant.py`<br>`python/sglang/srt/layers/moe/cutlass_w4a8_moe.py`<br>`python/sglang/srt/hardware_backend/npu/quantization/moe_methods.py` | Quantized MoE backends fuse SwiGLU / SiLU-and-mul with FP8 / FP4 / NPU re-quant before the second expert GEMM | If MoE traces show standalone activation then quant kernels, first check whether the quantized fused path is missing. |
| Fused MoE activation + quant / re-quant | `silu_and_mul_*quant*`<br>`npu_dequant_swiglu_quant`<br>`swiglu_quant` | `python/sglang/srt/layers/moe/ep_moe/kernels.py`<br>`python/sglang/kernels/ops/gemm/nvfp4_gemm_swiglu_nvfp4_quant.py`<br>`python/sglang/srt/layers/moe/cutlass_w4a8_moe.py`<br>`python/sglang/srt/hardware_backend/npu/quantization/moe_methods.py` | Quantized MoE backends fuse SwiGLU / SiLU-and-mul with FP8 / FP4 / NPU re-quant before the second expert GEMM | If MoE traces show standalone activation then quant kernels, first check whether the quantized fused path is missing. |
| DeepSeek comm-prep fused RMSNorm + quant / flatten-quant | `fused_rms_fp8_group_quant`<br>`fused_rms_mxfp4_quant`<br>`fused_flatten_fp8_group_quant`<br>`fused_flatten_mxfp4_quant` | `python/sglang/srt/layers/layer_boundary/residual/add_norm.py`<br>`python/sglang/srt/models/deepseek_common/attention_forward_methods/forward_mla.py`<br>`python/sglang/srt/models/deepseek_common/attention_forward_methods/forward_mha.py` | DeepSeek MLA / MHA ROCm paths fuse RMSNorm or flatten with FP8 / MXFP4 quantization for comm / attention prep | On DeepSeek quant traces, split norm + quant or flatten + quant is an existing family, not a new idea. |
| NSA fused top-k transform / page-table build | `fast_topk_transform_fused`<br>`fast_topk_transform_ragged_fused` | `python/sglang/srt/layers/attention/dsa_backend.py` | NSA can fuse top-k selection with paged / ragged index transform instead of separate top-k plus metadata scatter | If NSA top-k metadata work is split, check `SGLANG_DSA_FUSE_TOPK` and backend support first. |
| NSA fused quantize + indexed K-cache store | `fused_store_index_k_cache`<br>`act_quant`<br>`index_k_with_scale_buffer` | `python/sglang/kernels/ops/attention/fused_store_index_cache.py`<br>`python/sglang/srt/layers/attention/nsa/nsa_indexer.py` | Single JIT kernel quantizes bf16 K to fp8 + scale and writes directly into NSA index cache | Treat split `act_quant` + buffer-store on CUDA as missing an existing fused store path. |
| NSA fused quantize + indexed K-cache store | `fused_store_index_k_cache`<br>`act_quant`<br>`index_k_with_scale_buffer` | `python/sglang/kernels/ops/kvcache/dsa_fused_store_index_cache.py`<br>`python/sglang/srt/layers/attention/nsa/nsa_indexer.py` | Single JIT kernel quantizes bf16 K to fp8 + scale and writes directly into NSA index cache | Treat split `act_quant` + buffer-store on CUDA as missing an existing fused store path. |
| Fused sampling temperature + softmax | `fused_temperature_softmax*` | `python/sglang/srt/layers/fused_sampling.py`<br>`python/sglang/srt/layers/sampler.py` | Triton single-pass / multi-pass kernels fuse temperature scaling and softmax during decode | Separate temp-divide + softmax at decode batch sizes is often a missed existing fusion. |
| Fused logit softcap | `fused_softcap`<br>`final_logit_softcapping` | `python/sglang/srt/layers/elementwise.py`<br>`python/sglang/srt/layers/logits_processor.py` | Triton kernels fuse cast-to-float and softcap / tanh math for logits or generic elementwise softcapping | Treat exposed cast + softcap ladders as an existing Triton fuse family. |
| Linear-attention packed projection reshuffle | `fused_qkvzba_split_reshape_cat*`<br>`qkvz_proj`<br>`ba_proj`<br>`qkvabz_proj`<br>`fused_qkvbfg_a_proj` | `python/sglang/kernels/ops/attention/triton_gdn_fused_proj.py`<br>`python/sglang/srt/models/qwen3_next.py`<br>`python/sglang/srt/models/qwen3_5.py`<br>`python/sglang/srt/models/kimi_linear.py`<br>`python/sglang/srt/models/jet_nemotron.py` | GDN / Kimi / Jet-style linear-attn models pack multiple projections, then fuse split / reshape / cat into one kernel | Treat split reshape / transpose / cat ladders as an existing linear-attention fusion family. |
| Fused GDN gating prep | `fused_gdn_gating`<br>`softplus`<br>`beta_output` | `python/sglang/srt/layers/attention/fla/fused_gdn_gating.py` | Triton kernel computes GDN gate preparation such as `-exp(A_log) * softplus(...)` and `sigmoid(b)` together | On GDN traces, treat split gate-prep elementwise kernels as missing existing fusion first. |
| Fused GDN gating prep | `fused_gdn_gating`<br>`softplus`<br>`beta_output` | `python/sglang/kernels/ops/attention/linear/fused_gdn_gating.py` | Triton kernel computes GDN gate preparation such as `-exp(A_log) * softplus(...)` and `sigmoid(b)` together | On GDN traces, treat split gate-prep elementwise kernels as missing existing fusion first. |
| Fused RMSNorm-gated linear-attention output | `FusedRMSNormGated`<br>`layer_norm_gated_fwd` | `python/sglang/srt/layers/attention/fla/fused_norm_gate.py`<br>`python/sglang/srt/models/qwen3_next.py`<br>`python/sglang/srt/models/kimi_linear.py` | One Triton op covers residual-aware (RMS)Norm plus sigmoid / swish gating | If norm and output gate appear as separate kernels in GDN / Kimi-like blocks, first suspect a missing existing fusion. |
| Fused gated RMSNorm / LayerNorm | `rms_norm_gated`<br>`layer_norm_gated` | `python/sglang/srt/layers/attention/mamba/ops/layernorm_gated.py` | Mamba-derived kernels can fuse normalization with the gating branch `z * sigmoid(z)` | Treat split norm and gate post-processing on Mamba-style blocks as an existing fusion family. |
| Fused linear-attention chunk KKT + solve_tril | `chunk_gated_delta_rule_fwd_kkt_solve_kernel`<br>`scaled_dot_kkt`<br>`solve_tril`<br>`recompute_w_u` | `python/sglang/srt/layers/attention/fla/chunk_fwd.py`<br>`python/sglang/srt/layers/attention/fla/kda.py` | GDN / KDA chunk forward fuses `scaled_dot_kkt + solve_tril` in the prefill / intra-chunk path, then finishes `recompute_w_u` as the next step | Treat split KKT + triangular-solve ladders as an existing linear-attention fusion family first. |
| Fused linear-attention recurrent / KDA update | `fused_sigmoid_gating_delta_rule_update`<br>`fused_recurrent_gated_delta_rule_update`<br>`fused_kda_gate` | `python/sglang/srt/layers/attention/fla/fused_sigmoid_gating_recurrent.py`<br>`python/sglang/srt/layers/attention/fla/fused_recurrent.py`<br>`python/sglang/srt/models/kimi_linear.py`<br>`python/sglang/srt/models/jet_nemotron.py` | Triton / CuTeDSL kernels fuse gating math, optional QK l2norm, recurrent state update, and output generation | Treat split gating + recurrent-update chains as existing linear-attention fusion, not a novel opportunity. |
| Fused linear-attention recurrent / KDA update | `fused_sigmoid_gating_delta_rule_update`<br>`fused_recurrent_gated_delta_rule_update`<br>`fused_kda_gate` | `python/sglang/kernels/ops/attention/linear/fused_sigmoid_gating_recurrent.py`<br>`python/sglang/kernels/ops/attention/fla/fused_recurrent.py`<br>`python/sglang/srt/models/kimi_linear.py`<br>`python/sglang/srt/models/jet_nemotron.py` | Triton / CuTeDSL kernels fuse gating math, optional QK l2norm, recurrent state update, and output generation | Treat split gating + recurrent-update chains as existing linear-attention fusion, not a novel opportunity. |
| Fused Mamba state gather/scatter with mask | `fused_mamba_state_scatter_with_mask`<br>`index_elementwise_kernel` | `python/sglang/srt/layers/attention/mamba/mamba_state_scatter_triton.py` | Triton kernel replaces multiple masked gather / scatter index kernels with one fused update | If Mamba verify/update shows many tiny index kernels, first compare against this existing fused path. |
| Staging-buffer fused gather / scatter | `_fused_gather_to_staging_kernel`<br>`_fused_scatter_from_staging_kernel` | `python/sglang/srt/disaggregation/common/staging_buffer.py` | Triton kernels gather scattered KV slices into contiguous staging memory and scatter them back into KV cache on decode | Treat ladders of tiny gather/scatter/copy kernels in heterogeneous TP staging as missing an existing Triton fusion. |
@@ -146,7 +146,7 @@ Stable entries should be folded into the mainline family rows above.
| PR `#22005` fused add + RMSNorm + per-token FP8 quant | `fused_add_rmsnorm_per_token_quant`<br>`per_token_quant_fp8` | `PR #22005`<br>`python/sglang/kernels/jit/csrc/elementwise/fused_add_rmsnorm_per_token_quant.cuh`<br>`python/sglang/kernels/jit/fused_add_rmsnorm_per_token_quant.py` | CUDA JIT kernel keeps normed values in registers and emits BF16 + FP8 outputs plus per-token scales | If FP8 online-quant traces show add+norm followed by per-token quant, treat this as an in-flight upstream CUDA fuse family. |
| PR `#20667` Qwen3.5 fused QK norm + RoPE + KV cache write | `fused_qk_norm_rope_cache_pts_quant_shuffle`<br>`fused_qk_norm_mrope_3d_cache_pts_quant_shuffle`<br>`rotary_dim` | `PR #20667`<br>`python/sglang/srt/models/qwen3_5.py`<br>`python/sglang/srt/models/utils.py` | ROCm / AITER path fuses Q / K RMSNorm, partial or 3D RoPE, and direct KV cache write for Qwen3.5 attention | Treat split QK-norm + RoPE + cache-store on Qwen3.5 as a concrete in-flight upstream family, not a novel idea. |
| PR `#22392` CUTLASS FP8 GEMM replacing nvjet | `cutlass_scaled_mm`<br>`fp8_scaled_mm`<br>`nvjet`<br>`cudaMemsetAsync` | `PR #22392`<br>`python/sglang/kernels/aot/python/sgl_kernel/gemm.py`<br>`python/sglang/srt/layers/quantization/fp8_utils.py` | Runtime replacement swaps nvjet FP8 GEMMs for CUTLASS kernels, removing per-launch memset bubbles and extra output-copy kernels | Treat nvjet GEMM + memset bubble ladders as an in-flight SGLang linear-kernel family before calling them novel. |
| PR `#18612` NVFP4 CUTLASS MoE fused SiLU+Mul+quant | `silu_and_mul_scaled_nvfp4`<br>`nvfp4 expert quant`<br>`cutlass moe` | `PR #18612`<br>`python/sglang/srt/layers/moe/cutlass_w4a8_moe.py`<br>`python/sglang/kernels/ops/quantization/nvfp4_gemm_swiglu_nvfp4_quant.py` | Fuses MoE activation epilogue and NVFP4 expert quantization before the CUTLASS MoE second GEMM | Treat split SiLU+Mul then NVFP4 expert quant in CUTLASS MoE traces as an in-flight upstream SGLang family. |
| PR `#18612` NVFP4 CUTLASS MoE fused SiLU+Mul+quant | `silu_and_mul_scaled_nvfp4`<br>`nvfp4 expert quant`<br>`cutlass moe` | `PR #18612`<br>`python/sglang/srt/layers/moe/cutlass_w4a8_moe.py`<br>`python/sglang/kernels/ops/gemm/nvfp4_gemm_swiglu_nvfp4_quant.py` | Fuses MoE activation epilogue and NVFP4 expert quantization before the CUTLASS MoE second GEMM | Treat split SiLU+Mul then NVFP4 expert quant in CUTLASS MoE traces as an in-flight upstream SGLang family. |
| PR `#22918` FlashInfer per-token NVFP4 MoE | `per_token_nvfp4`<br>`trtllm_fp4_block_scale_moe`<br>`FlashInfer MoE` | `PR #22918`<br>`python/sglang/srt/layers/moe/fused_moe_triton/fused_moe.py` | Adds FlashInfer-backed per-token NVFP4 MoE execution so expert quant/dequant work can move into the fused MoE backend | Treat standalone per-token NVFP4 MoE support kernels as a candidate missing backend-selection path, not an automatically novel kernel idea. |
| PR `#22851` NSA top-k backend and FlashInfer / PyTorch top-k split | `nsa topk`<br>`flashinfer_topk`<br>`pytorch_topk`<br>`fast_topk_transform` | `PR #22851`<br>`python/sglang/srt/layers/attention/dsa_backend.py` | Makes NSA top-k backend selection explicit and aligns fused top-k transform with FlashInfer / PyTorch fallbacks | When NSA top-k dominates decode, first classify it as backend selection or fused-transform eligibility work. |
| PR `#24125` GLM5 NSA decode CatArrayBatchedCopy removal | `CatArrayBatchedCopy`<br>`GLM-5`<br>`NSA`<br>`TileLang decode` | `PR #24125`<br>`python/sglang/srt/layers/attention/dsa_backend.py` | Skips redundant cat/copy work in the GLM5 NSA TileLang decode path | Treat cat/copy bursts in GLM5 NSA decode as a concrete in-flight cleanup opportunity. |
@@ -18,7 +18,7 @@ Use these upstream files when the workflow or behavior needs to be justified fro
## Scheduler-side trace writing
- `python/sglang/srt/managers/scheduler_profiler_mixin.py`
- `python/sglang/srt/managers/scheduler_components/profiler_manager.py`
- actual trace start/stop behavior
- filename pattern for `TP/DP/PP/EP` and optional stage suffixes
- `CUDA_PROFILER` and torch profiler handling
@@ -37,6 +37,3 @@ Use these upstream files when the workflow or behavior needs to be justified fro
- `test/registered/profiling/test_start_profile.py`
- validates `/start_profile` behavior, including `CUDA_PROFILER`
- `test/registered/profiling/test_profile_v2.py`
- validates stage-scoped trace outputs under `SGLANG_PROFILE_V2`
@@ -482,7 +482,9 @@ FUSION_PATTERN_REGISTRY: Tuple[FusionPatternSpec, ...] = (
),
FusionPatternSpec(
pattern="Fused QK RoPE reshape + KV cache write",
candidate_path="python/sglang/srt/layers/attention/utils.py",
candidate_path=(
"python/sglang/kernels/ops/attention/fused_qk_rope_reshape_and_cache.py"
),
active_keywords=("fused_qk_rope_reshape_and_cache",),
split_groups=(
("rotary", "rope", "mrope"),
@@ -533,7 +535,7 @@ FUSION_PATTERN_REGISTRY: Tuple[FusionPatternSpec, ...] = (
),
FusionPatternSpec(
pattern="NSA fused metadata copy for graph replay",
candidate_path="python/sglang/kernels/ops/attention/fused_metadata_copy.py",
candidate_path="python/sglang/kernels/ops/attention/dsa/fused_metadata_copy.py",
active_keywords=(
"fused_metadata_copy",
"fused_metadata_copy_multi",
@@ -700,7 +702,7 @@ FUSION_PATTERN_REGISTRY: Tuple[FusionPatternSpec, ...] = (
pattern="Fused MoE activation + quant / re-quant",
candidate_path=(
"python/sglang/srt/layers/moe/ep_moe/kernels.py"
"<br>python/sglang/kernels/ops/quantization/nvfp4_gemm_swiglu_nvfp4_quant.py"
"<br>python/sglang/kernels/ops/gemm/nvfp4_gemm_swiglu_nvfp4_quant.py"
"<br>python/sglang/srt/layers/moe/cutlass_w4a8_moe.py"
),
active_keywords=(
@@ -761,7 +763,7 @@ FUSION_PATTERN_REGISTRY: Tuple[FusionPatternSpec, ...] = (
FusionPatternSpec(
pattern="NSA fused quantize + indexed K-cache store",
candidate_path=(
"python/sglang/kernels/ops/attention/fused_store_index_cache.py"
"python/sglang/kernels/ops/kvcache/dsa_fused_store_index_cache.py"
"<br>python/sglang/srt/layers/attention/nsa/nsa_indexer.py"
),
active_keywords=("fused_store_index_k_cache",),
+87 -78
View File
@@ -15,7 +15,7 @@ One container owns process-static runtime state: `sglang.srt.runtime_context.Run
| runtime flags | `get_flags()` | state that is *not* a pure function of config: `capture` (cuda-graph lifecycle), `moe` (ACTIVE backends, swappable), `dp` (DP-attention runtime flags) | materialized at subsystem init; groups offer `override()` for tests |
| resources | `get_resources()`, `get_stream(name)`, `get_buffer(name, factory)` | process-level handles: graph pools, EPLB state, EP dispatcher state, named side streams, workspace buffers | lazy; cleared by `reset_context()` |
| per-forward | `get_forward()` | forward-scoped flags (multi-stream switch, MoE output buffer, attn-TP inputs, extend-in-batch) | contextvar-backed; `scoped(**kw)` restores on exit; new threads see defaults |
| parallel | `get_parallel()` | one spelling per name: ranks and group handles are the live topology (`@property`, read-through); every other name, sizes included, is a leaf of the parallel config bag | ranks/groups: after dist init; leaves: after publish |
| parallel | `get_parallel()` | one spelling per name: scoped override, then permanent stamp, then published parallel config leaf | ranks: stamped at publish; group handles: stamped at distributed init; configured and derived widths: after publish |
`reset_context()` (unit-test teardown) drops the published config and installs fresh
flags/resources/forward tiers.
@@ -87,13 +87,12 @@ with what the operator typed, not with what resolution decided.**
`ModelRunner.update_model_fields` overrides `model_path` / `load_format` for
the target runner.
- **Late launcher-stage resolution (pre-publish)**: a few rules cannot run inside
`__post_init__` — LoRA normalization, and the auto-parser detection that needs a
`ServerArgs.resolve_once` — LoRA normalization, and the auto-parser detection that needs a
tokenizer/chat-template load. They are resolution, not mutation, and they
**declare** via `arg_groups.overrides.declare_resolution(server_args, source,
**fields)`, the same call the rest of the pipeline makes; there is no
`declare_late_resolution` any more. *When* a declaration is made is not
something the code marks — the guardrails that used to read that marker
cover these sites through the ordinary keyword scan instead. The declaration lands
something the code marks. The declaration lands
in the stash on that very object, so every holder of it carries the decision —
the HTTP server, the multi-tokenizer workers it is serialized for, the
schedulers it forks — and each of them publishes bags projected from it. The
@@ -135,16 +134,23 @@ bag to override at all.
instance: **constructor arguments** (`ModelRunner(draft_attention_backend=...)`,
`MMEncoder(gpu_id=...)`) and **runner attributes holding the resolved value**
(`model_runner.kv_cache_dtype_str`, `prefill_attention_backend_str`,
`num_fused_shared_experts`, `linear_attn_backends`) — threaded to consumers as
`linear_attn_backends`) — threaded to consumers as
arguments, never backfilled onto a shared object. A per-runner choice also stays
*out* of the bags: recording it there is how a second runner inherits the first
one's answer, which is exactly the bug `linear_attn_backends` replaced. The one sanctioned bend in that rule is
*scoped*: `ModelRunner._load_format_scope` exposes the draft's
`--speculative-draft-load-format` through `get_model().override(load_format=...)`
for exactly the duration of the draft build, because model construction
reads that bag leaf — the override restores on exit, so nothing outlives
the scope. When there is a runner in hand, read its
stamp; that is a different rule from "read the instance".
one's answer, which is exactly the bug `linear_attn_backends` replaced. The two
sanctioned bends in that rule are *scoped*. `ModelRunner._load_format_scope`
exposes the draft's `--speculative-draft-load-format` through
`get_model().override(load_format=...)` for exactly the duration of the draft
build, because model construction reads that bag leaf — the override restores
on exit, so nothing outlives the scope. The shared-expert layout is another scoped view:
`get_model().shared_experts` selects the immutable `SharedExpertsConfig` owned
by the loaded model (`model.shared_experts_config`). The loader binds it for
construction, and runners bind it for execution, warmup and graph capture.
Both the fused count and ordinary TopK slot count belong to this object;
neither is a separately mutable flag. A scope selects a built layout and
restores the previous reference on exit, including exceptions. It cannot
resize weights, buffers or a captured graph. When the model is in hand, read
its owned config directly (as the routed-experts capturer does).
- **Per-instance boundaries** — the tokenizer-manager family, everything under
`entrypoints/`, and the tokenizer-process multimodal processors read the bags.
The old justification for keeping them on `self.server_args` ("several
@@ -178,8 +184,8 @@ bag to override at all.
### Four ways a config sweep breaks something no test runs
Each of these shipped in a review round and cost a real defect; each now has a
guard, named here so the next sweep checks the same four things by hand first.
Each of these shipped in a review round and cost a real defect. Where a test
guards one it is named; the next sweep checks all four by hand first.
1. **The other implementations of an interface.** Dropping a parameter means
auditing implementers, not just callers: `CustomSpecAlgo` is the plugin
@@ -191,7 +197,8 @@ guard, named here so the next sweep checks the same four things by hand first.
`publish` says nothing about whether a given read runs before it. Spawned
workers (`MMEncoder` for encoder DP/TP, the Ray scheduler actor) start with
an empty context, so a bag read above the publish raises only there.
Guard: `test_publish_precedes_bag_reads.py`.
Guarded on the launch path only (`test_pre_publish_readers.py`); other
entries, spawned workers included, are checked by review.
3. **The role namespace a process publishes under.** `ROLE_NAMESPACE_SETS`
narrows what each role may read; the DP controller is audited for `exec`
alone. A helper that reaches for another namespace passes every default-mode
@@ -201,7 +208,9 @@ guard, named here so the next sweep checks the same four things by hand first.
means enumerating the others: HTTP, gRPC and in-process `Engine` each have
their own server-info and model-info, and each passes its own tests while
its users lose the field.
Guard: `test_effective_state_surfaces.py`.
Guarded for server info across HTTP, both gRPC roles and the Rust frontend
(`TestServerInfoTransportParity` in `entrypoints/test_server_info.py`);
`Engine.get_server_info` and model info are checked by review.
A fifth, from the same rounds: the accessor **name itself**. Called as an
object member (`manager.get_disagg()`), or shadowed by a same-named import
@@ -223,12 +232,13 @@ slack (`>= len(...) - 1`) or compared key names instead of value sources.
### `get_parallel()`: one spelling per name
**There is no `.config` hop.** Ranks and group handles are `@property`
read-through over the canonical getters, so they answer with the live process
groups. Everything else — `tp_size`, `pp_size`, `attn_cp_size`, `dcp_size`,
`moe_dp_size` included, alongside config-only leaves like `nccl_port`,
`dp_size`, `ep_size`, `dwdp_size` — is answered from the
published `parallel` bag. Reading a leaf before publish raises a `ValueError`
**There is no `.config` hop.** `ParallelContext._read` answers every name in
one order: a scoped override, then a permanent stamp, then the published
`parallel` bag leaf. `publish` stamps ranks from the launcher; distributed init
stamps group handles and validates their widths. Configured widths such as
`tp_size`, `pp_size`, `attn_cp_size`, `dcp_size` and `moe_dp_size`, and config-only
leaves such as `nccl_port`, `dp_size`, `ep_size` and `dwdp_size`, fall back to the
published bag. Reading a config leaf before publish raises a `ValueError`
naming the namespace; an unknown name is an `AttributeError`.
A size reads from the configuration because the groups are built at exactly the
@@ -245,38 +255,33 @@ configured widths — checked at every assignment to `_TP` / `_PP` / `_ATTN_CP`
coordinators keep their construction width. Those are different names, not two
answers to one name.
DCP keeps its own pair: `get_parallel().attn_dcp_size` / `.dcp_enabled` answer the
*effective* topology (`1` / `False` with no group installed), while `dcp_size` is
DCP keeps its own pair: `get_parallel().attn_dcp_size` / `.dcp_enabled` are derived
from the published configuration unless explicitly overridden; `dcp_size` is
what the launch requested.
A process-global seed field-read of one of these sizes
(`get_server_args().tp_size`, or an alias of it) is a read-ratchet failure. A
`server_args` the object was *handed* is a different thing and not a ratchet
matter — see "Reads that legitimately stay on a ServerArgs instance".
Fail-loud is narrower: before dist init, a live *rank/group* read raises. The six
Fail-loud is narrower: an unstamped rank or group handle with no declared default
raises; ranks are available after publish, before dist init. The six
parallel quotients are not live reads at all — `attn_tp_size`, `attn_dp_size`,
`attn_dcp_size`, `moe_ep_size`, `moe_tp_size`, `dcp_enabled` are a function of the
configured leaves, computed once at publish into bag leaves, and answered
override → stamp → published leaf. So `dcp_enabled` means "the launch configured
DCP" (`dcp_size > 1`), not "a DCP group is installed here"; in a scheduler the
stamp makes the two identical, in a process that publishes without dist init they
differ. `test_a_topology_is_stated_by_naming_the_width` and its neighbours in
`test_runtime_context.py` pin this; they replaced
DCP" (`dcp_size > 1`), not "a DCP group is installed here". Normal distributed
initialization builds the groups to match the configured widths; it does not
restamp these DCP leaves. `test_a_topology_is_stated_by_naming_the_width` and its
neighbours in `test_runtime_context.py` pin this; they replaced
`test_attn_dcp_defaults_when_group_is_uninitialized`. One consequence for tests:
overriding a leaf no longer moves its quotient — state a topology by publishing a
config, or by naming the width. After init,
only the DCP group is optional (`_DCP` exists only when `dcp_size > 1`; attn-CP and
moe-DP always install, as size-1 aliases if unused). The `config` hop is
deliberately dynamo-traceable (a plain property over a slot, no
`object.__getattribute__`); gate helpers like `enable_moe_dense_fully_dp()` run inside
moe-DP always install, as size-1 aliases if unused). Bare parallel leaf reads are
dynamo-traceable; gate helpers like `enable_moe_dense_fully_dp()` run inside
compiled model forwards (`test_parallel_config_leaves_trace_under_torch_compile` pins
this).
A third surface carries the same names: `ParallelState` (`self.ps` / `mr.ps`), the
frozen per-process snapshot built once in `Scheduler.__init__` from these configured
sizes plus this process's ranks, and handed down (draft runners included). Prefer it
where an object was handed one; it is not a global accessor.
### Reading config: the seed is off limits
`get_server_args().field` in business code is a ratchet failure. Read:
@@ -322,12 +327,13 @@ where an object was handed one; it is not a global accessor.
follows a post-publish override. Two questions are *not* that, and have their
own spelling: the width of the MoE communicator you are about to collectively
operate on is `get_moe_cp_size()` (the `_MOE_DP = _ATTN_CP` alias makes it
differ), and the effective DCP topology is `get_parallel().attn_dcp_size` /
`.dcp_enabled` (`1` / `False` when no group is installed), which does not need
dist init to answer.
differ), and DCP's derived width and enablement are
`get_parallel().attn_dcp_size` / `.dcp_enabled` (published configuration
unless explicitly overridden).
- **this runner's resolved value** → the runner
(`prefill_attention_backend_str`, `kv_cache_dtype_str`,
`draft_attention_backend`, `num_fused_shared_experts` on the model).
`draft_attention_backend`); for what the model's layers bake in at
construction, the layer (`num_fused_shared_experts` on a MoE block).
`self.server_args.field` is still right for handed per-instance config (see
"Reads that legitimately stay on a ServerArgs instance" above for the full set —
@@ -335,10 +341,9 @@ per-instance boundaries and whole-object passes; there are no per-runner config
copies to read any more). The allow-list is `GrammarManager` and `MMEncoder`;
what sits beside it is residue, not a family — and not for one single reason:
- the tokenizer-manager family and `entrypoints/` **read the bags**; what is
left of them in the exposure ratchet is a handful of individually-dispositioned
pairs, not a family awaiting conversion. Read the ratchet for the current set
rather than assuming a directory is off-limits;
- the tokenizer-manager family and `entrypoints/` **read the bags**; review a new
instance read there against the raw-input contract rather than assuming a
directory is off-limits;
- `GrammarManager` is a handed instance for its residual `self.server_args`
reads, but backend selection is **not** on the instance any more:
`create_grammar_backend` reads `get_exec().kernel.grammar_backend`, and
@@ -384,15 +389,15 @@ if you do it, say so in the test.
### Mid-resolution reads (inside the pipeline only)
Resolution runs in `__post_init__` and **writes nothing onto the record**: a
handler declares (`self._declare` / `declare_resolution`), the declaration goes
Resolution runs in `ServerArgs.resolve_once` and **writes nothing onto the record**: a
handler declares (`declare_resolution`), the declaration goes
into the stash, and the fields keep what the caller passed. So a mid-resolution
read of a field answers with the *raw input* — every reader in the pipeline goes
through a view instead:
- `resolving_view(server_args)` / `self._resolved()` — the live view (walks the
- `resolving_view(server_args)` — the live view (walks the
stash per read). This is what handlers and hooks bind, conventionally as
`cfg = resolving_view(self)` at the top of the handler.
`cfg = resolving_view(server_args)` at the top of the handler.
- `resolved_view(server_args)` — snapshots the overlay when built, which is what
a post-process pass wants: it reads the state at *its* slot.
@@ -451,13 +456,18 @@ Never assign `server_args` fields from model code. Declare instead
its checkpoint drives (shared-experts fusion) is a question the *loader* asks
the model class — `shared_experts_fusion_disable_reason(hf_config,
quant_config)`, a classmethod answering without an instance — at the single
model-instantiation point, and
`install_shared_experts_fusion_decision` writes the answer to the ACTIVE moe
flag before that model's layers build and read it
(`is_shared_experts_fusion_disabled`, config-intent fallback).
`draft_model_build_scope` brackets every draft build and routes the draft's
answer to the speculative leaf, so a draft's decision never overwrites the
target's. A process-level load-time fact (the sm80 dtype fallback —
model-instantiation point, together with how many shared experts it fuses
(`num_fusable_shared_experts(hf_config)`; an entry class without both hooks
fuses none). `resolve_shared_experts_config` returns one immutable config;
the loader scopes it through `get_model().override(shared_experts=config)`
while building and retains it on the resulting model. `num_shared_topk_slots`
derives from the fused count and frozen Waterfill policy, so ordinary routing
and Waterfill cannot both claim the shared slots. `TopK`/`HashTopK` retain the
same config; `top_k` counts only routed experts at construction.
`draft_model_build_scope` still selects draft boundary/MMA preferences; it
does not save shared-expert flags. Target and draft each own their layout.
These values have model lifetime, so they do not belong in `get_forward()`.
A process-level load-time fact (the sm80 dtype fallback —
device-driven, identical for every runner) records directly via
`get_context().override`.
@@ -466,12 +476,12 @@ dataclass. A declaration against a non-whitelisted field fails at its slot.
### Load-time vs resolution-time (critical)
`__post_init__` runs in the launcher process before any model/platform import. Logic that
`ServerArgs.resolve_once` runs in the launcher process before any model/platform import. Logic that
consults an **extensible registry** (e.g. out-of-tree platforms registering attention
backends in `init_backend()`, which runs at `model_runner` import) must stay at load time
(ModelRunner init), writing through `get_context().override()`. Before moving any
load-time logic into resolution, verify everything it reads is already complete at
construction time.
resolution time.
## Runtime flags (`get_flags()`)
@@ -499,8 +509,8 @@ Named slots + two keyed-lazy registries:
side stream is `"lora_side"`. Two call sites may share a name only if their work belongs
on one stream — sharing across roles serializes intended overlap.
- `get_buffer(name, factory)` — get-or-create a named persistent buffer. Grow-only or
per-device semantics manage their `resources.buffers` entries directly (see tokenspeed /
SM120 split / Marlin workspace). Buffer names are per-backend today; do not silently
per-device semantics manage their `resources.buffers` entries directly (see SM120
split / Marlin workspace). Buffer names are per-backend today; do not silently
share.
- Singletons with manager semantics (EP dispatcher buffers, EPLB recorder/metadata, graph
memory pool) keep their owning accessors/classes as facades; only the *state* lives in a
@@ -543,8 +553,9 @@ ONE thread — do not design for TBO threads that don't exist.
them explicitly on the mock; `MagicMock(spec=...)` raises on attributes that only
exist post-`__init__`, which is the fastest way to find a missed stub.
- `reset_context()` in teardown when a test publishes outside a scoped override.
- `ServerArgs(model_path="dummy")` early-returns the pipeline (few declarations, no
strict guard) — fine for lightweight fixtures.
- `ServerArgs(model_path="dummy")` resolves through the pipeline's early return (few
declarations) — fine for lightweight fixtures. The strict guard still applies once
`resolve_once` has run, as it does at `publish`.
- **Asserting what resolution decided reads `resolution_result(sa, "field")`**, not
`sa.field`: the field is the raw input. Assert the field only when the point of
the case *is* that the record stayed pristine (the FA4 page-size and waterfill
@@ -553,7 +564,7 @@ ONE thread — do not design for TBO threads that don't exist.
pytest run lets a context published by an earlier file mask a missing-publish bug in a
later one.
## Guardrails (these fail CI; what to do when they fire)
## Guardrails and review rules
1. **Strict mutation guard** (always on, and with no exception): bare
`server_args.x = ...` after resolution raises unconditionally in
@@ -563,10 +574,9 @@ ONE thread — do not design for TBO threads that don't exist.
resolved config with `get_context().override`; hand a per-runner value to its
runner as a constructor argument. Projected bags are sealed the same way (leaf
assignment raises).
2. **Mutation ratchet** (`test_server_args_mutation_ratchet.py`, exact pin 0 over the whole
package minus the pipeline / multimodal_gen): textual scan for assignment forms. Never
raise the baseline.
3. **No-copy contract** (`test_server_args_no_instance_mutation_entry.py`): neither
2. **No textual mutation ratchet**: the strict guard above is the only mutation
check.
3. **No-copy contract** (review only, no automated guard): neither
`ServerArgs.override` nor `ServerArgs.derive` exists, and nothing in the package
calls either form. Rerouting a writer to the bags means flipping **all its readers
in the same commit** (no transitional dual-write).
@@ -577,23 +587,23 @@ ONE thread — do not design for TBO threads that don't exist.
runner's stamp — not `get_server_args().field`, which the read ratchet below pins
at zero. `runtime_context.get_server_args()` is only for the
whole-object shapes (dumps, provenance, a hand-off to a callee that takes a config).
5. **Global config read ratchet** (`test_global_config_read_ratchet.py`): baselines are
**0** for both the direct `get_server_args().field` and the alias form (function-local
5. **Global config read check** (`test_global_config_read_ratchet.py`): no business code
reads a field off `get_server_args()`, directly or through an alias (function-local
— including local copies of an alias, `cfg = sa` — module-level, or parked on an
instance attribute, plus the `getattr(..., "field")` spelling of each; a name
computed at runtime or indirection deeper than a local name copy is census-tool
territory, per the test's docstring). The scanner matches `get_server_args` by its
computed at runtime or indirection deeper than a local name copy is outside the
check, per the test's docstring). The scanner matches `get_server_args` by its
literal name — bare or module-qualified (`ctx.get_server_args()`) — and
`TestNoRenamedAccessorImports` in the same file *bans* `import ... as` renames of it,
which is what makes literal-name matching sound. Exempt by owner
module only (`runtime_context.py`, `server_args.py`, `arg_groups/`). Two classes,
no more: `TestGlobalConfigReadRatchet` holds the two baselines and
no more: `TestGlobalConfigReads` holds the zero check and
`TestNoRenamedAccessorImports` holds the ban. There is no configured-size registry
here any longer — `get_parallel()` has one spelling per name, so a size read is not a
choice between two answers and nothing needs registering.
6. **Module-state ratchet** (`test_module_state_ratchet.py`): `global` statements in the
flag-owning layers are pinned by name. A new module-level runtime global belongs on a
flags group / resources slot instead; migrating a pinned survivor must shrink the pin.
6. **Module-state rule** (review only, no automated guard): a new module-level
runtime global in the flag-owning layers belongs on a flags group / resources slot
instead.
7. **Namespace coverage** (`test_server_args_namespaces.py`,
`test_runtime_context_config_bags.py`): every `ServerArgs` field resolves to a
namespace — from the `arg_groups/fields/` class that declares it — and the
@@ -649,8 +659,7 @@ Key source files: `python/sglang/srt/runtime_context.py` (the container, every t
`python/sglang/srt/arg_groups/overrides.py` (override registry, passes,
`declare_resolution` and the spellings around it), `python/sglang/srt/server_args.py` (`NS` metadata,
`Arg(..., resolvable=True)`, `__setattr__` strict guard), and the guardrail tests under
`test/registered/unit/` (`test_server_args_mutation_ratchet.py`,
`test_global_config_read_ratchet.py`,
`test_module_state_ratchet.py`, `test_server_args_namespaces.py`,
`test/registered/unit/` (`test_global_config_read_ratchet.py`,
`test_server_args_namespaces.py`,
`test_runtime_context.py` — the last one doubles
as executable documentation of every tier's semantics).
+1 -1
View File
@@ -36,4 +36,4 @@ a real `AttributeError` if the field is ever renamed:
revision=getattr(server_args, "revision", None), # BAD
revision=server_args.revision, # GOOD
```
(see `python/sglang/srt/managers/template_detection.py`)
(see `python/sglang/srt/parser/template_detection.py`)
+2 -2
View File
@@ -27,8 +27,8 @@ A new unit test case must fall into one of these categories:
3. **Critical-path bookkeeping.** Defends conventions that are easy to break by
forgetting to sync -- registry completeness, field lifecycle, serialization
compatibility. Enumerating assertions are fine here; the guarded failure
mode is "someone extended X without updating Y". Example: the ratchet tests
(`test/registered/unit/test_module_state_ratchet.py`).
mode is "someone extended X without updating Y". Example:
`test/registered/unit/test_global_config_read_ratchet.py`.
Not admissible:
+8
View File
@@ -1,12 +1,14 @@
.github @merrymercy @Fridge003 @ispobock @Kangyan-Zhou @HaiShaw @bingxche
/docker @Fridge003 @ispobock @HaiShaw @ishandhanani @yctseng0211 @sogalin
/docker/npu.Dockerfile @ping1jing2 @iforgetmyname @whybeyoung
/docker/sgl-router.Dockerfile @sherlockwu @Kangyan-Zhou @ShangmingCai
/docs @wisclmy0611 @zijiexia @sogalin
/docs @wisclmy0611 @zijiexia @Richardczl98 @JustinTong0323 @sogalin
/python/pyproject.toml @merrymercy @Fridge003 @ispobock
/python/sglang/kernels @DarkSharpness @HaiShaw @BBuf @celve @HydraQYH @yuan-luo
/python/sglang/kernels/ops/diffusion @yingluosanqian @BBuf @mickqian @triple-mu @niehen6174
/python/sglang/kernels/ops/attention/fla @yizhang2077 @hebiao064 @yuan-luo
/python/sglang/kernels/ops/attention/linear @yizhang2077 @hebiao064 @yuan-luo
/python/sglang/kernels/ops/lora @Ying1123 @Fridge003 @lifuhuang @yushengsu-thu @jybsuper
/python/sglang/kernels/jit/csrc/lora @Ying1123 @Fridge003 @lifuhuang @yushengsu-thu @jybsuper
/python/sglang/multimodal_gen @mickqian @ping1jing2 @HaiShaw @yichiche @AgainstEntropy @BBuf @triple-mu @kevin-mii @niehen6174
@@ -81,6 +83,11 @@
/rust/sglang-processor @sagearc @sherlockwu
/rust/sglang-radix-tree @Jialin @alphabetc1
/rust/sglang-server @rainj-me @merrymercy
/sgl-router @sherlockwu @Kangyan-Zhou @ShangmingCai @hzh0425
/sgl-router/sgl-kv-indexer @sherlockwu @lixiufei-leo @ShangmingCai @wuyl1 @hzh0425
/sgl-router/src/policies @sherlockwu @ShangmingCai @Bo-Vincent
/sgl-router/src/workers @sherlockwu @AMD-yanfeiwang
/sgl-router/src/protocol @sherlockwu @whybeyoung
/sgl-model-gateway @slin1237 @CatherineSue
/sgl-model-gateway/benches @slin1237
/sgl-model-gateway/bindings/python @CatherineSue @key4ng @slin1237
@@ -107,6 +114,7 @@
/python/sglang/srt/speculative/cpp_ngram @hnyls2002 @Qiaolin-Yu @kpham-sgl
/python/sglang/srt/speculative/frozen_kv_mtp_*.py @hnyls2002 @Qiaolin-Yu @kpham-sgl @pyc96
/python/sglang/kernels/ops/speculative/ngram_*.py @hnyls2002 @Qiaolin-Yu @kpham-sgl
/python/sglang/kernels/ops/embeddings/ngram_embedding.py @hnyls2002 @Qiaolin-Yu @kpham-sgl
/python/sglang/kernels/jit/csrc/ngram_corpus @hnyls2002 @Qiaolin-Yu @kpham-sgl
/python/sglang/srt/models/laguna.py @Jiminator
/python/sglang/srt/configs/laguna.py @Jiminator
+2 -2
View File
@@ -14,9 +14,9 @@ Recognized permission keys:
| --- | --- |
| `can_tag_run_ci_label` | `/tag-run-ci-label`, `/tag-and-rerun-ci`, `/run-full-ci`, `/run-extra-ci` |
| `can_rerun_failed_ci` | `/rerun-failed-ci`, `/tag-and-rerun-ci`, `/run-full-ci`, `/run-extra-ci` |
| `cooldown_interval_minutes` | rate limit in `pr-gate.yml`; `0` also grants `/rerun-test`, `/rerun-group` |
| `cooldown_interval_minutes` | rate limit in `pr-gate.yml`; `0` also grants `/rerun-test`, `/rerun-group`, `/run-cpu-ci` |
`/rerun-test` and `/rerun-group` are gated on the commenter alone: either
`/rerun-test`, `/rerun-group` and `/run-cpu-ci` are gated on the commenter alone: either
`cooldown_interval_minutes: 0`, or `write`/`admin` permission on the repo. Where
the PR comes from makes no difference, and authoring it grants nothing.
+2 -2
View File
@@ -11,7 +11,7 @@ inputs:
required: false
default: ''
cache_key_prefix:
description: 'Must match what _pr-test-rust-ext-build.yml saves under.'
description: 'Must match what _pr-test-rust-ext-build.yml saves under. ARM64 runners always restore rust-ext-aarch64-cp312.'
required: false
default: rust-ext-x86_64-cp312
@@ -53,7 +53,7 @@ runs:
path: |
python/sglang/srt/rust_extensions/_*.so
python/sglang/srt/mem_cache/rust_tree_core/mem_cache*.so
key: ${{ inputs.cache_key_prefix }}-${{ hashFiles('rust/**', 'proto/**', 'python/setup.py', 'python/pyproject.toml', 'python/sglang/srt/rust_extensions/torch_build.py', '.github/workflows/_pr-test-rust-ext-build.yml', 'scripts/ci/utils/stage_rust_ext_modules.sh') }}
key: ${{ runner.arch == 'ARM64' && 'rust-ext-aarch64-cp312' || inputs.cache_key_prefix }}-${{ hashFiles('rust/**', 'proto/**', 'python/setup.py', 'python/pyproject.toml', 'python/sglang/srt/rust_extensions/torch_build.py', '.github/workflows/_pr-test-rust-ext-build.yml', 'scripts/ci/utils/stage_rust_ext_modules.sh') }}
# Job-wide, but only setup.py reads it, and only while building.
# Whether the modules suit this interpreter is not decided here:
+22
View File
@@ -135,6 +135,28 @@ hicache:
- 'python/sglang/srt/mem_cache/**/*storage*'
- 'python/sglang/srt/mem_cache/l2_transfer.py'
# A *mooncake* filename glob also matches benchmark/datasets/mooncake.py.
mooncake:
- changed-files:
- any-glob-to-any-file:
# PD KV transfer.
- 'python/sglang/srt/disaggregation/mooncake/**/*'
- 'python/sglang/srt/distributed/device_communicators/mooncake_transfer_engine.py'
- 'python/sglang/srt/observability/mooncake_trace.py'
- 'test/registered/unit/disaggregation/**/*mooncake*'
- 'test/manual/kv_transfer/**/*mooncake*'
# HiCache store.
- 'python/sglang/srt/mem_cache/storage/mooncake_store/**/*'
- 'python/sglang/test/mooncake_utils.py'
- 'test/registered/unit/mem_cache/**/*mooncake*'
- 'test/registered/hicache/**/*mooncake*'
# MoE EP. elastic_ep.py also implements nixl.
- 'python/sglang/srt/layers/moe/token_dispatcher/mooncake.py'
- 'python/sglang/srt/elastic_ep/expert_backup_client.py'
- 'python/sglang/srt/elastic_ep/expert_backup_manager.py'
- 'test/registered/ep/**/*mooncake*'
- 'test/manual/ep/**/*mooncake*'
# Deterministic
deterministic:
- changed-files:
-133
View File
@@ -1,133 +0,0 @@
# Reusable workflow: cross-reference CI test failures with coverage-based
# test recommendations. Called by pr-test-npu.yml's analyze-failure-report job.
name: Analyze Failure Report
on:
workflow_call:
inputs:
log_artifact_pattern:
type: string
required: true
description: 'Glob pattern for test log artifacts (e.g. selected-test-logs-*)'
diffusion_artifact_pattern:
type: string
required: false
default: 'diffusion-failures-*'
description: 'Glob pattern for diffusion failure artifacts (consistency failures)'
recommendations_content:
type: string
required: false
default: ''
description: 'Content of recommendations from upstream job output'
defaults:
run:
shell: bash -el {0}
jobs:
analyze:
name: cross-reference failures with recommendations
continue-on-error: true # Non-blocking on parse/read failure.
runs-on: ubuntu-latest
steps:
- name: Checkout sglang repo
uses: actions/checkout@v4
# ----- Step 1: Locate recommendations -----
- name: Locate recommendations file
id: locate-recs
run: |
mkdir -p ./test-logs
# Always create the file so the analyzer can run on diffusion-only failures.
touch ./test-logs/_recommended.txt
echo "RECS_FILE=./test-logs/_recommended.txt" >> $GITHUB_ENV
if [ -n "${{ inputs.recommendations_content }}" ]; then
echo "has_recommendations=true" >> $GITHUB_OUTPUT
echo "Using workflow output recommendations"
echo "${{ inputs.recommendations_content }}" > ./test-logs/_recommended.txt
echo "RECS_SOURCE=output" >> $GITHUB_ENV
COUNT=$(wc -l < ./test-logs/_recommended.txt)
echo "Recommendations count: ${COUNT}"
else
echo "has_recommendations=false" >> $GITHUB_OUTPUT
echo "RECS_SOURCE=none" >> $GITHUB_ENV
echo "::notice::No recommended test cases detected; analyzing log and diffusion failures only."
fi
# ----- Step 2: Download test logs -----
# continue-on-error is a step keyword, not a download-artifact input.
- name: Download test logs
uses: actions/download-artifact@v7
with:
pattern: ${{ inputs.log_artifact_pattern }}
path: ./test-logs
merge-multiple: true
continue-on-error: true
# ----- Step 2b: Download diffusion failure artifacts -----
# Keep merge-multiple off so each partition's consistency_failures/summary.json
# lands in its own <artifact-name>/ directory instead of clobbering the others.
- name: Download diffusion failure artifacts
uses: actions/download-artifact@v7
with:
pattern: ${{ inputs.diffusion_artifact_pattern }}
path: ./diffusion-failures
continue-on-error: true
- name: Check for diffusion failure artifacts
id: check-diffusion
run: |
if compgen -G "./diffusion-failures/*/consistency_failures/summary.json" > /dev/null; then
echo "found=true" >> $GITHUB_OUTPUT
echo "Diffusion consistency failure artifacts found"
else
echo "found=false" >> $GITHUB_OUTPUT
echo "No diffusion consistency failure artifacts"
fi
- name: Show downloaded artifacts structure
run: |
echo "Downloaded log files:"
find ./test-logs -type f | head -50 || echo " (no logs found)"
echo "Downloaded diffusion failure artifacts:"
if compgen -G "./diffusion-failures/*" > /dev/null; then
find ./diffusion-failures -type f | head -50
else
echo " (no diffusion failure artifacts found)"
fi
# ----- Step 3: Run analysis -----
- name: Ensure regex is available
if: ${{ steps.locate-recs.outputs.has_recommendations != 'false' || steps.check-diffusion.outputs.found == 'true' }}
run: |
set -euo pipefail
if ! python3 -c "import regex" >/dev/null 2>&1; then
echo "regex not found for $(python3 -V); bootstrapping pip and installing regex"
curl -fsSL https://bootstrap.pypa.io/get-pip.py -o get-pip.py
python3 get-pip.py --break-system-packages
python3 -m pip install --break-system-packages regex
fi
python3 -c "import regex; print('regex ok:', regex.__file__)"
- name: Run failure analysis
if: ${{ steps.locate-recs.outputs.has_recommendations != 'false' || steps.check-diffusion.outputs.found == 'true' }}
id: analysis
continue-on-error: true
run: |
python3 scripts/ci/npu/precise-test/analyze_failure_report.py \
--log-dir ./test-logs \
--diffusion-dir ./diffusion-failures \
--recommendations-file "${RECS_FILE}" \
--recommendations-source "${RECS_SOURCE:-none}" \
--output ./test-logs/failure_report.md
# ----- Step 4: Upload report -----
- name: Upload failure analysis report
if: always()
uses: actions/upload-artifact@v7
with:
name: failure-analysis-report
path: ./test-logs/failure_report.md
if-no-files-found: ignore
retention-days: 14
+17 -14
View File
@@ -50,10 +50,10 @@ on:
description: 'If true, delegate test execution to scripts/ci/npu/precise-test/run_tests_with_coverage.sh instead of running via run_suite.py directly'
type: boolean
default: false
upload_test_logs:
description: 'If true, upload the test logs as a GitHub Actions artifact (only enabled by pr-test-npu)'
type: boolean
default: false
only_tests:
description: 'Coverage-recommended test files for this suite (one repo-relative path per line, from recommend-tests-from-coverage). Empty = run the full suite.'
type: string
default: ''
github-token:
description: 'GitHub token for API calls'
type: string
@@ -70,6 +70,7 @@ env:
jobs:
run:
name: ${{ inputs.use_coverage_runner && inputs.self_name || 'run' }} (${{ matrix.partition }})
runs-on: ${{ inputs.runner_config }}
container:
image: ${{ inputs.image }}
@@ -241,6 +242,7 @@ jobs:
env:
CONTINUE_ON_ERROR_FLAG: ${{ inputs.continue_on_error == 'true' && '--continue-on-error' || '' }}
TEST_LOG_DIR: /tmp/test-logs
ONLY_TESTS: ${{ inputs.only_tests }}
shell: bash
run: |
# Fail fast on any command error, undefined variable, or pipe failure.
@@ -284,6 +286,16 @@ jobs:
NIGHTLY_FLAG="--nightly --continue-on-error"
fi
# Precision-test mode: run only the coverage-recommended files for
# this suite. The bucketed list comes from recommend-tests-from-
# coverage; empty input keeps the full-suite behavior.
ONLY_FILE_ARGS=""
if [ -n "${ONLY_TESTS}" ]; then
printf '%s\n' "${ONLY_TESTS}" > /tmp/precision_tests.txt
ONLY_FILE_ARGS="--only-file-list /tmp/precision_tests.txt"
echo "Precision mode: running $(wc -l < /tmp/precision_tests.txt) recommended file(s)"
fi
# Optional: delegate execution to scripts/ci/npu/precise-test/run_tests_with_coverage.sh when the
# caller sets use_coverage_runner=true. list_tests.py (same directory) enumerates the
# selected test files; the original run_suite.py execution below is preserved
@@ -308,13 +320,4 @@ jobs:
--auto-partition-id ${{ matrix.partition }} \
--auto-partition-size ${{ fromJson(inputs.partitions).size }} \
${{ inputs.timeout_per_file && format('--timeout-per-file {0}', inputs.timeout_per_file) || '' }} \
$NIGHTLY_FLAG $CONTINUE_ON_ERROR_FLAG 2>&1 | tee "${LOG_FILE}"
- name: Upload test logs
if: always() && inputs.use_coverage_runner != true && inputs.upload_test_logs == true
uses: actions/upload-artifact@v7
with:
name: selected-test-logs-${{ inputs.self_name }}-part${{ matrix.partition }}
path: /tmp/test-logs/
if-no-files-found: ignore
retention-days: 14
$ONLY_FILE_ARGS $NIGHTLY_FLAG $CONTINUE_ON_ERROR_FLAG 2>&1 | tee "${LOG_FILE}"
@@ -71,10 +71,10 @@ on:
description: 'If true, delegate test execution to scripts/ci/npu/precise-test/run_tests_with_coverage.sh instead of running via run_suite.py directly'
type: boolean
default: false
upload_test_logs:
description: 'If true, upload the test logs as a GitHub Actions artifact (only enabled by pr-test-npu)'
type: boolean
default: false
only_tests:
description: 'Coverage-recommended test files for this suite (one repo-relative path per line, from recommend-tests-from-coverage). Empty = run the full suite.'
type: string
default: ''
env:
SKIP_PR_TEST_HEALTH_CHECK: ${{ inputs.skip_pr_test_health_check }}
@@ -85,7 +85,7 @@ concurrency:
jobs:
e2e:
name: ${{ inputs.test_suite }}
name: ${{ inputs.test_suite }}${{ inputs.use_coverage_runner && format(' ({0})', matrix.partition) || '' }}
strategy:
fail-fast: false
matrix:
@@ -238,6 +238,7 @@ jobs:
SGLANG_IS_IN_CI: true
TRANSFORMERS_VERBOSITY: "error"
GDN_ATTN_BACKEND_TRITON: 1
ONLY_TESTS: ${{ inputs.only_tests }}
shell: bash
run: |
# Fail fast on any command error, undefined variable, or pipe failure.
@@ -401,6 +402,16 @@ jobs:
PARTITION_ARGS="--auto-partition-id ${PARTITION_ID} --auto-partition-size ${PARTITION_SIZE}"
fi
# Precision-test mode: run only the coverage-recommended files for
# this suite. The bucketed list comes from recommend-tests-from-
# coverage; empty input keeps the full-suite behavior.
ONLY_FILE_ARGS=""
if [ -n "${ONLY_TESTS}" ]; then
printf '%s\n' "${ONLY_TESTS}" > /tmp/precision_tests.txt
ONLY_FILE_ARGS="--only-file-list /tmp/precision_tests.txt"
echo "Precision mode: running $(wc -l < /tmp/precision_tests.txt) recommended file(s)"
fi
# Append `--timeout-from-est‑time` when the job belongs to the nightly pipeline.
# per‑case timeout is derived from each case's est_time.
TIMEOUT_FLAG=""
@@ -435,7 +446,7 @@ jobs:
# Run NPU test suite with run_suite.py.
# Capture mode (--enable-retry --max-attempts 1) keeps test subprocesses off the
# tee pipe, so leftover server processes cannot block the pipeline on exit.
${PYTHON_FOR_SGLANG} -u run_suite.py --hw npu --suite ${test_suite} ${NIGHTLY_FLAG} ${PARTITION_ARGS} ${TIMEOUT_FLAG} --enable-retry --max-attempts 1 --timeout-per-file 7200 2>&1 | tee "${LOG_TEE_TARGETS[@]}" || test_exit_code=$?
${PYTHON_FOR_SGLANG} -u run_suite.py --hw npu --suite ${test_suite} ${NIGHTLY_FLAG} ${PARTITION_ARGS} ${ONLY_FILE_ARGS} ${TIMEOUT_FLAG} --enable-retry --max-attempts 1 --timeout-per-file 7200 2>&1 | tee "${LOG_TEE_TARGETS[@]}" || test_exit_code=$?
echo "Finished test: ${tc_name}"
@@ -467,12 +478,3 @@ jobs:
echo "" >> $GITHUB_STEP_SUMMARY
fi
exit ${test_exit_code}
- name: Upload test logs
if: always() && inputs.use_coverage_runner != true && inputs.upload_test_logs == true
uses: actions/upload-artifact@v7
with:
name: selected-test-logs-${{ inputs.test_suite }}-part${{ matrix.partition }}
path: /tmp/test-logs/
if-no-files-found: ignore
retention-days: 14
+2 -1
View File
@@ -95,7 +95,7 @@ jobs:
- "python/setup.py"
- "python/sglang/!(multimodal_gen|kernels)/**/!(*.md)"
# Diffusion-only kernels and tests are left to multimodal_gen and
# jit_kernel; ops/diffusion/common stays since srt imports it via kda_kernels.
# jit_kernel; ops/diffusion/common stays since kda_kernels (an srt path) imports it.
- "python/sglang/kernels/{,ops/,jit/,jit/csrc/}!(*.md)"
- "python/sglang/kernels/!(aot|ops|jit)/**/!(*.md)"
- "python/sglang/kernels/jit/!(csrc)/**/!(*.md)"
@@ -130,6 +130,7 @@ jobs:
simulator:
- ".github/workflows/pr-test-extra.yml"
- ".github/workflows/_pr-test-simulator-cpu.yml"
- ".github/workflows/_pr-test-simulator-gpu.yml"
- "benchmark/simulator/**/!(*.md)"
- "tools/sglang-simulator/**/!(*.md)"
multimodal_gen:
@@ -50,7 +50,7 @@ jobs:
strategy:
matrix:
include:
- python-version: "3.10"
- python-version: "3.11"
cuda-version: "13.0"
name: ${{ inputs.job_display_name }}
steps:
+2 -1
View File
@@ -70,7 +70,7 @@ jobs:
run: |
uv pip install -e "python" --index-strategy unsafe-best-match --prerelease allow
uv pip install pytest
uv pip install -e "tools/sglang-simulator[aic]"
uv pip install "./tools/sglang-simulator[aic]"
- name: Prebuild HiCache native hash extension
timeout-minutes: 5
@@ -91,6 +91,7 @@ jobs:
PYTEST_ADDOPTS: "-s"
TORCH_EXTENSIONS_DIR: ${{ runner.temp }}/torch-extensions
run: |
OMP_NUM_THREADS=2 python3 -m pytest -q tools/sglang-simulator/test/test_train_latency_model.py
python3 -m pytest -q tools/sglang-simulator/test/test_simulation_sglang_runner.py
python3 -m pytest -q tools/sglang-simulator/test/test_simulation_sglang_serving.py
python3 -m pytest -q tools/sglang-simulator/test/test_simulation_cache_hit_ratio.py
@@ -0,0 +1,68 @@
name: PR Test SGLang Simulator (GPU)
on:
workflow_call:
inputs:
check_changes:
description: 'toJson(needs.check-changes.outputs).'
type: string
required: true
caller_inputs:
description: 'toJson(inputs) from the caller workflow.'
type: string
required: true
rust_ext_artifact:
description: 'Artifact of prebuilt Rust extension modules.'
type: string
default: ''
env:
SGLANG_IS_IN_CI: true
SKIP_PR_TEST_HEALTH_CHECK: ${{ (fromJson(inputs.caller_inputs).skip_pr_test_health_check || fromJson(inputs.caller_inputs).test_parallel_dispatch || fromJson(inputs.caller_inputs).run_all_tests) && 'true' || 'false' }}
PR_TEST_BYPASS_MAINTENANCE_ON_MAIN: ${{ github.ref == 'refs/heads/main' && 'true' || 'false' }}
USE_VENV: "1"
jobs:
run:
name: simulator-test-gpu
if: fromJson(inputs.check_changes).main_package == 'true' || fromJson(inputs.check_changes).simulator == 'true' || fromJson(inputs.check_changes).sgl_kernel == 'true'
runs-on: ${{ fromJson(fromJson(inputs.check_changes).runs_on_map)['1-gpu-small'] }}
timeout-minutes: 40
steps:
- uses: actions/checkout@v4
with:
ref: ${{ fromJson(inputs.caller_inputs).git_ref || github.sha }}
- uses: ./.github/actions/check-pr-test-health
- uses: ./.github/actions/check-maintenance
- name: Download kernel wheels
if: fromJson(inputs.check_changes).sgl_kernel == 'true'
uses: actions/download-artifact@v4
with:
path: python/sglang/kernels/aot/dist/
merge-multiple: true
pattern: wheel-python3.11-cuda*
- uses: ./.github/actions/download-rust-ext
with:
artifact_name: ${{ inputs.rust_ext_artifact }}
- name: Install dependencies
timeout-minutes: 20
env:
CUSTOM_BUILD_SGL_KERNEL: ${{ fromJson(inputs.check_changes).sgl_kernel }}
run: bash scripts/ci/cuda/ci_install_dependency.sh
- name: Test baseline collection
timeout-minutes: 10
env:
PYTEST_DISABLE_PLUGIN_AUTOLOAD: "1"
run: |
uv pip install "./tools/sglang-simulator"
python3 -m pytest -q tools/sglang-simulator/test/test_baseline_collection_sglang.py
- name: Cleanup venv
if: always()
run: bash scripts/ci/cuda/ci_cleanup_venv.sh
+6 -3
View File
@@ -17,7 +17,7 @@ on:
type: string
required: true
caller_inputs:
description: 'toJson(inputs) from pr-test.yml. Read via fromJson(...).git_ref / skip_pr_test_health_check / test_parallel_dispatch / run_all_tests.'
description: 'toJson(inputs) from pr-test.yml. Read via fromJson(...).git_ref / skip_pr_test_health_check / test_parallel_dispatch / run_all_tests / skip_cache_save.'
type: string
required: true
partitions:
@@ -110,7 +110,7 @@ jobs:
with:
workspaces: rust
shared-key: "sglang-grpc-cpu"
save-if: ${{ matrix.partition == 0 }}
save-if: ${{ matrix.partition == 0 && !fromJson(inputs.caller_inputs).skip_cache_save }}
# uv pip targets a venv by default; setup-python has no venv — install into that interpreter (see UV_SYSTEM_PYTHON in https://docs.astral.sh/uv/guides/integration/github/)
- name: Install dependencies
@@ -156,9 +156,12 @@ jobs:
python3 run_suite.py --hw cpu --suite ${{ inputs.self_name }} --auto-partition-id ${{ matrix.partition }} --auto-partition-size ${{ fromJson(inputs.partitions)[inputs.self_name].size }} --partition-model-file /tmp/partition-model.json --fork-worker-batch-size 20 $CONTINUE_ON_ERROR_FLAG
# PRs save only what the base lacks; the implicit if: success() keeps a
# half-downloaded cache unpublished.
# half-downloaded cache unpublished. skip_cache_save is for callers that
# dispatch on main but test PR code (run-cpu-ci.yml): a save there would
# publish PR-written files under main's cache scope.
- name: Decide whether to save the HF cache
id: hf_cache_save
if: ${{ !fromJson(inputs.caller_inputs).skip_cache_save }}
run: |
size=$(du -sb "$HF_HOME" 2>/dev/null | cut -f1)
after=${size:-0}
+3 -3
View File
@@ -147,7 +147,7 @@ jobs:
with:
path: python/sglang/kernels/aot/dist/
merge-multiple: true
pattern: wheel-python3.10-cuda*
pattern: wheel-python3.11-cuda*
- name: Download artifacts (v6)
if: ${{ fromJson(inputs.check_changes).sgl_kernel == 'true' && steps.rc.outputs.artifact_version == 'v6' }}
@@ -155,7 +155,7 @@ jobs:
with:
path: python/sglang/kernels/aot/dist/
merge-multiple: true
pattern: wheel-python3.10-cuda*
pattern: wheel-python3.11-cuda*
- uses: ./.github/actions/download-rust-ext
if: ${{ !inputs.skip_prebuilt_rust_ext }}
@@ -216,7 +216,7 @@ jobs:
CONTINUE_ON_ERROR_FLAG: ${{ fromJson(inputs.check_changes).continue_on_error == 'true' && '--continue-on-error' || '' }}
RUNNER_LABELS: ${{ fromJson(fromJson(inputs.check_changes).runs_on_map)[inputs.runner_config] }}
GPU_CONFIG: ${{ inputs.runner_config }}
HF_MUSE_SPARK_TOKEN: ${{ inputs.self_name == 'base-c-test-8-gpu-b300' && secrets.HF_MUSE_SPARK_TOKEN || '' }}
HF_MUSE_SPARK_TOKEN: ${{ (inputs.self_name == 'base-c-test-8-gpu-b300' || inputs.self_name == 'extra-b-test-8-gpu-b300') && secrets.HF_MUSE_SPARK_TOKEN || '' }}
# Read only by test_runai_model_loader.py (nightly 1-gpu); inert elsewhere.
# Left unconditional because an `env:` key cannot be omitted, and an empty
# value would reach the runai streamer as a malformed limit.
+1 -5
View File
@@ -58,11 +58,7 @@ jobs:
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.10'
- name: Install Python dependencies
run: |
pip install tomli
python-version: '3.12'
- name: Configure Git and branch
run: |
@@ -24,11 +24,7 @@ jobs:
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.10'
- name: Install Python dependencies
run: |
pip install tomli
python-version: '3.12'
- name: Configure Git and branch
run: |
@@ -22,11 +22,7 @@ jobs:
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.10'
- name: Install Python dependencies
run: |
pip install tomli
python-version: '3.12'
- name: Check if sync is needed
id: check_sync
@@ -24,11 +24,7 @@ jobs:
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.10'
- name: Install Python dependencies
run: |
pip install tomli
python-version: '3.12'
- name: Configure Git and branch
run: |
@@ -26,11 +26,7 @@ jobs:
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.10'
- name: Install Python dependencies
run: |
pip install tomli
python-version: '3.12'
- name: Configure Git and branch
id: set_output
+5 -5
View File
@@ -25,7 +25,7 @@ jobs:
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.10'
python-version: '3.12'
- name: Generate Summary Report
run: |
@@ -41,7 +41,7 @@ jobs:
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.10'
python-version: '3.12'
- name: Generate Tests by Folder Report
run: |
@@ -57,7 +57,7 @@ jobs:
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.10'
python-version: '3.12'
- name: Generate Tests by Suite Report
run: |
@@ -73,7 +73,7 @@ jobs:
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.10'
python-version: '3.12'
- name: Generate Tests by GPU Count Report
run: |
@@ -184,7 +184,7 @@ jobs:
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.10'
python-version: '3.12'
- name: Generate JSON Report
run: |
+13 -11
View File
@@ -1,15 +1,17 @@
# Coverage data collection pipeline for the NPU precision test selection.
# Coverage baseline collection for the NPU precision test selection.
# Delegates to pr-test-npu.yml with coverage_mode=true, which switches:
# - Runners to the coverage pool (shared /root/.cache with setup-covstub)
# - use_coverage_runner=true on all test jobs
# - Skips pr-gate, multimodal-gen, recommend, analyze
# - Runs setup-covstub to assemble coverage data and publish it to the
# shared disk at a fixed path (/root/.cache/tests/precise-test/coverage)
# pr-test-npu-finish is the final gate in the called workflow.
#
# Manually triggered only (workflow_dispatch): PRs run the standard
# pr-test-npu.yml instead; this pipeline builds the full coverage baseline
# on demand.
# - Test jobs to the coverage pool with use_coverage_runner=true
# - base-a to a hosted runner (base-a-test-npu-cov, data merged via artifact)
# - Runs the multimodal-gen suites too. Their runner is asked for this
# shard's targets (case-level pytest node ids) and each target is run in
# its own coverage invocation, so diffusion coverage is attributed per
# test case instead of per shard.
# - Skips pr-gate, recommend
# - setup-covstub assembles and publishes the baseline to the shared disk
# at /root/.cache/tests/precise-test/coverage
# pr-test-npu-finish is the final gate in the called workflow. Manually
# triggered only (workflow_dispatch): PRs run the standard pr-test-npu.yml
# instead; this pipeline builds the full coverage baseline on demand.
name: Coverage Collection (NPU)
on:
+7 -7
View File
@@ -24,7 +24,7 @@ on:
default: ''
type: string
kernel_artifact_run_id:
description: "Optional sgl-kernel wheel source: GitHub Actions run ID of a pr-test.yml run on the SAME commit as 'ref' that uploaded wheel-python3.10-cuda13.0. Use only when this branch's torch/ABI change makes the PyPI sglang-kernel wheel incompatible. Find it under Actions > pr-test; artifacts expire after 90 days."
description: "Optional sgl-kernel wheel source: GitHub Actions run ID of a pr-test.yml run on the SAME commit as 'ref' that uploaded wheel-python3.11-cuda13.0. Use only when this branch's torch/ABI change makes the PyPI sglang-kernel wheel incompatible. Find it under Actions > pr-test; artifacts expire after 90 days."
required: false
default: ''
type: string
@@ -204,7 +204,7 @@ jobs:
with:
path: python/sglang/kernels/aot/dist/
merge-multiple: true
name: wheel-python3.10-cuda13.0
name: wheel-python3.11-cuda13.0
run-id: ${{ inputs.kernel_artifact_run_id }}
github-token: ${{ secrets.GITHUB_TOKEN }}
@@ -351,7 +351,7 @@ jobs:
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.10'
python-version: '3.12'
- name: Verify write access to sgl-project/ci-data-diffusion
if: ${{ inputs.publish }}
@@ -403,7 +403,7 @@ jobs:
with:
path: python/sglang/kernels/aot/dist/
merge-multiple: true
name: wheel-python3.10-cuda13.0
name: wheel-python3.11-cuda13.0
run-id: ${{ inputs.kernel_artifact_run_id }}
github-token: ${{ secrets.GITHUB_TOKEN }}
@@ -479,7 +479,7 @@ jobs:
with:
path: python/sglang/kernels/aot/dist/
merge-multiple: true
name: wheel-python3.10-cuda13.0
name: wheel-python3.11-cuda13.0
run-id: ${{ inputs.kernel_artifact_run_id }}
github-token: ${{ secrets.GITHUB_TOKEN }}
@@ -557,7 +557,7 @@ jobs:
with:
path: python/sglang/kernels/aot/dist/
merge-multiple: true
name: wheel-python3.10-cuda13.0
name: wheel-python3.11-cuda13.0
run-id: ${{ inputs.kernel_artifact_run_id }}
github-token: ${{ secrets.GITHUB_TOKEN }}
@@ -630,7 +630,7 @@ jobs:
with:
path: python/sglang/kernels/aot/dist/
merge-multiple: true
name: wheel-python3.10-cuda13.0
name: wheel-python3.11-cuda13.0
run-id: ${{ inputs.kernel_artifact_run_id }}
github-token: ${{ secrets.GITHUB_TOKEN }}
@@ -1,4 +1,4 @@
name: Nightly Experimental sgl-router Docker Image
name: Nightly sgl-router Docker Image
on:
schedule:
@@ -34,15 +34,18 @@ jobs:
- suite: nightly-stage-c-2-gpu-mi350
runner: linux-mi300-2gpu-sglang
timeout: 180
# Single-request decode of Qwen3.8-27B-FP8 runs at ~20 tok/s on MI300X, so long
# generations outlast the test client's 180 s request timeout.
skip_tests: tests/e2e/sglang/test_session_server_multi_role/test_qwen38small.py
# test_qwen38small: single-request decode of Qwen3.8-27B-FP8 runs at ~20 tok/s on
# MI300X, so long generations outlast the test client's 180 s request timeout.
# test_glm5_2_744b_a40b_5layer_ci: its tilelang indexer kernel needs 98 KiB of LDS
# per block; gfx942 has 64 KiB.
skip_tests: >-
tests/e2e/sglang/test_session_server_multi_role/test_qwen38small.py
tests/e2e/megatron/model_scripts/test_glm5_2_744b_a40b_5layer_ci.py
# The 4- and 8-GPU suites run past 180 min on MI300X; they take ~155 and ~112 min on MI355X.
- suite: nightly-stage-c-4-gpu-mi350
runner: linux-mi300-4gpu-sglang
timeout: 300
# Its tilelang indexer kernel needs 98 KiB of LDS per block; gfx942 has 64 KiB.
skip_tests: tests/e2e/megatron/model_scripts/test_glm5_2_744b_a40b_5layer_ci.py
skip_tests: ""
- suite: nightly-stage-c-8-gpu-mi350
runner: linux-mi300-8gpu-sglang
timeout: 300
@@ -24,15 +24,28 @@ concurrency:
jobs:
test-miles-mi35x:
if: github.repository == 'sgl-project/sglang'
name: test-miles-mi35x (${{ matrix.suite }}, ${{ matrix.runner }}${{ matrix.partition_size && format(', part {0}', matrix.part) || '' }})
strategy:
fail-fast: false
matrix:
# All suites use the 8-GPU MI35X pool. The 2-GPU suite includes the
# GLM-5.2 5-layer test, which takes the linux-mi35x-gpu-2 runner down
# ("runner lost communication") but passes on an 8-GPU runner.
include:
- suite: nightly-stage-c-2-gpu-mi350
runner: linux-mi35x-gpu-2
# Use 8-GPU MI35X pool for 4-GPU suite.
runner: linux-mi35x-gpu-8
# The 4-GPU suite is ~6 h of estimated test time; split it so each
# half fits in the step timeout.
- suite: nightly-stage-c-4-gpu-mi350
runner: linux-mi35x-gpu-8
partition_id: 0
partition_size: 2
part: 1/2
- suite: nightly-stage-c-4-gpu-mi350
runner: linux-mi35x-gpu-8
partition_id: 1
partition_size: 2
part: 2/2
- suite: nightly-stage-c-8-gpu-mi350
runner: linux-mi35x-gpu-8
runs-on: ${{ matrix.runner }}
@@ -88,6 +101,7 @@ jobs:
-e PYTHONPATH=/root/miles:/opt/tilelang \
-e GITHUB_STEP_SUMMARY=/sglang-checkout/github_summary.md \
python3 tests/ci/run_suite.py --hw rocm --suite ${{ matrix.suite }} --cadence nightly \
${{ matrix.partition_size && format('--auto-partition-id {0} --auto-partition-size {1}', matrix.partition_id, matrix.partition_size) || '' }} \
${{ (github.event_name != 'workflow_dispatch' || inputs.continue_on_error) && '--continue-on-error' || '' }} || TEST_EXIT_CODE=$?
cat "${GITHUB_WORKSPACE}/github_summary.md" >> "$GITHUB_STEP_SUMMARY" || true
exit ${TEST_EXIT_CODE}
+10
View File
@@ -1000,6 +1000,11 @@ jobs:
# HF download env vars:
# - HF_HUB_ENABLE_HF_TRANSFER=1: Use faster hf_transfer for downloads (if available)
# - HF_HUB_DISABLE_SYMLINKS_WARNING=1: Suppress symlink warnings
#
# Skips ltx_2_5_diffusion_decoder_2gpus: on mi300 its server has never
# finished warmup inside the 1200s startup deadline, and the harness
# retries that TimeoutError six times, running the shard into the
# step timeout.
docker exec \
-e HF_TOKEN \
-e HUGGING_FACE_HUB_TOKEN \
@@ -1019,6 +1024,7 @@ jobs:
--suite 2-gpu \
--partition-id ${{ matrix.part }} \
--total-partitions 2 \
-k "not ltx_2_5_diffusion_decoder" \
${{ needs.check-changes.outputs.continue_on_error == 'true' && '--continue-on-error' || '' }}
# Post-test diagnostics
@@ -1456,9 +1462,13 @@ jobs:
run: |
docker exec ci_sglang rocm-smi --showtopotype || true
docker exec ci_sglang bash -c 'strings $(python3 -c "import mooncake,os;print(os.path.dirname(mooncake.__file__))")/engine*.so | grep -iE "rdma,hip|Unsupported multi-protocol"' || true
# With Mooncake v0.3.14-rc1, registering the KV cache through HIP dmabuf
# leaves the server's next HIP stream creation failing (out of memory on
# rocm10, segfault on rocm720). Register through ibv_reg_mr instead.
bash scripts/ci/amd/amd_ci_exec.sh \
-e SGLANG_TEST_RDMA_DEVICE="${{ env.SGLANG_TEST_RDMA_DEVICE }}" \
-e MC_LOG_LEVEL=TRACE \
-e MOONCAKE_DISABLE_HIP_DMABUF=1 \
-e GLOG_logtostderr=1 \
-w "/sglang-checkout/test" python3 run_suite.py --hw amd --suite stage-b-test-large-8-gpu-mi35x-disaggregation-amd --timeout-per-file 1800 ${{ needs.check-changes.outputs.continue_on_error == 'true' && '--continue-on-error' || '' }}
+10
View File
@@ -150,6 +150,15 @@ jobs:
rust_ext_artifact: ${{ needs.rust-ext-build.outputs.artifact_name }}
# No `secrets: inherit`: this hosted CPU job has no secret consumer.
simulator-test-gpu:
needs: [check-changes, call-gate, sgl-kernel-build-wheels, rust-ext-build]
if: ${{ !failure() && !cancelled() && needs.check-changes.result == 'success' && (needs.call-gate.result == 'success' || needs.call-gate.result == 'skipped') }}
uses: ./.github/workflows/_pr-test-simulator-gpu.yml
with:
check_changes: ${{ toJson(needs.check-changes.outputs) }}
caller_inputs: ${{ toJson(inputs) }}
rust_ext_artifact: ${{ needs.rust-ext-build.outputs.artifact_name }}
# =============================================== extra-a / extra-b ===============================================
# `name:` keeps the job names wait-for-jobs and check-pr-test-health match on.
extra-test:
@@ -191,6 +200,7 @@ jobs:
sgl-kernel-build-wheels,
rust-ext-build,
simulator-test-cpu,
simulator-test-gpu,
extra-test,
]
if: always()
+1 -1
View File
@@ -99,7 +99,7 @@ jobs:
with:
path: python/sglang/kernels/aot/dist/
merge-multiple: true
pattern: wheel-python3.10-cuda13.0
pattern: wheel-python3.11-cuda13.0
- uses: ./.github/actions/download-rust-ext
with:
+38 -9
View File
@@ -65,7 +65,7 @@ jobs:
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.10'
python-version: '3.12'
- name: Compute partitions
id: compute
@@ -101,7 +101,7 @@ jobs:
with:
path: python/sglang/kernels/aot/dist/
merge-multiple: true
pattern: wheel-python3.10-cuda*
pattern: wheel-python3.11-cuda*
- name: Install dependencies
timeout-minutes: 20
@@ -168,13 +168,19 @@ jobs:
with:
path: python/sglang/kernels/aot/dist/
merge-multiple: true
pattern: wheel-python3.10-cuda*
pattern: wheel-python3.11-cuda*
- name: Install dependencies
timeout-minutes: 20
run: |
CUSTOM_BUILD_SGL_KERNEL=${{inputs.sgl_kernel}} bash scripts/ci/cuda/ci_install_dependency.sh diffusion
- name: Run SM120-only diffusion unit tests
timeout-minutes: 10
working-directory: python
run: |
python3 -m pytest -v sglang/multimodal_gen/test/unit/test_fp8_fa_sm120_attn.py
- name: Run 5090 diffusion canary tests
timeout-minutes: 120
env:
@@ -233,7 +239,7 @@ jobs:
with:
path: python/sglang/kernels/aot/dist/
merge-multiple: true
pattern: wheel-python3.10-cuda*
pattern: wheel-python3.11-cuda*
- name: Install dependencies
timeout-minutes: 20
@@ -298,7 +304,7 @@ jobs:
with:
path: python/sglang/kernels/aot/dist/
merge-multiple: true
pattern: wheel-python3.10-cuda*
pattern: wheel-python3.11-cuda*
- name: Install dependencies
timeout-minutes: 20
@@ -368,7 +374,7 @@ jobs:
with:
path: python/sglang/kernels/aot/dist/
merge-multiple: true
pattern: wheel-python3.10-cuda*
pattern: wheel-python3.11-cuda*
- name: Install dependencies
timeout-minutes: 20
@@ -413,7 +419,7 @@ jobs:
with:
path: python/sglang/kernels/aot/dist/
merge-multiple: true
pattern: wheel-python3.10-cuda*
pattern: wheel-python3.11-cuda*
- name: Install dependencies
timeout-minutes: 20
@@ -427,6 +433,22 @@ jobs:
run: |
python3 -m pytest -v python/sglang/multimodal_gen/test/single_test_file/test_wan_vae_parallel_groups_4_gpu.py
- name: Run Blackwell-only diffusion unit tests
timeout-minutes: 10
working-directory: python
run: |
python3 -m pytest -v \
sglang/multimodal_gen/test/unit/test_mxfp8_online_gemm.py \
sglang/multimodal_gen/test/unit/test_wan_gelu_mlp.py
- name: Run 4-GPU diffusion unit tests
timeout-minutes: 15
working-directory: python
run: |
python3 -m pytest -v \
sglang/multimodal_gen/test/unit/test_kandinsky6_sequence_parallel.py::test_joint_video_audio_tensor_sequence_parallel \
sglang/multimodal_gen/test/unit/test_kandinsky6_sr_sequence_parallel.py::test_sr_four_gpus
- name: Run diffusion server tests
timeout-minutes: 240
env:
@@ -483,13 +505,20 @@ jobs:
with:
path: python/sglang/kernels/aot/dist/
merge-multiple: true
pattern: wheel-python3.10-cuda*
pattern: wheel-python3.11-cuda*
- name: Install dependencies
timeout-minutes: 20
run: |
CUSTOM_BUILD_SGL_KERNEL=${{inputs.sgl_kernel}} bash scripts/ci/cuda/ci_install_dependency.sh diffusion
- name: Install Quant-VideoGen
timeout-minutes: 5
run: |
# quant-videogen pins an incompatible torch, so install it without deps
# plus the import-time deps of the diffusion-qvg extra.
uv pip install --no-deps quant-videogen==0.1.0 "loguru>=0.7" "termcolor>=2.3"
- name: Run diffusion unit tests
timeout-minutes: 60
run: |
@@ -519,7 +548,7 @@ jobs:
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.10'
python-version: '3.12'
- name: Download all execution reports
uses: actions/download-artifact@v4
+596 -108
View File
@@ -1,11 +1,13 @@
name: PR Test (NPU)
on:
push:
branches: [ main ]
schedule:
- cron: '0 12 * * *' # Run daily at 12:00 UTC
pull_request:
branches:
- main
- 'npu/**'
- 'npu-*'
workflow_dispatch:
inputs:
coverage_mode:
@@ -169,9 +171,84 @@ jobs:
cd test
python3 run_suite.py --hw npu --suite base-a-test-npu
# Coverage-mode-only counterpart of the base-a PR gate above. Stays on
# ubuntu-latest like the gate (its UTs mock the NPU stack and need a CPU
# triton; the NPU image's triton-ascend breaks those mocks), so the shared
# template (unconditional CANN container:) cannot be used and coverage
# data reaches setup-covstub via artifact instead of the shared disk.
base-a-test-npu-cov:
needs: [set-image-config]
if: ${{ inputs.coverage_mode == true }}
runs-on: ubuntu-latest
timeout-minutes: 45
env:
SGLANG_IS_IN_CI: true
steps:
- name: Checkout code
uses: actions/checkout@v4
with:
# Pin to the event SHA so this job tests the same commit as the rest of the run.
ref: ${{ inputs.ref || github.sha }}
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.12'
- name: Install uv
uses: astral-sh/setup-uv@v5
with:
enable-cache: true
cache-dependency-glob: 'python/pyproject_npu.toml'
- name: Install dependencies
timeout-minutes: 30
env:
# uv pip targets a venv by default; setup-python has no venv.
UV_SYSTEM_PYTHON: "1"
run: |
# Same dependency set as the base-a PR gate above (CPU torch + CPU
# triton, no Ascend wheels); keep the two blocks in sync.
rm -f python/pyproject.toml
mv python/pyproject_npu.toml python/pyproject.toml
uv pip install torch==2.10.0 torchvision==0.25.0 torchaudio==2.10.0 triton \
--index-url https://download.pytorch.org/whl/cpu \
--extra-index-url https://pypi.org/simple/ \
--index-strategy unsafe-best-match
uv pip install -e "python[dev_npu]" --index-strategy unsafe-best-match
pip install coverage==7.8.*
- name: Run test with coverage
timeout-minutes: 15
run: |
# Hosted runners cannot write the shared disk, so stage locally;
# run_tests_with_coverage.sh derives RUN_DIR/date tag from run-level
# env vars, so the artifact merges 1:1 into the shared-disk layout.
export PRECISE_TEST_CACHE_ROOT=/tmp/precise-test
cd test
python3 ../scripts/ci/npu/precise-test/list_tests.py --hw npu --suite base-a-test-npu \
-o /tmp/selected_tests.txt
echo "Selected test files:"
cat /tmp/selected_tests.txt
if [ -s /tmp/selected_tests.txt ]; then
bash ../scripts/ci/npu/precise-test/run_tests_with_coverage.sh $(cat /tmp/selected_tests.txt)
fi
- name: Upload coverage outputs
if: always()
uses: actions/upload-artifact@v4
with:
name: coverage-outputs-base-a-test-npu
path: /tmp/precise-test/
retention-days: 1
if-no-files-found: warn
base-b-test-1-npu-a3:
needs: [check-changes, pr-gate, set-image-config, base-a-test-npu]
if: ${{ !failure() && !cancelled() && (inputs.coverage_mode == true || needs.check-changes.outputs.main_package == 'true') }}
needs: [check-changes, pr-gate, set-image-config, base-a-test-npu, base-a-test-npu-cov, recommend-tests-from-coverage]
if: >-
${{ !failure() && !cancelled() &&
(inputs.coverage_mode == true || needs.check-changes.outputs.main_package == 'true') &&
needs.recommend-tests-from-coverage.outputs.should_run_base_b_test_1_npu_a3 != 'false' }}
uses: ./.github/workflows/_npu-pr-test-stage.yml
with:
self_name: base-b-test-1-npu-a3
@@ -179,13 +256,16 @@ jobs:
image: ${{ needs.set-image-config.outputs.CANN_image_a3 }}
run_timeout_minutes: '60'
timeout_per_file: '3600'
only_tests: ${{ needs.recommend-tests-from-coverage.outputs.only_tests_base_b_test_1_npu_a3 }}
use_coverage_runner: ${{ inputs.coverage_mode == true }}
upload_test_logs: ${{ inputs.coverage_mode != true }}
secrets: inherit
base-b-test-2-npu-a3:
needs: [check-changes, pr-gate, set-image-config, base-a-test-npu]
if: ${{ !failure() && !cancelled() && (inputs.coverage_mode == true || needs.check-changes.outputs.main_package == 'true') }}
needs: [check-changes, pr-gate, set-image-config, base-a-test-npu, base-a-test-npu-cov, recommend-tests-from-coverage]
if: >-
${{ !failure() && !cancelled() &&
(inputs.coverage_mode == true || needs.check-changes.outputs.main_package == 'true') &&
needs.recommend-tests-from-coverage.outputs.should_run_base_b_test_2_npu_a3 != 'false' }}
uses: ./.github/workflows/_npu-pr-test-stage.yml
with:
self_name: base-b-test-2-npu-a3
@@ -193,13 +273,16 @@ jobs:
image: ${{ needs.set-image-config.outputs.CANN_image_a3 }}
run_timeout_minutes: '60'
timeout_per_file: '3600'
only_tests: ${{ needs.recommend-tests-from-coverage.outputs.only_tests_base_b_test_2_npu_a3 }}
use_coverage_runner: ${{ inputs.coverage_mode == true }}
upload_test_logs: ${{ inputs.coverage_mode != true }}
secrets: inherit
base-b-test-4-npu-a3:
needs: [check-changes, pr-gate, set-image-config, base-a-test-npu]
if: ${{ !failure() && !cancelled() && (inputs.coverage_mode == true || needs.check-changes.outputs.main_package == 'true') }}
needs: [check-changes, pr-gate, set-image-config, base-a-test-npu, base-a-test-npu-cov, recommend-tests-from-coverage]
if: >-
${{ !failure() && !cancelled() &&
(inputs.coverage_mode == true || needs.check-changes.outputs.main_package == 'true') &&
needs.recommend-tests-from-coverage.outputs.should_run_base_b_test_4_npu_a3 != 'false' }}
uses: ./.github/workflows/_npu-pr-test-stage.yml
with:
self_name: base-b-test-4-npu-a3
@@ -208,13 +291,16 @@ jobs:
run_timeout_minutes: '120'
timeout_per_file: '3600'
partitions: '{"size":2,"arr":[0, 1]}'
only_tests: ${{ needs.recommend-tests-from-coverage.outputs.only_tests_base_b_test_4_npu_a3 }}
use_coverage_runner: ${{ inputs.coverage_mode == true }}
upload_test_logs: ${{ inputs.coverage_mode != true }}
secrets: inherit
base-b-test-8-npu-a3:
needs: [check-changes, pr-gate, set-image-config, base-a-test-npu]
if: ${{ !failure() && !cancelled() && (inputs.coverage_mode == true || needs.check-changes.outputs.main_package == 'true') }}
needs: [check-changes, pr-gate, set-image-config, base-a-test-npu, base-a-test-npu-cov, recommend-tests-from-coverage]
if: >-
${{ !failure() && !cancelled() &&
(inputs.coverage_mode == true || needs.check-changes.outputs.main_package == 'true') &&
needs.recommend-tests-from-coverage.outputs.should_run_base_b_test_8_npu_a3 != 'false' }}
uses: ./.github/workflows/_npu-pr-test-stage.yml
with:
self_name: base-b-test-8-npu-a3
@@ -222,13 +308,16 @@ jobs:
image: ${{ needs.set-image-config.outputs.CANN_image_a3 }}
run_timeout_minutes: '60'
timeout_per_file: '3600'
only_tests: ${{ needs.recommend-tests-from-coverage.outputs.only_tests_base_b_test_8_npu_a3 }}
use_coverage_runner: ${{ inputs.coverage_mode == true }}
upload_test_logs: ${{ inputs.coverage_mode != true }}
secrets: inherit
base-b-test-16-npu-a3:
needs: [check-changes, pr-gate, set-image-config, base-a-test-npu]
if: ${{ !failure() && !cancelled() && (inputs.coverage_mode == true || needs.check-changes.outputs.main_package == 'true') }}
needs: [check-changes, pr-gate, set-image-config, base-a-test-npu, base-a-test-npu-cov, recommend-tests-from-coverage]
if: >-
${{ !failure() && !cancelled() &&
(inputs.coverage_mode == true || needs.check-changes.outputs.main_package == 'true') &&
needs.recommend-tests-from-coverage.outputs.should_run_base_b_test_16_npu_a3 != 'false' }}
uses: ./.github/workflows/_npu-pr-test-stage.yml
with:
self_name: base-b-test-16-npu-a3
@@ -236,13 +325,24 @@ jobs:
image: ${{ needs.set-image-config.outputs.CANN_image_a3 }}
run_timeout_minutes: '120'
timeout_per_file: '3600'
only_tests: ${{ needs.recommend-tests-from-coverage.outputs.only_tests_base_b_test_16_npu_a3 }}
use_coverage_runner: ${{ inputs.coverage_mode == true }}
upload_test_logs: ${{ inputs.coverage_mode != true }}
secrets: inherit
multimodal-gen-test-1-npu-a3:
needs: [check-changes, pr-gate, set-image-config, base-a-test-npu]
if: ${{ !failure() && !cancelled() && inputs.coverage_mode != true && needs.check-changes.outputs.multimodal_gen == 'true' }}
needs:
[
check-changes,
pr-gate,
set-image-config,
base-a-test-npu,
base-a-test-npu-cov,
recommend-tests-from-coverage,
]
if: >-
${{ !failure() && !cancelled() &&
(inputs.coverage_mode == true || needs.check-changes.outputs.multimodal_gen == 'true') &&
needs.recommend-tests-from-coverage.outputs.should_run_multimodal_gen_test_1_npu_a3 != 'false' }}
runs-on: linux-aarch64-a3-800t-2
strategy:
fail-fast: false
@@ -290,7 +390,8 @@ jobs:
run: python3 scripts/ci/utils/diffusion/prefetch_input_image.py
- name: Run test
timeout-minutes: 60
timeout-minutes: ${{ inputs.coverage_mode == true && 150 || 60 }}
shell: bash
env:
SGLANG_USE_MODELSCOPE: true
SGLANG_IS_IN_CI: true
@@ -299,12 +400,75 @@ jobs:
PYTORCH_NPU_ALLOC_CONF: "expandable_segments:True"
STREAMS_PER_DEVICE: 32
SGLANG_DIFFUSION_ARTIFACT_DIR: ${{ github.workspace }}/diffusion-failures
ONLY_TESTS: ${{ needs.recommend-tests-from-coverage.outputs.only_tests_multimodal_gen_test_1_npu_a3 }}
run: |
# Coverage mode: enumerate this shard's targets with an NPU-only CI
# helper (it reuses the shared diffusion runner read-only) and run
# each target in its own `coverage run -m pytest` invocation. The
# runner normally batches a whole shard into one pytest call, which
# cannot be attributed per test case by the precise-test selector.
if [[ "${{ inputs.coverage_mode }}" == "true" ]]; then
set -euo pipefail
pip install coverage==7.8.*
# Coverage tracing slows down the Python glue, which shows up in
# wall-clock assertions. Relax only the timing tolerances for the
# collection run; PR runs (no coverage) keep the strict profile and
# the memory/VRAM checks are untouched.
export SGLANG_E2E_TOLERANCE=1.0
export SGLANG_STAGE_TIME_TOLERANCE=1.0
export SGLANG_NON_DENOISE_STAGE_TIME_TOLERANCE=2.0
export SGLANG_DENOISE_STEP_TOLERANCE=1.0
export SGLANG_DENOISE_AGG_TOLERANCE=1.0
python3 scripts/ci/npu/precise-test/list_multimodal_tests.py \
--suite 1-npu \
--partition-id ${{ matrix.part }} \
--total-partitions 2 \
-o /tmp/selected_tests.txt
echo "Selected test targets:"
cat /tmp/selected_tests.txt
if [ -s /tmp/selected_tests.txt ]; then
bash scripts/ci/npu/precise-test/run_tests_with_coverage.sh $(cat /tmp/selected_tests.txt)
fi
exit 0
fi
# Precision mode: ONLY_TESTS holds one pytest -k token per line
# ('[case_id]' for parametrized cases, a file stem for standalone
# files). It is empty when nothing was recommended, precision mode
# is off, or a safety gate forced the full suite; then no filter is
# applied and the shard runs its whole assignment.
FILTER=""
while IFS= read -r line; do
if [ -z "${line}" ]; then
continue
fi
# Every line must be a -k token ('[case_id]' or a standalone file
# stem). A path means the bucket contract drifted: with a path the
# -k expression matches nothing, pytest exits 5, and the shard
# would report success after running zero tests.
if [[ "${line}" == */* || "${line}" == *.py ]]; then
echo "::error::Unexpected only_tests entry '${line}' (expected a pytest -k token)"
exit 1
fi
if [ -n "${FILTER}" ]; then
FILTER="${FILTER} or "
fi
FILTER="${FILTER}${line}"
done <<< "${ONLY_TESTS}"
FILTER_ARGS=()
if [ -n "${FILTER}" ]; then
FILTER_ARGS+=(-k "${FILTER}")
echo "Precision mode: filtered to ${FILTER}"
else
echo "Precision mode: no filter (full suite)"
fi
cd python
python3 sglang/multimodal_gen/test/run_suite.py \
--suite 1-npu \
--partition-id ${{ matrix.part }} \
--total-partitions 2
--total-partitions 2 \
"${FILTER_ARGS[@]}"
- name: Upload diffusion failure artifacts
if: always()
@@ -316,8 +480,19 @@ jobs:
retention-days: 7
multimodal-gen-test-4-npu-a3:
needs: [check-changes, pr-gate, set-image-config, base-a-test-npu]
if: ${{ !failure() && !cancelled() && inputs.coverage_mode != true && needs.check-changes.outputs.multimodal_gen == 'true' }}
needs:
[
check-changes,
pr-gate,
set-image-config,
base-a-test-npu,
base-a-test-npu-cov,
recommend-tests-from-coverage,
]
if: >-
${{ !failure() && !cancelled() &&
(inputs.coverage_mode == true || needs.check-changes.outputs.multimodal_gen == 'true') &&
needs.recommend-tests-from-coverage.outputs.should_run_multimodal_gen_test_4_npu_a3 != 'false' }}
runs-on: linux-aarch64-a3-800t-4
strategy:
fail-fast: false
@@ -360,7 +535,8 @@ jobs:
python3 scripts/ci/utils/diffusion/prefetch_consistency_gt.py
- name: Run test
timeout-minutes: 60
timeout-minutes: ${{ inputs.coverage_mode == true && 150 || 60 }}
shell: bash
env:
SGLANG_USE_MODELSCOPE: true
SGLANG_IS_IN_CI: true
@@ -369,12 +545,75 @@ jobs:
PYTORCH_NPU_ALLOC_CONF: "expandable_segments:True"
STREAMS_PER_DEVICE: 32
SGLANG_DIFFUSION_ARTIFACT_DIR: ${{ github.workspace }}/diffusion-failures
ONLY_TESTS: ${{ needs.recommend-tests-from-coverage.outputs.only_tests_multimodal_gen_test_4_npu_a3 }}
run: |
# Coverage mode: enumerate this shard's targets with an NPU-only CI
# helper (it reuses the shared diffusion runner read-only) and run
# each target in its own `coverage run -m pytest` invocation. The
# runner normally batches a whole shard into one pytest call, which
# cannot be attributed per test case by the precise-test selector.
if [[ "${{ inputs.coverage_mode }}" == "true" ]]; then
set -euo pipefail
pip install coverage==7.8.*
# Coverage tracing slows down the Python glue, which shows up in
# wall-clock assertions. Relax only the timing tolerances for the
# collection run; PR runs (no coverage) keep the strict profile and
# the memory/VRAM checks are untouched.
export SGLANG_E2E_TOLERANCE=1.0
export SGLANG_STAGE_TIME_TOLERANCE=1.0
export SGLANG_NON_DENOISE_STAGE_TIME_TOLERANCE=2.0
export SGLANG_DENOISE_STEP_TOLERANCE=1.0
export SGLANG_DENOISE_AGG_TOLERANCE=1.0
python3 scripts/ci/npu/precise-test/list_multimodal_tests.py \
--suite 2-npu \
--partition-id ${{ matrix.part }} \
--total-partitions 2 \
-o /tmp/selected_tests.txt
echo "Selected test targets:"
cat /tmp/selected_tests.txt
if [ -s /tmp/selected_tests.txt ]; then
bash scripts/ci/npu/precise-test/run_tests_with_coverage.sh $(cat /tmp/selected_tests.txt)
fi
exit 0
fi
# Precision mode: ONLY_TESTS holds one pytest -k token per line
# ('[case_id]' for parametrized cases, a file stem for standalone
# files). It is empty when nothing was recommended, precision mode
# is off, or a safety gate forced the full suite; then no filter is
# applied and the shard runs its whole assignment.
FILTER=""
while IFS= read -r line; do
if [ -z "${line}" ]; then
continue
fi
# Every line must be a -k token ('[case_id]' or a standalone file
# stem). A path means the bucket contract drifted: with a path the
# -k expression matches nothing, pytest exits 5, and the shard
# would report success after running zero tests.
if [[ "${line}" == */* || "${line}" == *.py ]]; then
echo "::error::Unexpected only_tests entry '${line}' (expected a pytest -k token)"
exit 1
fi
if [ -n "${FILTER}" ]; then
FILTER="${FILTER} or "
fi
FILTER="${FILTER}${line}"
done <<< "${ONLY_TESTS}"
FILTER_ARGS=()
if [ -n "${FILTER}" ]; then
FILTER_ARGS+=(-k "${FILTER}")
echo "Precision mode: filtered to ${FILTER}"
else
echo "Precision mode: no filter (full suite)"
fi
cd python
python3 sglang/multimodal_gen/test/run_suite.py \
--suite 2-npu \
--partition-id ${{ matrix.part }} \
--total-partitions 2
--total-partitions 2 \
"${FILTER_ARGS[@]}"
- name: Upload diffusion failure artifacts
if: always()
@@ -387,8 +626,11 @@ jobs:
base-c-test-acc-2-npu-a3:
name: base-c-test-acc-2-npu-a3
needs: [ check-changes, pr-gate, set-image-config, base-b-test-1-npu-a3, base-b-test-2-npu-a3, base-b-test-4-npu-a3, base-b-test-8-npu-a3, base-b-test-16-npu-a3 ]
if: ${{ !failure() && !cancelled() && (inputs.coverage_mode == true || needs.check-changes.outputs.main_package == 'true') }}
needs: [ check-changes, pr-gate, set-image-config, base-b-test-1-npu-a3, base-b-test-2-npu-a3, base-b-test-4-npu-a3, base-b-test-8-npu-a3, base-b-test-16-npu-a3, recommend-tests-from-coverage ]
if: >-
${{ !failure() && !cancelled() &&
(inputs.coverage_mode == true || needs.check-changes.outputs.main_package == 'true') &&
needs.recommend-tests-from-coverage.outputs.should_run_base_c_test_acc_2_npu_a3 != 'false' }}
uses: ./.github/workflows/_npu-single-node-test-stage.yml
with:
runner: ${{ inputs.coverage_mode == true && 'linux-aarch64-a3-800t-2' || 'linux-aarch64-a3-2-' }}
@@ -398,13 +640,16 @@ jobs:
install_sglang_deps: true
device_type_for_deps: 'a3'
partitions: '{"size":2,"arr":[0,1]}'
only_tests: ${{ needs.recommend-tests-from-coverage.outputs.only_tests_base_c_test_acc_2_npu_a3 }}
use_coverage_runner: ${{ inputs.coverage_mode == true }}
upload_test_logs: ${{ inputs.coverage_mode != true }}
base-c-test-acc-16-npu-a3:
name: base-c-test-acc-16-npu-a3
needs: [ check-changes, pr-gate, set-image-config, base-b-test-1-npu-a3, base-b-test-2-npu-a3, base-b-test-4-npu-a3, base-b-test-8-npu-a3, base-b-test-16-npu-a3 ]
if: ${{ !failure() && !cancelled() && (inputs.coverage_mode == true || needs.check-changes.outputs.main_package == 'true') }}
needs: [ check-changes, pr-gate, set-image-config, base-b-test-1-npu-a3, base-b-test-2-npu-a3, base-b-test-4-npu-a3, base-b-test-8-npu-a3, base-b-test-16-npu-a3, recommend-tests-from-coverage ]
if: >-
${{ !failure() && !cancelled() &&
(inputs.coverage_mode == true || needs.check-changes.outputs.main_package == 'true') &&
needs.recommend-tests-from-coverage.outputs.should_run_base_c_test_acc_16_npu_a3 != 'false' }}
uses: ./.github/workflows/_npu-single-node-test-stage.yml
with:
runner: ${{ inputs.coverage_mode == true && 'linux-aarch64-a3-800t-16' || 'linux-aarch64-a3-16-' }}
@@ -413,13 +658,16 @@ jobs:
image: ${{ needs.set-image-config.outputs.CANN_image_a3 }}
install_sglang_deps: true
device_type_for_deps: 'a3'
only_tests: ${{ needs.recommend-tests-from-coverage.outputs.only_tests_base_c_test_acc_16_npu_a3 }}
use_coverage_runner: ${{ inputs.coverage_mode == true }}
upload_test_logs: ${{ inputs.coverage_mode != true }}
base-c-test-perf-2-npu-a3:
name: base-c-test-perf-2-npu-a3
needs: [ check-changes, pr-gate, set-image-config, base-c-test-acc-2-npu-a3, base-c-test-acc-16-npu-a3 ]
if: ${{ !failure() && !cancelled() && (inputs.coverage_mode == true || needs.check-changes.outputs.main_package == 'true') }}
needs: [ check-changes, pr-gate, set-image-config, base-c-test-acc-2-npu-a3, base-c-test-acc-16-npu-a3, recommend-tests-from-coverage ]
if: >-
${{ !failure() && !cancelled() &&
(inputs.coverage_mode == true || needs.check-changes.outputs.main_package == 'true') &&
needs.recommend-tests-from-coverage.outputs.should_run_base_c_test_perf_2_npu_a3 != 'false' }}
uses: ./.github/workflows/_npu-single-node-test-stage.yml
with:
runner: linux-aarch64-a3-800t-2
@@ -428,13 +676,16 @@ jobs:
image: ${{ needs.set-image-config.outputs.CANN_image_a3 }}
install_sglang_deps: true
device_type_for_deps: 'a3'
only_tests: ${{ needs.recommend-tests-from-coverage.outputs.only_tests_base_c_test_perf_2_npu_a3 }}
use_coverage_runner: ${{ inputs.coverage_mode == true }}
upload_test_logs: ${{ inputs.coverage_mode != true }}
base-c-test-perf-16-npu-a3:
name: base-c-test-perf-16-npu-a3
needs: [ check-changes, pr-gate, set-image-config, base-c-test-acc-2-npu-a3, base-c-test-acc-16-npu-a3 ]
if: ${{ !failure() && !cancelled() && (inputs.coverage_mode == true || needs.check-changes.outputs.main_package == 'true') }}
needs: [ check-changes, pr-gate, set-image-config, base-c-test-acc-2-npu-a3, base-c-test-acc-16-npu-a3, recommend-tests-from-coverage ]
if: >-
${{ !failure() && !cancelled() &&
(inputs.coverage_mode == true || needs.check-changes.outputs.main_package == 'true') &&
needs.recommend-tests-from-coverage.outputs.should_run_base_c_test_perf_16_npu_a3 != 'false' }}
uses: ./.github/workflows/_npu-single-node-test-stage.yml
with:
runner: linux-aarch64-a3-800t-16
@@ -444,8 +695,8 @@ jobs:
install_sglang_deps: true
device_type_for_deps: 'a3'
test_timeout_minutes: '180'
only_tests: ${{ needs.recommend-tests-from-coverage.outputs.only_tests_base_c_test_perf_16_npu_a3 }}
use_coverage_runner: ${{ inputs.coverage_mode == true }}
upload_test_logs: ${{ inputs.coverage_mode != true }}
# ==================== Recommend Tests from Coverage ==================== #
@@ -453,25 +704,63 @@ jobs:
# continue-on-error so it never blocks the PR gate.
recommend-tests-from-coverage:
name: Recommend tests from coverage
needs: [check-changes, pr-gate]
runs-on: ubuntu-latest
continue-on-error: true
# TEMP-DISABLED: OBS coverage baseline returns 403; re-enable once fixed.
if: false
# if: ${{ !cancelled() && github.event_name == 'pull_request' && inputs.coverage_mode != true }}
if: >-
${{ !failure() && !cancelled() &&
github.event_name == 'pull_request' &&
inputs.coverage_mode != true &&
needs.check-changes.outputs.changes_exist == 'true' }}
outputs:
coverage_paths: ${{ steps.export-paths.outputs.coverage_paths }}
should_run_base_b_test_1_npu_a3: ${{ steps.bucket-tests.outputs.should_run_base_b_test_1_npu_a3 }}
only_tests_base_b_test_1_npu_a3: ${{ steps.bucket-tests.outputs.only_tests_base_b_test_1_npu_a3 }}
should_run_base_b_test_2_npu_a3: ${{ steps.bucket-tests.outputs.should_run_base_b_test_2_npu_a3 }}
only_tests_base_b_test_2_npu_a3: ${{ steps.bucket-tests.outputs.only_tests_base_b_test_2_npu_a3 }}
should_run_base_b_test_4_npu_a3: ${{ steps.bucket-tests.outputs.should_run_base_b_test_4_npu_a3 }}
only_tests_base_b_test_4_npu_a3: ${{ steps.bucket-tests.outputs.only_tests_base_b_test_4_npu_a3 }}
should_run_base_b_test_8_npu_a3: ${{ steps.bucket-tests.outputs.should_run_base_b_test_8_npu_a3 }}
only_tests_base_b_test_8_npu_a3: ${{ steps.bucket-tests.outputs.only_tests_base_b_test_8_npu_a3 }}
should_run_base_b_test_16_npu_a3: ${{ steps.bucket-tests.outputs.should_run_base_b_test_16_npu_a3 }}
only_tests_base_b_test_16_npu_a3: ${{ steps.bucket-tests.outputs.only_tests_base_b_test_16_npu_a3 }}
should_run_base_c_test_acc_2_npu_a3: ${{ steps.bucket-tests.outputs.should_run_base_c_test_acc_2_npu_a3 }}
only_tests_base_c_test_acc_2_npu_a3: ${{ steps.bucket-tests.outputs.only_tests_base_c_test_acc_2_npu_a3 }}
should_run_base_c_test_acc_16_npu_a3: ${{ steps.bucket-tests.outputs.should_run_base_c_test_acc_16_npu_a3 }}
only_tests_base_c_test_acc_16_npu_a3: ${{ steps.bucket-tests.outputs.only_tests_base_c_test_acc_16_npu_a3 }}
should_run_base_c_test_perf_2_npu_a3: ${{ steps.bucket-tests.outputs.should_run_base_c_test_perf_2_npu_a3 }}
only_tests_base_c_test_perf_2_npu_a3: ${{ steps.bucket-tests.outputs.only_tests_base_c_test_perf_2_npu_a3 }}
should_run_base_c_test_perf_16_npu_a3: ${{ steps.bucket-tests.outputs.should_run_base_c_test_perf_16_npu_a3 }}
only_tests_base_c_test_perf_16_npu_a3: ${{ steps.bucket-tests.outputs.only_tests_base_c_test_perf_16_npu_a3 }}
should_run_multimodal_gen_test_1_npu_a3: ${{ steps.bucket-tests.outputs.should_run_multimodal_gen_test_1_npu_a3 }}
only_tests_multimodal_gen_test_1_npu_a3: ${{ steps.bucket-tests.outputs.only_tests_multimodal_gen_test_1_npu_a3 }}
should_run_multimodal_gen_test_4_npu_a3: ${{ steps.bucket-tests.outputs.should_run_multimodal_gen_test_4_npu_a3 }}
only_tests_multimodal_gen_test_4_npu_a3: ${{ steps.bucket-tests.outputs.only_tests_multimodal_gen_test_4_npu_a3 }}
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
steps:
# Every step carries step-level continue-on-error so the job conclusion
# is ALWAYS 'success' — the fast-fail health check in other jobs filters
# by job conclusion, and job-level continue-on-error still reports
# 'failure' via the API. Failed steps stay visible (red ✗ + annotation)
# but never cascade to test jobs.
- name: Checkout sglang scripts
continue-on-error: true
uses: actions/checkout@v4
with:
# test/registered + python/sglang/test/ci are needed by
# list_tests.py to enumerate each suite's test files for bucketing.
# The diffusion case config is read (AST only) by
# bucket_recommended_tests.py to enumerate multimodal cases.
sparse-checkout: |
scripts
test/registered
python/sglang/test/ci
python/sglang/multimodal_gen/test/server/ascend/testcase_configs_npu.py
sparse-checkout-cone-mode: false
- name: Download test case map
continue-on-error: true
run: |
set -euo pipefail
curl -fsSL \
@@ -485,6 +774,7 @@ jobs:
du -h test_case_map.json
- name: Download coverage package and extract covstub
continue-on-error: true
run: |
set -euo pipefail
curl -fsSL \
@@ -502,6 +792,7 @@ jobs:
find outputs/covstub -maxdepth 2 -type d | head -20 || true
- name: Recommend tests for current PR
continue-on-error: true
run: |
set -euo pipefail
if ! python3 -c "import regex" >/dev/null 2>&1; then
@@ -520,52 +811,42 @@ jobs:
--source-dir "$PWD/outputs/covstub" \
--map-file "$PWD/test_case_map.json"
- name: Export coverage paths output
id: export-paths
# Bucket the recommended tests per suite. Downstream jobs consume two
# outputs per suite: should_run_<suite> ('false' -> skip the job, e.g.
# empty bucket in precision mode) and only_tests_<suite> (non-empty only
# in precision mode, gated inside the script). On failure outputs stay
# empty -> callers run full suites.
- name: Bucket recommended tests per suite
id: bucket-tests
continue-on-error: true
if: always()
env:
# Precision-test switch: 'true' = the a3 suites run only the
# recommended subset, 'false' = full suites.
PRECISION_MODE: 'true'
run: |
FILE="scripts/ci/npu/precise-test/recommended_pytest_paths.txt"
if [ -f "$FILE" ]; then
{
echo "coverage_paths<<EOF"
cat "$FILE"
echo "EOF"
} >> "$GITHUB_OUTPUT"
echo "Exported $(wc -l < "$FILE") recommended paths"
else
echo "coverage_paths=" >> "$GITHUB_OUTPUT"
echo "::notice::recommended_pytest_paths.txt not found"
fi
set -euo pipefail
python3 scripts/ci/npu/precise-test/bucket_recommended_tests.py \
--recommended scripts/ci/npu/precise-test/recommended_pytest_paths.txt \
--changed-files scripts/ci/npu/precise-test/pr_changed_files.txt \
--suites "base-b-test-1-npu-a3,base-b-test-2-npu-a3,base-b-test-4-npu-a3,base-b-test-8-npu-a3,base-b-test-16-npu-a3,base-c-test-acc-2-npu-a3,base-c-test-acc-16-npu-a3,base-c-test-perf-2-npu-a3,base-c-test-perf-16-npu-a3" \
--diffusion-suites "multimodal-gen-test-1-npu-a3:1-npu,multimodal-gen-test-4-npu-a3:2-npu" \
--case-map "$PWD/test_case_map.json" \
--precision-mode "${PRECISION_MODE}" \
--github-output "${GITHUB_OUTPUT}" \
--github-summary "${GITHUB_STEP_SUMMARY:-}" \
--report-file recommended-tests-report.md
# ==================== Analyze Failure Report ==================== #
# Cross-references test failures with coverage recommendations.
# !cancelled() only: skipped jobs in needs poison success().
analyze-failure-report:
name: Analyze failure report
needs:
[
base-a-test-npu,
base-b-test-1-npu-a3,
base-b-test-2-npu-a3,
base-b-test-4-npu-a3,
base-b-test-8-npu-a3,
base-b-test-16-npu-a3,
multimodal-gen-test-1-npu-a3,
multimodal-gen-test-4-npu-a3,
base-c-test-acc-2-npu-a3,
base-c-test-acc-16-npu-a3,
base-c-test-perf-2-npu-a3,
base-c-test-perf-16-npu-a3,
recommend-tests-from-coverage,
]
if: ${{ !cancelled() && inputs.coverage_mode != true }}
uses: ./.github/workflows/_npu-analyze-failure.yml
with:
log_artifact_pattern: selected-test-logs-*
recommendations_content: ${{ needs.recommend-tests-from-coverage.outputs.coverage_paths || '' }}
# Same report as the step summary, downloadable from the run page.
- name: Upload recommended tests report
continue-on-error: true
if: always()
uses: actions/upload-artifact@v4
with:
name: recommended-tests-report
path: recommended-tests-report.md
if-no-files-found: warn
retention-days: 7
# ==================== Coverage Assembly & Publish ==================== #
# Assembles coverage data and publishes the baseline to the shared disk.
@@ -573,7 +854,7 @@ jobs:
setup-covstub:
needs:
[
set-image-config,
base-a-test-npu-cov,
base-b-test-1-npu-a3,
base-b-test-2-npu-a3,
base-b-test-4-npu-a3,
@@ -583,11 +864,13 @@ jobs:
base-c-test-acc-16-npu-a3,
base-c-test-perf-2-npu-a3,
base-c-test-perf-16-npu-a3,
multimodal-gen-test-1-npu-a3,
multimodal-gen-test-4-npu-a3,
]
if: ${{ !failure() && !cancelled() && inputs.coverage_mode == true }}
runs-on: linux-aarch64-a3-800t-2
runs-on: linux-amd64-cpu-4
container:
image: ${{ needs.set-image-config.outputs.CANN_image_a3 }}
image: swr.ap-southeast-1.myhuaweicloud.com/base_image/ascend-ci/sglang:main-x86
steps:
- name: Checkout code
uses: actions/checkout@v4
@@ -596,12 +879,31 @@ jobs:
# checks out the same commit even if the branch advances mid-run.
ref: ${{ inputs.ref || github.sha }}
- name: Collect base-a coverage artifact
# Runs on ubuntu-latest (no shared disk) — hence the artifact relay.
uses: actions/download-artifact@v4
with:
name: coverage-outputs-base-a-test-npu
path: /tmp/base-a-coverage
- name: Merge base-a coverage into shared run dir
run: |
# Artifact stages {RUN_DIR}/outputs/... — merges 1:1 into the shared layout.
CACHE_ROOT="/root/.cache/tests/precise-test"
mkdir -p "${CACHE_ROOT}"
cp -a /tmp/base-a-coverage/. "${CACHE_ROOT}/"
echo "Merged base-a coverage into ${CACHE_ROOT}:"
find "${CACHE_ROOT}" -maxdepth 3 -type d | head -20
- name: Copy sglang source to covstub
run: |
RUN_DIR="/root/.cache/tests/precise-test/${{ github.run_id }}-attempt-${{ github.run_attempt }}"
COVSTUB="${RUN_DIR}/outputs/covstub"
mkdir -p "${COVSTUB}"
cp -r python/sglang "${COVSTUB}/"
# _version.py is build-generated (git-ignored); the placeholder keeps
# coverage html from failing on a missing recorded source file.
echo '__version__ = "0.0.0"' > "${COVSTUB}/sglang/_version.py"
- name: Build test case map
shell: bash
@@ -609,11 +911,9 @@ jobs:
set -euo pipefail
RUN_DIR="/root/.cache/tests/precise-test/${{ github.run_id }}-attempt-${{ github.run_attempt }}"
# Date tag derivation must match scripts/ci/npu/precise-test/run_tests_with_coverage.sh.
# The custom k8s runner only injects explicitly-set env vars, so
# GITHUB_RUN_STARTED_AT may be absent (set -u would abort); fall back
# to local date exactly like run_tests_with_coverage.sh does, so both
# sides derive the same tag.
# Must match run_tests_with_coverage.sh: GITHUB_RUN_STARTED_AT may be
# unset on the k8s runner (only explicitly-set envs are injected), so
# fall back to local date the same way.
if [ -n "${GITHUB_RUN_STARTED_AT:-}" ]; then
COV_DATE_TAG="${GITHUB_RUN_STARTED_AT:0:10}"
COV_DATE_TAG="${COV_DATE_TAG//-/}"
@@ -622,14 +922,14 @@ jobs:
fi
COV_ROOT="${RUN_DIR}/outputs/sglang@${COV_DATE_TAG}"
COVSTUB="${RUN_DIR}/outputs/covstub"
# Placed outside outputs/ so it is NOT included in outputs.tar.gz.
# Standalone top-level deliverable, never packed inside outputs/.
MAP_FILE="${RUN_DIR}/test_case_map.json"
echo "Coverage dir: ${COV_ROOT}"
echo "Source dir: ${COVSTUB}"
# Coverage data may be absent when all test jobs were skipped
# (e.g. by the changes filter). Skip map building in that case.
# Coverage data may be absent when no test targets were enumerated
# (run_tests_with_coverage.sh never ran). Skip map building then.
if [ ! -d "${COV_ROOT}" ]; then
echo "::warning::Coverage dir not found: ${COV_ROOT}, skip building test case map"
exit 0
@@ -643,11 +943,11 @@ jobs:
fi
if ! python3 -c "import regex" 2>/dev/null; then
pip install regex \
python3 -m pip install regex \
-i http://cache-service.nginx-pypi-cache.svc.cluster.local/pypi/simple \
--trusted-host cache-service.nginx-pypi-cache.svc.cluster.local \
--retries 3 --timeout 60 \
|| pip install regex -i https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple
|| python3 -m pip install regex -i https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple
fi
python3 scripts/ci/npu/precise-test/test_selector.py \
@@ -658,10 +958,182 @@ jobs:
test -s "${MAP_FILE}"
du -h "${MAP_FILE}"
- name: Generate HTML coverage report
# Best-effort add-on: continue-on-error keeps the job green and later
# steps running, while a failed generation still shows a red ✗ in the UI.
continue-on-error: true
shell: bash
run: |
set -euo pipefail
RUN_DIR="/root/.cache/tests/precise-test/${{ github.run_id }}-attempt-${{ github.run_attempt }}"
# Date tag derivation must match run_tests_with_coverage.sh.
if [ -n "${GITHUB_RUN_STARTED_AT:-}" ]; then
COV_DATE_TAG="${GITHUB_RUN_STARTED_AT:0:10}"
COV_DATE_TAG="${COV_DATE_TAG//-/}"
else
COV_DATE_TAG="$(date +%Y%m%d)"
fi
COV_ROOT="${RUN_DIR}/outputs/sglang@${COV_DATE_TAG}"
COVSTUB="${RUN_DIR}/outputs/covstub"
# Standalone top-level dir, packaged into htmlcov.tar.gz at publish.
REPORT_DIR="${RUN_DIR}/htmlcov"
# Same skip conditions as map building: no data -> nothing to report.
if [ ! -d "${COV_ROOT}" ] \
|| [ "$(find "${COV_ROOT}" -name 'coverage.*' -type f | wc -l)" -eq 0 ]; then
echo "::warning::No coverage data found, skip HTML report"
exit 0
fi
# Failures below exit non-zero (errexit) — red ✗, job stays green.
# Do NOT convert failures to exit 0 behind an `if ! ( set -e; ... )`
# guard: bash ignores -e inside a subshell in a condition context,
# which silently swallowed three failures in run 38713909169.
# Pin matches the test jobs' coverage==7.8.* so combine can read
# their SQLite data.
if ! python3 -c "import coverage" 2>/dev/null; then
python3 -m pip install coverage==7.8.* \
-i http://cache-service.nginx-pypi-cache.svc.cluster.local/pypi/simple \
--trusted-host cache-service.nginx-pypi-cache.svc.cluster.local \
--retries 3 --timeout 60 \
|| python3 -m pip install coverage==7.8.* -i https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple
fi
# Alias runner-specific source paths onto covstub; first entry must
# be a real local path (no wildcards).
{
echo "[run]"
echo "branch = True"
echo "[paths]"
echo "sglang ="
echo " ${COVSTUB}/sglang/"
echo " */python/sglang/"
} > /tmp/report-rc
# --keep preserves the per-test raw coverage files: coverage-data.tar.gz
# archives them after this step for manual map rebuilds.
COVERAGE_FILE=/tmp/.coverage-all \
python3 -m coverage combine --rcfile=/tmp/report-rc --keep \
$(find "${COV_ROOT}" -name 'coverage.*' -type f)
rm -rf "${REPORT_DIR}"
COVERAGE_FILE=/tmp/.coverage-all \
python3 -m coverage html --rcfile=/tmp/report-rc \
--directory="${REPORT_DIR}"
# NPU-only view: [run] source does not filter at report time (a
# source-based rcfile yields the full report), so instead touch the
# unexecuted npu .py files into a data copy (os.walk, so the PEP 420
# namespace tree counts) and filter with --include, which works.
cp /tmp/.coverage-all /tmp/.coverage-npu
python3 - "${COVSTUB}" <<'PYEOF'
import os
import sys
from coverage import CoverageData
npu_root = os.path.join(sys.argv[1], "sglang", "srt", "hardware_backend", "npu")
data = CoverageData(basename="/tmp/.coverage-npu")
data.read()
measured = set(data.measured_files())
added = 0
for root, _, files in os.walk(npu_root):
for f in files:
if f.endswith(".py"):
p = os.path.join(root, f)
if p not in measured:
data.touch_file(p)
added += 1
data.write()
print(f"NPU view: {len(measured)} measured, touched {added} unexecuted files")
PYEOF
rm -rf "${REPORT_DIR}-npu"
COVERAGE_FILE=/tmp/.coverage-npu \
python3 -m coverage html --rcfile=/tmp/report-rc \
--include='*/sglang/srt/hardware_backend/npu/*' \
--directory="${REPORT_DIR}-npu"
# Surface the totals directly on the run page.
# GITHUB_STEP_SUMMARY may be unset on the k8s runner pool.
if [ -n "${GITHUB_STEP_SUMMARY:-}" ]; then
{
echo "### Coverage"
echo '```'
echo "python/sglang (measured files only):"
COVERAGE_FILE=/tmp/.coverage-all \
python3 -m coverage report --rcfile=/tmp/report-rc | tail -n 1
echo "python/sglang/srt/hardware_backend/npu (full-tree denominator):"
COVERAGE_FILE=/tmp/.coverage-npu \
python3 -m coverage report --rcfile=/tmp/report-rc \
--include='*/sglang/srt/hardware_backend/npu/*' | tail -n 1
echo '```'
echo "Full line-by-line report: download the \`coverage-html-report\` artifact and open \`htmlcov/index.html\` (NPU-only view: \`htmlcov-npu/index.html\`)."
} >> "${GITHUB_STEP_SUMMARY}"
fi
echo "HTML coverage report: ${REPORT_DIR}"
echo "NPU-only HTML coverage report: ${REPORT_DIR}-npu"
- name: Upload HTML coverage report
# Report upload must never block the baseline publish pipeline.
continue-on-error: true
uses: actions/upload-artifact@v4
with:
name: coverage-html-report
# Two paths -> artifact root becomes the run dir, so the archive
# holds htmlcov/ and htmlcov-npu/ side by side.
path: |
/root/.cache/tests/precise-test/${{ github.run_id }}-attempt-${{ github.run_attempt }}/htmlcov
/root/.cache/tests/precise-test/${{ github.run_id }}-attempt-${{ github.run_attempt }}/htmlcov-npu
if-no-files-found: ignore
retention-days: 14
- name: Archive outputs
run: |
RUN_DIR="/root/.cache/tests/precise-test/${{ github.run_id }}-attempt-${{ github.run_attempt }}"
tar -czf "${RUN_DIR}/outputs.tar.gz" -C "${RUN_DIR}" outputs
# Source-only baseline: recommend mode needs covstub only, and it
# shrinks the package from ~300MB to a few MB.
tar -czf "${RUN_DIR}/outputs.tar.gz" -C "${RUN_DIR}" outputs/covstub
# Raw coverage data for manual map rebuilds: extract both archives
# under one root to reconstruct outputs/. Absent when no data.
# Date tag derivation must match run_tests_with_coverage.sh.
if [ -n "${GITHUB_RUN_STARTED_AT:-}" ]; then
COV_DATE_TAG="${GITHUB_RUN_STARTED_AT:0:10}"
COV_DATE_TAG="${COV_DATE_TAG//-/}"
else
COV_DATE_TAG="$(date +%Y%m%d)"
fi
if [ -d "${RUN_DIR}/outputs/sglang@${COV_DATE_TAG}" ]; then
tar -czf "${RUN_DIR}/coverage-data.tar.gz" -C "${RUN_DIR}" "outputs/sglang@${COV_DATE_TAG}"
fi
- name: Upload outputs and test case map
# Never blocks the publish pipeline; must run before publish mv's
# the files to the fixed PUBLISH_DIR (races with concurrent runs).
continue-on-error: true
uses: actions/upload-artifact@v4
with:
name: precise-test-outputs
# Two paths -> artifact root becomes RUN_DIR, so the archive
# holds outputs.tar.gz and test_case_map.json side by side.
path: |
/root/.cache/tests/precise-test/${{ github.run_id }}-attempt-${{ github.run_attempt }}/outputs.tar.gz
/root/.cache/tests/precise-test/${{ github.run_id }}-attempt-${{ github.run_attempt }}/test_case_map.json
if-no-files-found: ignore
retention-days: 14
- name: Upload raw coverage data
# Separate artifact keeps the ~300MB bulk out of the baseline package;
# must run before publish mv's the file to the fixed PUBLISH_DIR.
continue-on-error: true
uses: actions/upload-artifact@v4
with:
name: precise-test-coverage-data
path: /root/.cache/tests/precise-test/${{ github.run_id }}-attempt-${{ github.run_attempt }}/coverage-data.tar.gz
# Absent when all test jobs were skipped (no coverage data).
if-no-files-found: ignore
retention-days: 14
- name: Publish outputs and map to shared disk
shell: bash
@@ -669,14 +1141,14 @@ jobs:
set -euo pipefail
RUN_DIR="/root/.cache/tests/precise-test/${{ github.run_id }}-attempt-${{ github.run_attempt }}"
# Fixed publish dir with no RUN_DIR layer: other pipelines have a
# different GITHUB_RUN_ID and must be able to read the baseline.
# Must match PUBLISH_DIR in the recommend-tests-from-coverage job.
# Fixed-name copy for manual use; the recommend job's baseline lives
# in OBS (upload outputs.tar.gz + test_case_map.json manually after
# each run). coverage-data.tar.gz has no automatic consumer — no OBS
# copy, only this dir and the precise-test-coverage-data artifact.
PUBLISH_DIR="/root/.cache/tests/precise-test/coverage"
# Skip publish when no coverage data was collected this run (map
# build was skipped), so we never overwrite the historical
# baseline on the shared disk with an incomplete one.
# Skip publish when no coverage data this run — never overwrite
# the historical baseline with an incomplete one.
if [ ! -s "${RUN_DIR}/test_case_map.json" ]; then
echo "::warning::test_case_map.json missing (no coverage data this run), skip publish"
exit 0
@@ -684,14 +1156,29 @@ jobs:
mkdir -p "${PUBLISH_DIR}"
# mv (rename) within the same shared filesystem is atomic, so a
# concurrent reader sees either the old or the new baseline,
# never a partial file.
# Same-filesystem mv is atomic per file, so readers never see a
# partial file (files from two concurrent publishes can interleave).
mv "${RUN_DIR}/outputs.tar.gz" "${PUBLISH_DIR}/outputs.tar.gz"
mv "${RUN_DIR}/test_case_map.json" "${PUBLISH_DIR}/test_case_map.json"
# Guarded: absent in the unlikely case this run collected no
# coverage data while still producing a map.
if [ -f "${RUN_DIR}/coverage-data.tar.gz" ]; then
mv "${RUN_DIR}/coverage-data.tar.gz" "${PUBLISH_DIR}/coverage-data.tar.gz"
fi
# HTML reports too, fixed names — latest wins, like the baseline.
# [ -d ] guards the continue-on-error generation step. A tar failure
# aborts red after the baseline mv completed; Cleanup stays skipped,
# keeping RUN_DIR for recovery.
if [ -d "${RUN_DIR}/htmlcov" ]; then
tar -czf "${PUBLISH_DIR}/htmlcov.tar.gz" -C "${RUN_DIR}" htmlcov
fi
if [ -d "${RUN_DIR}/htmlcov-npu" ]; then
tar -czf "${PUBLISH_DIR}/htmlcov-npu.tar.gz" -C "${RUN_DIR}" htmlcov-npu
fi
echo "Published baseline to ${PUBLISH_DIR}"
du -h "${PUBLISH_DIR}/outputs.tar.gz" "${PUBLISH_DIR}/test_case_map.json"
ls -lh "${PUBLISH_DIR}" | tail -n +2
- name: Clean up per-run directory
# Skipped on publish failure, keeping RUN_DIR for manual recovery.
@@ -708,6 +1195,7 @@ jobs:
check-changes,
base-a-test-npu,
base-a-test-npu-cov,
base-b-test-1-npu-a3,
base-b-test-2-npu-a3,
base-b-test-4-npu-a3,
+167 -53
View File
@@ -9,14 +9,14 @@ on:
workflow_dispatch:
inputs:
run_all_tests:
description: "Run the preflight even if the last commit touched no PPU path"
description: "Build and run both sanity files even without PPU path changes"
required: false
type: boolean
default: true
workflow_call:
inputs:
ref:
description: 'Git ref (branch, tag, or SHA) to test. If not provided, uses the default branch.'
description: "Git ref to test; defaults to the triggering event SHA"
required: false
type: string
default: ''
@@ -27,8 +27,7 @@ on:
default: false
concurrency:
# Newer runs cancel older runs for the same ref; unrelated label events
# use isolated groups to avoid interfering with active runs.
# Unrelated label events must not cancel or replace the active PPU checks.
group: >-
pr-test-ppu-${{
(github.event_name == 'pull_request' &&
@@ -40,11 +39,15 @@ concurrency:
}}
cancel-in-progress: true
defaults:
run:
shell: bash
env:
GIT_CONFIG_NOSYSTEM: "1"
jobs:
# ==================== Check Changes ==================== #
check-changes:
# Give skipped checks from unrelated labels distinct names so they cannot
# replace the corresponding checks from an active run.
name: >-
${{ (github.event_name == 'pull_request' && github.event.action == 'labeled' &&
github.event.label.name != 'run-ci' && github.event.label.name != 'run-ci-ppu') &&
@@ -59,25 +62,28 @@ jobs:
contents: read
pull-requests: read
outputs:
# Scope only; the shared pr-gate checks both required labels.
ppu: ${{ steps.filter.outputs.ppu == 'true' || steps.run-mode.outputs.run_all_tests == 'true' }}
source-sha: ${{ steps.source.outputs.sha }}
steps:
- name: Checkout code
uses: actions/checkout@v4
with:
# github.sha (not github.ref) so a runner that dequeues after main
# advances still validates the SHA that triggered the run.
ref: ${{ inputs.ref || github.sha }}
persist-credentials: false
- name: Pin the source revision for every job
id: source
run: printf 'sha=%s\n' "$(git rev-parse HEAD)" >> "$GITHUB_OUTPUT"
- name: Determine run mode
id: run-mode
env:
RUN_ALL_TESTS: ${{ inputs.run_all_tests }}
run: |
if [[ "${{ inputs.run_all_tests }}" == "true" ]]; then
if [[ "$RUN_ALL_TESTS" == "true" ]]; then
echo "run_all_tests=true" >> "$GITHUB_OUTPUT"
echo "Run mode: ALL TESTS"
else
echo "run_all_tests=false" >> "$GITHUB_OUTPUT"
echo "Run mode: FILTERED (triggered by ${{ github.event_name }})"
fi
- name: Detect file changes
@@ -85,16 +91,18 @@ jobs:
uses: dorny/paths-filter@v3
if: steps.run-mode.outputs.run_all_tests != 'true'
with:
# PPU-specific paths only.
filters: |
ppu:
- ".github/workflows/pr-test-ppu.yml"
- "scripts/ci/ppu/**"
- "python/sglang/srt/platforms/ppu.py"
- "python/sglang/srt/hardware_backend/ppu/**"
- "python/sglang/kernels/aot/csrc/ppu/**"
- "python/sglang/srt/platforms/sail.py"
- "python/sglang/kernels/aot/setup_sail.py"
- "python/sglang/kernels/aot/pyproject_sail.toml"
- "python/sglang/kernels/aot/csrc/common_extension_sail.cc"
- "python/sglang/kernels/jit/utils/compile/sailify.py"
- "test/registered/core/test_basic_sanity_eagle3.py"
- "test/registered/core/test_basic_sanity_dflash.py"
# ==================== PR Gate ==================== #
pr-gate:
name: >-
${{ (github.event_name == 'pull_request' && github.event.action == 'labeled' &&
@@ -108,62 +116,168 @@ jobs:
require-label: run-ci-ppu
secrets: inherit
# ==================== PPU Runner Preflight ==================== #
# Validates the runner contract only; the SRT platform and AOT kernels land
# in later PRs and build their first real test on top of this job.
ppu-preflight:
ppu-build:
name: >-
${{ (github.event_name == 'pull_request' && github.event.action == 'labeled' &&
github.event.label.name != 'run-ci' && github.event.label.name != 'run-ci-ppu') &&
'ignored-label-ppu-preflight' || 'ppu-preflight' }}
'ignored-label-ppu-build' || 'ppu-build' }}
needs: [check-changes, pr-gate]
if: needs.check-changes.outputs.ppu == 'true'
runs-on: 1-ppu-m890p
timeout-minutes: 60
runs-on: ubuntu-24.04
timeout-minutes: 120
permissions:
contents: read
container:
image: download.t-head.cn/docker_release/pytorch:sdk2.2.0-2.13.0-ubuntu24.04-hggcrt3-py312-20261005
env:
SGLANG_IS_IN_CI: true
PPU_CI_SOURCE_SHA: ${{ needs.check-changes.outputs.source-sha }}
MAX_JOBS: "2"
SGL_KERNEL_COMPILE_THREADS: "2"
outputs:
wheel-artifact-id: ${{ steps.wheels.outputs.artifact-id }}
steps:
- name: Checkout code
- name: Initialize private build directories and Git configuration
run: |
work="$(mktemp -d "${RUNNER_TEMP}/ppu-build.XXXXXX")"
touch "$work/gitconfig"
mkdir "$work/wheels" "$work/artifacts"
{
printf 'GIT_CONFIG_GLOBAL=%s\n' "$work/gitconfig"
printf 'PPU_CI_WHEEL_DIR=%s\n' "$work/wheels"
printf 'PPU_CI_ARTIFACT_DIR=%s\n' "$work/artifacts"
} >> "$GITHUB_ENV"
# Same ref as check-changes. build_and_test.sh rejects any HEAD other than
# PPU_CI_SOURCE_SHA, so a workflow_call branch that moves after
# check-changes fails the run instead of building different code.
- name: Checkout the triggering source revision
uses: actions/checkout@v4
with:
ref: ${{ inputs.ref || github.sha }}
persist-credentials: false
- name: Build native SAIL wheels
run: |
mkdir -p "$PPU_CI_WHEEL_DIR"
bash scripts/ci/ppu/build_and_test.sh build
- name: Upload PPU wheels
id: wheels
uses: actions/upload-artifact@v4
with:
name: ppu-wheels-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ env.PPU_CI_WHEEL_DIR }}
if-no-files-found: error
- name: Upload build diagnostics
if: always()
uses: actions/upload-artifact@v4
with:
name: ppu-build-logs-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ env.PPU_CI_ARTIFACT_DIR }}
if-no-files-found: warn
ppu-sanity:
name: >-
${{ (github.event_name == 'pull_request' && github.event.action == 'labeled' &&
github.event.label.name != 'run-ci' && github.event.label.name != 'run-ci-ppu') &&
format('ignored-label-ppu-sanity ({0})', matrix.runner) ||
format('ppu-sanity ({0})', matrix.runner) }}
needs: [check-changes, pr-gate, ppu-build]
if: needs.check-changes.outputs.ppu == 'true'
# Both PPU generations reuse the same wheels; each host supplies its own
# runtime image and device, while checkpoints come from the shared NAS.
strategy:
# A failure on one generation must not hide the other generation's result.
fail-fast: false
matrix:
runner: [8-ppu-m890p, 16-ppu-zw810e]
runs-on: ${{ matrix.runner }}
timeout-minutes: 60
permissions:
contents: read
actions: read
env:
PPU_CI_SOURCE_SHA: ${{ needs.check-changes.outputs.source-sha }}
steps:
- name: Initialize private runtime directories and Git configuration
run: |
work="$(mktemp -d "${RUNNER_TEMP}/ppu-runtime.XXXXXX")"
touch "$work/gitconfig"
mkdir "$work/wheels" "$work/artifacts"
{
printf 'GIT_CONFIG_GLOBAL=%s\n' "$work/gitconfig"
printf 'PPU_CI_WHEEL_DIR=%s\n' "$work/wheels"
printf 'PPU_CI_ARTIFACT_DIR=%s\n' "$work/artifacts"
} >> "$GITHUB_ENV"
# Same ref as check-changes; run_test.sh rejects any HEAD other than
# PPU_CI_SOURCE_SHA.
- name: Checkout the triggering source revision
timeout-minutes: 10
uses: actions/checkout@v4
with:
ref: ${{ inputs.ref || github.sha }}
persist-credentials: false
- name: Validate PPU runner contract
timeout-minutes: 10
run: python3 scripts/ci/ppu/check_ppu_environment.py
- name: Download the matching PPU wheel artifact
uses: actions/download-artifact@v4
with:
artifact-ids: ${{ needs.ppu-build.outputs.wheel-artifact-id }}
merge-multiple: true
path: ${{ env.PPU_CI_WHEEL_DIR }}
- name: Run both speculative sanity files in isolated Docker
env:
CONFIG_PPU_CI_DEVICE: ${{ vars.PPU_CI_DEVICE }}
CONFIG_PPU_CI_TARGET_MODEL_PATH: ${{ vars.PPU_CI_TARGET_MODEL_PATH }}
CONFIG_PPU_CI_EAGLE3_MODEL_PATH: ${{ vars.PPU_CI_EAGLE3_MODEL_PATH }}
CONFIG_PPU_CI_DFLASH_MODEL_PATH: ${{ vars.PPU_CI_DFLASH_MODEL_PATH }}
CONFIG_PPU_CI_EVAL_CACHE: ${{ vars.PPU_CI_EVAL_CACHE }}
run: |
for key in PPU_CI_DEVICE PPU_CI_TARGET_MODEL_PATH PPU_CI_EAGLE3_MODEL_PATH PPU_CI_DFLASH_MODEL_PATH PPU_CI_EVAL_CACHE; do
override="CONFIG_$key"
if [[ -n "${!override}" ]]; then
export "$key=${!override}"
fi
done
exec bash scripts/ci/ppu/run_test.sh
- name: Upload runtime diagnostics
if: always()
uses: actions/upload-artifact@v4
with:
name: ppu-runtime-logs-${{ matrix.runner }}-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ env.PPU_CI_ARTIFACT_DIR }}
if-no-files-found: warn
# ==================== Finish ==================== #
pr-test-ppu-finish:
name: >-
${{ (github.event_name == 'pull_request' && github.event.action == 'labeled' &&
github.event.label.name != 'run-ci' && github.event.label.name != 'run-ci-ppu') &&
'ignored-label-pr-test-ppu-finish' || 'pr-test-ppu-finish' }}
needs: [pr-gate, check-changes, ppu-preflight]
needs: [pr-gate, check-changes, ppu-build, ppu-sanity]
if: always() && needs.check-changes.result != 'skipped'
runs-on: ubuntu-latest
permissions: {}
steps:
- name: Check all dependent job statuses
- name: Require successful native build and both PPU sanity files when requested
env:
NEEDS_JSON: ${{ toJson(needs) }}
PPU_IN_SCOPE: ${{ needs.check-changes.outputs.ppu }}
BUILD_RESULT: ${{ needs.ppu-build.result }}
SANITY_RESULT: ${{ needs.ppu-sanity.result }}
run: |
json_needs='${{ toJson(needs) }}'
for job in $(echo "$json_needs" | jq -r 'keys_unsorted[]'); do
result=$(echo "$json_needs" | jq -r --arg j "$job" '.[$j].result')
echo "$job: $result"
if [[ "$result" == "failure" || "$result" == "cancelled" ]]; then
echo "The above jobs failed."
exit 1
fi
done
echo "All jobs completed successfully"
- name: Require the preflight when PPU is in scope
run: |
in_scope='${{ needs.check-changes.outputs.ppu }}'
preflight='${{ needs.ppu-preflight.result }}'
echo "PPU in scope: '$in_scope' / ppu-preflight: '$preflight'"
if [[ "$in_scope" == "true" && "$preflight" != "success" ]]; then
echo "PPU was in scope but ppu-preflight did not succeed."
if jq -e 'any(.[]; .result == "failure" or .result == "cancelled")' <<< "$NEEDS_JSON" >/dev/null; then
printf '%s\n' '::error::A PPU CI dependency failed or was cancelled.'
exit 1
fi
echo "Preflight status is consistent with the requested scope."
if [[ "$PPU_IN_SCOPE" == "true" ]]; then
if [[ "$BUILD_RESULT" != "success" || "$SANITY_RESULT" != "success" ]]; then
printf '%s\n' '::error::PPU scope requires a successful native build and both PPU sanity files.'
exit 1
fi
printf '%s\n' 'Native build and both PPU sanity files passed.'
else
printf '%s\n' 'PPU is outside this run scope; no PPU tests were requested.'
fi
+4 -4
View File
@@ -56,7 +56,7 @@ jobs:
with:
path: python/sglang/kernels/aot/dist/
merge-multiple: true
pattern: wheel-python3.10-cuda*
pattern: wheel-python3.11-cuda*
- uses: ./.github/actions/download-rust-ext
with:
@@ -95,7 +95,7 @@ jobs:
with:
path: python/sglang/kernels/aot/dist/
merge-multiple: true
pattern: wheel-python3.10-cuda*
pattern: wheel-python3.11-cuda*
- uses: ./.github/actions/download-rust-ext
with:
@@ -146,7 +146,7 @@ jobs:
with:
path: python/sglang/kernels/aot/dist/
merge-multiple: true
pattern: wheel-python3.10-cuda*
pattern: wheel-python3.11-cuda*
- uses: ./.github/actions/download-rust-ext
with:
@@ -181,7 +181,7 @@ jobs:
# with:
# path: python/sglang/kernels/aot/dist/
# merge-multiple: true
# pattern: wheel-python3.10-cuda*
# pattern: wheel-python3.11-cuda*
# - name: Install dependencies
# run: |
+108 -85
View File
@@ -6,10 +6,10 @@ name: PR Test (sgl-router)
# fires on every PR synchronize / push, so the run record always
# appears as a check on the PR (no more "workflow silently didn't
# fire" mystery debugging). Path-based skip decisions are made
# inside the `sgl-router-gate` job below, where we can log the
# inside the `sgl-router-changes` job below, where we can log the
# reason. Same for the `run-ci` label gate — moved off the
# workflow trigger and into the gate job so its outcome is
# observable.
# observable, and now fails (rather than skips) when it is missing.
#
# * `push` on `main` keeps firing so post-merge runs still record
# against the default branch.
@@ -29,21 +29,17 @@ env:
SGLANG_IS_IN_CI: true
jobs:
# Stage 0 — decide whether to run the heavy tiers. Always runs
# (cheap, ubuntu-latest), and emits a single `should_run` output
# consumed by every tier below. Reasons it might evict downstream
# work:
# * `pull_request` event without the `run-ci` label (budget gate)
# * No files matching the sgl-router paths changed in this PR
# Either outcome is logged on the gate job's page, so operators can
# see *why* the tiers were skipped instead of guessing.
sgl-router-gate:
name: gate
# Stage 0a — path detection. Always runs (cheap, ubuntu-latest) and
# emits a single `should_run` output consumed by every tier below.
# It only answers "does this PR touch sgl-router?"; the `run-ci`
# label is checked by the separate gate job so that a missing label
# fails that job instead of silently skipping the tiers (see below).
sgl-router-changes:
name: changes
runs-on: ubuntu-latest
timeout-minutes: 5
outputs:
should_run: ${{ steps.decide.outputs.should_run }}
paths_changed: ${{ steps.paths.outputs.sgl_router }}
has_run_ci_label: ${{ steps.label.outputs.has_run_ci }}
steps:
- uses: actions/checkout@v4
# `dorny/paths-filter` computes the diff against the PR base
@@ -53,53 +49,62 @@ jobs:
# the gate is observable.
- name: Detect sgl-router path changes
id: paths
if: github.event_name != 'workflow_dispatch'
uses: dorny/paths-filter@v3
with:
filters: |
sgl_router:
- 'experimental/sgl-router/**'
- 'sgl-router/**'
- 'rust/Cargo.toml'
- 'rust/sglang-processor/**'
- 'rust/sglang-api-types/**'
- 'proto/**'
- '.github/workflows/pr-test-sgl-router.yml'
- 'docker/sgl-router.Dockerfile'
- 'scripts/ci/cuda/ci_install_dependency.sh'
# `run-ci` label gate. Cheap workflow runs (e.g. docs-only PRs
# that happen to touch the sgl-router dir) still need the
# opt-in label before consuming the H100 / kind tiers below.
# `push` events on main and `workflow_dispatch` skip this gate.
- name: Check run-ci label
id: label
run: |
if [[ "${{ github.event_name }}" != "pull_request" ]]; then
echo "has_run_ci=true" >> "$GITHUB_OUTPUT"
echo "Non-PR event (${{ github.event_name }}); label gate bypassed."
exit 0
fi
if [[ "${{ contains(github.event.pull_request.labels.*.name, 'run-ci') }}" == "true" ]]; then
echo "has_run_ci=true" >> "$GITHUB_OUTPUT"
echo "PR has run-ci label; downstream tiers will run."
else
echo "has_run_ci=false" >> "$GITHUB_OUTPUT"
echo "::warning::PR is missing the 'run-ci' label; skipping sgl-router tiers. Apply the label to opt in."
fi
- name: Decide
id: decide
env:
EVENT_NAME: ${{ github.event_name }}
PATHS_CHANGED: ${{ steps.paths.outputs.sgl_router }}
run: |
paths='${{ steps.paths.outputs.sgl_router }}'
label='${{ steps.label.outputs.has_run_ci }}'
if [[ "$paths" == "true" && "$label" == "true" ]]; then
if [[ "$EVENT_NAME" == "workflow_dispatch" ]]; then
echo "should_run=true" >> "$GITHUB_OUTPUT"
echo "Both path filter and run-ci label match; running tiers."
else
echo "Manual dispatch; running sgl-router tiers."
elif [[ "$PATHS_CHANGED" != "true" ]]; then
echo "should_run=false" >> "$GITHUB_OUTPUT"
echo "Skipping tiers (paths_changed=$paths, has_run_ci=$label)."
echo "No sgl-router-related changes; skipping tiers."
else
echo "should_run=true" >> "$GITHUB_OUTPUT"
echo "sgl-router-related changes detected; running tiers."
fi
# Stage 0b — `run-ci` label gate, shared with `pr-test.yml`. Cheap
# workflow runs (e.g. docs-only PRs that happen to touch the
# sgl-router dir) still need the opt-in label before consuming the
# H100 / kind tiers below. `push` on main and `workflow_dispatch`
# pass straight through.
#
# `pr-gate.yml` fetches the PR's labels live and FAILS when `run-ci`
# is missing. Both matter for `/tag-and-rerun-ci`: the slash handler
# adds the label with GITHUB_TOKEN, which fires no `labeled` event,
# then reruns the PR's unsuccessful runs. A rerun replays the original
# event payload, so a gate reading `github.event.pull_request.labels`
# would skip again, and a run that concluded `success` with every
# tier skipped is never rerun at all. Failing here on the live label
# set makes the rerun both happen and pick the label up.
sgl-router-gate:
name: gate
needs: sgl-router-changes
if: needs.sgl-router-changes.outputs.should_run == 'true'
uses: ./.github/workflows/pr-gate.yml
with:
require-run-ci: true
sgl-router-lint:
name: tier-1 — lint
needs: sgl-router-gate
if: needs.sgl-router-gate.outputs.should_run == 'true'
needs: [sgl-router-changes, sgl-router-gate]
if: needs.sgl-router-changes.outputs.should_run == 'true'
runs-on: ubuntu-latest
env:
# Scoped per-job (not workflow-wide) to avoid SMG's documented breakage
@@ -133,22 +138,22 @@ jobs:
- name: Rust cache
uses: Swatinem/rust-cache@v2
with:
workspaces: experimental/sgl-router
workspaces: sgl-router
shared-key: "sgl-router-cache"
cache-all-crates: true
cache-on-failure: true
save-if: ${{ github.ref == 'refs/heads/main' }}
- name: cargo check
working-directory: experimental/sgl-router
working-directory: sgl-router
run: cargo check --locked --workspace --all-targets
- name: cargo clippy
working-directory: experimental/sgl-router
working-directory: sgl-router
run: cargo clippy --locked --workspace --all-targets -- -D warnings
- name: cargo fmt
working-directory: experimental/sgl-router
working-directory: sgl-router
run: |
rustup toolchain install nightly --profile minimal --component rustfmt
cargo +nightly fmt --all -- --check
@@ -158,7 +163,7 @@ jobs:
run: cargo install --locked cargo-deny@0.19.6
- name: cargo deny check
working-directory: experimental/sgl-router
working-directory: sgl-router
run: cargo deny check licenses bans sources advisories
- name: Show sccache stats
@@ -167,8 +172,8 @@ jobs:
sgl-router-build-and-test:
name: tier-2 — build + test
needs: [sgl-router-gate, sgl-router-lint]
if: needs.sgl-router-gate.outputs.should_run == 'true'
needs: [sgl-router-changes, sgl-router-gate, sgl-router-lint]
if: needs.sgl-router-changes.outputs.should_run == 'true'
runs-on: ubuntu-latest
env:
RUSTC_WRAPPER: sccache
@@ -206,14 +211,14 @@ jobs:
- name: Rust cache
uses: Swatinem/rust-cache@v2
with:
workspaces: experimental/sgl-router
workspaces: sgl-router
shared-key: "sgl-router-cache"
cache-all-crates: true
cache-on-failure: true
save-if: ${{ github.ref == 'refs/heads/main' }}
- name: cargo build (record wall time)
working-directory: experimental/sgl-router
working-directory: sgl-router
run: |
START=$(date +%s)
cargo build --locked --release --all-targets
@@ -223,7 +228,7 @@ jobs:
echo "::notice title=Tier-2 build duration::${BUILD_SECONDS}s"
- name: cargo test
working-directory: experimental/sgl-router
working-directory: sgl-router
# Skip the tokenizer parity_matrix test here: ubuntu-latest has
# no HuggingFace cache, so the matrix would hard-fail (see the
# test docstring). The e2e job runs it after pytest populates
@@ -245,13 +250,13 @@ jobs:
# through to a no-diff "pass" when no fixture was actually
# regenerated.
- name: kv_events hash parity fixture (drift check)
working-directory: experimental/sgl-router
working-directory: sgl-router
run: |
set -euo pipefail
python3 tests/scripts/generate_kv_events_hash_parity.py
if ! git diff --exit-code -- tests/fixtures/kv_events_hash_parity.json; then
echo "::error::kv_events hash parity fixture drifted." \
"Re-run experimental/sgl-router/tests/scripts/generate_kv_events_hash_parity.py" \
"Re-run sgl-router/tests/scripts/generate_kv_events_hash_parity.py" \
"and commit the updated tests/fixtures/kv_events_hash_parity.json."
exit 1
fi
@@ -268,8 +273,8 @@ jobs:
sgl-router-docker-build-test:
name: tier-3 — docker (placeholder)
needs: [sgl-router-gate, sgl-router-build-and-test]
if: needs.sgl-router-gate.outputs.should_run == 'true'
needs: [sgl-router-changes, sgl-router-gate, sgl-router-build-and-test]
if: needs.sgl-router-changes.outputs.should_run == 'true'
runs-on: ubuntu-24.04
timeout-minutes: 15
steps:
@@ -286,8 +291,8 @@ jobs:
# the GPU/cluster runners from spinning up.
sgl-router-k8s-integration:
name: tier-3 — k8s integration
needs: [sgl-router-gate, sgl-router-build-and-test]
if: needs.sgl-router-gate.outputs.should_run == 'true'
needs: [sgl-router-changes, sgl-router-gate, sgl-router-build-and-test]
if: needs.sgl-router-changes.outputs.should_run == 'true'
runs-on: ubuntu-22.04
timeout-minutes: 30
steps:
@@ -303,20 +308,20 @@ jobs:
curl -Lo /tmp/kubectl https://dl.k8s.io/release/v1.31.0/bin/linux/amd64/kubectl
chmod +x /tmp/kubectl && sudo mv /tmp/kubectl /usr/local/bin/kubectl
- name: Build router image
run: docker build -t sgl-router:e2e -f experimental/sgl-router/tests/e2e/k8s_integration/Dockerfile.router .
run: docker build -t sgl-router:e2e -f sgl-router/tests/e2e/k8s_integration/Dockerfile.router .
- name: Build fake_worker image
run: |
docker build -t sgl-router-fake-worker:e2e \
-f experimental/sgl-router/tests/e2e/k8s_integration/Dockerfile.fake_worker \
experimental/sgl-router/tests/e2e/k8s_integration/
-f sgl-router/tests/e2e/k8s_integration/Dockerfile.fake_worker \
sgl-router/tests/e2e/k8s_integration/
- name: Bootstrap kind + deploy
run: bash experimental/sgl-router/tests/e2e/k8s_integration/setup.sh
run: bash sgl-router/tests/e2e/k8s_integration/setup.sh
- name: Set up Python
run: |
python3 -m venv /tmp/e2e-venv
/tmp/e2e-venv/bin/pip install -r experimental/sgl-router/tests/e2e/k8s_integration/requirements.txt
/tmp/e2e-venv/bin/pip install -r sgl-router/tests/e2e/k8s_integration/requirements.txt
- name: Run E2E
run: /tmp/e2e-venv/bin/pytest experimental/sgl-router/tests/e2e/k8s_integration/ -v --tb=short
run: /tmp/e2e-venv/bin/pytest sgl-router/tests/e2e/k8s_integration/ -v --tb=short
- name: Dump cluster + router diagnostics on failure
if: failure()
run: |
@@ -333,8 +338,8 @@ jobs:
sgl-router-e2e:
name: tier-3 — e2e
needs: [sgl-router-gate, sgl-router-build-and-test]
if: needs.sgl-router-gate.outputs.should_run == 'true'
needs: [sgl-router-changes, sgl-router-gate, sgl-router-build-and-test]
if: needs.sgl-router-changes.outputs.should_run == 'true'
runs-on: 2-gpu-h100
timeout-minutes: 45
steps:
@@ -344,10 +349,10 @@ jobs:
- name: Rust cache
uses: Swatinem/rust-cache@v2
with:
workspaces: experimental/sgl-router
workspaces: sgl-router
shared-key: "sgl-router-cache"
- name: cargo build (release)
working-directory: experimental/sgl-router
working-directory: sgl-router
run: |
source "$HOME/.cargo/env"
cargo build --locked --release --workspace
@@ -365,7 +370,7 @@ jobs:
- name: Install e2e test dependencies
run: |
python3 -m pip install -r experimental/sgl-router/tests/e2e/requirements.txt
python3 -m pip install -r sgl-router/tests/e2e/requirements.txt
# IMPORTANT: --ignore=tests/e2e/k8s_integration. The k8s integration
# suite is owned by the `sgl-router-k8s-integration` job which has
@@ -376,7 +381,7 @@ jobs:
# image pull) must not surface as e2e failures on H100, and e2e
# flakes (model load, HF auth) must not surface as k8s failures.
- name: Run e2e
working-directory: experimental/sgl-router
working-directory: sgl-router
env:
HF_TOKEN: ${{ secrets.HF_TOKEN }}
run: |
@@ -384,7 +389,7 @@ jobs:
--ignore=tests/e2e/k8s_integration
- name: Tokenizer parity matrix (uses HF cache populated by e2e)
working-directory: experimental/sgl-router
working-directory: sgl-router
env:
HF_TOKEN: ${{ secrets.HF_TOKEN }}
run: |
@@ -394,25 +399,43 @@ jobs:
sgl-router-finish:
name: finish
needs:
- sgl-router-changes
- sgl-router-gate
- sgl-router-lint
- sgl-router-build-and-test
- sgl-router-docker-build-test
- sgl-router-k8s-integration
- sgl-router-e2e
# `always()` lets `finish` run after upstream `skipped` outcomes
# (when the gate decided to skip). The downstream-skipped path is
# an explicit success — the workflow was correctly bypassed for a
# PR that didn't need full CI. The downstream-failure /
# downstream-cancelled path is a real fail. We also accept the
# gate's `skipped` (impossible today, kept for symmetry) and
# require the gate itself to have succeeded.
if: |
always() &&
needs.sgl-router-gate.result == 'success' &&
!contains(needs.*.result, 'failure') &&
!contains(needs.*.result, 'cancelled')
# `always()` lets `finish` run after upstream `skipped` outcomes.
# Two outcomes are a success: every tier passed, or the PR touches
# no sgl-router path and every tier was skipped. Anything else,
# including a gate that failed for a missing `run-ci` label, fails
# the run, so `/tag-and-rerun-ci` sees an unsuccessful run to rerun.
if: always()
runs-on: ubuntu-latest
timeout-minutes: 5
steps:
- name: All required checks completed
run: echo "OK"
- name: Check required results
env:
CHANGES_RESULT: ${{ needs.sgl-router-changes.result }}
SHOULD_RUN: ${{ needs.sgl-router-changes.outputs.should_run }}
GATE_RESULT: ${{ needs.sgl-router-gate.result }}
LINT_RESULT: ${{ needs.sgl-router-lint.result }}
BUILD_RESULT: ${{ needs.sgl-router-build-and-test.result }}
DOCKER_RESULT: ${{ needs.sgl-router-docker-build-test.result }}
K8S_RESULT: ${{ needs.sgl-router-k8s-integration.result }}
E2E_RESULT: ${{ needs.sgl-router-e2e.result }}
run: |
if [[ "$CHANGES_RESULT" != "success" ]]; then
echo "::error::sgl-router path detection did not succeed ($CHANGES_RESULT)."
exit 1
fi
tiers="$GATE_RESULT $LINT_RESULT $BUILD_RESULT $DOCKER_RESULT $K8S_RESULT $E2E_RESULT"
if [[ "$SHOULD_RUN" == "true" && "$tiers" == "success success success success success success" ]]; then
echo "All sgl-router checks passed."
elif [[ "$SHOULD_RUN" == "false" && "$tiers" == "skipped skipped skipped skipped skipped skipped" ]]; then
echo "No sgl-router-related changes; tiers skipped."
else
echo "::error::Unexpected sgl-router results: should_run=$SHOULD_RUN, gate=$GATE_RESULT, lint=$LINT_RESULT, build=$BUILD_RESULT, docker=$DOCKER_RESULT, k8s=$K8S_RESULT, e2e=$E2E_RESULT."
exit 1
fi
+66 -43
View File
@@ -93,31 +93,27 @@ jobs:
suite: "stage-a-tp-test-cpu-intel"
name: gnr
# Three SPR boxes each run ONE full-machine container taking one
# partition of a 3-way split over the merged non-TP pool
# (base-b non-TP + base-c). Single container per box (no socket
# split): sglang auto-computes mem_fraction_static from host memory
# and does not honor cgroup limits, so two parallel containers on
# one box exhaust host RAM and OOM (tracked upstream). One container
# per box keeps each server within the box's memory budget.
# Each SPR box (2 sockets / 2 NUMA, 32 physical cores each) runs TWO
# socket-pinned containers in parallel, each taking one partition of a
# 4-way split over the merged non-TP pool. Each container is capped
# with --memory so sglang (which reads cgroup memory.max, PR #38260 +
# the empty-NUMA-node fix) sizes its KV pool against one socket's
# budget, not the whole host -- so two parallel servers no longer
# exhaust host RAM. 2 boxes x 2 sockets = 4 partitions.
- runner: xeon-spr
role: spr
suite: "stage-a-test-cpu-intel,stage-b-test-cpu-intel"
name: spr1
part_id: 0
part_size: 3
part_lo: 0
part_hi: 1
part_size: 4
- runner: xeon-spr
role: spr
suite: "stage-a-test-cpu-intel,stage-b-test-cpu-intel"
name: spr2
part_id: 1
part_size: 3
- runner: xeon-spr
role: spr
suite: "stage-a-test-cpu-intel,stage-b-test-cpu-intel"
name: spr3
part_id: 2
part_size: 3
part_lo: 2
part_hi: 3
part_size: 4
steps:
- name: Cleanup workspace
run: |
@@ -169,43 +165,70 @@ jobs:
timeout-minutes: 120
run: |
docker exec -w /sglang-checkout/ ci_sglang_${{ matrix.name }} \
bash -c "source /opt/.venv/bin/activate && cd ./test && python3 run_suite.py --hw cpu --suite ${{ matrix.suite }} --timeout-from-est-time"
bash -c "source /opt/.venv/bin/activate && hf auth login --token \$HF_TOKEN && cd ./test && python3 run_suite.py --hw cpu --suite ${{ matrix.suite }} --timeout-from-est-time"
# ============ SPR: single full-machine container, one partition ====== #
- name: Run container (SPR)
if: matrix.role == 'spr'
run: |
docker rm -f ci_sglang_${{ matrix.name }} || true
docker run -dt \
-v ${{ github.workspace }}:/sglang-checkout/ --shm-size=64g \
-v $HOME/.cache/huggingface:/root/.cache/huggingface \
-v $HOME/.cache/sgl_eval:/root/.cache/sgl_eval \
-e HF_TOKEN="$(cat ~/huggingface_token.txt)" \
--name ci_sglang_${{ matrix.name }} \
sglang_xeon
- name: Check AMX support (SPR)
if: matrix.role == 'spr'
timeout-minutes: 5
run: |
docker exec -w /sglang-checkout/ ci_sglang_${{ matrix.name }} \
bash -c "source /opt/.venv/bin/activate && python3 -c 'import torch; import sgl_kernel; assert torch.cpu._is_amx_tile_supported(); assert hasattr(torch.ops.sgl_kernel, \"convert_weight_packed\"); '"
- name: Run unit tests (SPR)
# ========= SPR: two socket-pinned containers, run in parallel ======== #
- name: Run unit tests (SPR, two sockets in parallel)
if: matrix.role == 'spr'
timeout-minutes: 120
run: |
docker exec -w /sglang-checkout/ ci_sglang_${{ matrix.name }} \
bash -c "source /opt/.venv/bin/activate && cd ./test && python3 run_suite.py --hw cpu --suite ${{ matrix.suite }} --auto-partition-id ${{ matrix.part_id }} --auto-partition-size ${{ matrix.part_size }} --timeout-from-est-time"
set -uo pipefail
HF_TOKEN_VAL="$(cat ~/huggingface_token.txt)"
# One socket-pinned container. Physical cores only (0-31 = socket0,
# 32-63 = socket1); logical/HT siblings (64-127) are never bound.
# --memory caps the cgroup so sglang sizes KV against one socket, and
# --cpuset-mems keeps allocations on the local memory bank.
# $1 socket id, $2 cpuset-cpus, $3 cpuset-mems, $4 partition id
run_socket () {
local sid=$1 cpus=$2 mems=$3 pid=$4
local cname=ci_sglang_${{ matrix.name }}_s${sid}
docker rm -f "$cname" >/dev/null 2>&1 || true
docker run -dt \
--cpuset-cpus="$cpus" --cpuset-mems="$mems" --memory=240g \
-v ${{ github.workspace }}:/sglang-checkout/ --shm-size=32g \
-v $HOME/.cache/huggingface:/root/.cache/huggingface \
-v $HOME/.cache/sgl_eval:/root/.cache/sgl_eval \
-e HF_TOKEN="$HF_TOKEN_VAL" \
-e SGLANG_CPU_OMP_THREADS_BIND="$cpus" \
--name "$cname" \
sglang_xeon
docker exec -w /sglang-checkout/ "$cname" bash -c \
"source /opt/.venv/bin/activate && python3 -c 'import torch; import sgl_kernel; assert torch.cpu._is_amx_tile_supported(); assert hasattr(torch.ops.sgl_kernel, \"convert_weight_packed\")' && \
hf auth login --token \$HF_TOKEN && \
cd ./test && python3 run_suite.py --hw cpu --suite ${{ matrix.suite }} --auto-partition-id $pid --auto-partition-size ${{ matrix.part_size }} --timeout-from-est-time"
}
# socket0 -> part_lo, socket1 -> part_hi; run concurrently, wait both.
run_socket 0 "0-31" 0 ${{ matrix.part_lo }} > s0.log 2>&1 &
P0=$!
run_socket 1 "32-63" 1 ${{ matrix.part_hi }} > s1.log 2>&1 &
P1=$!
rc=0
wait $P0 || rc=1
wait $P1 || rc=1
echo "==================== socket0 (partition ${{ matrix.part_lo }}) ===================="
cat s0.log || true
echo "==================== socket1 (partition ${{ matrix.part_hi }}) ===================="
cat s1.log || true
exit $rc
# ============================ Common ================================= #
- name: Change permission
if: always()
timeout-minutes: 2
run: |
docker exec -u root ci_sglang_${{ matrix.name }} bash -c "rm -rf /tmp/ci-home && chown -R $(id -u):$(id -g) /sglang-checkout/ 2>/dev/null || true" 2>/dev/null || true
# GNR has one container (ci_sglang_<name>); SPR has two socket
# containers (ci_sglang_<name>_s0/_s1). Chown from whichever exist.
for c in ci_sglang_${{ matrix.name }} ci_sglang_${{ matrix.name }}_s0 ci_sglang_${{ matrix.name }}_s1; do
docker exec -u root "$c" bash -c "rm -rf /tmp/ci-home && chown -R $(id -u):$(id -g) /sglang-checkout/ 2>/dev/null || true" 2>/dev/null || true
done
- name: Cleanup container
if: always()
run: |
docker rm -f ci_sglang_${{ matrix.name }} || true
for c in ci_sglang_${{ matrix.name }} ci_sglang_${{ matrix.name }}_s0 ci_sglang_${{ matrix.name }}_s1; do
docker rm -f "$c" || true
done
+1
View File
@@ -446,6 +446,7 @@ jobs:
fail-fast: false
matrix:
include:
- { runner_config: 2-gpu-gb300, run_timeout: '30', timeout_per_file: '1800' }
- { runner_config: 4-gpu-gb300, run_timeout: '30', timeout_per_file: '1800' }
uses: ./.github/workflows/_pr-test-stage.yml
with:
@@ -2,10 +2,20 @@ name: Release Docker Images Nightly gfx1151 (AMD)
on:
workflow_dispatch:
inputs:
push:
description: "Push the image to Docker Hub (uncheck to only check that the branch builds)"
required: false
type: boolean
default: true
schedule:
# Stagger this build from the CDNA ROCm nightlies at 12:00 UTC.
- cron: "0 14 * * *"
# Scheduled runs always push; a manual dispatch can turn the push off.
env:
PUSH_IMAGES: ${{ github.event_name != 'workflow_dispatch' || inputs.push }}
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
@@ -59,5 +69,9 @@ jobs:
--no-cache \
--progress=plain \
-t "rocm/sgl-dev:${IMAGE_TAG}"
docker push "rocm/sgl-dev:${IMAGE_TAG}"
echo "Published rocm/sgl-dev:${IMAGE_TAG}" >> "$GITHUB_STEP_SUMMARY"
if [ "${PUSH_IMAGES}" = "true" ]; then
docker push "rocm/sgl-dev:${IMAGE_TAG}"
echo "Published rocm/sgl-dev:${IMAGE_TAG}" >> "$GITHUB_STEP_SUMMARY"
else
echo "Build-only run: not pushing rocm/sgl-dev:${IMAGE_TAG}" >> "$GITHUB_STEP_SUMMARY"
fi
@@ -21,8 +21,17 @@ on:
- 'all'
- gfx942-rocm1000
- gfx950-rocm1000
push:
description: 'Push the images to Docker Hub (uncheck to only check that the branch builds)'
required: false
type: boolean
default: true
schedule:
- cron: '0 12 * * *'
- cron: '0 10 * * *' # before the ROCm 7.2 build, so it is not queued behind it
# Scheduled runs always push; a manual dispatch can turn the push off.
env:
PUSH_IMAGES: ${{ github.event_name != 'workflow_dispatch' || inputs.push }}
concurrency:
group: ${{ github.workflow }}-${{ github.event.number || github.sha }}
@@ -106,15 +115,21 @@ jobs:
--build-arg UBUNTU_MIRROR=https://archive.ubuntu.com \
-t "rocm/sgl-dev:${image_tag}" \
--no-cache
docker push "rocm/sgl-dev:${image_tag}"
if [ "${PUSH_IMAGES}" = "true" ]; then
docker push "rocm/sgl-dev:${image_tag}"
else
echo "Build-only run: not pushing rocm/sgl-dev:${image_tag}"
fi
- name: Login to Docker Hub (lmsys)
if: env.PUSH_IMAGES == 'true'
uses: docker/login-action@v3
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Push to lmsysorg/sglang-rocm
if: env.PUSH_IMAGES == 'true'
run: |
docker tag "rocm/sgl-dev:${IMAGE_TAG}" "lmsysorg/sglang-rocm:${IMAGE_TAG}"
docker push "lmsysorg/sglang-rocm:${IMAGE_TAG}"
@@ -178,15 +193,21 @@ jobs:
--build-arg UBUNTU_MIRROR=https://archive.ubuntu.com \
-t "rocm/sgl-dev:${image_tag}" \
--no-cache
docker push "rocm/sgl-dev:${image_tag}"
if [ "${PUSH_IMAGES}" = "true" ]; then
docker push "rocm/sgl-dev:${image_tag}"
else
echo "Build-only run: not pushing rocm/sgl-dev:${image_tag}"
fi
- name: Login to Docker Hub (lmsys)
if: env.PUSH_IMAGES == 'true'
uses: docker/login-action@v3
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Push to lmsysorg/sglang-rocm
if: env.PUSH_IMAGES == 'true'
run: |
docker tag "rocm/sgl-dev:${IMAGE_TAG}" "lmsysorg/sglang-rocm:${IMAGE_TAG}"
docker push "lmsysorg/sglang-rocm:${IMAGE_TAG}"
@@ -253,15 +274,21 @@ jobs:
--build-arg UBUNTU_MIRROR=https://archive.ubuntu.com \
-t "rocm/sgl-dev:${image_tag}" \
--no-cache
docker push "rocm/sgl-dev:${image_tag}"
if [ "${PUSH_IMAGES}" = "true" ]; then
docker push "rocm/sgl-dev:${image_tag}"
else
echo "Build-only run: not pushing rocm/sgl-dev:${image_tag}"
fi
- name: Login to Docker Hub (lmsys)
if: env.PUSH_IMAGES == 'true'
uses: docker/login-action@v3
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Push to lmsysorg/sglang-rocm
if: env.PUSH_IMAGES == 'true'
run: |
docker tag "rocm/sgl-dev:${IMAGE_TAG}" "lmsysorg/sglang-rocm:${IMAGE_TAG}"
docker push "lmsysorg/sglang-rocm:${IMAGE_TAG}"
@@ -21,8 +21,17 @@ on:
- gfx950-rocm724
- gfx942-rocm720
- gfx950-rocm720
push:
description: 'Push the images to Docker Hub (uncheck to only check that the branch builds)'
required: false
type: boolean
default: true
schedule:
- cron: '0 12 * * *'
- cron: '0 11 * * *' # an hour after ROCm 10, ahead of the 12:00 UTC MILES builds
# Scheduled runs always push; a manual dispatch can turn the push off.
env:
PUSH_IMAGES: ${{ github.event_name != 'workflow_dispatch' || inputs.push }}
concurrency:
# A PR number if a pull request and otherwise the commit hash. This cancels
@@ -112,18 +121,24 @@ jobs:
# UBUNTU_MIRROR forces apt over HTTPS to dodge port-80 reachability flakes
# to Canonical's archive.ubuntu.com mirror IPs from the amd-docker-scale runner.
docker build . -f docker/rocm.Dockerfile --build-arg SGL_BRANCH=${{ github.ref_name }} --build-arg BUILD_TYPE=${{ matrix.build_type }} --build-arg GPU_ARCH=${{ matrix.gpu_arch }} --build-arg ENABLE_MORI=1 --build-arg ENABLE_NIXL=1 --build-arg SETUPTOOLS_SCM_PRETEND_VERSION=${pretend_version} --build-arg UBUNTU_MIRROR=https://archive.ubuntu.com -t rocm/sgl-dev:${tag}-${{ env.DATE }} --no-cache
docker push rocm/sgl-dev:${tag}-${{ env.DATE }}
if [ "${PUSH_IMAGES}" = "true" ]; then
docker push rocm/sgl-dev:${tag}-${{ env.DATE }}
else
echo "Build-only run: not pushing rocm/sgl-dev:${tag}-${{ env.DATE }}"
fi
# Persist the tag right after rocm/sgl-dev push succeeds so the local
# registry mirror can run even if a later step in this job (lmsys push)
# fails. By default this step only runs when the previous step succeeded,
# so the artifact only exists when an image actually landed on Docker Hub.
- name: Save published image tag
if: env.PUSH_IMAGES == 'true'
run: |
mkdir -p image-tag
echo "${{ env.IMAGE_TAG }}" > "image-tag/${{ matrix.gpu_arch }}.txt"
- name: Upload image tag artifact
if: env.PUSH_IMAGES == 'true'
uses: actions/upload-artifact@v4
with:
name: image-tag-${{ matrix.gpu_arch }}
@@ -131,12 +146,14 @@ jobs:
retention-days: 1
- name: Login to Docker Hub (lmsys)
if: env.PUSH_IMAGES == 'true'
uses: docker/login-action@v2
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Push to lmsysorg/sglang-rocm
if: env.PUSH_IMAGES == 'true'
run: |
docker tag rocm/sgl-dev:${{ env.IMAGE_TAG }} lmsysorg/sglang-rocm:${{ env.IMAGE_TAG }}
docker push lmsysorg/sglang-rocm:${{ env.IMAGE_TAG }}
@@ -57,6 +57,7 @@ jobs:
kernel_source=wheel
kernel_ref=main
latest_tag=latest
image_tag="${image_tag}-main-whl"
else
# Pin the kernel commit so the tag names exactly what was built.
kernel_ref=$(git ls-remote https://github.com/sgl-project/sgl-kernel-xpu.git refs/heads/main | cut -f1)
@@ -66,7 +67,7 @@ jobs:
fi
kernel_source=source
latest_tag=latest-kernel-main
image_tag="${image_tag}-kernel-main-${kernel_ref:0:7}"
image_tag="${image_tag}-main-main-${kernel_ref:0:7}"
fi
echo "DATE=${date_tag}" >> $GITHUB_ENV
echo "commit_hash=${commit_hash}" >> $GITHUB_OUTPUT
@@ -83,7 +83,7 @@ jobs:
provenance: false
sbom: false
build-args: |
SGLANG_KERNEL_NPU_TAG=2026.9.0.post9
SGLANG_KERNEL_NPU_TAG=2026.9.0.post10
CANN_VERSION=${{ matrix.cann_version }}
DEVICE_TYPE=${{ matrix.device_type }}
+1 -1
View File
@@ -87,7 +87,7 @@ jobs:
push: ${{ github.repository == 'sgl-project/sglang' && github.event_name != 'pull_request' }}
provenance: false
build-args: |
SGLANG_KERNEL_NPU_TAG=2026.9.0.post9
SGLANG_KERNEL_NPU_TAG=2026.9.0.post10
CANN_VERSION=${{ matrix.cann_version }}
DEVICE_TYPE=${{ matrix.device_type }}
SGLANG_TAG=${{ steps.version.outputs.version }}
+1 -1
View File
@@ -161,7 +161,7 @@ jobs:
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: "3.10"
python-version: "3.12"
- name: Update wheel index
run: |
+1 -1
View File
@@ -170,7 +170,7 @@ jobs:
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: "3.10"
python-version: "3.12"
- name: Update wheel index
run: |
+1 -1
View File
@@ -104,7 +104,7 @@ jobs:
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: "3.10"
python-version: "3.12"
- name: Upload to pypi
run: |
+1 -1
View File
@@ -26,7 +26,7 @@ jobs:
strategy:
fail-fast: false
matrix:
python-version: ["3.10", "3.11", "3.12", "3.13"]
python-version: ["3.11", "3.12", "3.13"]
arch: [x86_64, aarch64]
include:
- arch: x86_64
@@ -168,7 +168,6 @@ jobs:
# so bind eagerly to fail here on any missing symbol.
LD_BIND_NOW=1 python3 - <<'PY'
import ctypes
import sys
import torch
import deep_ep
import deep_gemm
@@ -179,9 +178,6 @@ jobs:
assert ctypes.CDLL("libnccl.so.2").ncclGetVersion(ctypes.byref(nccl_version)) == 0
assert nccl_version.value == 23007, nccl_version.value
assert hasattr(deep_ep, "ElasticBuffer"), "DeepEP ElasticBuffer is required"
sys.path.insert(0, "DeepGEMM/third-party")
from tilelang_ops import ref_mhc
assert ref_mhc.has_baseline(), "Mega mHC reference operators are unavailable"
print("DeepGEMM test dependencies ready; NCCL:", nccl_version.value)
PY
+252
View File
@@ -0,0 +1,252 @@
name: Release sgl-flash-mla
on:
workflow_dispatch:
inputs:
version:
description: "Wheel version (e.g. 1.0.0, 1.0.1rc0)"
type: string
required: true
branch:
description: "FlashMLA branch to build from (default: sgl)"
type: string
required: false
default: 'sgl'
validation_only:
description: "Build and test without publishing wheels"
type: boolean
required: false
default: false
skip_tests:
description: "Skip the GPU test suite (publishes untested wheels)"
type: boolean
required: false
default: false
concurrency:
group: release-sgl-flash-mla-${{ github.ref }}
cancel-in-progress: true
env:
# The wheel pins this; it is also what the test jobs install.
# The extensions use PyTorch's stable C++ ABI and would load against any 2.10+,
# but a release is tested against one version.
TORCH_VER: "2.14.1"
jobs:
build:
if: github.repository == 'sgl-project/sglang'
strategy:
matrix:
# One wheel per architecture. No Python matrix: the wheel is abi3
# (cp310-abi3) and serves CPython 3.10+, so 3.12 only drives the build.
python-version: ["3.12"]
cuda-version: ["13.4"]
arch: [x86_64, aarch64]
include:
- arch: x86_64
runner: x64-kernel-build-node
- arch: aarch64
runner: arm-kernel-build-node
runs-on: ${{ matrix.runner }}
steps:
- name: Clean workspace (remove root-owned files from prior runs)
run: |
docker run --rm -v "${{ github.workspace }}:/workspace" alpine:3 \
sh -c 'rm -rf /workspace/..?* /workspace/.[!.]* /workspace/*' || true
- uses: actions/checkout@v4
- name: Checkout FlashMLA
uses: actions/checkout@v4
with:
repository: sgl-project/FlashMLA
ref: ${{ inputs.branch || 'sgl' }}
path: FlashMLA
submodules: recursive
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
- name: Set wheel version
run: |
echo -n "${{ inputs.version }}" > FlashMLA/flash_mla/VERSION
cat FlashMLA/flash_mla/VERSION
- name: Build wheel
run: |
chmod +x ./scripts/build_sgl_flash_mla.sh ./scripts/rename_sgl_flash_mla_whl.sh
./scripts/build_sgl_flash_mla.sh "${{ matrix.python-version }}" "${{ matrix.cuda-version }}" "${{ github.workspace }}/FlashMLA" "${{ matrix.arch }}"
- name: Upload artifacts
uses: actions/upload-artifact@v4
with:
name: flashmla-wheel-${{ matrix.arch }}
path: FlashMLA/dist/*.whl
- name: Upload PyPI artifacts
uses: actions/upload-artifact@v4
with:
name: flashmla-pypi-${{ matrix.arch }}
path: FlashMLA/dist-pypi/*.whl
test:
needs: build
if: ${{ !inputs.skip_tests }}
strategy:
fail-fast: false
matrix:
include:
# SM90 runs only the frozen kernels: the current ones are sm_100a/sm_103a.
# The frozen ones are built for sm_90a/sm_100a/sm_103a, so SM100 runs both.
- arch_label: sm90
runner: 8-gpu-h200
wheel_arch: x86_64
tests: legacy
timeout: 90
- arch_label: sm100
runner: 8-gpu-b200
wheel_arch: x86_64
tests: current legacy
timeout: 180
- arch_label: sm103-aarch64
runner: 4-gpu-gb300
wheel_arch: aarch64
tests: current legacy
timeout: 180
runs-on: ${{ matrix.runner }}
timeout-minutes: ${{ matrix.timeout }}
steps:
- uses: actions/checkout@v4
- name: Checkout FlashMLA
uses: actions/checkout@v4
with:
repository: sgl-project/FlashMLA
ref: ${{ inputs.branch || 'sgl' }}
path: FlashMLA
submodules: recursive
# The tests need only tests/. The source package would shadow the installed
# wheel (its directory has no built extensions), so it is removed.
- name: Remove the FlashMLA source package
run: rm -rf FlashMLA/flash_mla
- name: Download wheel
uses: actions/download-artifact@v4
with:
path: dist/
merge-multiple: true
pattern: flashmla-wheel-${{ matrix.wheel_arch }}
- name: Install wheel and deps
run: |
python3 -m pip install --upgrade pip
python3 -m pip install "torch==${TORCH_VER}" --index-url https://download.pytorch.org/whl/cu130 --force-reinstall
python3 -m pip install numpy einops rich pytest
# --force-reinstall: a base image may already carry this version, in
# which case pip skips the wheel and the job tests a stale extension.
python3 -m pip install dist/*.whl --force-reinstall --no-deps
# Both extensions register through STABLE_TORCH_LIBRARY at dlopen time,
# so a missing symbol surfaces here rather than at the first launch.
python3 -c "
import flash_mla
from flash_mla._loader import get_ops
print('flash_mla:', flash_mla.__file__)
import os
assert not flash_mla.__file__.startswith(os.environ['GITHUB_WORKSPACE']), 'not the installed wheel'
print('version:', flash_mla.__version__)
print('has_legacy:', flash_mla.has_legacy())
assert get_ops(use_legacy=False).sparse_decode_fwd is not None
assert get_ops(use_legacy=True).dense_decode_fwd is not None
"
- name: Run tests (current kernels)
if: contains(matrix.tests, 'current')
working-directory: FlashMLA
env:
PYTHONPATH: ${{ github.workspace }}/FlashMLA/tests
run: |
python3 -c "import os, flash_mla; assert not flash_mla.__file__.startswith(os.environ['GITHUB_WORKSPACE']), flash_mla.__file__"
python3 tests/test-sparse-prefill.py
python3 tests/test-sparse-decode.py
- name: Run tests (frozen kernels)
if: contains(matrix.tests, 'legacy')
working-directory: FlashMLA
env:
# conftest.py only fixes sys.path under pytest; these are scripts.
PYTHONPATH: ${{ github.workspace }}/FlashMLA/tests
run: |
python3 -c "import os, flash_mla; assert not flash_mla.__file__.startswith(os.environ['GITHUB_WORKSPACE']), flash_mla.__file__"
python3 -m pytest tests/legacy/ -q
# Dense MLA decoding is SM90-only.
if [ "${{ matrix.arch_label }}" = sm90 ]; then
python3 tests/legacy/test_flash_mla_dense_decoding.py
fi
python3 tests/legacy/test_flash_mla_sparse_prefill.py
python3 tests/legacy/test_flash_mla_sparse_decoding.py
python3 tests/legacy/test_flash_mla_sparse_decoding_v32_bit_equal.py
release:
if: |
!cancelled() && !inputs.validation_only &&
needs.build.result == 'success' &&
(needs.test.result == 'success' || needs.test.result == 'skipped')
needs: [build, test]
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Download artifacts
uses: actions/download-artifact@v4
with:
path: dist/
merge-multiple: true
pattern: flashmla-wheel-*
- name: Download PyPI artifacts
uses: actions/download-artifact@v4
with:
path: dist-pypi/
merge-multiple: true
pattern: flashmla-pypi-*
- name: Record that the GPU test suite was skipped
if: ${{ inputs.skip_tests }}
run: |
echo "Releasing v${{ inputs.version }} with the GPU test suite skipped." \
>> "$GITHUB_STEP_SUMMARY"
- name: Upload to PyPI
run: |
pip install --upgrade twine "packaging>=24.2"
python3 -m twine upload --skip-existing dist-pypi/* -u __token__ -p ${{ secrets.SGL_FLASH_MLA_PYPI_TOKEN }}
- name: Release
uses: softprops/action-gh-release@v2
with:
tag_name: v${{ inputs.version }}
repository: sgl-project/whl
token: ${{ secrets.GH_PAT_FOR_WHL_RELEASE }}
files: |
dist/*
- name: Clone wheel index
run: git clone https://oauth2:${WHL_TOKEN}@github.com/sgl-project/whl.git sgl-whl
env:
WHL_TOKEN: ${{ secrets.GH_PAT_FOR_WHL_RELEASE }}
- name: Update wheel index
run: python3 scripts/update_flashmla_whl_index.py --cuda 130
- name: Push wheel index
run: |
cd sgl-whl
git config --local user.name "sglang-bot"
git config --local user.email "sglangbot@gmail.com"
git add -A
git commit -m "update sgl-flash-mla whl index for v${{ inputs.version }}${{ inputs.skip_tests && ' (GPU tests skipped)' || '' }}"
git push
+4 -3
View File
@@ -39,7 +39,7 @@ jobs:
(github.event_name == 'push' || github.event.inputs.target == 'all' || github.event.inputs.target == 'cu130')
strategy:
matrix:
python-version: ["3.10"]
python-version: ["3.11"]
cuda-version: ["13.0"]
arch: [x86_64, aarch64]
include:
@@ -174,8 +174,9 @@ jobs:
matrix:
python-version: ["3.10"]
# ROCm 10.0.0 builds inside a Python 3.12 container; the wheel is still
# tagged cp310-abi3 (wheel.py-api in pyproject.toml), so it stays under
# the same python-version leg as the 7.x wheels.
# tagged cp310-abi3 (build_rocm.sh sets wheel.py-api to cp310 for every
# ROCm flavor), so it stays under the same python-version leg as the 7.x
# wheels.
#
# A dispatch that names one flavor builds only that flavor. Every extra
# leg is another image build on the shared AMD node, and its release job
+1 -1
View File
@@ -183,7 +183,7 @@ jobs:
env:
TEST_COMMAND: ${{ matrix.test_command }}
REQUESTED_TEST_COMMAND: ${{ inputs.test_command }}
HF_MUSE_SPARK_TOKEN: ${{ inputs.runs_on == '8-gpu-b300' && inputs.pr_head_sha == '' && contains(matrix.test_command, 'registered/e2e/models/test_muse_spark_12_b300_pd.py') && secrets.HF_MUSE_SPARK_TOKEN || '' }}
HF_MUSE_SPARK_TOKEN: ${{ inputs.pr_head_sha == '' && inputs.runs_on == '8-gpu-b300' && (contains(matrix.test_command, 'registered/e2e/models/test_muse_spark_12_b300_pd.py') || contains(matrix.test_command, 'registered/e2e/models/test_muse_spark_unified_memory.py')) && secrets.HF_MUSE_SPARK_TOKEN || '' }}
SGLANG_PRECISION_HF_TOKEN: ${{ inputs.refresh_precision_baseline && inputs.mode == 'cuda' && inputs.runs_on == '8-gpu-h200' && inputs.test_command == 'registered/debug_utils/test_nightly_precision_regression.py' && inputs.pr_head_sha == '' && secrets.HF_TOKEN_PRECISION_STORE || '' }}
run: |
if [[ "${{ inputs.refresh_precision_baseline }}" == "true" ]]; then
+69
View File
@@ -0,0 +1,69 @@
# Runs only the per-commit CPU unit suite (base-a-test-cpu) against a PR head,
# dispatched by the `/run-cpu-ci` slash command. A cheap check for small PRs:
# pair it with `/rerun-group <module>` instead of a full `run-ci` round.
#
# Always dispatched on main, so the workflow definition and every reusable
# workflow it calls are trusted; only the code under test comes from the PR.
# That code runs on hosted runners with a read-only token and no secrets
# (nothing here inherits them), the same exposure as `/rerun-test` cpu mode.
name: Run CPU CI
run-name: "[run-cpu-ci] ${{ inputs.git_ref }}"
on:
workflow_dispatch:
inputs:
git_ref:
description: "PR head SHA to test"
required: true
type: string
pr_number:
description: "PR number (informational)"
required: false
type: string
default: ""
# Read by _pr-test-stage-cpu.yml via caller_inputs. Its fail-fast health
# check inspects sibling jobs of a pull_request run, which this is not.
skip_pr_test_health_check:
description: "Skip the PR test health check fail-fast"
required: false
type: boolean
default: true
# Also read via caller_inputs. This run is on main's ref, so a cache it
# saved would be restored by main and every PR, yet hold PR-written files.
skip_cache_save:
description: "Do not save the HF / Rust caches"
required: false
type: boolean
default: true
concurrency:
group: run-cpu-ci-${{ inputs.git_ref }}
cancel-in-progress: true
permissions:
contents: read
issues: read
pull-requests: read
jobs:
# pr_test_yml stays pr-test.yml: compute_partitions.py reads it from the PR
# checkout, which may predate this file, and sizing shards off the per-commit
# stage's run_timeout_minutes keeps the split identical to PR CI.
check-changes:
uses: ./.github/workflows/_pr-test-check-changes.yml
with:
git_ref: ${{ inputs.git_ref }}
run_all_tests: true
# self_name must be the suite name: the stage passes it to run_suite.py.
# No rust_ext_artifact: its compile job runs on a self-hosted build node, so
# the stage falls back to its cache and compiles on a miss.
base-a-test-cpu:
needs: check-changes
uses: ./.github/workflows/_pr-test-stage-cpu.yml
with:
self_name: base-a-test-cpu
check_changes: ${{ toJson(needs.check-changes.outputs) }}
caller_inputs: ${{ toJson(inputs) }}
partitions: ${{ needs.check-changes.outputs.partitions }}
run_timeout_minutes: '15'
+3 -2
View File
@@ -7,6 +7,7 @@ on:
paths:
- '.github/workflows/runner-utilization.yml'
- 'scripts/ci/utils/runner_utilization_report.py'
- 'scripts/ci/utils/test_runner_utilization_report.py'
workflow_dispatch:
inputs:
hours:
@@ -50,7 +51,7 @@ jobs:
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.10'
python-version: '3.12'
- name: Unit-test report logic
# Pure-logic tests (stdlib only, no API calls) — guards the
@@ -70,7 +71,7 @@ jobs:
run: |
# PR-trigger is just a script smoke check; scan a 20-minute window
# (0.34h) to bound API cost. schedule / dispatch run the full report.
python scripts/ci/utils/runner_utilization_report.py \
python -u scripts/ci/utils/runner_utilization_report.py \
--repo ${{ github.repository }} \
--hours ${{ (github.event_name == 'pull_request' && '0.34') || inputs.hours || '24' }} \
${{ inputs.filter && format('--filter {0}', inputs.filter) || '' }} \
+3 -1
View File
@@ -27,6 +27,8 @@ jobs:
contains(github.event.comment.body, '/run-extra-ci') ||
contains(github.event.comment.body, '/rerun-full-ci') ||
contains(github.event.comment.body, '/rerun-extra-ci') ||
contains(github.event.comment.body, '/run-cpu-ci') ||
contains(github.event.comment.body, '/rerun-cpu-ci') ||
contains(github.event.comment.body, '/rerun-group') ||
contains(github.event.comment.body, '/rerun-test') ||
startsWith(github.event.comment.body, '/tag-') ||
@@ -92,7 +94,7 @@ jobs:
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.10'
python-version: '3.12'
- name: Install dependencies
run: |
+1 -1
View File
@@ -21,7 +21,7 @@ jobs:
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.10'
python-version: '3.12'
- name: Update est_time values
env:
+3 -2
View File
@@ -73,7 +73,8 @@ instance/
# PyBuilder
.pybuilder/
target/
# `**/` because .dockerignore links here and Docker anchors bare patterns at the context root.
**/target/
# Jupyter Notebook
.ipynb_checkpoints
@@ -232,7 +233,7 @@ work_dirs/
Cargo.lock
!rust/Cargo.lock
!rust/sglang-radix-tree/Cargo.lock
!experimental/sgl-router/Cargo.lock
!sgl-router/Cargo.lock
# Other repos
lmms-eval
+15 -7
View File
@@ -11,10 +11,6 @@ repos:
- id: end-of-file-fixer
- id: check-yaml
args: [--allow-multiple-documents]
# Helm chart templates embed Go template syntax ({{- ... -}})
# that is not valid YAML on its own; the rendered output is
# validated by `helm template` / `helm lint`.
exclude: ^experimental/sgl-router/helm/.*/templates/.*\.(yaml|tpl)$
- id: check-toml
- id: check-ast
- id: check-added-large-files
@@ -74,6 +70,12 @@ repos:
- '--extra-keys=metadata.kernelspec metadata.language_info.version'
- repo: local
hooks:
- id: forbid-test-srt-path
name: forbid test/srt/ (use test/registered/ or test/manual/)
entry: >-
test/srt/ is retired. Move these tests under test/registered/ or test/manual/ instead.
language: fail
files: ^test/srt/
- id: forbid-docs-new-path
name: forbid docs_new/ (renamed to docs/)
# docs_new/ was renamed to docs/ in #32123. `language: fail` fires only
@@ -110,6 +112,12 @@ repos:
language: system
files: ^(\.github/actions/download-rust-ext/action\.yml|\.github/workflows/_pr-test-rust-ext-build\.yml)$
pass_filenames: false
- id: check-dockerfile-wheel-pins
name: check Dockerfile sgl-deep-gemm/sgl-deep-ep versions match python/pyproject.toml
entry: python3 scripts/lint/check_dockerfile_wheel_pins.py
language: system
files: ^(python/pyproject\.toml|docker/.*Dockerfile.*|scripts/lint/check_dockerfile_wheel_pins\.py)$
pass_filenames: false
- id: check-no-bare-pytest-main
name: reject bare pytest.main calls in __main__ blocks
entry: python3 scripts/lint/check_no_bare_pytest_main.py
@@ -134,10 +142,10 @@ repos:
files: ^sgl-model-gateway/.*\.rs$
pass_filenames: false
- id: rustfmt-sgl-router
name: rustfmt experimental/sgl-router
entry: bash -c 'cd experimental/sgl-router && cargo fmt --all -- --check'
name: rustfmt sgl-router
entry: bash -c 'cd sgl-router && cargo fmt --all -- --check'
language: system
files: ^experimental/sgl-router/.*\.rs$
files: ^sgl-router/.*\.rs$
pass_filenames: false
- id: clippy-rust-workspace
name: clippy rust/ workspace (auto-fix)
+15 -1
View File
@@ -56,6 +56,19 @@ if [[ "${ROCM_VERSION}" == "1000" ]]; then
WHEEL_ROCM_VERSION="1000"
fi
# The build reuses the CUDA pyproject.toml, which requires Python 3.11 and tags
# its wheel cp311-abi3. The ROCm 7.2.0 image and amd-sglang still run on Python
# 3.10, so the ROCm wheels keep the Python 3.10 floor and the cp310-abi3 tag.
sed -i \
-e 's/^requires-python = ">=3.11"$/requires-python = ">=3.10"/' \
-e 's/^wheel.py-api = "cp311"$/wheel.py-api = "cp310"/' \
pyproject.toml
if ! grep -qx 'requires-python = ">=3.10"' pyproject.toml \
|| ! grep -qx 'wheel.py-api = "cp310"' pyproject.toml; then
echo "ERROR: could not set the ROCm Python 3.10 / cp310 metadata in pyproject.toml." >&2
exit 1
fi
# Pull and run the latest image
echo "Pulling Docker image: ${IMAGE}"
docker pull "${IMAGE}"
@@ -86,5 +99,6 @@ ${ROCM_SETUP}
rm -rf CMakeLists.txt && mv CMakeLists_rocm.txt CMakeLists.txt && \
${PYTHON_ROOT_PATH}/python rocm_hipify.py && \
${PYTHON_ROOT_PATH}/python -m uv build --wheel -Cbuild-dir=build . --color=always --no-build-isolation && \
./rename_wheels_rocm.sh ${WHEEL_ROCM_VERSION}
./rename_wheels_rocm.sh ${WHEEL_ROCM_VERSION} && \
${PYTHON_ROOT_PATH}/pip install --dry-run --no-deps dist/*.whl
"
+8 -1
View File
@@ -22,6 +22,11 @@ SGLang is an open-source inference framework for large language, vision-language
👋 Get started below, or meet the community at [SGLang Events](https://www.sglang.io/events), including meetups, developer meetings, workshops, and office hours.
> **[SGLang Summit 2026: Meet the builders of open AI](https://www.sglang.io/summit)**<br>
> **Our flagship annual gathering · November 12–13 · San Francisco**
>
> Go deep on inference, training, and AI hardware through technical talks, live demos, and hands-on workshops. Bring your deployment challenges and meet the engineers and researchers building the stack.
## Get Started
Pull the Docker image, which includes SGLang and its dependencies:
@@ -49,7 +54,7 @@ SGLang supports a wide range of GPUs, TPUs, NPUs, CPUs, and Apple Silicon platfo
| --- | --- |
| [NVIDIA](https://docs.sglang.io/docs/hardware-platforms/nvidia-gpus) | B200/B300/GB200/GB300; H100/H200/H800/H20; A100; RTX 30/40/50 series, RTX 6000 Ada / PRO 6000; DGX Spark, Jetson Orin |
| [AMD](https://docs.sglang.io/docs/hardware-platforms/amd_gpu) | Instinct MI300X, MI325X, MI350X, MI355X |
| [Google TPU](https://docs.sglang.io/docs/hardware-platforms/tpu) | v6e, v7; [SGL-JAX](https://github.com/sgl-project/sglang-jax) / [SGL-torchtpu](https://lmsys.org/blog/2026-07-30-sglang-google-tpu/) |
| [Google TPU](https://docs.sglang.io/docs/hardware-platforms/tpu) | v6e, v7; [SGLang-TorchTPU](https://github.com/sgl-project/sglang-torchtpu) / [SGL-JAX](https://github.com/sgl-project/sglang-jax) |
| Intel ([GPU](https://docs.sglang.io/docs/hardware-platforms/xpu) / [CPU](https://docs.sglang.io/docs/hardware-platforms/cpu_server)) | Arc / Arc Pro B-Series GPUs, Xeon CPUs |
| [Apple Silicon](https://docs.sglang.io/docs/hardware-platforms/apple_metal) | Macs via Metal / MLX |
| [Huawei Ascend](https://docs.sglang.io/docs/hardware-platforms/ascend-npus/getting-started/installation) | A2, A3, 950PR/DT NPUs |
@@ -57,6 +62,8 @@ SGLang supports a wide range of GPUs, TPUs, NPUs, CPUs, and Apple Silicon platfo
Integrations in progress: AWS Trainium, [Alibaba T-Head PPU](https://github.com/sgl-project/sglang/issues/37519), [Cambricon MLU](https://github.com/sgl-project/sglang/pull/26898), Qualcomm QAIC, MetaX, Hygon HCU/DCU, Iluvatar CoreX, and more.
SGLang-TorchTPU brings the SGLang serving engine to Google TPUs through PyTorch, torch_tpu, and Pallas kernels. It supports LLM and vision-language serving on TPU7x (Ironwood), including tensor and expert parallelism across hosts. SGL-JAX provides the JAX-based TPU serving path.
See the [Cookbook](https://cookbook.sglang.io/) and platform guides for model compatibility and setup.
## SGL Ecosystem
@@ -20,10 +20,10 @@ import triton
from sglang.kernels.ops.attention.cutedsl_kda import (
cutedsl_fused_sigmoid_gating_kda_update,
)
from sglang.kernels.ops.attention.fla.fused_sigmoid_gating_recurrent import (
from sglang.kernels.ops.attention.fla.kda import chunk_kda
from sglang.kernels.ops.attention.linear.fused_sigmoid_gating_recurrent import (
fused_sigmoid_gating_delta_rule_update,
)
from sglang.kernels.ops.attention.fla.kda import chunk_kda
def make_inputs(
@@ -32,7 +32,7 @@ import triton
from sglang.kernels.ops.attention.fla.fused_recurrent import (
fused_recurrent_gated_delta_rule_packed_decode,
)
from sglang.kernels.ops.attention.fla.fused_sigmoid_gating_recurrent import (
from sglang.kernels.ops.attention.linear.fused_sigmoid_gating_recurrent import (
fused_sigmoid_gating_delta_rule_update,
)
@@ -26,12 +26,12 @@ import torch
from sglang.kernels.ops.attention.fla.fused_recurrent import (
fused_recurrent_kda_packed_decode,
)
from sglang.kernels.ops.attention.fla.fused_sigmoid_gating_recurrent import (
fused_sigmoid_gating_delta_rule_update,
)
from sglang.kernels.ops.attention.helion.kda_decode import (
helion_fused_recurrent_kda_packed_decode,
)
from sglang.kernels.ops.attention.linear.fused_sigmoid_gating_recurrent import (
fused_sigmoid_gating_delta_rule_update,
)
def make_inputs(
@@ -37,7 +37,7 @@ import triton
from sglang.kernels.ops.attention.fla.fused_recurrent import (
fused_recurrent_gated_delta_rule_packed_decode,
)
from sglang.kernels.ops.attention.fla.fused_recurrent_linear_replayssm import (
from sglang.kernels.ops.attention.linear.fused_recurrent_linear_replayssm import (
fused_recurrent_gdn_replayssm_decode,
)
+3 -3
View File
@@ -22,11 +22,11 @@ import argparse
import torch
import sglang.kernels.ops.attention.fla.fused_kda_conv_recurrent_verify as fused_mod
from sglang.kernels.ops.attention.fla.fused_kda_conv_recurrent_verify import (
import sglang.kernels.ops.attention.linear.fused_kda_conv_recurrent_verify as fused_mod
from sglang.kernels.ops.attention.linear.fused_kda_conv_recurrent_verify import (
fused_kda_conv_gating_verify,
)
from sglang.kernels.ops.attention.fla.fused_sigmoid_gating_recurrent import (
from sglang.kernels.ops.attention.linear.fused_sigmoid_gating_recurrent import (
fused_sigmoid_gating_delta_rule_update,
)
from sglang.kernels.ops.mamba.causal_conv1d_triton import causal_conv1d_update
@@ -3,7 +3,7 @@
Compares the legacy gather (``gather_dequant_requant_fp8_paged_legacy``:
fresh ``torch.zeros`` destination + one program per (token, 128-elem
slice)) against the new gather (``gather_dequant_requant_fp8_paged``:
slice)) against the new gather (``gather_dequant_requant_fp8_paged_dsa``:
no pre-zeroing needed, fused pad-row zero-fill, TOKENS_PER_PROG tokens
per program with 16B-vectorized access), in three flavors:
@@ -43,6 +43,8 @@ import sys
from pathlib import Path
import torch
import triton
import triton.language as tl
_REPO_ROOT = Path(__file__).resolve().parents[3]
_MOD_PATH = _REPO_ROOT / "python/sglang/kernels/ops/attention/dsa/dequant_k_cache.py"
@@ -54,8 +56,127 @@ _spec = importlib.util.spec_from_file_location("dequant_k_cache", _MOD_PATH)
_mod = importlib.util.module_from_spec(_spec)
_spec.loader.exec_module(_mod)
gather_new = _mod.gather_dequant_requant_fp8_paged
gather_legacy = _mod.gather_dequant_requant_fp8_paged_legacy
gather_new = _mod.gather_dequant_requant_fp8_paged_dsa
_infer_dsa_dims = _mod._infer_dsa_dims
def gather_dequant_requant_fp8_paged_legacy(
quant_k_cache: torch.Tensor,
page_table_1_flattened: torch.Tensor,
group_size: int = 128,
extra_rows: int = 0,
) -> torch.Tensor:
"""Legacy (pre-vectorization) gather + dequant + requant.
Kept as the bit-exactness / performance reference for
``gather_dequant_requant_fp8_paged_dsa`` (see
``benchmark/kernels/deepseek/benchmark_q8kv8_kv_gather.py``). Allocates and zero-fills the
full destination each call, then launches one program per
(token, 128-elem slice).
"""
dim_quant = quant_k_cache.shape[-1]
dim_nope, dim_rope = _infer_dsa_dims(dim_quant)
quant_k_cache = quant_k_cache.view((-1, dim_quant))
num_tokens = page_table_1_flattened.shape[0]
assert quant_k_cache.dtype == torch.float8_e4m3fn
num_tiles = dim_nope // group_size
out_dim = dim_nope + dim_rope
assert num_tiles * group_size == dim_nope
total_rows = num_tokens + extra_rows
# Allocate a fresh zero-filled buffer. The extra landing-pad rows at
# the tail must read as zeros (the kernel may over-read past
# num_tokens for masked indices).
output = torch.zeros(
(total_rows, 1, out_dim),
dtype=torch.float8_e4m3fn,
device=quant_k_cache.device,
)
num_blocks_per_token = num_tiles + triton.cdiv(dim_rope, group_size)
input_nope_q = quant_k_cache[:, :dim_nope]
input_nope_s = quant_k_cache[:, dim_nope : dim_nope + num_tiles * 4].view(
torch.float32
)
input_rope = quant_k_cache[:, dim_nope + num_tiles * 4 :].view(torch.bfloat16)
_gather_dequant_requant_fp8_paged_kernel[(num_tokens, num_blocks_per_token)](
output,
input_nope_q,
input_nope_s,
input_rope,
page_table_1_flattened,
output.stride(0),
input_nope_q.stride(0),
input_nope_s.stride(0),
input_rope.stride(0),
NUM_NOPE_BLOCKS=num_tiles,
GROUP_SIZE=group_size,
DIM_NOPE=dim_nope,
DIM_ROPE=dim_rope,
)
return output
@triton.jit
def _gather_dequant_requant_fp8_paged_kernel(
output_ptr,
input_nope_q_ptr,
input_nope_s_ptr,
input_rope_ptr,
page_table_1_ptr,
output_stride_0: int,
input_nope_q_stride_0: int,
input_nope_s_stride_0: int,
input_rope_stride_0: int,
NUM_NOPE_BLOCKS: tl.constexpr,
GROUP_SIZE: tl.constexpr,
DIM_NOPE: tl.constexpr,
DIM_ROPE: tl.constexpr,
):
"""Fused gather + dequant(per-group) + requant(per-tensor) -> fp8."""
token_id = tl.program_id(0)
token_id_paged = tl.load(page_table_1_ptr + token_id).to(tl.int32)
raw_block_id = tl.program_id(1)
if raw_block_id < NUM_NOPE_BLOCKS:
# nope: read fp8, mul group scale -> f32, cast to fp8_e4m3fn
effective_block_id = raw_block_id
offs_q = effective_block_id * GROUP_SIZE + tl.arange(0, GROUP_SIZE)
mask = offs_q < DIM_NOPE
ptr_q = input_nope_q_ptr + token_id_paged * input_nope_q_stride_0 + offs_q
ptr_s = (
input_nope_s_ptr
+ token_id_paged * input_nope_s_stride_0
+ effective_block_id
)
y_q = tl.load(ptr_q, mask=mask, other=0.0).to(tl.float32)
y_s = tl.load(ptr_s)
# dequant -> f32 -> requant to fp8
y = (y_q * y_s).to(tl.float8e4nv)
dst_ptr = output_ptr + token_id * output_stride_0 + offs_q
tl.store(dst_ptr, y, mask=mask)
else:
# rope: read bf16, cast to fp8
effective_block_id = raw_block_id - NUM_NOPE_BLOCKS
offs = effective_block_id * GROUP_SIZE + tl.arange(0, GROUP_SIZE)
mask = offs < DIM_ROPE
src_ptr = input_rope_ptr + token_id_paged * input_rope_stride_0 + offs
dst_ptr = output_ptr + token_id * output_stride_0 + DIM_NOPE + offs
data = tl.load(src_ptr, mask=mask).to(tl.float8e4nv)
tl.store(dst_ptr, data, mask=mask)
gather_legacy = gather_dequant_requant_fp8_paged_legacy
PAGE_SIZE = 64
DIM_QUANT = 656 # 512 nope fp8 + 16 scale bytes + 128 rope bytes
@@ -54,7 +54,7 @@ import argparse
import torch
from sglang.kernels.ops.kvcache.cache_ops import (
from sglang.kernels.ops.attention.mla_qk_prep import (
absorbed_bmm_concat_cast_q_fp8,
concat_and_cast_q_fp8_pad,
)
@@ -1,5 +1,5 @@
"""Tune the gfx950 native MXFP8 dense route for one (N, K) and write its rows into
python/sglang/kernels/ops/quantization/mxfp8_gemv_gfx95_configs.json.
python/sglang/kernels/ops/gemm/mxfp8_gemv_gfx95_configs.json.
Two tables: the scaled-MFMA GEMV config per M bucket (1 ... 32) and the dot_scaled tile per
M bucket (64 ... 16384). Each candidate is timed by HIP-graph replay rotating over weight
@@ -18,7 +18,7 @@ import torch
from triton.runtime.errors import OutOfResources
from sglang.kernels.jit.utils import empty_sentinel
from sglang.kernels.ops.quantization import mxfp8_native_amd_gfx95 as native
from sglang.kernels.ops.gemm import mxfp8_native_amd_gfx95 as native
from sglang.kernels.ops.quantization.mxfp8_amd_gfx95 import fp8_grid_quantize
GRAPH_CALLS = 20
@@ -0,0 +1,89 @@
"""Benchmark seeded versus unseeded EAGLE draft Gumbel sampling.
Example:
python benchmark/kernels/sampling/bench_deterministic_fast_sample.py \
--batch-sizes 1 8 32 128 --vocab-sizes 32000 128256 163840
"""
import argparse
import torch
import triton
from sglang.srt.speculative.spec_utils import fast_sample
def parse_args():
parser = argparse.ArgumentParser()
parser.add_argument("--batch-sizes", type=int, nargs="+", default=[1, 8, 32, 128])
parser.add_argument(
"--vocab-sizes", type=int, nargs="+", default=[32000, 128256, 163840]
)
parser.add_argument("--warmup-ms", type=int, default=100)
parser.add_argument("--rep-ms", type=int, default=500)
parser.add_argument(
"--modes", nargs="+", choices=("eager", "graph"), default=["graph"]
)
return parser.parse_args()
def capture(fn):
fn()
torch.cuda.synchronize()
graph = torch.cuda.CUDAGraph()
with torch.cuda.graph(graph):
fn()
return graph.replay
def main():
args = parse_args()
device = torch.device("cuda")
print(f"device={torch.cuda.get_device_name(device)}")
print("mode\tbatch\tvocab\tunseeded_us\tseeded_us\tslowdown")
for vocab_size in args.vocab_sizes:
for batch_size in args.batch_sizes:
logits = torch.randn(
(batch_size, vocab_size), device=device, dtype=torch.float32
)
probs = torch.softmax(logits, dim=-1)
seeds = torch.arange(batch_size, device=device, dtype=torch.int64) + 12345
positions = (
torch.arange(batch_size, device=device, dtype=torch.int64) + 4096
)
def unseeded():
return fast_sample(probs)
def seeded():
return fast_sample(
probs,
sampling_seed=seeds,
positions=positions,
draft_step=1,
)
# Compile/JIT and populate allocator caches before timing.
unseeded()
seeded()
torch.cuda.synchronize()
for mode in args.modes:
unseeded_fn = capture(unseeded) if mode == "graph" else unseeded
seeded_fn = capture(seeded) if mode == "graph" else seeded
unseeded_ms = triton.testing.do_bench(
unseeded_fn, warmup=args.warmup_ms, rep=args.rep_ms
)
seeded_ms = triton.testing.do_bench(
seeded_fn, warmup=args.warmup_ms, rep=args.rep_ms
)
print(
f"{mode}\t{batch_size}\t{vocab_size}\t"
f"{unseeded_ms * 1000:.1f}\t{seeded_ms * 1000:.1f}\t"
f"{seeded_ms / unseeded_ms:.2f}x"
)
if __name__ == "__main__":
main()
@@ -0,0 +1,25 @@
# L3 SSD image: recipe-ns engine + official mooncake 0.3.14 (CUDA 13, cp312).
#
# Build context: this directory plus the 0.3.14 wheel
# mooncake_transfer_engine_cuda13-0.3.14-cp312-cp312-manylinux_2_28_x86_64.whl
# Use build-l3ssd-image.sh, which downloads and verifies the wheel.
#
# Base tag recipe-ns-433257c7b-baked-20261114 = b300/integration @ 433257c7b
# (recipe namespace/chat-ns + gateway binary baked). Rebuild it from that commit
# if the local tag is missing.
FROM sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-433257c7b-baked-20261114
COPY mooncake_transfer_engine_cuda13-0.3.14-cp312-cp312-manylinux_2_28_x86_64.whl /tmp/
RUN python3 -m pip install --no-deps --force-reinstall \
/tmp/mooncake_transfer_engine_cuda13-0.3.14-cp312-cp312-manylinux_2_28_x86_64.whl \
&& rm -f /tmp/mooncake_transfer_engine_cuda13-0.3.14-cp312-cp312-manylinux_2_28_x86_64.whl \
# The wheel's master binary uses $ORIGIN/../mooncake_transfer_engine_cuda13.libs;
# keep it in place and expose a wrapper at the legacy path used by the manifests.
&& printf '%s\n' \
'#!/bin/sh' \
'export LD_LIBRARY_PATH=/opt/sglang/lib/python3.12/site-packages/mooncake_transfer_engine_cuda13.libs:$LD_LIBRARY_PATH' \
'exec /opt/sglang/lib/python3.12/site-packages/mooncake/mooncake_master "$@"' \
> /opt/sglang/bin/mooncake_master \
&& chmod +x /opt/sglang/bin/mooncake_master \
&& python3 -c "import importlib.metadata as m; print('mooncake:', m.version('mooncake-transfer-engine-cuda13'))" \
&& /opt/sglang/bin/mooncake_master --version | head -1
+139
View File
@@ -0,0 +1,139 @@
# B300 L3 Mooncake SSD 基线部署(2026-11-15)
> 以同事在 `ds41-gist-exp` 上验证的 **L3 Mooncake SSD** 配置为基础,使用我们的
> `b300/integration` 引擎代码 + 官方 **mooncake 0.3.14**,不再依赖 hostPath
> `/opt/mooncake-latest`。
## 1. 版本要求(重要)
- **mooncake `0.3.14`**(`mooncake-transfer-engine-cuda13`,CUDA 13 / cp312 wheel)。
- 镜像内原为 `0.3.13.post1`;0.3.14 含上游 SSD offload 相关修复/增强。
- wheel 与 sha256 见 `requirements-mooncake.txt`,安装必须 `--no-deps`(基底已有运行时依赖)。
- 引擎:`b300/integration` @ `433257c7b`(recipe namespace/chat-ns + 新 gateway)。
- 镜像:`sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115`
(40.1GB,已 `ctr -n k8s.io images import`;本地 docker tag `ymkymx/sglang:...` 同名)。
- 构建/装载:`./build-l3ssd-image.sh`(下载校验 wheel → docker build → ctr import)。
镜像内 `/opt/sglang/bin/mooncake_master` 是 wrapper:设置 wheel `.libs` 的
`LD_LIBRARY_PATH` 后 exec 包内 master(已验证 `0.3.14 (git e1ea5d3)`)。
## 2. L3 SSD 配置
`mooncake-store-l3ssd.json`:
```json
{"master_server_address":"127.0.0.1:50051","metadata_server":"P2PHANDSHAKE",
"global_segment_size":"200gb","protocol":"tcp",
"enable_ssd_offload":true,"ssd_offload_path":"/data/ymk/mooncake-ssd"}
```
- 部署时把该文件放到节点 `/data/ymk/ds41/mooncake-store-l3ssd.json`,SSD 目录
`/data/ymk/mooncake-ssd` 需存在。
- `MOONCAKE_OFFLOAD_*`:`bucket_storage_backend` / 512GiB bucket 上限 / `fifo` /
`flush_partial=0`(见 manifest)。
- master 参数必须带
`--enable_offload=true --offload_on_evict=true --promotion_on_hit=false --offload_force_evict=false`。
## 3. manifest
| 文件 | 用途 |
|---|---|
| `ds41-l3ssd-test-baked-20261115.yaml` | 新建 cw `ds41-l3ssd-test`(推荐先做基线验证) |
| `ds41-gist-exp-l3ssd-baked-20261115.yaml` | 原名 `ds41-gist-exp`,用于直接更新原栈 |
已通过 `kmctl create/update --dry-run` 校验。启动前先停掉占用 8 卡的 stack
(`ds41-recipe-ns-test` 或旧 `ds41-gist-exp`),确认 GPU 归零:
```bash
kmctl create cw ds41-l3ssd-test --node-group B300 \
-f deploy/b300-k8s/ds41-l3ssd-test-baked-20261115.yaml
# 或替换原栈:
# kmctl update cw ds41-gist-exp -f deploy/b300-k8s/ds41-gist-exp-l3ssd-baked-20261115.yaml
```
端口/角色与旧 `ds41-gist-exp` 一致:master 50051、P 30000、D 30300、gateway 30400。
启动约 10-15 分钟;核对 P/D `max_total_num_tokens` 和 master 日志中的 SSD offload 初始化。
## 4. 与上一版配置的差异
P/D 的 `sglang serve` 参数**完全一致**;差异只在 mooncake:
| 项 | 旧(11-09) | 本基线 |
|---|---|---|
| mooncake | 镜像内 0.3.13.post1 | 镜像内 **0.3.14** |
| master | `/opt/sglang/bin/mooncake_master`,无 offload | 同路径(0.3.14 wrapper),带 offload 参数 |
| P/D PYTHONPATH | 无 | 无(0.3.14 已烘入) |
| L3 配置 | 纯 TCP,无 SSD | `enable_ssd_offload` + `ssd_offload_path` |
| `MOONCAKE_OFFLOAD_*` | 无 | bucket backend / 512GiB / fifo / flush_partial=0 |
| TMPDIR/XDG/HF | 默认/`/data/ymk` | `/data/ymk/l3ssd/{master,p,d}-*` |
## 5. 注意
- 同事原配置使用 `SGLANG_TRACE_LOOP_SAMPLE=1.0`(全量 detail tracing,约 2% 吞吐 +
p95 开销)。本基线为对齐其配置保留 1.0;正式上线建议改回 `0.05`。
- `global_segment_size=200gb` + 512GiB SSD offload:上线前按宿主内存总账复核。
- SSD 目录增长需监控(`/data` 当前约 7.6T 可用)。
- 旧镜像 `recipe-ns-433257c7b-baked-20261114` 与 `recipe-ns-a7c668a93-baked-20261114`
保留作回滚/对照。
## 6. Tokenizer A/B(2026-11-15,c580 / 1200s)
同一 `ds41-gist-exp`(L3 SSD + mooncake 0.3.14)整栈重启,只切 P/D
`--tokenizer-backend`;AgentX c580 / 1200s / warmup 256 / seed 42,经 30400。
| 臂 | tokenizer | rps | in tok/s | out tok/s | TTFT p50/p95/p99 | ITL p50/p95/p99 | requests |
|---|---|---:|---:|---:|---|---|---:|
| A | `huggingface`(默认) | 23.789 | 2,257,661 | 22,224 | 2.043 / 4.176 / 7.579 s | 8.02 / 14.48 / 20.91 ms | 29,498 |
| B | **`fastokens`** | **24.045** | **2,289,885** | **22,475** | **1.746 / 3.676 / 6.820 s** | 8.13 / 11.65 / 14.87 ms | 29,816 |
- 结论:fastokens 臂 **rps +1.1%**、TTFT p50/p95/p99 **−14.5% / −12.0% / −10.0%**;
ITL p95 的差异大概率是 run-to-run 噪声(decode 不走 tokenizer),不作为判据。
- #48928 的分段编码已生效:容器内 `DeepSeekRecipe.from_hf_tokenizer(..., fastokens tokenizer)`
的 `_fast_prompt_tokenizer` 非 None(`fastokens.Tokenizer`);服务端未打印
“fastokens segmented” 日志,属日志路径差异,不影响结论。
- **当前线上实验栈保留 fastokens 臂**;要回默认用
`ds41-gist-exp-l3ssd-baked-20261115.yaml` 更新并整栈重启。
- 单轮对比,c580 本身有方差;如需正式定论建议每臂 ≥3 轮。
- SSD offload 已配置(master 报 `SSD Storage: 0 B / 2.00 TB`),本次 c580 工作集未触发
SSD 逐出(`/data/ymk/mooncake-ssd` 约 20K),不是配置未生效。
## 7. 多轮验证(2026-11-15,每轮整栈重启)
> 之前 B2/B3 在同一实例上连跑、未重启,数据受实例状态影响;下面只列**整栈重启后**的轮次。
> 整栈重启 = `kmctl stop` + `update`(换臂),或同臂时删除 p/d/gateway/master 四个 Pod。
| 臂 | 轮次 | rps | TTFT p50/p95/p99 (s) | ITL p95 (ms) | requests |
|---|---|---:|---|---:|---:|
| A(huggingface) | A1 | 23.789 | 2.043 / 4.176 / 7.579 | 14.48 | 29,498 |
| A | A2 | 23.552 | 2.305 / 5.894 / 8.600 | 15.05 | 29,205 |
| B(fastokens) | B1 | 24.045 | 1.746 / 3.676 / 6.820 | 11.65 | 29,816 |
| B | **B2(fresh)** | **24.040** | **1.708 / 3.451 / 7.105** | 11.71 | 29,810 |
- **B2 满足判据:TTFT p95 = 3.451s < 5s**,且与 B1(3.676s)一致;
A 臂 fresh 轮 p95 为 4.176 / 5.894s。
- fresh 轮均值:A rps 23.67 / p95 5.04s;B rps 24.04 / p95 3.56s
(B +1.6% rps、p95 −29%)。
- 结论:保留 fastokens;之前未重启的 B2/B3(p95 5.99/6.69s)仅作历史参考、不纳入。
## 8. L3 SSD 4TB + offload_on_evict=false(2026-11-15)
- 启动清单 `ds41-gist-exp-l3ssd-fastokens-20261115.yaml` 已改:master
`--offload_on_evict=false`;P 的 `MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE` /
`MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES` 改为 1 TiB/rank,4 rank 合计
master 报 `SSD Storage: 0 B / 4.00 TB`。
- 每次实验前必须重置 L3:stop cw → 确认 Terminated 0/0 + GPU 0 MiB →
`bash clean_l3ssd_cache.sh`(删除 `/data/ymk/mooncake-ssd` 下的 rank_*)→ update/start。
本次清理释放 1.2T,`/data` 可用 6.4T→7.5T。
- c580(1200s,warmup 256,seed 42,fastokens,干净 L3):
rps 24.003、in 2.285M tok/s、out 22,447、TTFT p50/p95/p99 1.768/3.500/7.549s、
ITL 8.15/11.74/15.31ms、29,764 请求、0 错误、valid。
对比 B1/B2:24.045/1.746/3.676/6.820 与 24.040/1.708/3.451/7.105——
吞吐与尾延迟均在方差内,**没有可归因于 offload_on_evict/4TB/清理的收益或回归**。
- **重要:`offload_on_evict=false` 并没有停止 SSD 写入**。run 后 master
`SSD Storage: 533.81 GB / 4.00 TB`,`du /data/ymk/mooncake-ssd = 535G`;
这是 `--hicache-write-policy write_through` 的 backup 写入,和内存 eviction 无关。
若要真正避免 SSD 写入,需要改 `write_back` 或关闭 `--enable_offload`,但那会改变
L3 语义,需另做 A/B。
- run 后 P cgroup `memory.current=2,198,904,565,760` ≈ 2047.5 GiB,已顶到 2 TiB
limit;宿主 MemAvailable 1.2 TiB。下一次实验前必须先停/清/重启;不要在同一实例
上连续跑,否则 L3 与 cgroup 都处于顶格状态。
@@ -0,0 +1,10 @@
# 2026-11-17 c580 基线:D tokenizer 16 + fastokens
- manifest: `manifest.yaml`(= `ds41-gist-exp-20261117-dtok16-fastokens.yaml`)
- P: cache / 无 optimistic / tokenizer-worker 16 / fastokens / hicache ratio 4 / L3 SSD 4T
- D: tokenizer-worker 16 / fastokens
- Gateway: 正常(无 P admission cap,health/circuit 开启,decode-max-running 512)
- L3:每轮前清空;完整重启
- c580 结果:rps 24.045,in 2.290M tok/s,out 22.449k tok/s,
TTFT 1.743/3.772/6.580 s,ITL p95 10.74 ms,29,816 req,valid=true。
- 原始结果:`experiments/agentx-align/results-c580-dtok16-fastokens-20261117/`
@@ -0,0 +1,488 @@
apiVersion: workloads.x-k8s.io/v1alpha2
kind: RoleBasedGroup
metadata:
annotations:
rbg.workloads.x-k8s.io/discovery-config-mode: refine
labels:
app: ds41-gist-exp
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
sglang-model: ds-v41-flash
name: ds41-gist-exp
namespace: prj-d7te67bj0sq2fgdrmbig
spec:
roles:
- minReadySeconds: 0
name: mooncake-master
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '19003'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-mooncake
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "# 18081: 宿主机 18080 被 AgentX fake upstream(./fakeup)长期占用;\n# 我们的 P/D 用 metadata_server=P2PHANDSHAKE,此 HTTP 端口只是 master 可选项。\nexec /opt/sglang/bin/mooncake_master \\\n --port 50051 --enable_http_metadata_server=true \\\n --http_metadata_server_port=18081 --metrics_port=19003 --eviction_high_watermark_ratio=0.90 --enable_offload=true --offload_on_evict=false --promotion_on_hit=false --offload_force_evict=false\n"
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: mooncake-master
ports:
- containerPort: 19003
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 60
initialDelaySeconds: 5
periodSeconds: 5
tcpSocket:
port: 50051
resources:
limits:
cpu: '4'
memory: 64Gi
requests:
cpu: '1'
memory: 8Gi
env:
- name: TMPDIR
value: /data/ymk/l3ssd/master-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/master-xdg
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
- dependencies:
- mooncake-master
minReadySeconds: 0
name: p
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '30000'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-p
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 32 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r32_ep4.json \\\n --mem-fraction-static 0.80 \\\n --chunked-prefill-size 131072 --max-prefill-tokens 131072 \\\n --max-running-requests 512 --tokenizer-worker-num 16 \\\n --load-balance-method total_tokens \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --enable-decoder-swa-bounded-replay --swa-full-tokens-ratio 0.12 \\\n --cuda-graph-max-bs-decode 1 --disable-decode-cuda-graph \\\n --cuda-graph-backend-prefill breakable \\\n --cuda-graph-bs-prefill 1024 2048 4096 8192 16384 32768 \\\n --cuda-graph-prefill-max-context 262144 --cuda-graph-max-seq-len-prefill 262144 \\\n --enable-hierarchical-cache --hicache-ratio 4 \\\n --hicache-mem-layout page_first_direct --hicache-io-backend direct \\\n --hicache-write-policy write_through --hicache-storage-backend mooncake \\\n --hicache-storage-prefetch-policy wait_complete \\\n --disaggregation-mode prefill --disaggregation-transfer-backend mooncake \\\n --disaggregation-bootstrap-port 8998 \\\n --expert-balancedness-report-mode prometheus \\\n --expert-distribution-recorder-buffer-size 64 \\\n --host 0.0.0.0 --port 30000 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-p \\\n --watchdog-timeout 3600 --enable-dp-attention-local-control-broadcast\n"
env:
- name: CUDA_VISIBLE_DEVICES
value: 0,1,2,3
- name: SGLANG_LOG_MS
value: '1'
- name: SGLANG_OPT_DSV41_ENGRAM_PEER_SHARDS
value: '1'
- name: SGLANG_OPT_DSV41_MEGA_MHC_PREFILL
value: '1'
- name: SGLANG_OPT_DSV41_BF16_INDEX_LOGITS
value: '1'
- name: SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK
value: '65536'
- name: SGLANG_DSV4_KV_LAYOUT
value: v41_new
- name: SGLANG_DP_USE_GATHERV
value: '1'
- name: SGLANG_RAGGED_VERIFY_MODE
value: static
- name: SGLANG_ENABLE_DP_SPEC_PREFILL_COORDINATION
value: '0'
- name: SGLANG_NUMA_BIND_V2
value: '0'
- name: SGLANG_DEFAULT_THINKING
value: '1'
- name: SGLANG_DSV41_REASONING_EFFORT
value: high
- name: SGLANG_TIMEOUT_KEEP_ALIVE
value: '900'
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
value: '1'
- name: MC_INTRANODE_NVLINK
value: 'true'
- name: MC_INTRA_NVLINK
value: 'true'
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
value: 'true'
- name: SGLANG_HICACHE_MOONCAKE_CONFIG_PATH
value: /data/ymk/ds41/mooncake-store-l3ssd.json
- name: SGLANG_HICACHE_PREFETCH_IO_THREADS
value: '4'
- name: SGLANG_HICACHE_BACKUP_IO_THREADS
value: '4'
- name: SGLANG_HICACHE_TIMING
value: '1'
- name: SGLANG_BATCH_TELEMETRY
value: '1'
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
value: '1'
- name: SGLANG_ENABLE_METRICS_DEVICE_TIMER
value: '1'
- name: SGLANG_EPLB_HEATMAP_COLLECTION_INTERVAL
value: '100'
- name: SGLANG_MM_TRACE
value: '1'
- name: SGLANG_BATCH_SPAN_MIN_MS
value: '200'
- name: SGLANG_BATCH_SPAN_SAMPLE
value: '0.05'
- name: SGLANG_BATCH_SPAN_MAX_LINKS
value: '8'
- name: SGLANG_BATCH_STAGE_TIMING
value: '1'
- name: SGLANG_BATCH_PASS_SPAN
value: '1'
- name: SGLANG_BATCH_PASS_SPAN_SAMPLE
value: '1.0'
- name: SGLANG_RECV_SPAN
value: '1'
- name: SGLANG_RECV_SPAN_SAMPLE
value: '1.0'
- name: SGLANG_MM_TRACE_SYNC
value: '0'
- name: SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK
value: '1'
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
value: '600'
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
value: '16'
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
value: '32'
- name: TZ
value: Asia/Shanghai
- name: SGLANG_TRACE_LOOP_SPAN
value: '1'
- name: SGLANG_TRACE_LOOP_SAMPLE
value: '1.0'
- name: SGLANG_FWD_TIMELINE
value: '0'
- name: SGLANG_TRACE_LOOP_DETAIL
value: '1'
- name: SGLANG_TRACE_MOONCAKE_DETAIL
value: '0'
- name: SGLANG_TRACE_BACKUP_DETAIL
value: '1'
- name: SGLANG_ASYNC_BACKUP_SUBMIT
value: '1'
- name: SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH
value: '1'
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
value: '1024'
- name: SGLANG_MOONCAKE_CLIENT_METRICS_PORT_BASE
value: '9400'
- name: MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR
value: bucket_storage_backend
- name: MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE
value: '1099511627776'
- name: MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES
value: '1099511627776'
- name: MOONCAKE_OFFLOAD_BUCKET_EVICTION_POLICY
value: fifo
- name: TMPDIR
value: /data/ymk/l3ssd/p-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/p-xdg
- name: HF_HOME
value: /data/ymk/l3ssd/p-hf
- name: MOONCAKE_OFFLOAD_FLUSH_PARTIAL_BUCKETS
value: '0'
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: sglang
ports:
- containerPort: 30000
name: metrics
protocol: TCP
- containerPort: 8998
name: bootstrap
protocol: TCP
- name: mc-0
containerPort: 9400
protocol: TCP
- name: mc-1
containerPort: 9401
protocol: TCP
- name: mc-2
containerPort: 9402
protocol: TCP
- name: mc-3
containerPort: 9403
protocol: TCP
readinessProbe:
failureThreshold: 120
httpGet:
path: /health
port: metrics
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '96'
memory: 2Ti
nvidia.com/gpu: '4'
requests:
cpu: '32'
memory: 512Gi
nvidia.com/gpu: '4'
securityContext:
privileged: true
volumeMounts:
- mountPath: /data
name: data
- mountPath: /root/.cache/sglang
name: sglang-cache
- mountPath: /dev/shm
name: dshm
- mountPath: /root/.cargo/bin/cargo
name: fake-cargo
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
hostPID: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/cache/sglang
type: Directory
name: sglang-cache
- emptyDir:
medium: Memory
sizeLimit: 64Gi
name: dshm
- hostPath:
path: /data/ymk/ds41/fake-cargo
type: File
name: fake-cargo
- dependencies:
- mooncake-master
minReadySeconds: 0
name: d
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '30300'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-d
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --enable-fp32-lm-head --enable-dp-attention-local-control-broadcast \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 64 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r64_ep4.json \\\n --mem-fraction-static 0.80 \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --stream-interval 32 --swa-prefix-tails 2048 --enable-decoder-swa-bounded-replay \\\n --max-running-requests 512 --cuda-graph-max-bs-decode 512 --tokenizer-worker-num 16 \\\n --chunked-prefill-size 32768 --max-prefill-tokens 8192 \\\n --disaggregation-mode decode --disaggregation-transfer-backend mooncake \\\n --host 0.0.0.0 --port 30300 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-d \\\n --watchdog-timeout 3600\n"
env:
- name: CUDA_VISIBLE_DEVICES
value: 4,5,6,7
- name: SGLANG_RAGGED_VERIFY_MODE
value: static
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
value: '1'
- name: SGLANG_DSV4_KV_LAYOUT
value: v41_new
- name: SGLANG_DEFAULT_THINKING
value: '1'
- name: SGLANG_DSV41_REASONING_EFFORT
value: high
- name: MC_INTRANODE_NVLINK
value: 'true'
- name: MC_INTRA_NVLINK
value: 'true'
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
value: 'true'
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
value: '1'
- name: SGLANG_BATCH_TELEMETRY
value: '1'
- name: SGLANG_BATCH_SPAN_SAMPLE
value: '0.1'
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
value: '600'
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
value: '16'
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
value: '32'
- name: TZ
value: Asia/Shanghai
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
value: '1024'
- name: TMPDIR
value: /data/ymk/l3ssd/d-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/d-xdg
- name: HF_HOME
value: /data/ymk/l3ssd/d-hf
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: sglang
ports:
- containerPort: 30300
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 150
httpGet:
path: /health
port: metrics
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '96'
memory: 550Gi
nvidia.com/gpu: '4'
requests:
cpu: '32'
memory: 256Gi
nvidia.com/gpu: '4'
securityContext:
privileged: true
volumeMounts:
- mountPath: /data
name: data
- mountPath: /root/.cache/sglang
name: sglang-cache
- mountPath: /dev/shm
name: dshm
- mountPath: /root/.cargo/bin/cargo
name: fake-cargo
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
hostPID: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/cache/sglang
type: Directory
name: sglang-cache
- emptyDir:
medium: Memory
sizeLimit: 64Gi
name: dshm
- hostPath:
path: /data/ymk/ds41/fake-cargo
type: File
name: fake-cargo
- dependencies:
- p
- d
minReadySeconds: 0
name: gateway
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '29000'
prometheus.io/scheme: http
prometheus.io/scrape: 'true'
labels:
app: ds41-l3ssd-gw
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec /opt/sglang/bin/sgl-model-gateway launch \\\n --pd-disaggregation \\\n --prefill http://127.0.0.1:30000 8998 \\\n --decode http://127.0.0.1:30300 \\\n --prefill-policy cache_aware --decode-policy cache_aware \\\n --g9i-enabled --g9i-max-new-tokens 4096 \\\n \\\n \\\n --g9i-regular-p-queue-reject-secs 0 \\\n --g9i-decode-max-running 512 \\\n --health-check-timeout-secs 30 --health-check-interval-secs 30 \\\n --cb-failure-threshold 50 \\\n --model-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --tokenizer-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --host 0.0.0.0 --port 30400\n"
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: gateway
ports:
- containerPort: 30400
name: http
protocol: TCP
- containerPort: 29000
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 60
httpGet:
path: /health
port: http
initialDelaySeconds: 20
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '16'
memory: 32Gi
requests:
cpu: '2'
memory: 8Gi
volumeMounts:
- mountPath: /data
name: data
- mountPath: /opt/sglang/bin/sgl-model-gateway
name: gw-bin
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- name: gw-bin
hostPath:
path: /data/ymk/cache/gw-est-td/sgl-model-gateway
type: File
@@ -0,0 +1 @@
{"rps": 24.04512015569562, "in_tput": 2290117.7612668574, "out_tput": 22449.32127276801, "ttft_p50": 1743.335193, "ttft_p95": 3771.88552025, "ttft_p99": 6579.86958935, "itl_p50": 8.162268671611512, "itl_p95": 10.741317697249674, "itl_p99": 11.806237803568294, "lat_p95": 31884.86991075, "requests": 29816.0, "isl_p50": 79909.0, "osl_p50": 339.0, "cache_pct": NaN, "valid": true}
+20
View File
@@ -0,0 +1,20 @@
#!/usr/bin/env bash
# Build and load the L3 SSD image (mooncake 0.3.14) on b300-01.
set -euo pipefail
cd "$(dirname "$0")"
WHEEL=mooncake_transfer_engine_cuda13-0.3.14-cp312-cp312-manylinux_2_28_x86_64.whl
URL=https://files.pythonhosted.org/packages/b9/bf/a117c9c9f892e86893caf0dd8b5bfae1d11edbfb0173dcd30108a7168fc3/$WHEEL
SHA=74fa5f0684b143167503e33f85cccb9ca687d3f9843d4b594dbd5342c7df3e17
TAG=recipe-ns-l3ssd-433257c7b-baked-20261115
REPO=sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang
if [ ! -f "$WHEEL" ]; then
curl -fL -o "$WHEEL" "$URL"
fi
echo "$SHA $WHEEL" | sha256sum -c -
docker build -t "ymkymx/sglang:$TAG" -f Dockerfile.mooncake-0.3.14 .
docker tag "ymkymx/sglang:$TAG" "$REPO:$TAG"
docker save "$REPO:$TAG" | ctr -n k8s.io images import -
echo "loaded $REPO:$TAG"
+30
View File
@@ -0,0 +1,30 @@
#!/usr/bin/env bash
# Reset the mooncake L3 SSD KV cache before an experiment.
#
# MUST run on b300-01 only after the ds41-gist-exp workload is fully stopped:
# 1) kmctl stop cw ds41-gist-exp --wait ...
# 2) kubectl get cw ... -> Terminated 0/0, nvidia-smi -> 0 MiB on GPU0-7
# 3) bash clean_l3ssd_cache.sh
# 4) kmctl update/start the new manifest
#
# The script refuses to delete anything while a mooncake master or sglang
# server process is still visible on the host.
set -euo pipefail
SSD_DIR="${SSD_DIR:-/data/ymk/mooncake-ssd}"
if pgrep -af '[m]ooncake_master|[s]glang serve' >/dev/null 2>&1; then
echo "refusing: mooncake_master or sglang serve is still running" >&2
pgrep -af '[m]ooncake_master|[s]glang serve' >&2 || true
exit 1
fi
if [ ! -d "$SSD_DIR" ]; then
echo "SSD cache dir does not exist: $SSD_DIR"
exit 0
fi
echo "before: $(du -sh "$SSD_DIR" | awk '{print $1}') ($SSD_DIR)"
find "$SSD_DIR" -mindepth 1 -maxdepth 1 -exec rm -rf {} +
echo "after: $(du -sh "$SSD_DIR" | awk '{print $1}') ($SSD_DIR)"
df -h /data | tail -1
@@ -0,0 +1,542 @@
apiVersion: workloads.x-k8s.io/v1alpha2
kind: RoleBasedGroup
metadata:
annotations:
rbg.workloads.x-k8s.io/discovery-config-mode: refine
labels:
app: ds41-gist-exp
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
sglang-model: ds-v41-flash
name: ds41-gist-exp
namespace: prj-d7te67bj0sq2fgdrmbig
spec:
roles:
- minReadySeconds: 0
name: mooncake-master
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '19003'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-mooncake
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "# 18081: 宿主机 18080 被 AgentX fake upstream(./fakeup)长期占用;\n# 我们的 P/D\
\ 用 metadata_server=P2PHANDSHAKE,此 HTTP 端口只是 master 可选项。\nexec /opt/sglang/bin/mooncake_master\
\ \\\n --port 50051 --enable_http_metadata_server=true \\\n --http_metadata_server_port=18081\
\ --metrics_port=19003 --eviction_high_watermark_ratio=0.90 --enable_offload=true\
\ --offload_on_evict=false --promotion_on_hit=false --offload_force_evict=false\n"
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: mooncake-master
ports:
- containerPort: 19003
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 60
initialDelaySeconds: 5
periodSeconds: 5
tcpSocket:
port: 50051
resources:
limits:
cpu: '4'
memory: 64Gi
requests:
cpu: '1'
memory: 8Gi
env:
- name: TMPDIR
value: /data/ymk/l3ssd/master-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/master-xdg
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
- dependencies:
- mooncake-master
minReadySeconds: 0
name: p
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '30000'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-p
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens\
\ --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name\
\ /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser\
\ auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"\
reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4\
\ --enable-dp-attention --enable-dp-lm-head \\\n --moe-a2a-backend\
\ megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts\
\ 32 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r32_ep4.json\
\ \\\n --mem-fraction-static 0.80 \\\n --chunked-prefill-size 131072\
\ --max-prefill-tokens 131072 \\\n --max-running-requests 512 --tokenizer-worker-num\
\ 16 \\\n --load-balance-method total_tokens \\\n --speculative-algorithm\
\ DSPARK --speculative-dspark-block-size 3 \\\n --enable-decoder-swa-bounded-replay\
\ --swa-full-tokens-ratio 0.12 \\\n --cuda-graph-max-bs-decode 1 --disable-decode-cuda-graph\
\ \\\n --cuda-graph-backend-prefill breakable \\\n --cuda-graph-bs-prefill\
\ 1024 2048 4096 8192 16384 32768 \\\n --cuda-graph-prefill-max-context\
\ 262144 --cuda-graph-max-seq-len-prefill 262144 \\\n --enable-hierarchical-cache\
\ --hicache-ratio 4 \\\n --hicache-mem-layout page_first_direct --hicache-io-backend\
\ direct \\\n --hicache-write-policy write_through --hicache-storage-backend\
\ mooncake \\\n --hicache-storage-prefetch-policy wait_complete \\\n\
\ --disaggregation-mode prefill --disaggregation-transfer-backend mooncake\
\ \\\n --disaggregation-bootstrap-port 8998 \\\n --expert-balancedness-report-mode\
\ prometheus \\\n --expert-distribution-recorder-buffer-size 64 \\\n\
\ --host 0.0.0.0 --port 30000 --enable-metrics --enable-cache-report\
\ \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint\
\ 127.0.0.1:4317 --otlp-service-name gist-exp-p \\\n --watchdog-timeout\
\ 3600 --enable-dp-attention-local-control-broadcast\n"
env:
- name: CUDA_VISIBLE_DEVICES
value: 0,1,2,3
- name: SGLANG_LOG_MS
value: '1'
- name: SGLANG_OPT_DSV41_ENGRAM_PEER_SHARDS
value: '1'
- name: SGLANG_OPT_DSV41_MEGA_MHC_PREFILL
value: '1'
- name: SGLANG_OPT_DSV41_BF16_INDEX_LOGITS
value: '1'
- name: SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK
value: '65536'
- name: SGLANG_DSV4_KV_LAYOUT
value: v41_new
- name: SGLANG_DP_USE_GATHERV
value: '1'
- name: SGLANG_RAGGED_VERIFY_MODE
value: static
- name: SGLANG_ENABLE_DP_SPEC_PREFILL_COORDINATION
value: '0'
- name: SGLANG_NUMA_BIND_V2
value: '0'
- name: SGLANG_DEFAULT_THINKING
value: '1'
- name: SGLANG_DSV41_REASONING_EFFORT
value: high
- name: SGLANG_TIMEOUT_KEEP_ALIVE
value: '900'
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
value: '1'
- name: MC_INTRANODE_NVLINK
value: 'true'
- name: MC_INTRA_NVLINK
value: 'true'
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
value: 'true'
- name: SGLANG_HICACHE_MOONCAKE_CONFIG_PATH
value: /data/ymk/ds41/mooncake-store-l3ssd.json
- name: SGLANG_HICACHE_PREFETCH_IO_THREADS
value: '4'
- name: SGLANG_HICACHE_BACKUP_IO_THREADS
value: '4'
- name: SGLANG_HICACHE_TIMING
value: '1'
- name: SGLANG_BATCH_TELEMETRY
value: '1'
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
value: '1'
- name: SGLANG_ENABLE_METRICS_DEVICE_TIMER
value: '1'
- name: SGLANG_EPLB_HEATMAP_COLLECTION_INTERVAL
value: '100'
- name: SGLANG_MM_TRACE
value: '1'
- name: SGLANG_BATCH_SPAN_MIN_MS
value: '200'
- name: SGLANG_BATCH_SPAN_SAMPLE
value: '0.05'
- name: SGLANG_BATCH_SPAN_MAX_LINKS
value: '8'
- name: SGLANG_BATCH_STAGE_TIMING
value: '1'
- name: SGLANG_BATCH_PASS_SPAN
value: '1'
- name: SGLANG_BATCH_PASS_SPAN_SAMPLE
value: '1.0'
- name: SGLANG_RECV_SPAN
value: '1'
- name: SGLANG_RECV_SPAN_SAMPLE
value: '1.0'
- name: SGLANG_MM_TRACE_SYNC
value: '0'
- name: SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK
value: '1'
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
value: '600'
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
value: '16'
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
value: '32'
- name: TZ
value: Asia/Shanghai
- name: SGLANG_TRACE_LOOP_SPAN
value: '1'
- name: SGLANG_TRACE_LOOP_SAMPLE
value: '1.0'
- name: SGLANG_FWD_TIMELINE
value: '0'
- name: SGLANG_TRACE_LOOP_DETAIL
value: '1'
- name: SGLANG_TRACE_MOONCAKE_DETAIL
value: '0'
- name: SGLANG_TRACE_BACKUP_DETAIL
value: '1'
- name: SGLANG_ASYNC_BACKUP_SUBMIT
value: '1'
- name: SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH
value: '1'
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
value: '1024'
- name: SGLANG_MOONCAKE_CLIENT_METRICS_PORT_BASE
value: '9400'
- name: MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR
value: bucket_storage_backend
- name: MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE
value: '1099511627776'
- name: MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES
value: '1099511627776'
- name: MOONCAKE_OFFLOAD_BUCKET_EVICTION_POLICY
value: fifo
- name: TMPDIR
value: /data/ymk/l3ssd/p-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/p-xdg
- name: HF_HOME
value: /data/ymk/l3ssd/p-hf
- name: MOONCAKE_OFFLOAD_FLUSH_PARTIAL_BUCKETS
value: '0'
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: sglang
ports:
- containerPort: 30000
name: metrics
protocol: TCP
- containerPort: 8998
name: bootstrap
protocol: TCP
- name: mc-0
containerPort: 9400
protocol: TCP
- name: mc-1
containerPort: 9401
protocol: TCP
- name: mc-2
containerPort: 9402
protocol: TCP
- name: mc-3
containerPort: 9403
protocol: TCP
readinessProbe:
failureThreshold: 120
httpGet:
path: /health
port: metrics
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '96'
memory: 2Ti
nvidia.com/gpu: '4'
requests:
cpu: '32'
memory: 512Gi
nvidia.com/gpu: '4'
securityContext:
privileged: true
volumeMounts:
- mountPath: /data
name: data
- mountPath: /root/.cache/sglang
name: sglang-cache
- mountPath: /dev/shm
name: dshm
- mountPath: /root/.cargo/bin/cargo
name: fake-cargo
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
hostPID: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/cache/sglang
type: Directory
name: sglang-cache
- emptyDir:
medium: Memory
sizeLimit: 64Gi
name: dshm
- hostPath:
path: /data/ymk/ds41/fake-cargo
type: File
name: fake-cargo
- dependencies:
- mooncake-master
minReadySeconds: 0
name: d
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '30300'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-d
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens\
\ --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name\
\ /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser\
\ auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"\
reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4\
\ --enable-dp-attention --enable-dp-lm-head \\\n --enable-fp32-lm-head\
\ --enable-dp-attention-local-control-broadcast \\\n --moe-a2a-backend\
\ megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts\
\ 64 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r64_ep4.json\
\ \\\n --mem-fraction-static 0.80 \\\n --speculative-algorithm DSPARK\
\ --speculative-dspark-block-size 3 \\\n --stream-interval 32 --swa-prefix-tails\
\ 2048 --enable-decoder-swa-bounded-replay \\\n --max-running-requests\
\ 512 --cuda-graph-max-bs-decode 512 \\\n --chunked-prefill-size 32768\
\ --max-prefill-tokens 8192 \\\n --disaggregation-mode decode --disaggregation-transfer-backend\
\ mooncake \\\n --host 0.0.0.0 --port 30300 --enable-metrics --enable-cache-report\
\ \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint\
\ 127.0.0.1:4317 --otlp-service-name gist-exp-d \\\n --watchdog-timeout\
\ 3600\n"
env:
- name: CUDA_VISIBLE_DEVICES
value: 4,5,6,7
- name: SGLANG_RAGGED_VERIFY_MODE
value: static
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
value: '1'
- name: SGLANG_DSV4_KV_LAYOUT
value: v41_new
- name: SGLANG_DEFAULT_THINKING
value: '1'
- name: SGLANG_DSV41_REASONING_EFFORT
value: high
- name: MC_INTRANODE_NVLINK
value: 'true'
- name: MC_INTRA_NVLINK
value: 'true'
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
value: 'true'
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
value: '1'
- name: SGLANG_BATCH_TELEMETRY
value: '1'
- name: SGLANG_BATCH_SPAN_SAMPLE
value: '0.1'
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
value: '600'
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
value: '16'
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
value: '32'
- name: TZ
value: Asia/Shanghai
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
value: '1024'
- name: TMPDIR
value: /data/ymk/l3ssd/d-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/d-xdg
- name: HF_HOME
value: /data/ymk/l3ssd/d-hf
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: sglang
ports:
- containerPort: 30300
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 150
httpGet:
path: /health
port: metrics
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '96'
memory: 550Gi
nvidia.com/gpu: '4'
requests:
cpu: '32'
memory: 256Gi
nvidia.com/gpu: '4'
securityContext:
privileged: true
volumeMounts:
- mountPath: /data
name: data
- mountPath: /root/.cache/sglang
name: sglang-cache
- mountPath: /dev/shm
name: dshm
- mountPath: /root/.cargo/bin/cargo
name: fake-cargo
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
hostPID: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/cache/sglang
type: Directory
name: sglang-cache
- emptyDir:
medium: Memory
sizeLimit: 64Gi
name: dshm
- hostPath:
path: /data/ymk/ds41/fake-cargo
type: File
name: fake-cargo
- dependencies:
- p
- d
minReadySeconds: 0
name: gateway
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '29000'
prometheus.io/scheme: http
prometheus.io/scrape: 'true'
labels:
app: ds41-l3ssd-gw
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec /opt/sglang/bin/sgl-model-gateway launch \\\n --pd-disaggregation\
\ \\\n --prefill http://127.0.0.1:30000 8998 \\\n --decode http://127.0.0.1:30300\
\ \\\n --prefill-policy cache_aware --decode-policy cache_aware \\\n\
\ --g9i-enabled --g9i-max-new-tokens 4096 \\\n \\\n \\\n --g9i-regular-p-queue-reject-secs\
\ 32 \\\n --g9i-decode-max-running 512 \\\n --health-check-timeout-secs\
\ 30 --health-check-interval-secs 30 \\\n --cb-failure-threshold 50\
\ \\\n --model-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --tokenizer-path\
\ /data/models/DeepSeek-V4.1-Flash-1017 \\\n --host 0.0.0.0 --port\
\ 30400\n"
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: gateway
ports:
- containerPort: 30400
name: http
protocol: TCP
- containerPort: 29000
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 60
httpGet:
path: /health
port: http
initialDelaySeconds: 20
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '16'
memory: 32Gi
requests:
cpu: '2'
memory: 8Gi
volumeMounts:
- mountPath: /data
name: data
- mountPath: /opt/sglang/bin/sgl-model-gateway
name: gw-bin
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- name: gw-bin
hostPath:
path: /data/ymk/cache/gw-est-td/sgl-model-gateway
type: File
@@ -0,0 +1,544 @@
apiVersion: workloads.x-k8s.io/v1alpha2
kind: RoleBasedGroup
metadata:
annotations:
rbg.workloads.x-k8s.io/discovery-config-mode: refine
labels:
app: ds41-gist-exp
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
sglang-model: ds-v41-flash
name: ds41-gist-exp
namespace: prj-d7te67bj0sq2fgdrmbig
spec:
roles:
- minReadySeconds: 0
name: mooncake-master
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '19003'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-mooncake
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "# 18081: 宿主机 18080 被 AgentX fake upstream(./fakeup)长期占用;\n# 我们的 P/D\
\ 用 metadata_server=P2PHANDSHAKE,此 HTTP 端口只是 master 可选项。\nexec /opt/sglang/bin/mooncake_master\
\ \\\n --port 50051 --enable_http_metadata_server=true \\\n --http_metadata_server_port=18081\
\ --metrics_port=19003 --eviction_high_watermark_ratio=0.90 --enable_offload=true\
\ --offload_on_evict=false --promotion_on_hit=false --offload_force_evict=false\n"
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: mooncake-master
ports:
- containerPort: 19003
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 60
initialDelaySeconds: 5
periodSeconds: 5
tcpSocket:
port: 50051
resources:
limits:
cpu: '4'
memory: 64Gi
requests:
cpu: '1'
memory: 8Gi
env:
- name: TMPDIR
value: /data/ymk/l3ssd/master-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/master-xdg
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
- dependencies:
- mooncake-master
minReadySeconds: 0
name: p
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '30000'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-p
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens\
\ --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name\
\ /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser\
\ auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"\
reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4\
\ --enable-dp-attention --enable-dp-lm-head \\\n --moe-a2a-backend\
\ megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts\
\ 32 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r32_ep4.json\
\ \\\n --mem-fraction-static 0.80 \\\n --chunked-prefill-size 131072\
\ --max-prefill-tokens 131072 \\\n --max-running-requests 512 --tokenizer-worker-num\
\ 16 \\\n --load-balance-method total_tokens \\\n --speculative-algorithm\
\ DSPARK --speculative-dspark-block-size 3 \\\n --enable-decoder-swa-bounded-replay\
\ --swa-full-tokens-ratio 0.12 \\\n --cuda-graph-max-bs-decode 1 --disable-decode-cuda-graph\
\ \\\n --cuda-graph-backend-prefill breakable \\\n --cuda-graph-bs-prefill\
\ 1024 2048 4096 8192 16384 32768 \\\n --cuda-graph-prefill-max-context\
\ 262144 --cuda-graph-max-seq-len-prefill 262144 \\\n --enable-hierarchical-cache\
\ --hicache-ratio 4 \\\n --hicache-mem-layout page_first_direct --hicache-io-backend\
\ direct \\\n --hicache-write-policy write_through --hicache-storage-backend\
\ mooncake \\\n --hicache-storage-prefetch-policy wait_complete \\\n\
\ --disaggregation-mode prefill --disaggregation-transfer-backend mooncake\
\ \\\n --disaggregation-bootstrap-port 8998 \\\n --expert-balancedness-report-mode\
\ prometheus \\\n --expert-distribution-recorder-buffer-size 64 \\\n\
\ --host 0.0.0.0 --port 30000 --enable-metrics --enable-cache-report\
\ \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint\
\ 127.0.0.1:4317 --otlp-service-name gist-exp-p \\\n --watchdog-timeout\
\ 3600 --enable-dp-attention-local-control-broadcast\n"
env:
- name: CUDA_VISIBLE_DEVICES
value: 0,1,2,3
- name: SGLANG_LOG_MS
value: '1'
- name: SGLANG_OPT_DSV41_ENGRAM_PEER_SHARDS
value: '1'
- name: SGLANG_OPT_DSV41_MEGA_MHC_PREFILL
value: '1'
- name: SGLANG_OPT_DSV41_BF16_INDEX_LOGITS
value: '1'
- name: SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK
value: '65536'
- name: SGLANG_DSV4_KV_LAYOUT
value: v41_new
- name: SGLANG_DP_USE_GATHERV
value: '1'
- name: SGLANG_RAGGED_VERIFY_MODE
value: static
- name: SGLANG_ENABLE_DP_SPEC_PREFILL_COORDINATION
value: '0'
- name: SGLANG_NUMA_BIND_V2
value: '0'
- name: SGLANG_DEFAULT_THINKING
value: '1'
- name: SGLANG_DSV41_REASONING_EFFORT
value: high
- name: SGLANG_TIMEOUT_KEEP_ALIVE
value: '900'
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
value: '1'
- name: MC_INTRANODE_NVLINK
value: 'true'
- name: MC_INTRA_NVLINK
value: 'true'
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
value: 'true'
- name: SGLANG_HICACHE_MOONCAKE_CONFIG_PATH
value: /data/ymk/ds41/mooncake-store-l3ssd.json
- name: SGLANG_HICACHE_PREFETCH_IO_THREADS
value: '4'
- name: SGLANG_HICACHE_BACKUP_IO_THREADS
value: '4'
- name: SGLANG_HICACHE_TIMING
value: '1'
- name: SGLANG_BATCH_TELEMETRY
value: '1'
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
value: '1'
- name: SGLANG_ENABLE_METRICS_DEVICE_TIMER
value: '1'
- name: SGLANG_EPLB_HEATMAP_COLLECTION_INTERVAL
value: '100'
- name: SGLANG_MM_TRACE
value: '1'
- name: SGLANG_BATCH_SPAN_MIN_MS
value: '200'
- name: SGLANG_BATCH_SPAN_SAMPLE
value: '0.05'
- name: SGLANG_BATCH_SPAN_MAX_LINKS
value: '8'
- name: SGLANG_BATCH_STAGE_TIMING
value: '1'
- name: SGLANG_BATCH_PASS_SPAN
value: '1'
- name: SGLANG_BATCH_PASS_SPAN_SAMPLE
value: '1.0'
- name: SGLANG_RECV_SPAN
value: '1'
- name: SGLANG_RECV_SPAN_SAMPLE
value: '1.0'
- name: SGLANG_MM_TRACE_SYNC
value: '0'
- name: SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK
value: '1'
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
value: '600'
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
value: '16'
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
value: '32'
- name: TZ
value: Asia/Shanghai
- name: SGLANG_TRACE_LOOP_SPAN
value: '1'
- name: SGLANG_TRACE_LOOP_SAMPLE
value: '1.0'
- name: SGLANG_FWD_TIMELINE
value: '0'
- name: SGLANG_TRACE_LOOP_DETAIL
value: '1'
- name: SGLANG_TRACE_MOONCAKE_DETAIL
value: '0'
- name: SGLANG_TRACE_BACKUP_DETAIL
value: '1'
- name: SGLANG_ASYNC_BACKUP_SUBMIT
value: '1'
- name: SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH
value: '1'
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
value: '1024'
- name: SGLANG_MOONCAKE_CLIENT_METRICS_PORT_BASE
value: '9400'
- name: MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR
value: bucket_storage_backend
- name: MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE
value: '1099511627776'
- name: MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES
value: '1099511627776'
- name: MOONCAKE_OFFLOAD_BUCKET_EVICTION_POLICY
value: fifo
- name: TMPDIR
value: /data/ymk/l3ssd/p-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/p-xdg
- name: HF_HOME
value: /data/ymk/l3ssd/p-hf
- name: MOONCAKE_OFFLOAD_FLUSH_PARTIAL_BUCKETS
value: '0'
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: sglang
ports:
- containerPort: 30000
name: metrics
protocol: TCP
- containerPort: 8998
name: bootstrap
protocol: TCP
- name: mc-0
containerPort: 9400
protocol: TCP
- name: mc-1
containerPort: 9401
protocol: TCP
- name: mc-2
containerPort: 9402
protocol: TCP
- name: mc-3
containerPort: 9403
protocol: TCP
readinessProbe:
failureThreshold: 120
httpGet:
path: /health
port: metrics
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '96'
memory: 2Ti
nvidia.com/gpu: '4'
requests:
cpu: '32'
memory: 512Gi
nvidia.com/gpu: '4'
securityContext:
privileged: true
volumeMounts:
- mountPath: /data
name: data
- mountPath: /root/.cache/sglang
name: sglang-cache
- mountPath: /dev/shm
name: dshm
- mountPath: /root/.cargo/bin/cargo
name: fake-cargo
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
hostPID: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/cache/sglang
type: Directory
name: sglang-cache
- emptyDir:
medium: Memory
sizeLimit: 64Gi
name: dshm
- hostPath:
path: /data/ymk/ds41/fake-cargo
type: File
name: fake-cargo
- dependencies:
- mooncake-master
minReadySeconds: 0
name: d
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '30300'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-d
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens\
\ --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name\
\ /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser\
\ auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"\
reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4\
\ --enable-dp-attention --enable-dp-lm-head \\\n --enable-fp32-lm-head\
\ --enable-dp-attention-local-control-broadcast \\\n --moe-a2a-backend\
\ megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts\
\ 64 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r64_ep4.json\
\ \\\n --mem-fraction-static 0.80 \\\n --speculative-algorithm DSPARK\
\ --speculative-dspark-block-size 3 \\\n --stream-interval 32 --swa-prefix-tails\
\ 2048 --enable-decoder-swa-bounded-replay \\\n --max-running-requests\
\ 512 --cuda-graph-max-bs-decode 512 \\\n --chunked-prefill-size 32768\
\ --max-prefill-tokens 8192 \\\n --disaggregation-mode decode --disaggregation-transfer-backend\
\ mooncake \\\n --host 0.0.0.0 --port 30300 --enable-metrics --enable-cache-report\
\ \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint\
\ 127.0.0.1:4317 --otlp-service-name gist-exp-d \\\n --watchdog-timeout\
\ 3600\n"
env:
- name: CUDA_VISIBLE_DEVICES
value: 4,5,6,7
- name: SGLANG_RAGGED_VERIFY_MODE
value: static
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
value: '1'
- name: SGLANG_DSV4_KV_LAYOUT
value: v41_new
- name: SGLANG_DEFAULT_THINKING
value: '1'
- name: SGLANG_DSV41_REASONING_EFFORT
value: high
- name: MC_INTRANODE_NVLINK
value: 'true'
- name: MC_INTRA_NVLINK
value: 'true'
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
value: 'true'
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
value: '1'
- name: SGLANG_BATCH_TELEMETRY
value: '1'
- name: SGLANG_BATCH_SPAN_SAMPLE
value: '0.1'
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
value: '600'
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
value: '16'
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
value: '32'
- name: TZ
value: Asia/Shanghai
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
value: '1024'
- name: TMPDIR
value: /data/ymk/l3ssd/d-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/d-xdg
- name: HF_HOME
value: /data/ymk/l3ssd/d-hf
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: sglang
ports:
- containerPort: 30300
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 150
httpGet:
path: /health
port: metrics
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '96'
memory: 550Gi
nvidia.com/gpu: '4'
requests:
cpu: '32'
memory: 256Gi
nvidia.com/gpu: '4'
securityContext:
privileged: true
volumeMounts:
- mountPath: /data
name: data
- mountPath: /root/.cache/sglang
name: sglang-cache
- mountPath: /dev/shm
name: dshm
- mountPath: /root/.cargo/bin/cargo
name: fake-cargo
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
hostPID: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/cache/sglang
type: Directory
name: sglang-cache
- emptyDir:
medium: Memory
sizeLimit: 64Gi
name: dshm
- hostPath:
path: /data/ymk/ds41/fake-cargo
type: File
name: fake-cargo
- dependencies:
- p
- d
minReadySeconds: 0
name: gateway
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '29000'
prometheus.io/scheme: http
prometheus.io/scrape: 'true'
labels:
app: ds41-l3ssd-gw
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec /opt/sglang/bin/sgl-model-gateway launch \\\n --pd-disaggregation\
\ \\\n --prefill http://127.0.0.1:30000 8998 \\\n --decode http://127.0.0.1:30300\
\ \\\n --prefill-policy cache_aware --decode-policy cache_aware \\\n\
\ --g9i-enabled --g9i-max-new-tokens 4096 --g9i-est-td-enabled --g9i-shadow-enabled\
\ --g9i-td-admit enforce --g9i-p-queue-reqs-cap 32 --g9i-p-queue-token-cap\
\ 16000000 \\\n \\\n \\\n --g9i-regular-p-queue-reject-secs 0 \\\n\
\ --g9i-decode-max-running 512 \\\n --health-check-timeout-secs 30\
\ --health-check-interval-secs 30 \\\n --cb-failure-threshold 50 \\\
\n --model-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --tokenizer-path\
\ /data/models/DeepSeek-V4.1-Flash-1017 \\\n --host 0.0.0.0 --port\
\ 30400\n"
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: gateway
ports:
- containerPort: 30400
name: http
protocol: TCP
- containerPort: 29000
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 60
httpGet:
path: /health
port: http
initialDelaySeconds: 20
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '16'
memory: 32Gi
requests:
cpu: '2'
memory: 8Gi
volumeMounts:
- mountPath: /data
name: data
- mountPath: /opt/sglang/bin/sgl-model-gateway
name: gw-bin
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- name: gw-bin
hostPath:
path: /data/ymk/gw-ns-est-td/sgl-model-gateway
type: File
@@ -0,0 +1,544 @@
apiVersion: workloads.x-k8s.io/v1alpha2
kind: RoleBasedGroup
metadata:
annotations:
rbg.workloads.x-k8s.io/discovery-config-mode: refine
labels:
app: ds41-gist-exp
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
sglang-model: ds-v41-flash
name: ds41-gist-exp
namespace: prj-d7te67bj0sq2fgdrmbig
spec:
roles:
- minReadySeconds: 0
name: mooncake-master
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '19003'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-mooncake
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "# 18081: 宿主机 18080 被 AgentX fake upstream(./fakeup)长期占用;\n# 我们的 P/D\
\ 用 metadata_server=P2PHANDSHAKE,此 HTTP 端口只是 master 可选项。\nexec /opt/sglang/bin/mooncake_master\
\ \\\n --port 50051 --enable_http_metadata_server=true \\\n --http_metadata_server_port=18081\
\ --metrics_port=19003 --eviction_high_watermark_ratio=0.90 --enable_offload=true\
\ --offload_on_evict=false --promotion_on_hit=false --offload_force_evict=false\n"
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: mooncake-master
ports:
- containerPort: 19003
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 60
initialDelaySeconds: 5
periodSeconds: 5
tcpSocket:
port: 50051
resources:
limits:
cpu: '4'
memory: 64Gi
requests:
cpu: '1'
memory: 8Gi
env:
- name: TMPDIR
value: /data/ymk/l3ssd/master-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/master-xdg
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
- dependencies:
- mooncake-master
minReadySeconds: 0
name: p
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '30000'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-p
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens\
\ --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name\
\ /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser\
\ auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"\
reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4\
\ --enable-dp-attention --enable-dp-lm-head \\\n --moe-a2a-backend\
\ megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts\
\ 32 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r32_ep4.json\
\ \\\n --mem-fraction-static 0.80 \\\n --chunked-prefill-size 131072\
\ --max-prefill-tokens 131072 \\\n --max-running-requests 512 --tokenizer-worker-num\
\ 16 \\\n --load-balance-method total_tokens \\\n --speculative-algorithm\
\ DSPARK --speculative-dspark-block-size 3 \\\n --enable-decoder-swa-bounded-replay\
\ --swa-full-tokens-ratio 0.12 \\\n --cuda-graph-max-bs-decode 1 --disable-decode-cuda-graph\
\ \\\n --cuda-graph-backend-prefill breakable \\\n --cuda-graph-bs-prefill\
\ 1024 2048 4096 8192 16384 32768 \\\n --cuda-graph-prefill-max-context\
\ 262144 --cuda-graph-max-seq-len-prefill 262144 \\\n --enable-hierarchical-cache\
\ --hicache-ratio 4 \\\n --hicache-mem-layout page_first_direct --hicache-io-backend\
\ direct \\\n --hicache-write-policy write_through --hicache-storage-backend\
\ mooncake \\\n --hicache-storage-prefetch-policy wait_complete \\\n\
\ --disaggregation-mode prefill --disaggregation-transfer-backend mooncake\
\ \\\n --disaggregation-bootstrap-port 8998 \\\n --expert-balancedness-report-mode\
\ prometheus \\\n --expert-distribution-recorder-buffer-size 64 \\\n\
\ --host 0.0.0.0 --port 30000 --enable-metrics --enable-cache-report\
\ \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint\
\ 127.0.0.1:4317 --otlp-service-name gist-exp-p \\\n --watchdog-timeout\
\ 3600 --enable-dp-attention-local-control-broadcast\n"
env:
- name: CUDA_VISIBLE_DEVICES
value: 0,1,2,3
- name: SGLANG_LOG_MS
value: '1'
- name: SGLANG_OPT_DSV41_ENGRAM_PEER_SHARDS
value: '1'
- name: SGLANG_OPT_DSV41_MEGA_MHC_PREFILL
value: '1'
- name: SGLANG_OPT_DSV41_BF16_INDEX_LOGITS
value: '1'
- name: SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK
value: '65536'
- name: SGLANG_DSV4_KV_LAYOUT
value: v41_new
- name: SGLANG_DP_USE_GATHERV
value: '1'
- name: SGLANG_RAGGED_VERIFY_MODE
value: static
- name: SGLANG_ENABLE_DP_SPEC_PREFILL_COORDINATION
value: '0'
- name: SGLANG_NUMA_BIND_V2
value: '0'
- name: SGLANG_DEFAULT_THINKING
value: '1'
- name: SGLANG_DSV41_REASONING_EFFORT
value: high
- name: SGLANG_TIMEOUT_KEEP_ALIVE
value: '900'
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
value: '1'
- name: MC_INTRANODE_NVLINK
value: 'true'
- name: MC_INTRA_NVLINK
value: 'true'
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
value: 'true'
- name: SGLANG_HICACHE_MOONCAKE_CONFIG_PATH
value: /data/ymk/ds41/mooncake-store-l3ssd.json
- name: SGLANG_HICACHE_PREFETCH_IO_THREADS
value: '4'
- name: SGLANG_HICACHE_BACKUP_IO_THREADS
value: '4'
- name: SGLANG_HICACHE_TIMING
value: '1'
- name: SGLANG_BATCH_TELEMETRY
value: '1'
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
value: '1'
- name: SGLANG_ENABLE_METRICS_DEVICE_TIMER
value: '1'
- name: SGLANG_EPLB_HEATMAP_COLLECTION_INTERVAL
value: '100'
- name: SGLANG_MM_TRACE
value: '1'
- name: SGLANG_BATCH_SPAN_MIN_MS
value: '200'
- name: SGLANG_BATCH_SPAN_SAMPLE
value: '0.05'
- name: SGLANG_BATCH_SPAN_MAX_LINKS
value: '8'
- name: SGLANG_BATCH_STAGE_TIMING
value: '1'
- name: SGLANG_BATCH_PASS_SPAN
value: '1'
- name: SGLANG_BATCH_PASS_SPAN_SAMPLE
value: '1.0'
- name: SGLANG_RECV_SPAN
value: '1'
- name: SGLANG_RECV_SPAN_SAMPLE
value: '1.0'
- name: SGLANG_MM_TRACE_SYNC
value: '0'
- name: SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK
value: '1'
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
value: '600'
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
value: '16'
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
value: '32'
- name: TZ
value: Asia/Shanghai
- name: SGLANG_TRACE_LOOP_SPAN
value: '1'
- name: SGLANG_TRACE_LOOP_SAMPLE
value: '1.0'
- name: SGLANG_FWD_TIMELINE
value: '0'
- name: SGLANG_TRACE_LOOP_DETAIL
value: '1'
- name: SGLANG_TRACE_MOONCAKE_DETAIL
value: '0'
- name: SGLANG_TRACE_BACKUP_DETAIL
value: '1'
- name: SGLANG_ASYNC_BACKUP_SUBMIT
value: '1'
- name: SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH
value: '1'
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
value: '1024'
- name: SGLANG_MOONCAKE_CLIENT_METRICS_PORT_BASE
value: '9400'
- name: MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR
value: bucket_storage_backend
- name: MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE
value: '1099511627776'
- name: MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES
value: '1099511627776'
- name: MOONCAKE_OFFLOAD_BUCKET_EVICTION_POLICY
value: fifo
- name: TMPDIR
value: /data/ymk/l3ssd/p-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/p-xdg
- name: HF_HOME
value: /data/ymk/l3ssd/p-hf
- name: MOONCAKE_OFFLOAD_FLUSH_PARTIAL_BUCKETS
value: '0'
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: sglang
ports:
- containerPort: 30000
name: metrics
protocol: TCP
- containerPort: 8998
name: bootstrap
protocol: TCP
- name: mc-0
containerPort: 9400
protocol: TCP
- name: mc-1
containerPort: 9401
protocol: TCP
- name: mc-2
containerPort: 9402
protocol: TCP
- name: mc-3
containerPort: 9403
protocol: TCP
readinessProbe:
failureThreshold: 120
httpGet:
path: /health
port: metrics
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '96'
memory: 2Ti
nvidia.com/gpu: '4'
requests:
cpu: '32'
memory: 512Gi
nvidia.com/gpu: '4'
securityContext:
privileged: true
volumeMounts:
- mountPath: /data
name: data
- mountPath: /root/.cache/sglang
name: sglang-cache
- mountPath: /dev/shm
name: dshm
- mountPath: /root/.cargo/bin/cargo
name: fake-cargo
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
hostPID: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/cache/sglang
type: Directory
name: sglang-cache
- emptyDir:
medium: Memory
sizeLimit: 64Gi
name: dshm
- hostPath:
path: /data/ymk/ds41/fake-cargo
type: File
name: fake-cargo
- dependencies:
- mooncake-master
minReadySeconds: 0
name: d
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '30300'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-d
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens\
\ --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name\
\ /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser\
\ auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"\
reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4\
\ --enable-dp-attention --enable-dp-lm-head \\\n --enable-fp32-lm-head\
\ --enable-dp-attention-local-control-broadcast \\\n --moe-a2a-backend\
\ megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts\
\ 64 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r64_ep4.json\
\ \\\n --mem-fraction-static 0.80 \\\n --speculative-algorithm DSPARK\
\ --speculative-dspark-block-size 3 \\\n --stream-interval 32 --swa-prefix-tails\
\ 2048 --enable-decoder-swa-bounded-replay \\\n --max-running-requests\
\ 512 --cuda-graph-max-bs-decode 512 \\\n --chunked-prefill-size 32768\
\ --max-prefill-tokens 8192 \\\n --disaggregation-mode decode --disaggregation-transfer-backend\
\ mooncake \\\n --host 0.0.0.0 --port 30300 --enable-metrics --enable-cache-report\
\ \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint\
\ 127.0.0.1:4317 --otlp-service-name gist-exp-d \\\n --watchdog-timeout\
\ 3600\n"
env:
- name: CUDA_VISIBLE_DEVICES
value: 4,5,6,7
- name: SGLANG_RAGGED_VERIFY_MODE
value: static
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
value: '1'
- name: SGLANG_DSV4_KV_LAYOUT
value: v41_new
- name: SGLANG_DEFAULT_THINKING
value: '1'
- name: SGLANG_DSV41_REASONING_EFFORT
value: high
- name: MC_INTRANODE_NVLINK
value: 'true'
- name: MC_INTRA_NVLINK
value: 'true'
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
value: 'true'
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
value: '1'
- name: SGLANG_BATCH_TELEMETRY
value: '1'
- name: SGLANG_BATCH_SPAN_SAMPLE
value: '0.1'
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
value: '600'
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
value: '16'
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
value: '32'
- name: TZ
value: Asia/Shanghai
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
value: '1024'
- name: TMPDIR
value: /data/ymk/l3ssd/d-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/d-xdg
- name: HF_HOME
value: /data/ymk/l3ssd/d-hf
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: sglang
ports:
- containerPort: 30300
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 150
httpGet:
path: /health
port: metrics
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '96'
memory: 550Gi
nvidia.com/gpu: '4'
requests:
cpu: '32'
memory: 256Gi
nvidia.com/gpu: '4'
securityContext:
privileged: true
volumeMounts:
- mountPath: /data
name: data
- mountPath: /root/.cache/sglang
name: sglang-cache
- mountPath: /dev/shm
name: dshm
- mountPath: /root/.cargo/bin/cargo
name: fake-cargo
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
hostPID: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/cache/sglang
type: Directory
name: sglang-cache
- emptyDir:
medium: Memory
sizeLimit: 64Gi
name: dshm
- hostPath:
path: /data/ymk/ds41/fake-cargo
type: File
name: fake-cargo
- dependencies:
- p
- d
minReadySeconds: 0
name: gateway
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '29000'
prometheus.io/scheme: http
prometheus.io/scrape: 'true'
labels:
app: ds41-l3ssd-gw
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec /opt/sglang/bin/sgl-model-gateway launch \\\n --pd-disaggregation\
\ \\\n --prefill http://127.0.0.1:30000 8998 \\\n --decode http://127.0.0.1:30300\
\ \\\n --prefill-policy cache_aware --decode-policy cache_aware \\\n\
\ --g9i-enabled --g9i-max-new-tokens 4096 --g9i-est-td-enabled --g9i-shadow-enabled\
\ --g9i-td-admit enforce --g9i-p-queue-reqs-cap 32 --g9i-p-queue-token-cap\
\ 16000000 --g9i-p-qps 20 --g9i-p-refresh-ms 250 \\\n \\\n \\\n --g9i-regular-p-queue-reject-secs\
\ 0 \\\n --g9i-decode-max-running 512 \\\n --health-check-timeout-secs\
\ 30 --health-check-interval-secs 30 \\\n --cb-failure-threshold 50\
\ \\\n --model-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --tokenizer-path\
\ /data/models/DeepSeek-V4.1-Flash-1017 \\\n --host 0.0.0.0 --port\
\ 30400\n"
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: gateway
ports:
- containerPort: 30400
name: http
protocol: TCP
- containerPort: 29000
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 60
httpGet:
path: /health
port: http
initialDelaySeconds: 20
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '16'
memory: 32Gi
requests:
cpu: '2'
memory: 8Gi
volumeMounts:
- mountPath: /data
name: data
- mountPath: /opt/sglang/bin/sgl-model-gateway
name: gw-bin
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- name: gw-bin
hostPath:
path: /data/ymk/gw-ns-est-td/sgl-model-gateway
type: File
@@ -0,0 +1,488 @@
apiVersion: workloads.x-k8s.io/v1alpha2
kind: RoleBasedGroup
metadata:
annotations:
rbg.workloads.x-k8s.io/discovery-config-mode: refine
labels:
app: ds41-gist-exp
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
sglang-model: ds-v41-flash
name: ds41-gist-exp
namespace: prj-d7te67bj0sq2fgdrmbig
spec:
roles:
- minReadySeconds: 0
name: mooncake-master
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '19003'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-mooncake
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "# 18081: 宿主机 18080 被 AgentX fake upstream(./fakeup)长期占用;\n# 我们的 P/D 用 metadata_server=P2PHANDSHAKE,此 HTTP 端口只是 master 可选项。\nexec /opt/sglang/bin/mooncake_master \\\n --port 50051 --enable_http_metadata_server=true \\\n --http_metadata_server_port=18081 --metrics_port=19003 --eviction_high_watermark_ratio=0.90 --enable_offload=true --offload_on_evict=false --promotion_on_hit=false --offload_force_evict=false\n"
env:
- name: TMPDIR
value: /data/ymk/l3ssd/master-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/master-xdg
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: mooncake-master
ports:
- containerPort: 19003
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 60
initialDelaySeconds: 5
periodSeconds: 5
tcpSocket:
port: 50051
resources:
limits:
cpu: '4'
memory: 64Gi
requests:
cpu: '1'
memory: 8Gi
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
- dependencies:
- mooncake-master
minReadySeconds: 0
name: p
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '30000'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-p
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 32 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r32_ep4.json \\\n --mem-fraction-static 0.80 \\\n --chunked-prefill-size 131072 --max-prefill-tokens 131072 \\\n --max-running-requests 512 --tokenizer-worker-num 16 \\\n --load-balance-method total_tokens \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --enable-decoder-swa-bounded-replay --swa-full-tokens-ratio 0.12 \\\n --cuda-graph-max-bs-decode 1 --disable-decode-cuda-graph \\\n --cuda-graph-backend-prefill breakable \\\n --cuda-graph-bs-prefill 1024 2048 4096 8192 16384 32768 \\\n --cuda-graph-prefill-max-context 262144 --cuda-graph-max-seq-len-prefill 262144 \\\n --enable-hierarchical-cache --hicache-ratio 4 --hicache-host-memory-mode buffer_only \\\n --hicache-mem-layout page_first_direct --hicache-io-backend direct \\\n --hicache-write-policy write_through --hicache-storage-backend mooncake \\\n --hicache-storage-prefetch-policy wait_complete --optimistic-prefill-attempts 4 \\\n --disaggregation-mode prefill --disaggregation-transfer-backend mooncake \\\n --disaggregation-bootstrap-port 8998 \\\n --expert-balancedness-report-mode prometheus \\\n --expert-distribution-recorder-buffer-size 64 \\\n --host 0.0.0.0 --port 30000 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-p \\\n --watchdog-timeout 3600 --enable-dp-attention-local-control-broadcast\n"
env:
- name: CUDA_VISIBLE_DEVICES
value: 0,1,2,3
- name: SGLANG_LOG_MS
value: '1'
- name: SGLANG_OPT_DSV41_ENGRAM_PEER_SHARDS
value: '1'
- name: SGLANG_OPT_DSV41_MEGA_MHC_PREFILL
value: '1'
- name: SGLANG_OPT_DSV41_BF16_INDEX_LOGITS
value: '1'
- name: SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK
value: '65536'
- name: SGLANG_DSV4_KV_LAYOUT
value: v41_new
- name: SGLANG_DP_USE_GATHERV
value: '1'
- name: SGLANG_RAGGED_VERIFY_MODE
value: static
- name: SGLANG_ENABLE_DP_SPEC_PREFILL_COORDINATION
value: '0'
- name: SGLANG_NUMA_BIND_V2
value: '0'
- name: SGLANG_DEFAULT_THINKING
value: '1'
- name: SGLANG_DSV41_REASONING_EFFORT
value: high
- name: SGLANG_TIMEOUT_KEEP_ALIVE
value: '900'
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
value: '1'
- name: MC_INTRANODE_NVLINK
value: 'true'
- name: MC_INTRA_NVLINK
value: 'true'
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
value: 'true'
- name: SGLANG_HICACHE_MOONCAKE_CONFIG_PATH
value: /data/ymk/ds41/mooncake-store-l3ssd.json
- name: SGLANG_HICACHE_PREFETCH_IO_THREADS
value: '4'
- name: SGLANG_HICACHE_BACKUP_IO_THREADS
value: '4'
- name: SGLANG_HICACHE_TIMING
value: '1'
- name: SGLANG_BATCH_TELEMETRY
value: '1'
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
value: '1'
- name: SGLANG_ENABLE_METRICS_DEVICE_TIMER
value: '1'
- name: SGLANG_EPLB_HEATMAP_COLLECTION_INTERVAL
value: '100'
- name: SGLANG_MM_TRACE
value: '1'
- name: SGLANG_BATCH_SPAN_MIN_MS
value: '200'
- name: SGLANG_BATCH_SPAN_SAMPLE
value: '0.05'
- name: SGLANG_BATCH_SPAN_MAX_LINKS
value: '8'
- name: SGLANG_BATCH_STAGE_TIMING
value: '1'
- name: SGLANG_BATCH_PASS_SPAN
value: '1'
- name: SGLANG_BATCH_PASS_SPAN_SAMPLE
value: '1.0'
- name: SGLANG_RECV_SPAN
value: '1'
- name: SGLANG_RECV_SPAN_SAMPLE
value: '1.0'
- name: SGLANG_MM_TRACE_SYNC
value: '0'
- name: SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK
value: '1'
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
value: '600'
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
value: '16'
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
value: '32'
- name: TZ
value: Asia/Shanghai
- name: SGLANG_TRACE_LOOP_SPAN
value: '1'
- name: SGLANG_TRACE_LOOP_SAMPLE
value: '1.0'
- name: SGLANG_FWD_TIMELINE
value: '0'
- name: SGLANG_TRACE_LOOP_DETAIL
value: '1'
- name: SGLANG_TRACE_MOONCAKE_DETAIL
value: '0'
- name: SGLANG_TRACE_BACKUP_DETAIL
value: '1'
- name: SGLANG_ASYNC_BACKUP_SUBMIT
value: '1'
- name: SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH
value: '1'
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
value: '1024'
- name: SGLANG_MOONCAKE_CLIENT_METRICS_PORT_BASE
value: '9400'
- name: MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR
value: bucket_storage_backend
- name: MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE
value: '1099511627776'
- name: MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES
value: '1099511627776'
- name: MOONCAKE_OFFLOAD_BUCKET_EVICTION_POLICY
value: fifo
- name: TMPDIR
value: /data/ymk/l3ssd/p-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/p-xdg
- name: HF_HOME
value: /data/ymk/l3ssd/p-hf
- name: MOONCAKE_OFFLOAD_FLUSH_PARTIAL_BUCKETS
value: '0'
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: sglang
ports:
- containerPort: 30000
name: metrics
protocol: TCP
- containerPort: 8998
name: bootstrap
protocol: TCP
- containerPort: 9400
name: mc-0
protocol: TCP
- containerPort: 9401
name: mc-1
protocol: TCP
- containerPort: 9402
name: mc-2
protocol: TCP
- containerPort: 9403
name: mc-3
protocol: TCP
readinessProbe:
failureThreshold: 120
httpGet:
path: /health
port: metrics
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '96'
memory: 2Ti
nvidia.com/gpu: '4'
requests:
cpu: '32'
memory: 512Gi
nvidia.com/gpu: '4'
securityContext:
privileged: true
volumeMounts:
- mountPath: /data
name: data
- mountPath: /root/.cache/sglang
name: sglang-cache
- mountPath: /dev/shm
name: dshm
- mountPath: /root/.cargo/bin/cargo
name: fake-cargo
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
hostPID: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/cache/sglang
type: Directory
name: sglang-cache
- emptyDir:
medium: Memory
sizeLimit: 64Gi
name: dshm
- hostPath:
path: /data/ymk/ds41/fake-cargo
type: File
name: fake-cargo
- dependencies:
- mooncake-master
minReadySeconds: 0
name: d
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '30300'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-d
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --enable-fp32-lm-head --enable-dp-attention-local-control-broadcast \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 64 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r64_ep4.json \\\n --mem-fraction-static 0.80 \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --stream-interval 32 --swa-prefix-tails 2048 --enable-decoder-swa-bounded-replay \\\n --max-running-requests 512 --cuda-graph-max-bs-decode 512 --tokenizer-worker-num 16 \\\n --chunked-prefill-size 32768 --max-prefill-tokens 8192 \\\n --disaggregation-mode decode --disaggregation-transfer-backend mooncake \\\n --host 0.0.0.0 --port 30300 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-d \\\n --watchdog-timeout 3600\n"
env:
- name: CUDA_VISIBLE_DEVICES
value: 4,5,6,7
- name: SGLANG_RAGGED_VERIFY_MODE
value: static
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
value: '1'
- name: SGLANG_DSV4_KV_LAYOUT
value: v41_new
- name: SGLANG_DEFAULT_THINKING
value: '1'
- name: SGLANG_DSV41_REASONING_EFFORT
value: high
- name: MC_INTRANODE_NVLINK
value: 'true'
- name: MC_INTRA_NVLINK
value: 'true'
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
value: 'true'
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
value: '1'
- name: SGLANG_BATCH_TELEMETRY
value: '1'
- name: SGLANG_BATCH_SPAN_SAMPLE
value: '0.1'
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
value: '600'
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
value: '16'
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
value: '32'
- name: TZ
value: Asia/Shanghai
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
value: '1024'
- name: TMPDIR
value: /data/ymk/l3ssd/d-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/d-xdg
- name: HF_HOME
value: /data/ymk/l3ssd/d-hf
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: sglang
ports:
- containerPort: 30300
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 150
httpGet:
path: /health
port: metrics
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '96'
memory: 550Gi
nvidia.com/gpu: '4'
requests:
cpu: '32'
memory: 256Gi
nvidia.com/gpu: '4'
securityContext:
privileged: true
volumeMounts:
- mountPath: /data
name: data
- mountPath: /root/.cache/sglang
name: sglang-cache
- mountPath: /dev/shm
name: dshm
- mountPath: /root/.cargo/bin/cargo
name: fake-cargo
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
hostPID: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/cache/sglang
type: Directory
name: sglang-cache
- emptyDir:
medium: Memory
sizeLimit: 64Gi
name: dshm
- hostPath:
path: /data/ymk/ds41/fake-cargo
type: File
name: fake-cargo
- dependencies:
- p
- d
minReadySeconds: 0
name: gateway
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '29000'
prometheus.io/scheme: http
prometheus.io/scrape: 'true'
labels:
app: ds41-l3ssd-gw
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec /opt/sglang/bin/sgl-model-gateway launch \\\n --pd-disaggregation \\\n --prefill http://127.0.0.1:30000 8998 \\\n --decode http://127.0.0.1:30300 \\\n --prefill-policy cache_aware --decode-policy cache_aware \\\n --g9i-enabled --g9i-max-new-tokens 4096 --g9i-est-td-enabled --g9i-shadow-enabled --g9i-td-admit enforce --g9i-p-queue-reqs-cap 64 --g9i-p-queue-token-cap 32000000 --g9i-p-qps 40 --g9i-p-refresh-ms 250 --g9i-window-fixed 256 \\\n \\\n \\\n --g9i-regular-p-queue-reject-secs 0 \\\n --g9i-decode-max-running 512 \\\n --pool-idle-timeout-secs 2 --pool-max-idle-per-host 0 \\\n --health-check-timeout-secs 30 --health-check-interval-secs 30 \\\n --cb-failure-threshold 50 \\\n --model-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --tokenizer-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --host 0.0.0.0 --port 30400\n"
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: gateway
ports:
- containerPort: 30400
name: http
protocol: TCP
- containerPort: 29000
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 60
httpGet:
path: /health
port: http
initialDelaySeconds: 20
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '16'
memory: 32Gi
requests:
cpu: '2'
memory: 8Gi
volumeMounts:
- mountPath: /data
name: data
- mountPath: /opt/sglang/bin/sgl-model-gateway
name: gw-bin
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/gw-ns-est-td/sgl-model-gateway
type: File
name: gw-bin
@@ -0,0 +1,488 @@
apiVersion: workloads.x-k8s.io/v1alpha2
kind: RoleBasedGroup
metadata:
annotations:
rbg.workloads.x-k8s.io/discovery-config-mode: refine
labels:
app: ds41-gist-exp
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
sglang-model: ds-v41-flash
name: ds41-gist-exp
namespace: prj-d7te67bj0sq2fgdrmbig
spec:
roles:
- minReadySeconds: 0
name: mooncake-master
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '19003'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-mooncake
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "# 18081: 宿主机 18080 被 AgentX fake upstream(./fakeup)长期占用;\n# 我们的 P/D 用 metadata_server=P2PHANDSHAKE,此 HTTP 端口只是 master 可选项。\nexec /opt/sglang/bin/mooncake_master \\\n --port 50051 --enable_http_metadata_server=true \\\n --http_metadata_server_port=18081 --metrics_port=19003 --eviction_high_watermark_ratio=0.90 --enable_offload=true --offload_on_evict=false --promotion_on_hit=false --offload_force_evict=false\n"
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: mooncake-master
ports:
- containerPort: 19003
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 60
initialDelaySeconds: 5
periodSeconds: 5
tcpSocket:
port: 50051
resources:
limits:
cpu: '4'
memory: 64Gi
requests:
cpu: '1'
memory: 8Gi
env:
- name: TMPDIR
value: /data/ymk/l3ssd/master-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/master-xdg
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
- dependencies:
- mooncake-master
minReadySeconds: 0
name: p
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '30000'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-p
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 32 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r32_ep4.json \\\n --mem-fraction-static 0.80 \\\n --chunked-prefill-size 131072 --max-prefill-tokens 131072 \\\n --max-running-requests 512 --tokenizer-worker-num 16 \\\n --load-balance-method total_tokens \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --enable-decoder-swa-bounded-replay --swa-full-tokens-ratio 0.12 \\\n --cuda-graph-max-bs-decode 1 --disable-decode-cuda-graph \\\n --cuda-graph-backend-prefill breakable \\\n --cuda-graph-bs-prefill 1024 2048 4096 8192 16384 32768 \\\n --cuda-graph-prefill-max-context 262144 --cuda-graph-max-seq-len-prefill 262144 \\\n --enable-hierarchical-cache --hicache-ratio 4 --hicache-host-memory-mode buffer_only \\\n --hicache-mem-layout page_first_direct --hicache-io-backend direct \\\n --hicache-write-policy write_through --hicache-storage-backend mooncake \\\n --hicache-storage-prefetch-policy wait_complete \\\n --disaggregation-mode prefill --disaggregation-transfer-backend mooncake \\\n --disaggregation-bootstrap-port 8998 \\\n --expert-balancedness-report-mode prometheus \\\n --expert-distribution-recorder-buffer-size 64 \\\n --host 0.0.0.0 --port 30000 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-p \\\n --watchdog-timeout 3600 --enable-dp-attention-local-control-broadcast\n"
env:
- name: CUDA_VISIBLE_DEVICES
value: 0,1,2,3
- name: SGLANG_LOG_MS
value: '1'
- name: SGLANG_OPT_DSV41_ENGRAM_PEER_SHARDS
value: '1'
- name: SGLANG_OPT_DSV41_MEGA_MHC_PREFILL
value: '1'
- name: SGLANG_OPT_DSV41_BF16_INDEX_LOGITS
value: '1'
- name: SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK
value: '65536'
- name: SGLANG_DSV4_KV_LAYOUT
value: v41_new
- name: SGLANG_DP_USE_GATHERV
value: '1'
- name: SGLANG_RAGGED_VERIFY_MODE
value: static
- name: SGLANG_ENABLE_DP_SPEC_PREFILL_COORDINATION
value: '0'
- name: SGLANG_NUMA_BIND_V2
value: '0'
- name: SGLANG_DEFAULT_THINKING
value: '1'
- name: SGLANG_DSV41_REASONING_EFFORT
value: high
- name: SGLANG_TIMEOUT_KEEP_ALIVE
value: '900'
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
value: '1'
- name: MC_INTRANODE_NVLINK
value: 'true'
- name: MC_INTRA_NVLINK
value: 'true'
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
value: 'true'
- name: SGLANG_HICACHE_MOONCAKE_CONFIG_PATH
value: /data/ymk/ds41/mooncake-store-l3ssd.json
- name: SGLANG_HICACHE_PREFETCH_IO_THREADS
value: '4'
- name: SGLANG_HICACHE_BACKUP_IO_THREADS
value: '4'
- name: SGLANG_HICACHE_TIMING
value: '1'
- name: SGLANG_BATCH_TELEMETRY
value: '1'
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
value: '1'
- name: SGLANG_ENABLE_METRICS_DEVICE_TIMER
value: '1'
- name: SGLANG_EPLB_HEATMAP_COLLECTION_INTERVAL
value: '100'
- name: SGLANG_MM_TRACE
value: '1'
- name: SGLANG_BATCH_SPAN_MIN_MS
value: '200'
- name: SGLANG_BATCH_SPAN_SAMPLE
value: '0.05'
- name: SGLANG_BATCH_SPAN_MAX_LINKS
value: '8'
- name: SGLANG_BATCH_STAGE_TIMING
value: '1'
- name: SGLANG_BATCH_PASS_SPAN
value: '1'
- name: SGLANG_BATCH_PASS_SPAN_SAMPLE
value: '1.0'
- name: SGLANG_RECV_SPAN
value: '1'
- name: SGLANG_RECV_SPAN_SAMPLE
value: '1.0'
- name: SGLANG_MM_TRACE_SYNC
value: '0'
- name: SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK
value: '1'
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
value: '600'
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
value: '16'
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
value: '32'
- name: TZ
value: Asia/Shanghai
- name: SGLANG_TRACE_LOOP_SPAN
value: '1'
- name: SGLANG_TRACE_LOOP_SAMPLE
value: '1.0'
- name: SGLANG_FWD_TIMELINE
value: '0'
- name: SGLANG_TRACE_LOOP_DETAIL
value: '1'
- name: SGLANG_TRACE_MOONCAKE_DETAIL
value: '0'
- name: SGLANG_TRACE_BACKUP_DETAIL
value: '1'
- name: SGLANG_ASYNC_BACKUP_SUBMIT
value: '1'
- name: SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH
value: '1'
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
value: '1024'
- name: SGLANG_MOONCAKE_CLIENT_METRICS_PORT_BASE
value: '9400'
- name: MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR
value: bucket_storage_backend
- name: MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE
value: '1099511627776'
- name: MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES
value: '1099511627776'
- name: MOONCAKE_OFFLOAD_BUCKET_EVICTION_POLICY
value: fifo
- name: TMPDIR
value: /data/ymk/l3ssd/p-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/p-xdg
- name: HF_HOME
value: /data/ymk/l3ssd/p-hf
- name: MOONCAKE_OFFLOAD_FLUSH_PARTIAL_BUCKETS
value: '0'
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: sglang
ports:
- containerPort: 30000
name: metrics
protocol: TCP
- containerPort: 8998
name: bootstrap
protocol: TCP
- name: mc-0
containerPort: 9400
protocol: TCP
- name: mc-1
containerPort: 9401
protocol: TCP
- name: mc-2
containerPort: 9402
protocol: TCP
- name: mc-3
containerPort: 9403
protocol: TCP
readinessProbe:
failureThreshold: 120
httpGet:
path: /health
port: metrics
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '96'
memory: 2Ti
nvidia.com/gpu: '4'
requests:
cpu: '32'
memory: 512Gi
nvidia.com/gpu: '4'
securityContext:
privileged: true
volumeMounts:
- mountPath: /data
name: data
- mountPath: /root/.cache/sglang
name: sglang-cache
- mountPath: /dev/shm
name: dshm
- mountPath: /root/.cargo/bin/cargo
name: fake-cargo
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
hostPID: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/cache/sglang
type: Directory
name: sglang-cache
- emptyDir:
medium: Memory
sizeLimit: 64Gi
name: dshm
- hostPath:
path: /data/ymk/ds41/fake-cargo
type: File
name: fake-cargo
- dependencies:
- mooncake-master
minReadySeconds: 0
name: d
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '30300'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-d
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --enable-fp32-lm-head --enable-dp-attention-local-control-broadcast \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 64 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r64_ep4.json \\\n --mem-fraction-static 0.80 \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --stream-interval 32 --swa-prefix-tails 2048 --enable-decoder-swa-bounded-replay \\\n --max-running-requests 512 --cuda-graph-max-bs-decode 512 --tokenizer-worker-num 16 \\\n --chunked-prefill-size 32768 --max-prefill-tokens 8192 \\\n --disaggregation-mode decode --disaggregation-transfer-backend mooncake \\\n --host 0.0.0.0 --port 30300 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-d \\\n --watchdog-timeout 3600\n"
env:
- name: CUDA_VISIBLE_DEVICES
value: 4,5,6,7
- name: SGLANG_RAGGED_VERIFY_MODE
value: static
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
value: '1'
- name: SGLANG_DSV4_KV_LAYOUT
value: v41_new
- name: SGLANG_DEFAULT_THINKING
value: '1'
- name: SGLANG_DSV41_REASONING_EFFORT
value: high
- name: MC_INTRANODE_NVLINK
value: 'true'
- name: MC_INTRA_NVLINK
value: 'true'
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
value: 'true'
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
value: '1'
- name: SGLANG_BATCH_TELEMETRY
value: '1'
- name: SGLANG_BATCH_SPAN_SAMPLE
value: '0.1'
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
value: '600'
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
value: '16'
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
value: '32'
- name: TZ
value: Asia/Shanghai
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
value: '1024'
- name: TMPDIR
value: /data/ymk/l3ssd/d-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/d-xdg
- name: HF_HOME
value: /data/ymk/l3ssd/d-hf
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: sglang
ports:
- containerPort: 30300
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 150
httpGet:
path: /health
port: metrics
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '96'
memory: 550Gi
nvidia.com/gpu: '4'
requests:
cpu: '32'
memory: 256Gi
nvidia.com/gpu: '4'
securityContext:
privileged: true
volumeMounts:
- mountPath: /data
name: data
- mountPath: /root/.cache/sglang
name: sglang-cache
- mountPath: /dev/shm
name: dshm
- mountPath: /root/.cargo/bin/cargo
name: fake-cargo
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
hostPID: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/cache/sglang
type: Directory
name: sglang-cache
- emptyDir:
medium: Memory
sizeLimit: 64Gi
name: dshm
- hostPath:
path: /data/ymk/ds41/fake-cargo
type: File
name: fake-cargo
- dependencies:
- p
- d
minReadySeconds: 0
name: gateway
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '29000'
prometheus.io/scheme: http
prometheus.io/scrape: 'true'
labels:
app: ds41-l3ssd-gw
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec /opt/sglang/bin/sgl-model-gateway launch \\\n --pd-disaggregation \\\n --prefill http://127.0.0.1:30000 8998 \\\n --decode http://127.0.0.1:30300 \\\n --prefill-policy cache_aware --decode-policy cache_aware \\\n --g9i-enabled --g9i-max-new-tokens 4096 \\\n \\\n \\\n --g9i-regular-p-queue-reject-secs 0 \\\n --g9i-decode-max-running 512 \\\n --health-check-timeout-secs 30 --health-check-interval-secs 30 \\\n --cb-failure-threshold 50 \\\n --model-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --tokenizer-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --host 0.0.0.0 --port 30400\n"
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: gateway
ports:
- containerPort: 30400
name: http
protocol: TCP
- containerPort: 29000
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 60
httpGet:
path: /health
port: http
initialDelaySeconds: 20
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '16'
memory: 32Gi
requests:
cpu: '2'
memory: 8Gi
volumeMounts:
- mountPath: /data
name: data
- mountPath: /opt/sglang/bin/sgl-model-gateway
name: gw-bin
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- name: gw-bin
hostPath:
path: /data/ymk/cache/gw-est-td/sgl-model-gateway
type: File
@@ -0,0 +1,488 @@
apiVersion: workloads.x-k8s.io/v1alpha2
kind: RoleBasedGroup
metadata:
annotations:
rbg.workloads.x-k8s.io/discovery-config-mode: refine
labels:
app: ds41-gist-exp
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
sglang-model: ds-v41-flash
name: ds41-gist-exp
namespace: prj-d7te67bj0sq2fgdrmbig
spec:
roles:
- minReadySeconds: 0
name: mooncake-master
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '19003'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-mooncake
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "# 18081: 宿主机 18080 被 AgentX fake upstream(./fakeup)长期占用;\n# 我们的 P/D 用 metadata_server=P2PHANDSHAKE,此 HTTP 端口只是 master 可选项。\nexec /opt/sglang/bin/mooncake_master \\\n --port 50051 --enable_http_metadata_server=true \\\n --http_metadata_server_port=18081 --metrics_port=19003 --eviction_high_watermark_ratio=0.90 --enable_offload=true --offload_on_evict=false --promotion_on_hit=false --offload_force_evict=false\n"
env:
- name: TMPDIR
value: /data/ymk/l3ssd/master-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/master-xdg
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: mooncake-master
ports:
- containerPort: 19003
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 60
initialDelaySeconds: 5
periodSeconds: 5
tcpSocket:
port: 50051
resources:
limits:
cpu: '4'
memory: 64Gi
requests:
cpu: '1'
memory: 8Gi
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
- dependencies:
- mooncake-master
minReadySeconds: 0
name: p
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '30000'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-p
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 32 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r32_ep4.json \\\n --mem-fraction-static 0.80 \\\n --chunked-prefill-size 131072 --max-prefill-tokens 131072 \\\n --max-running-requests 512 --tokenizer-worker-num 16 \\\n --load-balance-method total_tokens \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --enable-decoder-swa-bounded-replay --swa-full-tokens-ratio 0.12 \\\n --cuda-graph-max-bs-decode 1 --disable-decode-cuda-graph \\\n --cuda-graph-backend-prefill breakable \\\n --cuda-graph-bs-prefill 1024 2048 4096 8192 16384 32768 \\\n --cuda-graph-prefill-max-context 262144 --cuda-graph-max-seq-len-prefill 262144 \\\n --enable-hierarchical-cache --hicache-ratio 4 \\\n --hicache-mem-layout page_first_direct --hicache-io-backend direct \\\n --hicache-write-policy write_through --hicache-storage-backend mooncake \\\n --hicache-storage-prefetch-policy wait_complete \\\n --disaggregation-mode prefill --disaggregation-transfer-backend mooncake \\\n --disaggregation-bootstrap-port 8998 \\\n --expert-balancedness-report-mode prometheus \\\n --expert-distribution-recorder-buffer-size 64 \\\n --host 0.0.0.0 --port 30000 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-p \\\n --watchdog-timeout 3600 --enable-dp-attention-local-control-broadcast\n"
env:
- name: CUDA_VISIBLE_DEVICES
value: 0,1,2,3
- name: SGLANG_LOG_MS
value: '1'
- name: SGLANG_OPT_DSV41_ENGRAM_PEER_SHARDS
value: '1'
- name: SGLANG_OPT_DSV41_MEGA_MHC_PREFILL
value: '1'
- name: SGLANG_OPT_DSV41_BF16_INDEX_LOGITS
value: '1'
- name: SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK
value: '65536'
- name: SGLANG_DSV4_KV_LAYOUT
value: v41_new
- name: SGLANG_DP_USE_GATHERV
value: '1'
- name: SGLANG_RAGGED_VERIFY_MODE
value: static
- name: SGLANG_ENABLE_DP_SPEC_PREFILL_COORDINATION
value: '0'
- name: SGLANG_NUMA_BIND_V2
value: '0'
- name: SGLANG_DEFAULT_THINKING
value: '1'
- name: SGLANG_DSV41_REASONING_EFFORT
value: high
- name: SGLANG_TIMEOUT_KEEP_ALIVE
value: '900'
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
value: '1'
- name: MC_INTRANODE_NVLINK
value: 'true'
- name: MC_INTRA_NVLINK
value: 'true'
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
value: 'true'
- name: SGLANG_HICACHE_MOONCAKE_CONFIG_PATH
value: /data/ymk/ds41/mooncake-store-l3ssd.json
- name: SGLANG_HICACHE_PREFETCH_IO_THREADS
value: '4'
- name: SGLANG_HICACHE_BACKUP_IO_THREADS
value: '4'
- name: SGLANG_HICACHE_TIMING
value: '1'
- name: SGLANG_BATCH_TELEMETRY
value: '1'
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
value: '1'
- name: SGLANG_ENABLE_METRICS_DEVICE_TIMER
value: '1'
- name: SGLANG_EPLB_HEATMAP_COLLECTION_INTERVAL
value: '100'
- name: SGLANG_MM_TRACE
value: '1'
- name: SGLANG_BATCH_SPAN_MIN_MS
value: '200'
- name: SGLANG_BATCH_SPAN_SAMPLE
value: '0.05'
- name: SGLANG_BATCH_SPAN_MAX_LINKS
value: '8'
- name: SGLANG_BATCH_STAGE_TIMING
value: '1'
- name: SGLANG_BATCH_PASS_SPAN
value: '1'
- name: SGLANG_BATCH_PASS_SPAN_SAMPLE
value: '1.0'
- name: SGLANG_RECV_SPAN
value: '1'
- name: SGLANG_RECV_SPAN_SAMPLE
value: '1.0'
- name: SGLANG_MM_TRACE_SYNC
value: '0'
- name: SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK
value: '1'
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
value: '600'
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
value: '16'
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
value: '32'
- name: TZ
value: Asia/Shanghai
- name: SGLANG_TRACE_LOOP_SPAN
value: '1'
- name: SGLANG_TRACE_LOOP_SAMPLE
value: '1.0'
- name: SGLANG_FWD_TIMELINE
value: '0'
- name: SGLANG_TRACE_LOOP_DETAIL
value: '1'
- name: SGLANG_TRACE_MOONCAKE_DETAIL
value: '0'
- name: SGLANG_TRACE_BACKUP_DETAIL
value: '1'
- name: SGLANG_ASYNC_BACKUP_SUBMIT
value: '1'
- name: SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH
value: '1'
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
value: '1024'
- name: SGLANG_MOONCAKE_CLIENT_METRICS_PORT_BASE
value: '9400'
- name: MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR
value: bucket_storage_backend
- name: MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE
value: '1099511627776'
- name: MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES
value: '1099511627776'
- name: MOONCAKE_OFFLOAD_BUCKET_EVICTION_POLICY
value: fifo
- name: TMPDIR
value: /data/ymk/l3ssd/p-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/p-xdg
- name: HF_HOME
value: /data/ymk/l3ssd/p-hf
- name: MOONCAKE_OFFLOAD_FLUSH_PARTIAL_BUCKETS
value: '0'
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: sglang
ports:
- containerPort: 30000
name: metrics
protocol: TCP
- containerPort: 8998
name: bootstrap
protocol: TCP
- containerPort: 9400
name: mc-0
protocol: TCP
- containerPort: 9401
name: mc-1
protocol: TCP
- containerPort: 9402
name: mc-2
protocol: TCP
- containerPort: 9403
name: mc-3
protocol: TCP
readinessProbe:
failureThreshold: 120
httpGet:
path: /health
port: metrics
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '96'
memory: 2Ti
nvidia.com/gpu: '4'
requests:
cpu: '32'
memory: 512Gi
nvidia.com/gpu: '4'
securityContext:
privileged: true
volumeMounts:
- mountPath: /data
name: data
- mountPath: /root/.cache/sglang
name: sglang-cache
- mountPath: /dev/shm
name: dshm
- mountPath: /root/.cargo/bin/cargo
name: fake-cargo
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
hostPID: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/cache/sglang
type: Directory
name: sglang-cache
- emptyDir:
medium: Memory
sizeLimit: 64Gi
name: dshm
- hostPath:
path: /data/ymk/ds41/fake-cargo
type: File
name: fake-cargo
- dependencies:
- mooncake-master
minReadySeconds: 0
name: d
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '30300'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-d
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --enable-fp32-lm-head --enable-dp-attention-local-control-broadcast \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 64 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r64_ep4.json \\\n --mem-fraction-static 0.80 \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --stream-interval 32 --swa-prefix-tails 2048 --enable-decoder-swa-bounded-replay \\\n --max-running-requests 512 --cuda-graph-max-bs-decode 512 --tokenizer-worker-num 16 \\\n --chunked-prefill-size 32768 --max-prefill-tokens 8192 \\\n --disaggregation-mode decode --disaggregation-transfer-backend mooncake \\\n --host 0.0.0.0 --port 30300 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-d \\\n --watchdog-timeout 3600\n"
env:
- name: CUDA_VISIBLE_DEVICES
value: 4,5,6,7
- name: SGLANG_RAGGED_VERIFY_MODE
value: static
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
value: '1'
- name: SGLANG_DSV4_KV_LAYOUT
value: v41_new
- name: SGLANG_DEFAULT_THINKING
value: '1'
- name: SGLANG_DSV41_REASONING_EFFORT
value: high
- name: MC_INTRANODE_NVLINK
value: 'true'
- name: MC_INTRA_NVLINK
value: 'true'
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
value: 'true'
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
value: '1'
- name: SGLANG_BATCH_TELEMETRY
value: '1'
- name: SGLANG_BATCH_SPAN_SAMPLE
value: '0.1'
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
value: '600'
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
value: '16'
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
value: '32'
- name: TZ
value: Asia/Shanghai
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
value: '1024'
- name: TMPDIR
value: /data/ymk/l3ssd/d-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/d-xdg
- name: HF_HOME
value: /data/ymk/l3ssd/d-hf
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: sglang
ports:
- containerPort: 30300
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 150
httpGet:
path: /health
port: metrics
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '96'
memory: 550Gi
nvidia.com/gpu: '4'
requests:
cpu: '32'
memory: 256Gi
nvidia.com/gpu: '4'
securityContext:
privileged: true
volumeMounts:
- mountPath: /data
name: data
- mountPath: /root/.cache/sglang
name: sglang-cache
- mountPath: /dev/shm
name: dshm
- mountPath: /root/.cargo/bin/cargo
name: fake-cargo
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
hostPID: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/cache/sglang
type: Directory
name: sglang-cache
- emptyDir:
medium: Memory
sizeLimit: 64Gi
name: dshm
- hostPath:
path: /data/ymk/ds41/fake-cargo
type: File
name: fake-cargo
- dependencies:
- p
- d
minReadySeconds: 0
name: gateway
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '29000'
prometheus.io/scheme: http
prometheus.io/scrape: 'true'
labels:
app: ds41-l3ssd-gw
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec /opt/sglang/bin/sgl-model-gateway launch \\\n --pd-disaggregation \\\n --prefill http://127.0.0.1:30000 8998 \\\n --decode http://127.0.0.1:30300 \\\n --prefill-policy cache_aware --decode-policy cache_aware \\\n --g9i-enabled --g9i-max-new-tokens 4096 --g9i-est-td-enabled --g9i-shadow-enabled --g9i-td-admit off --g9i-p-queue-reqs-cap 0 --g9i-p-queue-token-cap 0 --g9i-p-qps 0 --g9i-p-refresh-ms 250 --g9i-window-fixed 256 \\\n \\\n \\\n --g9i-regular-p-queue-reject-secs 0 \\\n --g9i-decode-max-running 1000000 \\\n --pool-idle-timeout-secs 2 --pool-max-idle-per-host 0 \\\n --health-check-timeout-secs 30 --health-check-interval-secs 30 \\\n --cb-failure-threshold 50 --disable-circuit-breaker --disable-health-check \\\n --model-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --tokenizer-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --host 0.0.0.0 --port 30400\n"
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: gateway
ports:
- containerPort: 30400
name: http
protocol: TCP
- containerPort: 29000
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 60
httpGet:
path: /health
port: http
initialDelaySeconds: 20
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '16'
memory: 32Gi
requests:
cpu: '2'
memory: 8Gi
volumeMounts:
- mountPath: /data
name: data
- mountPath: /opt/sglang/bin/sgl-model-gateway
name: gw-bin
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/gw-ns-est-td/sgl-model-gateway
type: File
name: gw-bin
@@ -0,0 +1,488 @@
apiVersion: workloads.x-k8s.io/v1alpha2
kind: RoleBasedGroup
metadata:
annotations:
rbg.workloads.x-k8s.io/discovery-config-mode: refine
labels:
app: ds41-gist-exp
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
sglang-model: ds-v41-flash
name: ds41-gist-exp
namespace: prj-d7te67bj0sq2fgdrmbig
spec:
roles:
- minReadySeconds: 0
name: mooncake-master
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '19003'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-mooncake
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "# 18081: 宿主机 18080 被 AgentX fake upstream(./fakeup)长期占用;\n# 我们的 P/D 用 metadata_server=P2PHANDSHAKE,此 HTTP 端口只是 master 可选项。\nexec /opt/sglang/bin/mooncake_master \\\n --port 50051 --enable_http_metadata_server=true \\\n --http_metadata_server_port=18081 --metrics_port=19003 --eviction_high_watermark_ratio=0.90 --enable_offload=true --offload_on_evict=false --promotion_on_hit=false --offload_force_evict=false\n"
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: mooncake-master
ports:
- containerPort: 19003
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 60
initialDelaySeconds: 5
periodSeconds: 5
tcpSocket:
port: 50051
resources:
limits:
cpu: '4'
memory: 64Gi
requests:
cpu: '1'
memory: 8Gi
env:
- name: TMPDIR
value: /data/ymk/l3ssd/master-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/master-xdg
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
- dependencies:
- mooncake-master
minReadySeconds: 0
name: p
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '30000'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-p
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 32 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r32_ep4.json \\\n --mem-fraction-static 0.80 \\\n --chunked-prefill-size 131072 --max-prefill-tokens 131072 \\\n --max-running-requests 512 --tokenizer-worker-num 16 \\\n --load-balance-method total_tokens \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --enable-decoder-swa-bounded-replay --swa-full-tokens-ratio 0.12 \\\n --cuda-graph-max-bs-decode 1 --disable-decode-cuda-graph \\\n --cuda-graph-backend-prefill breakable \\\n --cuda-graph-bs-prefill 1024 2048 4096 8192 16384 32768 \\\n --cuda-graph-prefill-max-context 262144 --cuda-graph-max-seq-len-prefill 262144 \\\n --enable-hierarchical-cache --hicache-ratio 4 \\\n --hicache-mem-layout page_first_direct --hicache-io-backend direct \\\n --hicache-write-policy write_through --hicache-storage-backend mooncake \\\n --hicache-storage-prefetch-policy wait_complete \\\n --disaggregation-mode prefill --disaggregation-transfer-backend mooncake \\\n --disaggregation-bootstrap-port 8998 \\\n --expert-balancedness-report-mode prometheus \\\n --expert-distribution-recorder-buffer-size 64 \\\n --host 0.0.0.0 --port 30000 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-p \\\n --watchdog-timeout 3600 --enable-dp-attention-local-control-broadcast\n"
env:
- name: CUDA_VISIBLE_DEVICES
value: 0,1,2,3
- name: SGLANG_LOG_MS
value: '1'
- name: SGLANG_OPT_DSV41_ENGRAM_PEER_SHARDS
value: '1'
- name: SGLANG_OPT_DSV41_MEGA_MHC_PREFILL
value: '1'
- name: SGLANG_OPT_DSV41_BF16_INDEX_LOGITS
value: '1'
- name: SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK
value: '65536'
- name: SGLANG_DSV4_KV_LAYOUT
value: v41_new
- name: SGLANG_DP_USE_GATHERV
value: '1'
- name: SGLANG_RAGGED_VERIFY_MODE
value: static
- name: SGLANG_ENABLE_DP_SPEC_PREFILL_COORDINATION
value: '0'
- name: SGLANG_NUMA_BIND_V2
value: '0'
- name: SGLANG_DEFAULT_THINKING
value: '1'
- name: SGLANG_DSV41_REASONING_EFFORT
value: high
- name: SGLANG_TIMEOUT_KEEP_ALIVE
value: '900'
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
value: '1'
- name: MC_INTRANODE_NVLINK
value: 'true'
- name: MC_INTRA_NVLINK
value: 'true'
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
value: 'true'
- name: SGLANG_HICACHE_MOONCAKE_CONFIG_PATH
value: /data/ymk/ds41/mooncake-store-l3ssd.json
- name: SGLANG_HICACHE_PREFETCH_IO_THREADS
value: '4'
- name: SGLANG_HICACHE_BACKUP_IO_THREADS
value: '4'
- name: SGLANG_HICACHE_TIMING
value: '1'
- name: SGLANG_BATCH_TELEMETRY
value: '1'
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
value: '1'
- name: SGLANG_ENABLE_METRICS_DEVICE_TIMER
value: '1'
- name: SGLANG_EPLB_HEATMAP_COLLECTION_INTERVAL
value: '100'
- name: SGLANG_MM_TRACE
value: '1'
- name: SGLANG_BATCH_SPAN_MIN_MS
value: '200'
- name: SGLANG_BATCH_SPAN_SAMPLE
value: '0.05'
- name: SGLANG_BATCH_SPAN_MAX_LINKS
value: '8'
- name: SGLANG_BATCH_STAGE_TIMING
value: '1'
- name: SGLANG_BATCH_PASS_SPAN
value: '1'
- name: SGLANG_BATCH_PASS_SPAN_SAMPLE
value: '1.0'
- name: SGLANG_RECV_SPAN
value: '1'
- name: SGLANG_RECV_SPAN_SAMPLE
value: '1.0'
- name: SGLANG_MM_TRACE_SYNC
value: '0'
- name: SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK
value: '1'
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
value: '600'
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
value: '16'
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
value: '32'
- name: TZ
value: Asia/Shanghai
- name: SGLANG_TRACE_LOOP_SPAN
value: '1'
- name: SGLANG_TRACE_LOOP_SAMPLE
value: '1.0'
- name: SGLANG_FWD_TIMELINE
value: '0'
- name: SGLANG_TRACE_LOOP_DETAIL
value: '1'
- name: SGLANG_TRACE_MOONCAKE_DETAIL
value: '0'
- name: SGLANG_TRACE_BACKUP_DETAIL
value: '1'
- name: SGLANG_ASYNC_BACKUP_SUBMIT
value: '1'
- name: SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH
value: '1'
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
value: '1024'
- name: SGLANG_MOONCAKE_CLIENT_METRICS_PORT_BASE
value: '9400'
- name: MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR
value: bucket_storage_backend
- name: MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE
value: '1099511627776'
- name: MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES
value: '1099511627776'
- name: MOONCAKE_OFFLOAD_BUCKET_EVICTION_POLICY
value: fifo
- name: TMPDIR
value: /data/ymk/l3ssd/p-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/p-xdg
- name: HF_HOME
value: /data/ymk/l3ssd/p-hf
- name: MOONCAKE_OFFLOAD_FLUSH_PARTIAL_BUCKETS
value: '0'
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: sglang
ports:
- containerPort: 30000
name: metrics
protocol: TCP
- containerPort: 8998
name: bootstrap
protocol: TCP
- name: mc-0
containerPort: 9400
protocol: TCP
- name: mc-1
containerPort: 9401
protocol: TCP
- name: mc-2
containerPort: 9402
protocol: TCP
- name: mc-3
containerPort: 9403
protocol: TCP
readinessProbe:
failureThreshold: 120
httpGet:
path: /health
port: metrics
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '96'
memory: 2Ti
nvidia.com/gpu: '4'
requests:
cpu: '32'
memory: 512Gi
nvidia.com/gpu: '4'
securityContext:
privileged: true
volumeMounts:
- mountPath: /data
name: data
- mountPath: /root/.cache/sglang
name: sglang-cache
- mountPath: /dev/shm
name: dshm
- mountPath: /root/.cargo/bin/cargo
name: fake-cargo
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
hostPID: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/cache/sglang
type: Directory
name: sglang-cache
- emptyDir:
medium: Memory
sizeLimit: 64Gi
name: dshm
- hostPath:
path: /data/ymk/ds41/fake-cargo
type: File
name: fake-cargo
- dependencies:
- mooncake-master
minReadySeconds: 0
name: d
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '30300'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-d
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --enable-fp32-lm-head --enable-dp-attention-local-control-broadcast \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 64 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r64_ep4.json \\\n --mem-fraction-static 0.80 \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --stream-interval 32 --swa-prefix-tails 2048 --enable-decoder-swa-bounded-replay \\\n --max-running-requests 512 --cuda-graph-max-bs-decode 512 --tokenizer-worker-num 16 \\\n --chunked-prefill-size 32768 --max-prefill-tokens 8192 \\\n --disaggregation-mode decode --disaggregation-transfer-backend mooncake \\\n --host 0.0.0.0 --port 30300 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-d \\\n --watchdog-timeout 3600\n"
env:
- name: CUDA_VISIBLE_DEVICES
value: 4,5,6,7
- name: SGLANG_RAGGED_VERIFY_MODE
value: static
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
value: '1'
- name: SGLANG_DSV4_KV_LAYOUT
value: v41_new
- name: SGLANG_DEFAULT_THINKING
value: '1'
- name: SGLANG_DSV41_REASONING_EFFORT
value: high
- name: MC_INTRANODE_NVLINK
value: 'true'
- name: MC_INTRA_NVLINK
value: 'true'
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
value: 'true'
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
value: '1'
- name: SGLANG_BATCH_TELEMETRY
value: '1'
- name: SGLANG_BATCH_SPAN_SAMPLE
value: '0.1'
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
value: '600'
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
value: '16'
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
value: '32'
- name: TZ
value: Asia/Shanghai
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
value: '1024'
- name: TMPDIR
value: /data/ymk/l3ssd/d-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/d-xdg
- name: HF_HOME
value: /data/ymk/l3ssd/d-hf
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: sglang
ports:
- containerPort: 30300
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 150
httpGet:
path: /health
port: metrics
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '96'
memory: 550Gi
nvidia.com/gpu: '4'
requests:
cpu: '32'
memory: 256Gi
nvidia.com/gpu: '4'
securityContext:
privileged: true
volumeMounts:
- mountPath: /data
name: data
- mountPath: /root/.cache/sglang
name: sglang-cache
- mountPath: /dev/shm
name: dshm
- mountPath: /root/.cargo/bin/cargo
name: fake-cargo
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
hostPID: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/cache/sglang
type: Directory
name: sglang-cache
- emptyDir:
medium: Memory
sizeLimit: 64Gi
name: dshm
- hostPath:
path: /data/ymk/ds41/fake-cargo
type: File
name: fake-cargo
- dependencies:
- p
- d
minReadySeconds: 0
name: gateway
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '29000'
prometheus.io/scheme: http
prometheus.io/scrape: 'true'
labels:
app: ds41-l3ssd-gw
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec /opt/sglang/bin/sgl-model-gateway launch \\\n --pd-disaggregation \\\n --prefill http://127.0.0.1:30000 8998 \\\n --decode http://127.0.0.1:30300 \\\n --prefill-policy cache_aware --decode-policy cache_aware \\\n --g9i-enabled --g9i-max-new-tokens 4096 \\\n \\\n \\\n --g9i-regular-p-queue-reject-secs 0 \\\n --g9i-decode-max-running 512 \\\n --health-check-timeout-secs 30 --health-check-interval-secs 30 \\\n --cb-failure-threshold 50 \\\n --model-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --tokenizer-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --host 0.0.0.0 --port 30400\n"
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: gateway
ports:
- containerPort: 30400
name: http
protocol: TCP
- containerPort: 29000
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 60
httpGet:
path: /health
port: http
initialDelaySeconds: 20
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '16'
memory: 32Gi
requests:
cpu: '2'
memory: 8Gi
volumeMounts:
- mountPath: /data
name: data
- mountPath: /opt/sglang/bin/sgl-model-gateway
name: gw-bin
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- name: gw-bin
hostPath:
path: /data/ymk/cache/gw-est-td/sgl-model-gateway
type: File
@@ -0,0 +1,488 @@
apiVersion: workloads.x-k8s.io/v1alpha2
kind: RoleBasedGroup
metadata:
annotations:
rbg.workloads.x-k8s.io/discovery-config-mode: refine
labels:
app: ds41-gist-exp
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
sglang-model: ds-v41-flash
name: ds41-gist-exp
namespace: prj-d7te67bj0sq2fgdrmbig
spec:
roles:
- minReadySeconds: 0
name: mooncake-master
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '19003'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-mooncake
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "# 18081: 宿主机 18080 被 AgentX fake upstream(./fakeup)长期占用;\n# 我们的 P/D 用 metadata_server=P2PHANDSHAKE,此 HTTP 端口只是 master 可选项。\nexec /opt/sglang/bin/mooncake_master \\\n --port 50051 --enable_http_metadata_server=true \\\n --http_metadata_server_port=18081 --metrics_port=19003 --eviction_high_watermark_ratio=0.90 --enable_offload=true --offload_on_evict=false --promotion_on_hit=false --offload_force_evict=false\n"
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: mooncake-master
ports:
- containerPort: 19003
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 60
initialDelaySeconds: 5
periodSeconds: 5
tcpSocket:
port: 50051
resources:
limits:
cpu: '4'
memory: 64Gi
requests:
cpu: '1'
memory: 8Gi
env:
- name: TMPDIR
value: /data/ymk/l3ssd/master-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/master-xdg
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
- dependencies:
- mooncake-master
minReadySeconds: 0
name: p
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '30000'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-p
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend huggingface --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 32 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r32_ep4.json \\\n --mem-fraction-static 0.80 \\\n --chunked-prefill-size 131072 --max-prefill-tokens 131072 \\\n --max-running-requests 512 --tokenizer-worker-num 16 \\\n --load-balance-method total_tokens \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --enable-decoder-swa-bounded-replay --swa-full-tokens-ratio 0.12 \\\n --cuda-graph-max-bs-decode 1 --disable-decode-cuda-graph \\\n --cuda-graph-backend-prefill breakable \\\n --cuda-graph-bs-prefill 1024 2048 4096 8192 16384 32768 \\\n --cuda-graph-prefill-max-context 262144 --cuda-graph-max-seq-len-prefill 262144 \\\n --enable-hierarchical-cache --hicache-ratio 4 \\\n --hicache-mem-layout page_first_direct --hicache-io-backend direct \\\n --hicache-write-policy write_through --hicache-storage-backend mooncake \\\n --hicache-storage-prefetch-policy wait_complete \\\n --disaggregation-mode prefill --disaggregation-transfer-backend mooncake \\\n --disaggregation-bootstrap-port 8998 \\\n --expert-balancedness-report-mode prometheus \\\n --expert-distribution-recorder-buffer-size 64 \\\n --host 0.0.0.0 --port 30000 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-p \\\n --watchdog-timeout 3600 --enable-dp-attention-local-control-broadcast\n"
env:
- name: CUDA_VISIBLE_DEVICES
value: 0,1,2,3
- name: SGLANG_LOG_MS
value: '1'
- name: SGLANG_OPT_DSV41_ENGRAM_PEER_SHARDS
value: '1'
- name: SGLANG_OPT_DSV41_MEGA_MHC_PREFILL
value: '1'
- name: SGLANG_OPT_DSV41_BF16_INDEX_LOGITS
value: '1'
- name: SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK
value: '65536'
- name: SGLANG_DSV4_KV_LAYOUT
value: v41_new
- name: SGLANG_DP_USE_GATHERV
value: '1'
- name: SGLANG_RAGGED_VERIFY_MODE
value: static
- name: SGLANG_ENABLE_DP_SPEC_PREFILL_COORDINATION
value: '0'
- name: SGLANG_NUMA_BIND_V2
value: '0'
- name: SGLANG_DEFAULT_THINKING
value: '1'
- name: SGLANG_DSV41_REASONING_EFFORT
value: high
- name: SGLANG_TIMEOUT_KEEP_ALIVE
value: '900'
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
value: '1'
- name: MC_INTRANODE_NVLINK
value: 'true'
- name: MC_INTRA_NVLINK
value: 'true'
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
value: 'true'
- name: SGLANG_HICACHE_MOONCAKE_CONFIG_PATH
value: /data/ymk/ds41/mooncake-store-l3ssd.json
- name: SGLANG_HICACHE_PREFETCH_IO_THREADS
value: '4'
- name: SGLANG_HICACHE_BACKUP_IO_THREADS
value: '4'
- name: SGLANG_HICACHE_TIMING
value: '1'
- name: SGLANG_BATCH_TELEMETRY
value: '1'
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
value: '1'
- name: SGLANG_ENABLE_METRICS_DEVICE_TIMER
value: '1'
- name: SGLANG_EPLB_HEATMAP_COLLECTION_INTERVAL
value: '100'
- name: SGLANG_MM_TRACE
value: '1'
- name: SGLANG_BATCH_SPAN_MIN_MS
value: '200'
- name: SGLANG_BATCH_SPAN_SAMPLE
value: '0.05'
- name: SGLANG_BATCH_SPAN_MAX_LINKS
value: '8'
- name: SGLANG_BATCH_STAGE_TIMING
value: '1'
- name: SGLANG_BATCH_PASS_SPAN
value: '1'
- name: SGLANG_BATCH_PASS_SPAN_SAMPLE
value: '1.0'
- name: SGLANG_RECV_SPAN
value: '1'
- name: SGLANG_RECV_SPAN_SAMPLE
value: '1.0'
- name: SGLANG_MM_TRACE_SYNC
value: '0'
- name: SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK
value: '1'
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
value: '600'
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
value: '16'
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
value: '32'
- name: TZ
value: Asia/Shanghai
- name: SGLANG_TRACE_LOOP_SPAN
value: '1'
- name: SGLANG_TRACE_LOOP_SAMPLE
value: '1.0'
- name: SGLANG_FWD_TIMELINE
value: '0'
- name: SGLANG_TRACE_LOOP_DETAIL
value: '1'
- name: SGLANG_TRACE_MOONCAKE_DETAIL
value: '0'
- name: SGLANG_TRACE_BACKUP_DETAIL
value: '1'
- name: SGLANG_ASYNC_BACKUP_SUBMIT
value: '1'
- name: SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH
value: '1'
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
value: '1024'
- name: SGLANG_MOONCAKE_CLIENT_METRICS_PORT_BASE
value: '9400'
- name: MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR
value: bucket_storage_backend
- name: MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE
value: '1099511627776'
- name: MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES
value: '1099511627776'
- name: MOONCAKE_OFFLOAD_BUCKET_EVICTION_POLICY
value: fifo
- name: TMPDIR
value: /data/ymk/l3ssd/p-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/p-xdg
- name: HF_HOME
value: /data/ymk/l3ssd/p-hf
- name: MOONCAKE_OFFLOAD_FLUSH_PARTIAL_BUCKETS
value: '0'
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: sglang
ports:
- containerPort: 30000
name: metrics
protocol: TCP
- containerPort: 8998
name: bootstrap
protocol: TCP
- name: mc-0
containerPort: 9400
protocol: TCP
- name: mc-1
containerPort: 9401
protocol: TCP
- name: mc-2
containerPort: 9402
protocol: TCP
- name: mc-3
containerPort: 9403
protocol: TCP
readinessProbe:
failureThreshold: 120
httpGet:
path: /health
port: metrics
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '96'
memory: 2Ti
nvidia.com/gpu: '4'
requests:
cpu: '32'
memory: 512Gi
nvidia.com/gpu: '4'
securityContext:
privileged: true
volumeMounts:
- mountPath: /data
name: data
- mountPath: /root/.cache/sglang
name: sglang-cache
- mountPath: /dev/shm
name: dshm
- mountPath: /root/.cargo/bin/cargo
name: fake-cargo
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
hostPID: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/cache/sglang
type: Directory
name: sglang-cache
- emptyDir:
medium: Memory
sizeLimit: 64Gi
name: dshm
- hostPath:
path: /data/ymk/ds41/fake-cargo
type: File
name: fake-cargo
- dependencies:
- mooncake-master
minReadySeconds: 0
name: d
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '30300'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-d
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend huggingface --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --enable-fp32-lm-head --enable-dp-attention-local-control-broadcast \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 64 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r64_ep4.json \\\n --mem-fraction-static 0.80 \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --stream-interval 32 --swa-prefix-tails 2048 --enable-decoder-swa-bounded-replay \\\n --max-running-requests 512 --cuda-graph-max-bs-decode 512 --tokenizer-worker-num 16 \\\n --chunked-prefill-size 32768 --max-prefill-tokens 8192 \\\n --disaggregation-mode decode --disaggregation-transfer-backend mooncake \\\n --host 0.0.0.0 --port 30300 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-d \\\n --watchdog-timeout 3600\n"
env:
- name: CUDA_VISIBLE_DEVICES
value: 4,5,6,7
- name: SGLANG_RAGGED_VERIFY_MODE
value: static
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
value: '1'
- name: SGLANG_DSV4_KV_LAYOUT
value: v41_new
- name: SGLANG_DEFAULT_THINKING
value: '1'
- name: SGLANG_DSV41_REASONING_EFFORT
value: high
- name: MC_INTRANODE_NVLINK
value: 'true'
- name: MC_INTRA_NVLINK
value: 'true'
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
value: 'true'
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
value: '1'
- name: SGLANG_BATCH_TELEMETRY
value: '1'
- name: SGLANG_BATCH_SPAN_SAMPLE
value: '0.1'
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
value: '600'
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
value: '16'
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
value: '32'
- name: TZ
value: Asia/Shanghai
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
value: '1024'
- name: TMPDIR
value: /data/ymk/l3ssd/d-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/d-xdg
- name: HF_HOME
value: /data/ymk/l3ssd/d-hf
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: sglang
ports:
- containerPort: 30300
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 150
httpGet:
path: /health
port: metrics
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '96'
memory: 550Gi
nvidia.com/gpu: '4'
requests:
cpu: '32'
memory: 256Gi
nvidia.com/gpu: '4'
securityContext:
privileged: true
volumeMounts:
- mountPath: /data
name: data
- mountPath: /root/.cache/sglang
name: sglang-cache
- mountPath: /dev/shm
name: dshm
- mountPath: /root/.cargo/bin/cargo
name: fake-cargo
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
hostPID: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/cache/sglang
type: Directory
name: sglang-cache
- emptyDir:
medium: Memory
sizeLimit: 64Gi
name: dshm
- hostPath:
path: /data/ymk/ds41/fake-cargo
type: File
name: fake-cargo
- dependencies:
- p
- d
minReadySeconds: 0
name: gateway
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '29000'
prometheus.io/scheme: http
prometheus.io/scrape: 'true'
labels:
app: ds41-l3ssd-gw
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec /opt/sglang/bin/sgl-model-gateway launch \\\n --pd-disaggregation \\\n --prefill http://127.0.0.1:30000 8998 \\\n --decode http://127.0.0.1:30300 \\\n --prefill-policy cache_aware --decode-policy cache_aware \\\n --g9i-enabled --g9i-max-new-tokens 4096 \\\n \\\n \\\n --g9i-regular-p-queue-reject-secs 0 \\\n --g9i-decode-max-running 512 \\\n --health-check-timeout-secs 30 --health-check-interval-secs 30 \\\n --cb-failure-threshold 50 \\\n --model-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --tokenizer-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --host 0.0.0.0 --port 30400\n"
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: gateway
ports:
- containerPort: 30400
name: http
protocol: TCP
- containerPort: 29000
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 60
httpGet:
path: /health
port: http
initialDelaySeconds: 20
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '16'
memory: 32Gi
requests:
cpu: '2'
memory: 8Gi
volumeMounts:
- mountPath: /data
name: data
- mountPath: /opt/sglang/bin/sgl-model-gateway
name: gw-bin
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- name: gw-bin
hostPath:
path: /data/ymk/cache/gw-est-td/sgl-model-gateway
type: File
@@ -0,0 +1,488 @@
apiVersion: workloads.x-k8s.io/v1alpha2
kind: RoleBasedGroup
metadata:
annotations:
rbg.workloads.x-k8s.io/discovery-config-mode: refine
labels:
app: ds41-gist-exp
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
sglang-model: ds-v41-flash
name: ds41-gist-exp
namespace: prj-d7te67bj0sq2fgdrmbig
spec:
roles:
- minReadySeconds: 0
name: mooncake-master
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '19003'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-mooncake
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "# 18081: 宿主机 18080 被 AgentX fake upstream(./fakeup)长期占用;\n# 我们的 P/D 用 metadata_server=P2PHANDSHAKE,此 HTTP 端口只是 master 可选项。\nexec /opt/sglang/bin/mooncake_master \\\n --port 50051 --enable_http_metadata_server=true \\\n --http_metadata_server_port=18081 --metrics_port=19003 --eviction_high_watermark_ratio=0.90 --enable_offload=true --offload_on_evict=true --promotion_on_hit=false --offload_force_evict=false\n"
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: mooncake-master
ports:
- containerPort: 19003
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 60
initialDelaySeconds: 5
periodSeconds: 5
tcpSocket:
port: 50051
resources:
limits:
cpu: '4'
memory: 64Gi
requests:
cpu: '1'
memory: 8Gi
env:
- name: TMPDIR
value: /data/ymk/l3ssd/master-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/master-xdg
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
- dependencies:
- mooncake-master
minReadySeconds: 0
name: p
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '30000'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-p
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec sglang serve \\\n --trust-remote-code --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 32 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r32_ep4.json \\\n --mem-fraction-static 0.80 \\\n --chunked-prefill-size 131072 --max-prefill-tokens 131072 \\\n --max-running-requests 512 --tokenizer-worker-num 16 \\\n --load-balance-method total_tokens \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --enable-decoder-swa-bounded-replay --swa-full-tokens-ratio 0.12 \\\n --cuda-graph-max-bs-decode 1 --disable-decode-cuda-graph \\\n --cuda-graph-backend-prefill breakable \\\n --cuda-graph-bs-prefill 1024 2048 4096 8192 16384 32768 \\\n --cuda-graph-prefill-max-context 262144 --cuda-graph-max-seq-len-prefill 262144 \\\n --enable-hierarchical-cache --hicache-ratio 4 \\\n --hicache-mem-layout page_first_direct --hicache-io-backend direct \\\n --hicache-write-policy write_through --hicache-storage-backend mooncake \\\n --hicache-storage-prefetch-policy wait_complete \\\n --disaggregation-mode prefill --disaggregation-transfer-backend mooncake \\\n --disaggregation-bootstrap-port 8998 \\\n --expert-balancedness-report-mode prometheus \\\n --expert-distribution-recorder-buffer-size 64 \\\n --host 0.0.0.0 --port 30000 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-p \\\n --watchdog-timeout 3600 --enable-dp-attention-local-control-broadcast\n"
env:
- name: CUDA_VISIBLE_DEVICES
value: 0,1,2,3
- name: SGLANG_LOG_MS
value: '1'
- name: SGLANG_OPT_DSV41_ENGRAM_PEER_SHARDS
value: '1'
- name: SGLANG_OPT_DSV41_MEGA_MHC_PREFILL
value: '1'
- name: SGLANG_OPT_DSV41_BF16_INDEX_LOGITS
value: '1'
- name: SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK
value: '65536'
- name: SGLANG_DSV4_KV_LAYOUT
value: v41_new
- name: SGLANG_DP_USE_GATHERV
value: '1'
- name: SGLANG_RAGGED_VERIFY_MODE
value: static
- name: SGLANG_ENABLE_DP_SPEC_PREFILL_COORDINATION
value: '0'
- name: SGLANG_NUMA_BIND_V2
value: '0'
- name: SGLANG_DEFAULT_THINKING
value: '1'
- name: SGLANG_DSV41_REASONING_EFFORT
value: high
- name: SGLANG_TIMEOUT_KEEP_ALIVE
value: '900'
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
value: '1'
- name: MC_INTRANODE_NVLINK
value: 'true'
- name: MC_INTRA_NVLINK
value: 'true'
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
value: 'true'
- name: SGLANG_HICACHE_MOONCAKE_CONFIG_PATH
value: /data/ymk/ds41/mooncake-store-l3ssd.json
- name: SGLANG_HICACHE_PREFETCH_IO_THREADS
value: '4'
- name: SGLANG_HICACHE_BACKUP_IO_THREADS
value: '4'
- name: SGLANG_HICACHE_TIMING
value: '1'
- name: SGLANG_BATCH_TELEMETRY
value: '1'
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
value: '1'
- name: SGLANG_ENABLE_METRICS_DEVICE_TIMER
value: '1'
- name: SGLANG_EPLB_HEATMAP_COLLECTION_INTERVAL
value: '100'
- name: SGLANG_MM_TRACE
value: '1'
- name: SGLANG_BATCH_SPAN_MIN_MS
value: '200'
- name: SGLANG_BATCH_SPAN_SAMPLE
value: '0.05'
- name: SGLANG_BATCH_SPAN_MAX_LINKS
value: '8'
- name: SGLANG_BATCH_STAGE_TIMING
value: '1'
- name: SGLANG_BATCH_PASS_SPAN
value: '1'
- name: SGLANG_BATCH_PASS_SPAN_SAMPLE
value: '1.0'
- name: SGLANG_RECV_SPAN
value: '1'
- name: SGLANG_RECV_SPAN_SAMPLE
value: '1.0'
- name: SGLANG_MM_TRACE_SYNC
value: '0'
- name: SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK
value: '1'
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
value: '600'
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
value: '16'
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
value: '32'
- name: TZ
value: Asia/Shanghai
- name: SGLANG_TRACE_LOOP_SPAN
value: '1'
- name: SGLANG_TRACE_LOOP_SAMPLE
value: '1.0'
- name: SGLANG_FWD_TIMELINE
value: '0'
- name: SGLANG_TRACE_LOOP_DETAIL
value: '1'
- name: SGLANG_TRACE_MOONCAKE_DETAIL
value: '0'
- name: SGLANG_TRACE_BACKUP_DETAIL
value: '1'
- name: SGLANG_ASYNC_BACKUP_SUBMIT
value: '1'
- name: SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH
value: '1'
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
value: '1024'
- name: SGLANG_MOONCAKE_CLIENT_METRICS_PORT_BASE
value: '9400'
- name: MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR
value: bucket_storage_backend
- name: MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE
value: '549755813888'
- name: MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES
value: '549755813888'
- name: MOONCAKE_OFFLOAD_BUCKET_EVICTION_POLICY
value: fifo
- name: TMPDIR
value: /data/ymk/l3ssd/p-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/p-xdg
- name: HF_HOME
value: /data/ymk/l3ssd/p-hf
- name: MOONCAKE_OFFLOAD_FLUSH_PARTIAL_BUCKETS
value: '0'
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: sglang
ports:
- containerPort: 30000
name: metrics
protocol: TCP
- containerPort: 8998
name: bootstrap
protocol: TCP
- name: mc-0
containerPort: 9400
protocol: TCP
- name: mc-1
containerPort: 9401
protocol: TCP
- name: mc-2
containerPort: 9402
protocol: TCP
- name: mc-3
containerPort: 9403
protocol: TCP
readinessProbe:
failureThreshold: 120
httpGet:
path: /health
port: metrics
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '96'
memory: 2Ti
nvidia.com/gpu: '4'
requests:
cpu: '32'
memory: 512Gi
nvidia.com/gpu: '4'
securityContext:
privileged: true
volumeMounts:
- mountPath: /data
name: data
- mountPath: /root/.cache/sglang
name: sglang-cache
- mountPath: /dev/shm
name: dshm
- mountPath: /root/.cargo/bin/cargo
name: fake-cargo
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
hostPID: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/cache/sglang
type: Directory
name: sglang-cache
- emptyDir:
medium: Memory
sizeLimit: 64Gi
name: dshm
- hostPath:
path: /data/ymk/ds41/fake-cargo
type: File
name: fake-cargo
- dependencies:
- mooncake-master
minReadySeconds: 0
name: d
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '30300'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-d
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec sglang serve \\\n --trust-remote-code --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --enable-fp32-lm-head --enable-dp-attention-local-control-broadcast \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 64 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r64_ep4.json \\\n --mem-fraction-static 0.80 \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --stream-interval 32 --swa-prefix-tails 2048 --enable-decoder-swa-bounded-replay \\\n --max-running-requests 512 --cuda-graph-max-bs-decode 512 \\\n --chunked-prefill-size 32768 --max-prefill-tokens 8192 \\\n --disaggregation-mode decode --disaggregation-transfer-backend mooncake \\\n --host 0.0.0.0 --port 30300 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-d \\\n --watchdog-timeout 3600\n"
env:
- name: CUDA_VISIBLE_DEVICES
value: 4,5,6,7
- name: SGLANG_RAGGED_VERIFY_MODE
value: static
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
value: '1'
- name: SGLANG_DSV4_KV_LAYOUT
value: v41_new
- name: SGLANG_DEFAULT_THINKING
value: '1'
- name: SGLANG_DSV41_REASONING_EFFORT
value: high
- name: MC_INTRANODE_NVLINK
value: 'true'
- name: MC_INTRA_NVLINK
value: 'true'
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
value: 'true'
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
value: '1'
- name: SGLANG_BATCH_TELEMETRY
value: '1'
- name: SGLANG_BATCH_SPAN_SAMPLE
value: '0.1'
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
value: '600'
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
value: '16'
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
value: '32'
- name: TZ
value: Asia/Shanghai
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
value: '1024'
- name: TMPDIR
value: /data/ymk/l3ssd/d-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/d-xdg
- name: HF_HOME
value: /data/ymk/l3ssd/d-hf
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: sglang
ports:
- containerPort: 30300
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 150
httpGet:
path: /health
port: metrics
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '96'
memory: 550Gi
nvidia.com/gpu: '4'
requests:
cpu: '32'
memory: 256Gi
nvidia.com/gpu: '4'
securityContext:
privileged: true
volumeMounts:
- mountPath: /data
name: data
- mountPath: /root/.cache/sglang
name: sglang-cache
- mountPath: /dev/shm
name: dshm
- mountPath: /root/.cargo/bin/cargo
name: fake-cargo
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
hostPID: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/cache/sglang
type: Directory
name: sglang-cache
- emptyDir:
medium: Memory
sizeLimit: 64Gi
name: dshm
- hostPath:
path: /data/ymk/ds41/fake-cargo
type: File
name: fake-cargo
- dependencies:
- p
- d
minReadySeconds: 0
name: gateway
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '29000'
prometheus.io/scheme: http
prometheus.io/scrape: 'true'
labels:
app: ds41-l3ssd-gw
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec /opt/sglang/bin/sgl-model-gateway launch \\\n --pd-disaggregation \\\n --prefill http://127.0.0.1:30000 8998 \\\n --decode http://127.0.0.1:30300 \\\n --prefill-policy cache_aware --decode-policy cache_aware \\\n --g9i-enabled --g9i-max-new-tokens 4096 \\\n \\\n \\\n --g9i-regular-p-queue-reject-secs 0 \\\n --g9i-decode-max-running 512 \\\n --health-check-timeout-secs 30 --health-check-interval-secs 30 \\\n --cb-failure-threshold 50 \\\n --model-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --tokenizer-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --host 0.0.0.0 --port 30400\n"
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: gateway
ports:
- containerPort: 30400
name: http
protocol: TCP
- containerPort: 29000
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 60
httpGet:
path: /health
port: http
initialDelaySeconds: 20
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '16'
memory: 32Gi
requests:
cpu: '2'
memory: 8Gi
volumeMounts:
- mountPath: /data
name: data
- mountPath: /opt/sglang/bin/sgl-model-gateway
name: gw-bin
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- name: gw-bin
hostPath:
path: /data/ymk/cache/gw-est-td/sgl-model-gateway
type: File
@@ -0,0 +1,488 @@
apiVersion: workloads.x-k8s.io/v1alpha2
kind: RoleBasedGroup
metadata:
annotations:
rbg.workloads.x-k8s.io/discovery-config-mode: refine
labels:
app: ds41-gist-exp
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
sglang-model: ds-v41-flash
name: ds41-gist-exp
namespace: prj-d7te67bj0sq2fgdrmbig
spec:
roles:
- minReadySeconds: 0
name: mooncake-master
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '19003'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-mooncake
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "# 18081: 宿主机 18080 被 AgentX fake upstream(./fakeup)长期占用;\n# 我们的 P/D 用 metadata_server=P2PHANDSHAKE,此 HTTP 端口只是 master 可选项。\nexec /opt/sglang/bin/mooncake_master \\\n --port 50051 --enable_http_metadata_server=true \\\n --http_metadata_server_port=18081 --metrics_port=19003 --eviction_high_watermark_ratio=0.90 --enable_offload=true --offload_on_evict=false --promotion_on_hit=false --offload_force_evict=false\n"
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: mooncake-master
ports:
- containerPort: 19003
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 60
initialDelaySeconds: 5
periodSeconds: 5
tcpSocket:
port: 50051
resources:
limits:
cpu: '4'
memory: 64Gi
requests:
cpu: '1'
memory: 8Gi
env:
- name: TMPDIR
value: /data/ymk/l3ssd/master-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/master-xdg
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
- dependencies:
- mooncake-master
minReadySeconds: 0
name: p
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '30000'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-p
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 32 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r32_ep4.json \\\n --mem-fraction-static 0.80 \\\n --chunked-prefill-size 131072 --max-prefill-tokens 131072 \\\n --max-running-requests 512 --tokenizer-worker-num 16 \\\n --load-balance-method total_tokens \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --enable-decoder-swa-bounded-replay --swa-full-tokens-ratio 0.12 \\\n --cuda-graph-max-bs-decode 1 --disable-decode-cuda-graph \\\n --cuda-graph-backend-prefill breakable \\\n --cuda-graph-bs-prefill 1024 2048 4096 8192 16384 32768 \\\n --cuda-graph-prefill-max-context 262144 --cuda-graph-max-seq-len-prefill 262144 \\\n --enable-hierarchical-cache --hicache-ratio 4 \\\n --hicache-mem-layout page_first_direct --hicache-io-backend direct \\\n --hicache-write-policy write_through --hicache-storage-backend mooncake \\\n --hicache-storage-prefetch-policy wait_complete \\\n --disaggregation-mode prefill --disaggregation-transfer-backend mooncake \\\n --disaggregation-bootstrap-port 8998 \\\n --expert-balancedness-report-mode prometheus \\\n --expert-distribution-recorder-buffer-size 64 \\\n --host 0.0.0.0 --port 30000 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-p \\\n --watchdog-timeout 3600 --enable-dp-attention-local-control-broadcast\n"
env:
- name: CUDA_VISIBLE_DEVICES
value: 0,1,2,3
- name: SGLANG_LOG_MS
value: '1'
- name: SGLANG_OPT_DSV41_ENGRAM_PEER_SHARDS
value: '1'
- name: SGLANG_OPT_DSV41_MEGA_MHC_PREFILL
value: '1'
- name: SGLANG_OPT_DSV41_BF16_INDEX_LOGITS
value: '1'
- name: SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK
value: '65536'
- name: SGLANG_DSV4_KV_LAYOUT
value: v41_new
- name: SGLANG_DP_USE_GATHERV
value: '1'
- name: SGLANG_RAGGED_VERIFY_MODE
value: static
- name: SGLANG_ENABLE_DP_SPEC_PREFILL_COORDINATION
value: '0'
- name: SGLANG_NUMA_BIND_V2
value: '0'
- name: SGLANG_DEFAULT_THINKING
value: '1'
- name: SGLANG_DSV41_REASONING_EFFORT
value: high
- name: SGLANG_TIMEOUT_KEEP_ALIVE
value: '900'
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
value: '1'
- name: MC_INTRANODE_NVLINK
value: 'true'
- name: MC_INTRA_NVLINK
value: 'true'
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
value: 'true'
- name: SGLANG_HICACHE_MOONCAKE_CONFIG_PATH
value: /data/ymk/ds41/mooncake-store-l3ssd.json
- name: SGLANG_HICACHE_PREFETCH_IO_THREADS
value: '4'
- name: SGLANG_HICACHE_BACKUP_IO_THREADS
value: '4'
- name: SGLANG_HICACHE_TIMING
value: '1'
- name: SGLANG_BATCH_TELEMETRY
value: '1'
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
value: '1'
- name: SGLANG_ENABLE_METRICS_DEVICE_TIMER
value: '1'
- name: SGLANG_EPLB_HEATMAP_COLLECTION_INTERVAL
value: '100'
- name: SGLANG_MM_TRACE
value: '1'
- name: SGLANG_BATCH_SPAN_MIN_MS
value: '200'
- name: SGLANG_BATCH_SPAN_SAMPLE
value: '0.05'
- name: SGLANG_BATCH_SPAN_MAX_LINKS
value: '8'
- name: SGLANG_BATCH_STAGE_TIMING
value: '1'
- name: SGLANG_BATCH_PASS_SPAN
value: '1'
- name: SGLANG_BATCH_PASS_SPAN_SAMPLE
value: '1.0'
- name: SGLANG_RECV_SPAN
value: '1'
- name: SGLANG_RECV_SPAN_SAMPLE
value: '1.0'
- name: SGLANG_MM_TRACE_SYNC
value: '0'
- name: SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK
value: '1'
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
value: '600'
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
value: '16'
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
value: '32'
- name: TZ
value: Asia/Shanghai
- name: SGLANG_TRACE_LOOP_SPAN
value: '1'
- name: SGLANG_TRACE_LOOP_SAMPLE
value: '1.0'
- name: SGLANG_FWD_TIMELINE
value: '0'
- name: SGLANG_TRACE_LOOP_DETAIL
value: '1'
- name: SGLANG_TRACE_MOONCAKE_DETAIL
value: '0'
- name: SGLANG_TRACE_BACKUP_DETAIL
value: '1'
- name: SGLANG_ASYNC_BACKUP_SUBMIT
value: '1'
- name: SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH
value: '1'
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
value: '1024'
- name: SGLANG_MOONCAKE_CLIENT_METRICS_PORT_BASE
value: '9400'
- name: MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR
value: bucket_storage_backend
- name: MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE
value: '1099511627776'
- name: MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES
value: '1099511627776'
- name: MOONCAKE_OFFLOAD_BUCKET_EVICTION_POLICY
value: fifo
- name: TMPDIR
value: /data/ymk/l3ssd/p-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/p-xdg
- name: HF_HOME
value: /data/ymk/l3ssd/p-hf
- name: MOONCAKE_OFFLOAD_FLUSH_PARTIAL_BUCKETS
value: '0'
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: sglang
ports:
- containerPort: 30000
name: metrics
protocol: TCP
- containerPort: 8998
name: bootstrap
protocol: TCP
- name: mc-0
containerPort: 9400
protocol: TCP
- name: mc-1
containerPort: 9401
protocol: TCP
- name: mc-2
containerPort: 9402
protocol: TCP
- name: mc-3
containerPort: 9403
protocol: TCP
readinessProbe:
failureThreshold: 120
httpGet:
path: /health
port: metrics
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '96'
memory: 2Ti
nvidia.com/gpu: '4'
requests:
cpu: '32'
memory: 512Gi
nvidia.com/gpu: '4'
securityContext:
privileged: true
volumeMounts:
- mountPath: /data
name: data
- mountPath: /root/.cache/sglang
name: sglang-cache
- mountPath: /dev/shm
name: dshm
- mountPath: /root/.cargo/bin/cargo
name: fake-cargo
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
hostPID: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/cache/sglang
type: Directory
name: sglang-cache
- emptyDir:
medium: Memory
sizeLimit: 64Gi
name: dshm
- hostPath:
path: /data/ymk/ds41/fake-cargo
type: File
name: fake-cargo
- dependencies:
- mooncake-master
minReadySeconds: 0
name: d
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '30300'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-d
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --enable-fp32-lm-head --enable-dp-attention-local-control-broadcast \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 64 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r64_ep4.json \\\n --mem-fraction-static 0.80 \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --stream-interval 32 --swa-prefix-tails 2048 --enable-decoder-swa-bounded-replay \\\n --max-running-requests 512 --cuda-graph-max-bs-decode 512 \\\n --chunked-prefill-size 32768 --max-prefill-tokens 8192 \\\n --disaggregation-mode decode --disaggregation-transfer-backend mooncake \\\n --host 0.0.0.0 --port 30300 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-d \\\n --watchdog-timeout 3600\n"
env:
- name: CUDA_VISIBLE_DEVICES
value: 4,5,6,7
- name: SGLANG_RAGGED_VERIFY_MODE
value: static
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
value: '1'
- name: SGLANG_DSV4_KV_LAYOUT
value: v41_new
- name: SGLANG_DEFAULT_THINKING
value: '1'
- name: SGLANG_DSV41_REASONING_EFFORT
value: high
- name: MC_INTRANODE_NVLINK
value: 'true'
- name: MC_INTRA_NVLINK
value: 'true'
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
value: 'true'
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
value: '1'
- name: SGLANG_BATCH_TELEMETRY
value: '1'
- name: SGLANG_BATCH_SPAN_SAMPLE
value: '0.1'
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
value: '600'
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
value: '16'
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
value: '32'
- name: TZ
value: Asia/Shanghai
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
value: '1024'
- name: TMPDIR
value: /data/ymk/l3ssd/d-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/d-xdg
- name: HF_HOME
value: /data/ymk/l3ssd/d-hf
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: sglang
ports:
- containerPort: 30300
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 150
httpGet:
path: /health
port: metrics
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '96'
memory: 550Gi
nvidia.com/gpu: '4'
requests:
cpu: '32'
memory: 256Gi
nvidia.com/gpu: '4'
securityContext:
privileged: true
volumeMounts:
- mountPath: /data
name: data
- mountPath: /root/.cache/sglang
name: sglang-cache
- mountPath: /dev/shm
name: dshm
- mountPath: /root/.cargo/bin/cargo
name: fake-cargo
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
hostPID: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/cache/sglang
type: Directory
name: sglang-cache
- emptyDir:
medium: Memory
sizeLimit: 64Gi
name: dshm
- hostPath:
path: /data/ymk/ds41/fake-cargo
type: File
name: fake-cargo
- dependencies:
- p
- d
minReadySeconds: 0
name: gateway
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '29000'
prometheus.io/scheme: http
prometheus.io/scrape: 'true'
labels:
app: ds41-l3ssd-gw
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec /opt/sglang/bin/sgl-model-gateway launch \\\n --pd-disaggregation \\\n --prefill http://127.0.0.1:30000 8998 \\\n --decode http://127.0.0.1:30300 \\\n --prefill-policy cache_aware --decode-policy cache_aware \\\n --g9i-enabled --g9i-max-new-tokens 4096 \\\n \\\n \\\n --g9i-regular-p-queue-reject-secs 0 \\\n --g9i-decode-max-running 512 \\\n --health-check-timeout-secs 30 --health-check-interval-secs 30 \\\n --cb-failure-threshold 50 \\\n --model-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --tokenizer-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --host 0.0.0.0 --port 30400\n"
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: gateway
ports:
- containerPort: 30400
name: http
protocol: TCP
- containerPort: 29000
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 60
httpGet:
path: /health
port: http
initialDelaySeconds: 20
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '16'
memory: 32Gi
requests:
cpu: '2'
memory: 8Gi
volumeMounts:
- mountPath: /data
name: data
- mountPath: /opt/sglang/bin/sgl-model-gateway
name: gw-bin
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- name: gw-bin
hostPath:
path: /data/ymk/cache/gw-est-td/sgl-model-gateway
type: File
@@ -0,0 +1,488 @@
apiVersion: workloads.x-k8s.io/v1alpha2
kind: RoleBasedGroup
metadata:
annotations:
rbg.workloads.x-k8s.io/discovery-config-mode: refine
labels:
app: ds41-gist-exp
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
sglang-model: ds-v41-flash
name: ds41-l3ssd-test
namespace: prj-d7te67bj0sq2fgdrmbig
spec:
roles:
- minReadySeconds: 0
name: mooncake-master
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '19003'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-mooncake
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "# 18081: 宿主机 18080 被 AgentX fake upstream(./fakeup)长期占用;\n# 我们的 P/D 用 metadata_server=P2PHANDSHAKE,此 HTTP 端口只是 master 可选项。\nexec /opt/sglang/bin/mooncake_master \\\n --port 50051 --enable_http_metadata_server=true \\\n --http_metadata_server_port=18081 --metrics_port=19003 --eviction_high_watermark_ratio=0.90 --enable_offload=true --offload_on_evict=true --promotion_on_hit=false --offload_force_evict=false\n"
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: mooncake-master
ports:
- containerPort: 19003
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 60
initialDelaySeconds: 5
periodSeconds: 5
tcpSocket:
port: 50051
resources:
limits:
cpu: '4'
memory: 64Gi
requests:
cpu: '1'
memory: 8Gi
env:
- name: TMPDIR
value: /data/ymk/l3ssd/master-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/master-xdg
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
- dependencies:
- mooncake-master
minReadySeconds: 0
name: p
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '30000'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-p
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec sglang serve \\\n --trust-remote-code --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 32 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r32_ep4.json \\\n --mem-fraction-static 0.80 \\\n --chunked-prefill-size 131072 --max-prefill-tokens 131072 \\\n --max-running-requests 512 --tokenizer-worker-num 16 \\\n --load-balance-method total_tokens \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --enable-decoder-swa-bounded-replay --swa-full-tokens-ratio 0.12 \\\n --cuda-graph-max-bs-decode 1 --disable-decode-cuda-graph \\\n --cuda-graph-backend-prefill breakable \\\n --cuda-graph-bs-prefill 1024 2048 4096 8192 16384 32768 \\\n --cuda-graph-prefill-max-context 262144 --cuda-graph-max-seq-len-prefill 262144 \\\n --enable-hierarchical-cache --hicache-ratio 4 \\\n --hicache-mem-layout page_first_direct --hicache-io-backend direct \\\n --hicache-write-policy write_through --hicache-storage-backend mooncake \\\n --hicache-storage-prefetch-policy wait_complete \\\n --disaggregation-mode prefill --disaggregation-transfer-backend mooncake \\\n --disaggregation-bootstrap-port 8998 \\\n --expert-balancedness-report-mode prometheus \\\n --expert-distribution-recorder-buffer-size 64 \\\n --host 0.0.0.0 --port 30000 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-p \\\n --watchdog-timeout 3600 --enable-dp-attention-local-control-broadcast\n"
env:
- name: CUDA_VISIBLE_DEVICES
value: 0,1,2,3
- name: SGLANG_LOG_MS
value: '1'
- name: SGLANG_OPT_DSV41_ENGRAM_PEER_SHARDS
value: '1'
- name: SGLANG_OPT_DSV41_MEGA_MHC_PREFILL
value: '1'
- name: SGLANG_OPT_DSV41_BF16_INDEX_LOGITS
value: '1'
- name: SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK
value: '65536'
- name: SGLANG_DSV4_KV_LAYOUT
value: v41_new
- name: SGLANG_DP_USE_GATHERV
value: '1'
- name: SGLANG_RAGGED_VERIFY_MODE
value: static
- name: SGLANG_ENABLE_DP_SPEC_PREFILL_COORDINATION
value: '0'
- name: SGLANG_NUMA_BIND_V2
value: '0'
- name: SGLANG_DEFAULT_THINKING
value: '1'
- name: SGLANG_DSV41_REASONING_EFFORT
value: high
- name: SGLANG_TIMEOUT_KEEP_ALIVE
value: '900'
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
value: '1'
- name: MC_INTRANODE_NVLINK
value: 'true'
- name: MC_INTRA_NVLINK
value: 'true'
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
value: 'true'
- name: SGLANG_HICACHE_MOONCAKE_CONFIG_PATH
value: /data/ymk/ds41/mooncake-store-l3ssd.json
- name: SGLANG_HICACHE_PREFETCH_IO_THREADS
value: '4'
- name: SGLANG_HICACHE_BACKUP_IO_THREADS
value: '4'
- name: SGLANG_HICACHE_TIMING
value: '1'
- name: SGLANG_BATCH_TELEMETRY
value: '1'
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
value: '1'
- name: SGLANG_ENABLE_METRICS_DEVICE_TIMER
value: '1'
- name: SGLANG_EPLB_HEATMAP_COLLECTION_INTERVAL
value: '100'
- name: SGLANG_MM_TRACE
value: '1'
- name: SGLANG_BATCH_SPAN_MIN_MS
value: '200'
- name: SGLANG_BATCH_SPAN_SAMPLE
value: '0.05'
- name: SGLANG_BATCH_SPAN_MAX_LINKS
value: '8'
- name: SGLANG_BATCH_STAGE_TIMING
value: '1'
- name: SGLANG_BATCH_PASS_SPAN
value: '1'
- name: SGLANG_BATCH_PASS_SPAN_SAMPLE
value: '1.0'
- name: SGLANG_RECV_SPAN
value: '1'
- name: SGLANG_RECV_SPAN_SAMPLE
value: '1.0'
- name: SGLANG_MM_TRACE_SYNC
value: '0'
- name: SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK
value: '1'
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
value: '600'
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
value: '16'
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
value: '32'
- name: TZ
value: Asia/Shanghai
- name: SGLANG_TRACE_LOOP_SPAN
value: '1'
- name: SGLANG_TRACE_LOOP_SAMPLE
value: '1.0'
- name: SGLANG_FWD_TIMELINE
value: '0'
- name: SGLANG_TRACE_LOOP_DETAIL
value: '1'
- name: SGLANG_TRACE_MOONCAKE_DETAIL
value: '0'
- name: SGLANG_TRACE_BACKUP_DETAIL
value: '1'
- name: SGLANG_ASYNC_BACKUP_SUBMIT
value: '1'
- name: SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH
value: '1'
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
value: '1024'
- name: SGLANG_MOONCAKE_CLIENT_METRICS_PORT_BASE
value: '9400'
- name: MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR
value: bucket_storage_backend
- name: MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE
value: '549755813888'
- name: MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES
value: '549755813888'
- name: MOONCAKE_OFFLOAD_BUCKET_EVICTION_POLICY
value: fifo
- name: TMPDIR
value: /data/ymk/l3ssd/p-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/p-xdg
- name: HF_HOME
value: /data/ymk/l3ssd/p-hf
- name: MOONCAKE_OFFLOAD_FLUSH_PARTIAL_BUCKETS
value: '0'
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: sglang
ports:
- containerPort: 30000
name: metrics
protocol: TCP
- containerPort: 8998
name: bootstrap
protocol: TCP
- name: mc-0
containerPort: 9400
protocol: TCP
- name: mc-1
containerPort: 9401
protocol: TCP
- name: mc-2
containerPort: 9402
protocol: TCP
- name: mc-3
containerPort: 9403
protocol: TCP
readinessProbe:
failureThreshold: 120
httpGet:
path: /health
port: metrics
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '96'
memory: 2Ti
nvidia.com/gpu: '4'
requests:
cpu: '32'
memory: 512Gi
nvidia.com/gpu: '4'
securityContext:
privileged: true
volumeMounts:
- mountPath: /data
name: data
- mountPath: /root/.cache/sglang
name: sglang-cache
- mountPath: /dev/shm
name: dshm
- mountPath: /root/.cargo/bin/cargo
name: fake-cargo
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
hostPID: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/cache/sglang
type: Directory
name: sglang-cache
- emptyDir:
medium: Memory
sizeLimit: 64Gi
name: dshm
- hostPath:
path: /data/ymk/ds41/fake-cargo
type: File
name: fake-cargo
- dependencies:
- mooncake-master
minReadySeconds: 0
name: d
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '30300'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-d
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec sglang serve \\\n --trust-remote-code --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --enable-fp32-lm-head --enable-dp-attention-local-control-broadcast \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 64 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r64_ep4.json \\\n --mem-fraction-static 0.80 \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --stream-interval 32 --swa-prefix-tails 2048 --enable-decoder-swa-bounded-replay \\\n --max-running-requests 512 --cuda-graph-max-bs-decode 512 \\\n --chunked-prefill-size 32768 --max-prefill-tokens 8192 \\\n --disaggregation-mode decode --disaggregation-transfer-backend mooncake \\\n --host 0.0.0.0 --port 30300 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-d \\\n --watchdog-timeout 3600\n"
env:
- name: CUDA_VISIBLE_DEVICES
value: 4,5,6,7
- name: SGLANG_RAGGED_VERIFY_MODE
value: static
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
value: '1'
- name: SGLANG_DSV4_KV_LAYOUT
value: v41_new
- name: SGLANG_DEFAULT_THINKING
value: '1'
- name: SGLANG_DSV41_REASONING_EFFORT
value: high
- name: MC_INTRANODE_NVLINK
value: 'true'
- name: MC_INTRA_NVLINK
value: 'true'
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
value: 'true'
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
value: '1'
- name: SGLANG_BATCH_TELEMETRY
value: '1'
- name: SGLANG_BATCH_SPAN_SAMPLE
value: '0.1'
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
value: '600'
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
value: '16'
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
value: '32'
- name: TZ
value: Asia/Shanghai
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
value: '1024'
- name: TMPDIR
value: /data/ymk/l3ssd/d-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/d-xdg
- name: HF_HOME
value: /data/ymk/l3ssd/d-hf
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: sglang
ports:
- containerPort: 30300
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 150
httpGet:
path: /health
port: metrics
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '96'
memory: 550Gi
nvidia.com/gpu: '4'
requests:
cpu: '32'
memory: 256Gi
nvidia.com/gpu: '4'
securityContext:
privileged: true
volumeMounts:
- mountPath: /data
name: data
- mountPath: /root/.cache/sglang
name: sglang-cache
- mountPath: /dev/shm
name: dshm
- mountPath: /root/.cargo/bin/cargo
name: fake-cargo
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
hostPID: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/cache/sglang
type: Directory
name: sglang-cache
- emptyDir:
medium: Memory
sizeLimit: 64Gi
name: dshm
- hostPath:
path: /data/ymk/ds41/fake-cargo
type: File
name: fake-cargo
- dependencies:
- p
- d
minReadySeconds: 0
name: gateway
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '29000'
prometheus.io/scheme: http
prometheus.io/scrape: 'true'
labels:
app: ds41-l3ssd-gw
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec /opt/sglang/bin/sgl-model-gateway launch \\\n --pd-disaggregation \\\n --prefill http://127.0.0.1:30000 8998 \\\n --decode http://127.0.0.1:30300 \\\n --prefill-policy cache_aware --decode-policy cache_aware \\\n --g9i-enabled --g9i-max-new-tokens 4096 \\\n \\\n \\\n --g9i-regular-p-queue-reject-secs 0 \\\n --g9i-decode-max-running 512 \\\n --health-check-timeout-secs 30 --health-check-interval-secs 30 \\\n --cb-failure-threshold 50 \\\n --model-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --tokenizer-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --host 0.0.0.0 --port 30400\n"
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: gateway
ports:
- containerPort: 30400
name: http
protocol: TCP
- containerPort: 29000
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 60
httpGet:
path: /health
port: http
initialDelaySeconds: 20
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '16'
memory: 32Gi
requests:
cpu: '2'
memory: 8Gi
volumeMounts:
- mountPath: /data
name: data
- mountPath: /opt/sglang/bin/sgl-model-gateway
name: gw-bin
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- name: gw-bin
hostPath:
path: /data/ymk/cache/gw-est-td/sgl-model-gateway
type: File
@@ -0,0 +1,62 @@
<h1>当前基线:ds41-gist-exp(4T L3 SSD,2026-11-17)</h1>
<callout emoji="📌" background-color="light-blue"><p>本节是 2026-11-17 定稿的 c580/c620 基线,后续 Dynamo 叠加以此为底座。旧的 ds41-gist-1p1d 章节已归档,不再作为当前部署。</p></callout>
<h2>1. 基线身份与镜像</h2>
<ul>
<li><b>工作负载</b>:<code>ds41-gist-exp</code> / <code>cw-dblft6sgrhagmpdt8ef0</code>,namespace <code>prj-d7te67bj0sq2fgdrmbig</code>,b300-01。</li>
<li><b>镜像</b>:<code>sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115</code>(40.1GB,含 Mooncake 0.3.14 + L3 SSD 支持)。</li>
<li><b>manifest</b>:<code>deploy/b300-k8s/ds41-gist-exp-20261117-dtok16-fastokens.yaml</code>;基线副本 <code>deploy/b300-k8s/baselines/20261117-dtok16-fastokens/manifest.yaml</code>;b300 副本在 <code>/data/ymk/deploy/b300-k8s/</code>。</li>
<li><b>形态</b>:RBG 三角色 <code>p</code> / <code>d</code> / <code>gateway</code> + <code>mooncake-master</code>;全 hostNetwork / hostPID / privileged;P=GPU0-3,D=GPU4-7。</li>
</ul>
<h2>2. 部署方式与端口</h2>
<table>
<thead><tr><th><p>角色</p></th><th><p>端口</p></th><th><p>说明</p></th></tr></thead>
<tbody>
<tr><td><p>P</p></td><td><p>30000 / 8998 / 9400-9403</p></td><td><p>HTTP / bootstrap / Mooncake client metrics</p></td></tr>
<tr><td><p>D</p></td><td><p>30300</p></td><td><p>HTTP</p></td></tr>
<tr><td><p>gateway</p></td><td><p>30400 / 29000</p></td><td><p>业务 / metrics</p></td></tr>
<tr><td><p>mooncake-master</p></td><td><p>50051 / 18081 / 19003</p></td><td><p>store / metadata / metrics</p></td></tr>
</tbody>
</table>
<p>默认入口:<code>http://127.0.0.1:30400</code>;当前 new-api/AgentX 直接使用该入口。</p>
<h2>3. 关键参数</h2>
<p><b>P(prefill)</b>:tp4 / dp4 / ep4,megamoe + flashmla_mega,ep-redundant 32,DSpark block 3,decoder SWA bounded replay;chunked-prefill 131072 / max-prefill 131072,max-running 512;fastokens + tokenizer-worker 16;HiCache ratio 4、page_first_direct、direct、write_through、Mooncake L3;<b>cache 模式,无 buffer_only、无 optimistic</b>;mem-fraction 0.80。</p>
<p><b>D(decode)</b>:tp4 / dp4 / ep4,ep-redundant 64,DSpark block 3,stream-interval 32,swa-prefix-tails 2048;max-running 512,cuda-graph-max-bs-decode 512;chunked 32768 / max-prefill 8192;fastokens + tokenizer-worker 16;mem-fraction 0.80。</p>
<p><b>gateway</b>:镜像内 <code>sgl-model-gateway</code>;<b>无 P admission cap</b>(无 p-qps / p-queue-reqs-cap / p-queue-token-cap,td-admit 默认 off);保留 health check + circuit breaker;<code>--g9i-decode-max-running 512</code>。这是 c580 能跑通的关键。</p>
<p><b>L3</b>:Mooncake master 4TB SSD offload;<code>offload_on_evict=false</code>;L3 配置见 <code>/data/ymk/ds41/mooncake-store-l3ssd.json</code>。</p>
<h2>4. 启动 / 重启流程</h2>
<pre lang="bash"><code># 1) 停完整工作负载并确认 GPU 0
MSYS_NO_PATHCONV=1 wsl -e /home/ubuntu/.local/bin/kmctl stop cw ds41-gist-exp --wait --timeout 10m --request-timeout 180s
ssh 24.125.93.200.207 "nvidia-smi --query-gpu=index,memory.used --format=csv,noheader"
# 2) 清 L3 SSD(实验/基线前必须)
ssh 24.125.93.200.207 'bash /data/ymk/deploy/b300-k8s/clean_l3ssd_cache.sh'
# 3) 用完整平台 manifest 启动(不要只 kubectl patch gateway)
MSYS_NO_PATHCONV=1 wsl -e /home/ubuntu/.local/bin/kmctl update cw ds41-gist-exp \
-f /mnt/d/B300/deploy/b300-k8s/ds41-gist-exp-20261117-dtok16-fastokens.yaml
# 4) 等 4 Pod 1/1,检查 P/D/gateway health 200,再 smoke
curl -sS http://127.0.0.1:30400/v1/chat/completions -H 'Content-Type: application/json' \
-d '{"model":"/data/models/DeepSeek-V4.1-Flash","messages":[{"role":"user","content":"1+1=?"}],"max_tokens":8}'
</code></pre>
<p>启动基线:P <code>max_total_num_tokens≈16,158,208</code> / 128 per-rank / avail ≈23GB;D <code>58,859,520</code> / 128 per-rank / avail ≈79GB。正常 warmup 6,385 条约 6.5 分钟。</p>
<h2>5. 基线结果</h2>
<table>
<thead><tr><th><p>场景</p></th><th><p>RPS</p></th><th><p>TTFT p50/p95/p99</p></th><th><p>ITL p95</p></th><th><p>结论</p></th></tr></thead>
<tbody>
<tr><td><p>c580 基线</p></td><td><p>24.045</p></td><td><p>1.743 / 3.772 / 6.580 s</p></td><td><p>10.74 ms</p></td><td><p>推荐工作点</p></td></tr>
<tr><td><p>c620 首轮(清 L3)</p></td><td><p>25.998</p></td><td><p>1.814 / 4.041 / 8.257 s</p></td><td><p>11.69 ms</p></td><td><p>吞吐 +8%,尾部变差</p></td></tr>
<tr><td><p>c620 不重启 r2/r3</p></td><td><p>25.85 / 25.61</p></td><td><p>p95 5.39 / 6.44 s,p99 9.56 / 10.79 s</p></td><td><p>10.96 / 10.68 ms</p></td><td><p>D KV 接近/打满,TTFT 尾部持续劣化,不建议连测</p></td></tr>
<tr><td><p>buffer_only</p></td><td><p>23.749</p></td><td><p>2.153 / 4.904 / 10.483 s</p></td><td><p>10.32 ms</p></td><td><p>吞吐 -1.2%、TTFT 更差,不采用</p></td></tr>
</tbody>
</table>
<p>结论:<b>c580 是稳定基线</b>;c620 只适合短时冲吞吐,且每轮之间必须重启/清池。buffer_only 不作为基线。</p>
<h2>6. 作为 Dynamo 底座</h2>
<p>当前镜像是 <b>SGLang + Mooncake 0.3.14 + L3 SSD</b> 基线,<b>不包含 Dynamo/nixl</b>。后续套 Dynamo 需要以本镜像为 base 另装 Dynamo 1.6.x / nixl,旧 <code>gist-nvlink-7f6fe2244-dynamo-local-20261026</code> 是更早的 10-23 代码,不是本基线。</p>
<p>Dynamo 叠加时保留本页的 P/D 参数、L3 SSD 和 gateway 入口;先做单机 4P4D 冒烟,再评估 Dynamo 的 P/D 角色、discovery 和 KV 传输配置。</p>
@@ -0,0 +1,8 @@
{
"master_server_address": "127.0.0.1:50051",
"metadata_server": "P2PHANDSHAKE",
"global_segment_size": "200gb",
"protocol": "tcp",
"enable_ssd_offload": true,
"ssd_offload_path": "/data/ymk/mooncake-ssd"
}
@@ -0,0 +1,6 @@
# L3 Mooncake SSD baseline requires the official 0.3.14 wheel.
# Previous image shipped 0.3.13.post1; 0.3.14 contains the upstream SSD-offload fixes.
# Wheel: https://files.pythonhosted.org/packages/b9/bf/a117c9c9f892e86893caf0dd8b5bfae1d11edbfb0173dcd30108a7168fc3/mooncake_transfer_engine_cuda13-0.3.14-cp312-cp312-manylinux_2_28_x86_64.whl
# sha256: 74fa5f0684b143167503e33f85cccb9ca687d3f9843d4b594dbd5342c7df3e17
# Installed with --no-deps; the base image already provides the runtime deps.
mooncake-transfer-engine-cuda13==0.3.14
@@ -0,0 +1,44 @@
# 2026-11-16 buffer_only + optimistic 首启记录
## 变更(相对 `stable-20261116-pre-optimistic`)
P:
- 增加 `--hicache-host-memory-mode buffer_only`(保留 `--hicache-write-policy write_through`、
`--hicache-ratio 4`、mooncake L3)
- 增加 `--optimistic-prefill-attempts 4`
D:
- 增加 `--tokenizer-worker-num 16`(与 P 相同)
其余角色/参数不变;网关保持 live 的 `p-queue-reqs-cap 64 / token-cap 32M / p-qps 40`。
## 启动前清理
- `kmctl stop cw ds41-gist-exp --wait --timeout 10m --request-timeout 180s` → `Terminated 0/0`,GPU 0-7 全 0 MiB
- `/data/ymk/mooncake-ssd`:2.3T → 4.0K;`/data` 7.5T available
- `kmctl update cw ds41-gist-exp -f ds41-gist-exp-20261116-bufferonly-optimistic.yaml`
## 首启验证(2026-11-16 22:00 CST)
- 4 个 Pod 全 `1/1 Running`;P/D/gateway `/health` 全 200
- P `server_args`:`hicache_host_memory_mode='buffer_only'`、`optimistic_prefill_attempts=4`;
没有 "Optimistic prefill supports ..." 的禁用 warning
- D `server_args`:`tokenizer_worker_num=16`;日志有 16 个 multi-tokenizer worker
- 启动基线:P `max_total_num_tokens=16,158,208` / 128 per-rank / `available_gpu_mem=23.00 GB`;
D `58,859,520` / 128 per-rank / `available_gpu_mem=79.27 GB`
- 冒烟:chat `/v1/chat/completions` 200,回答正确;16 并发 4k 字符请求全 200
- **optimistic 已生效**:P 日志出现 `#optimistic-req: 1`(其余为 0)
- P/D 日志无 Traceback / ERROR / assert
## 回滚
```bash
MSYS_NO_PATHCONV=1 wsl -e /home/ubuntu/.local/bin/kmctl stop cw ds41-gist-exp --wait --timeout 10m --request-timeout 180s
ssh 24.125.93.200.207 'bash /data/ymk/deploy/b300-k8s/clean_l3ssd_cache.sh'
MSYS_NO_PATHCONV=1 wsl -e /home/ubuntu/.local/bin/kmctl update cw ds41-gist-exp \
-f /mnt/d/B300/sglang/deploy/b300-k8s/snapshots/20261116-pre-optimistic/cw-ds41-gist-exp-11-16-pre-optimistic.yaml
```
代码仓库标签:`stable-20261116-pre-optimistic`。
@@ -0,0 +1,608 @@
apiVersion: workloads.x-k8s.io/v1alpha2
kind: RoleBasedGroup
metadata:
annotations:
rbg.workloads.x-k8s.io/discovery-config-mode: refine
creationTimestamp: "2026-11-16T13:54:16Z"
generation: 1
labels:
app: ds41-gist-exp
kongming.io/history-revision: hrev-a416985ca3a8cd59f808d12dbcd3426b833d85b0
kongming.io/history-run: hrun-4ca5f278e80f5e20e03b98351874ad7cd859d5e7
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-id: cw-dblft6sgrhagmpdt8ef0
kongming.io/resource-type: custom-workload
kongming.io/user-id: usr-d7upbr3j0sq2fgdrmcr0
sglang-model: ds-v41-flash
name: ds41-gist-exp
namespace: prj-d7te67bj0sq2fgdrmbig
resourceVersion: "30768532"
uid: 242c2504-0c97-4414-ae60-8376f7d63de0
spec:
roles:
- minReadySeconds: 0
name: mooncake-master
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: "19003"
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-mooncake
kongming.io/history-revision: hrev-a416985ca3a8cd59f808d12dbcd3426b833d85b0
kongming.io/history-run: hrun-4ca5f278e80f5e20e03b98351874ad7cd859d5e7
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-id: cw-dblft6sgrhagmpdt8ef0
kongming.io/resource-type: custom-workload
kongming.io/user-id: usr-d7upbr3j0sq2fgdrmcr0
spec:
containers:
- command:
- /bin/bash
- -c
- |
# 18081: 宿主机 18080 被 AgentX fake upstream(./fakeup)长期占用;
# 我们的 P/D 用 metadata_server=P2PHANDSHAKE,此 HTTP 端口只是 master 可选项。
exec /opt/sglang/bin/mooncake_master \
--port 50051 --enable_http_metadata_server=true \
--http_metadata_server_port=18081 --metrics_port=19003 --eviction_high_watermark_ratio=0.90 --enable_offload=true --offload_on_evict=false --promotion_on_hit=false --offload_force_evict=false
env:
- name: TMPDIR
value: /data/ymk/l3ssd/master-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/master-xdg
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: mooncake-master
ports:
- containerPort: 19003
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 60
initialDelaySeconds: 5
periodSeconds: 5
tcpSocket:
port: 50051
resources:
limits:
cpu: "4"
memory: 64Gi
requests:
cpu: "1"
memory: 8Gi
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
- dependencies:
- mooncake-master
minReadySeconds: 0
name: p
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: "30000"
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-p
kongming.io/history-revision: hrev-a416985ca3a8cd59f808d12dbcd3426b833d85b0
kongming.io/history-run: hrun-4ca5f278e80f5e20e03b98351874ad7cd859d5e7
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-id: cw-dblft6sgrhagmpdt8ef0
kongming.io/resource-type: custom-workload
kongming.io/user-id: usr-d7upbr3j0sq2fgdrmcr0
spec:
containers:
- command:
- /bin/bash
- -c
- |
exec sglang serve \
--trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \
--reasoning-parser auto --tool-call-parser auto \
--default-chat-template-kwargs '{"thinking": true, "reasoning_effort": "high"}' \
--tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \
--moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \
--ep-num-redundant-experts 32 --ep-dispatch-algorithm dynamic \
--init-expert-location /data/ymk/gist-dsv41/placement_r32_ep4.json \
--mem-fraction-static 0.80 \
--chunked-prefill-size 131072 --max-prefill-tokens 131072 \
--max-running-requests 512 --tokenizer-worker-num 16 \
--load-balance-method total_tokens \
--speculative-algorithm DSPARK --speculative-dspark-block-size 3 \
--enable-decoder-swa-bounded-replay --swa-full-tokens-ratio 0.12 \
--cuda-graph-max-bs-decode 1 --disable-decode-cuda-graph \
--cuda-graph-backend-prefill breakable \
--cuda-graph-bs-prefill 1024 2048 4096 8192 16384 32768 \
--cuda-graph-prefill-max-context 262144 --cuda-graph-max-seq-len-prefill 262144 \
--enable-hierarchical-cache --hicache-ratio 4 --hicache-host-memory-mode buffer_only \
--hicache-mem-layout page_first_direct --hicache-io-backend direct \
--hicache-write-policy write_through --hicache-storage-backend mooncake \
--hicache-storage-prefetch-policy wait_complete --optimistic-prefill-attempts 4 \
--disaggregation-mode prefill --disaggregation-transfer-backend mooncake \
--disaggregation-bootstrap-port 8998 \
--expert-balancedness-report-mode prometheus \
--expert-distribution-recorder-buffer-size 64 \
--host 0.0.0.0 --port 30000 --enable-metrics --enable-cache-report \
--enable-trace --trace-modules request,mooncake \
--otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-p \
--watchdog-timeout 3600 --enable-dp-attention-local-control-broadcast
env:
- name: CUDA_VISIBLE_DEVICES
value: 0,1,2,3
- name: SGLANG_LOG_MS
value: "1"
- name: SGLANG_OPT_DSV41_ENGRAM_PEER_SHARDS
value: "1"
- name: SGLANG_OPT_DSV41_MEGA_MHC_PREFILL
value: "1"
- name: SGLANG_OPT_DSV41_BF16_INDEX_LOGITS
value: "1"
- name: SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK
value: "65536"
- name: SGLANG_DSV4_KV_LAYOUT
value: v41_new
- name: SGLANG_DP_USE_GATHERV
value: "1"
- name: SGLANG_RAGGED_VERIFY_MODE
value: static
- name: SGLANG_ENABLE_DP_SPEC_PREFILL_COORDINATION
value: "0"
- name: SGLANG_NUMA_BIND_V2
value: "0"
- name: SGLANG_DEFAULT_THINKING
value: "1"
- name: SGLANG_DSV41_REASONING_EFFORT
value: high
- name: SGLANG_TIMEOUT_KEEP_ALIVE
value: "900"
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
value: "1"
- name: MC_INTRANODE_NVLINK
value: "true"
- name: MC_INTRA_NVLINK
value: "true"
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
value: "true"
- name: SGLANG_HICACHE_MOONCAKE_CONFIG_PATH
value: /data/ymk/ds41/mooncake-store-l3ssd.json
- name: SGLANG_HICACHE_PREFETCH_IO_THREADS
value: "4"
- name: SGLANG_HICACHE_BACKUP_IO_THREADS
value: "4"
- name: SGLANG_HICACHE_TIMING
value: "1"
- name: SGLANG_BATCH_TELEMETRY
value: "1"
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
value: "1"
- name: SGLANG_ENABLE_METRICS_DEVICE_TIMER
value: "1"
- name: SGLANG_EPLB_HEATMAP_COLLECTION_INTERVAL
value: "100"
- name: SGLANG_MM_TRACE
value: "1"
- name: SGLANG_BATCH_SPAN_MIN_MS
value: "200"
- name: SGLANG_BATCH_SPAN_SAMPLE
value: "0.05"
- name: SGLANG_BATCH_SPAN_MAX_LINKS
value: "8"
- name: SGLANG_BATCH_STAGE_TIMING
value: "1"
- name: SGLANG_BATCH_PASS_SPAN
value: "1"
- name: SGLANG_BATCH_PASS_SPAN_SAMPLE
value: "1.0"
- name: SGLANG_RECV_SPAN
value: "1"
- name: SGLANG_RECV_SPAN_SAMPLE
value: "1.0"
- name: SGLANG_MM_TRACE_SYNC
value: "0"
- name: SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK
value: "1"
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
value: "600"
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
value: "16"
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
value: "32"
- name: TZ
value: Asia/Shanghai
- name: SGLANG_TRACE_LOOP_SPAN
value: "1"
- name: SGLANG_TRACE_LOOP_SAMPLE
value: "1.0"
- name: SGLANG_FWD_TIMELINE
value: "0"
- name: SGLANG_TRACE_LOOP_DETAIL
value: "1"
- name: SGLANG_TRACE_MOONCAKE_DETAIL
value: "0"
- name: SGLANG_TRACE_BACKUP_DETAIL
value: "1"
- name: SGLANG_ASYNC_BACKUP_SUBMIT
value: "1"
- name: SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH
value: "1"
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
value: "1024"
- name: SGLANG_MOONCAKE_CLIENT_METRICS_PORT_BASE
value: "9400"
- name: MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR
value: bucket_storage_backend
- name: MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE
value: "1099511627776"
- name: MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES
value: "1099511627776"
- name: MOONCAKE_OFFLOAD_BUCKET_EVICTION_POLICY
value: fifo
- name: TMPDIR
value: /data/ymk/l3ssd/p-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/p-xdg
- name: HF_HOME
value: /data/ymk/l3ssd/p-hf
- name: MOONCAKE_OFFLOAD_FLUSH_PARTIAL_BUCKETS
value: "0"
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: sglang
ports:
- containerPort: 30000
name: metrics
protocol: TCP
- containerPort: 8998
name: bootstrap
protocol: TCP
- containerPort: 9400
name: mc-0
protocol: TCP
- containerPort: 9401
name: mc-1
protocol: TCP
- containerPort: 9402
name: mc-2
protocol: TCP
- containerPort: 9403
name: mc-3
protocol: TCP
readinessProbe:
failureThreshold: 120
httpGet:
path: /health
port: metrics
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: "96"
memory: 2Ti
nvidia.com/gpu: "4"
requests:
cpu: "32"
memory: 512Gi
nvidia.com/gpu: "4"
securityContext:
privileged: true
volumeMounts:
- mountPath: /data
name: data
- mountPath: /root/.cache/sglang
name: sglang-cache
- mountPath: /dev/shm
name: dshm
- mountPath: /root/.cargo/bin/cargo
name: fake-cargo
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
hostPID: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/cache/sglang
type: Directory
name: sglang-cache
- emptyDir:
medium: Memory
sizeLimit: 64Gi
name: dshm
- hostPath:
path: /data/ymk/ds41/fake-cargo
type: File
name: fake-cargo
- dependencies:
- mooncake-master
minReadySeconds: 0
name: d
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: "30300"
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-d
kongming.io/history-revision: hrev-a416985ca3a8cd59f808d12dbcd3426b833d85b0
kongming.io/history-run: hrun-4ca5f278e80f5e20e03b98351874ad7cd859d5e7
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-id: cw-dblft6sgrhagmpdt8ef0
kongming.io/resource-type: custom-workload
kongming.io/user-id: usr-d7upbr3j0sq2fgdrmcr0
spec:
containers:
- command:
- /bin/bash
- -c
- |
exec sglang serve \
--trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \
--reasoning-parser auto --tool-call-parser auto \
--default-chat-template-kwargs '{"thinking": true, "reasoning_effort": "high"}' \
--tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \
--enable-fp32-lm-head --enable-dp-attention-local-control-broadcast \
--moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \
--ep-num-redundant-experts 64 --ep-dispatch-algorithm dynamic \
--init-expert-location /data/ymk/gist-dsv41/placement_r64_ep4.json \
--mem-fraction-static 0.80 \
--speculative-algorithm DSPARK --speculative-dspark-block-size 3 \
--stream-interval 32 --swa-prefix-tails 2048 --enable-decoder-swa-bounded-replay \
--max-running-requests 512 --cuda-graph-max-bs-decode 512 --tokenizer-worker-num 16 \
--chunked-prefill-size 32768 --max-prefill-tokens 8192 \
--disaggregation-mode decode --disaggregation-transfer-backend mooncake \
--host 0.0.0.0 --port 30300 --enable-metrics --enable-cache-report \
--enable-trace --trace-modules request,mooncake \
--otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-d \
--watchdog-timeout 3600
env:
- name: CUDA_VISIBLE_DEVICES
value: 4,5,6,7
- name: SGLANG_RAGGED_VERIFY_MODE
value: static
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
value: "1"
- name: SGLANG_DSV4_KV_LAYOUT
value: v41_new
- name: SGLANG_DEFAULT_THINKING
value: "1"
- name: SGLANG_DSV41_REASONING_EFFORT
value: high
- name: MC_INTRANODE_NVLINK
value: "true"
- name: MC_INTRA_NVLINK
value: "true"
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
value: "true"
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
value: "1"
- name: SGLANG_BATCH_TELEMETRY
value: "1"
- name: SGLANG_BATCH_SPAN_SAMPLE
value: "0.1"
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
value: "600"
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
value: "16"
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
value: "32"
- name: TZ
value: Asia/Shanghai
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
value: "1024"
- name: TMPDIR
value: /data/ymk/l3ssd/d-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/d-xdg
- name: HF_HOME
value: /data/ymk/l3ssd/d-hf
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: sglang
ports:
- containerPort: 30300
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 150
httpGet:
path: /health
port: metrics
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: "96"
memory: 550Gi
nvidia.com/gpu: "4"
requests:
cpu: "32"
memory: 256Gi
nvidia.com/gpu: "4"
securityContext:
privileged: true
volumeMounts:
- mountPath: /data
name: data
- mountPath: /root/.cache/sglang
name: sglang-cache
- mountPath: /dev/shm
name: dshm
- mountPath: /root/.cargo/bin/cargo
name: fake-cargo
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
hostPID: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/cache/sglang
type: Directory
name: sglang-cache
- emptyDir:
medium: Memory
sizeLimit: 64Gi
name: dshm
- hostPath:
path: /data/ymk/ds41/fake-cargo
type: File
name: fake-cargo
- dependencies:
- p
- d
minReadySeconds: 0
name: gateway
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: "29000"
prometheus.io/scheme: http
prometheus.io/scrape: "true"
labels:
app: ds41-l3ssd-gw
kongming.io/history-revision: hrev-a416985ca3a8cd59f808d12dbcd3426b833d85b0
kongming.io/history-run: hrun-4ca5f278e80f5e20e03b98351874ad7cd859d5e7
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-id: cw-dblft6sgrhagmpdt8ef0
kongming.io/resource-type: custom-workload
kongming.io/user-id: usr-d7upbr3j0sq2fgdrmcr0
spec:
containers:
- command:
- /bin/bash
- -c
- |
exec /opt/sglang/bin/sgl-model-gateway launch \
--pd-disaggregation \
--prefill http://127.0.0.1:30000 8998 \
--decode http://127.0.0.1:30300 \
--prefill-policy cache_aware --decode-policy cache_aware \
--g9i-enabled --g9i-max-new-tokens 4096 --g9i-est-td-enabled --g9i-shadow-enabled --g9i-td-admit enforce --g9i-p-queue-reqs-cap 64 --g9i-p-queue-token-cap 32000000 --g9i-p-qps 40 --g9i-p-refresh-ms 250 --g9i-window-fixed 256 \
\
\
--g9i-regular-p-queue-reject-secs 0 \
--g9i-decode-max-running 512 \
--pool-idle-timeout-secs 2 --pool-max-idle-per-host 0 \
--health-check-timeout-secs 30 --health-check-interval-secs 30 \
--cb-failure-threshold 50 \
--model-path /data/models/DeepSeek-V4.1-Flash-1017 \
--tokenizer-path /data/models/DeepSeek-V4.1-Flash-1017 \
--host 0.0.0.0 --port 30400
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: gateway
ports:
- containerPort: 30400
name: http
protocol: TCP
- containerPort: 29000
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 60
httpGet:
path: /health
port: http
initialDelaySeconds: 20
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: "16"
memory: 32Gi
requests:
cpu: "2"
memory: 8Gi
volumeMounts:
- mountPath: /data
name: data
- mountPath: /opt/sglang/bin/sgl-model-gateway
name: gw-bin
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/gw-ns-est-td/sgl-model-gateway
type: File
name: gw-bin
status:
conditions:
- lastTransitionTime: "2026-11-16T14:01:52Z"
message: All roles are ready
observedGeneration: 1
reason: AllRolesReady
status: "True"
type: Ready
observedGeneration: 1
roleStatuses:
- name: mooncake-master
readyReplicas: 1
replicas: 1
updatedReplicas: 1
- name: p
readyReplicas: 1
replicas: 1
updatedReplicas: 1
- name: d
readyReplicas: 1
replicas: 1
updatedReplicas: 1
- name: gateway
readyReplicas: 1
replicas: 1
updatedReplicas: 1
@@ -0,0 +1,68 @@
# 11/16 上线前稳定配置(ds41-gist-exp)
记录时间:2026-11-16 21:49 CST(13:49Z),在切换 `buffer_only + optimistic` 之前。
## 身份
- 平台 cw:`ds41-gist-exp` / `cw-dblft6sgrhagmpdt8ef0`
- RBG:`ds41-gist-exp`(namespace `prj-d7te67bj0sq2fgdrmbig`)
- 角色:`mooncake-master` / `p` / `d` / `gateway`
- 镜像:`sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115`
- P:GPU 0-3;D:GPU 4-7;全 hostNetwork / hostPID / privileged,钉 b300-01
## 文件
- `rbg-live-raw.yaml`:`kubectl get rbg -o yaml` 原始快照(含 status,仅存档)
- `cw-ds41-gist-exp-11-16-pre-optimistic.yaml`:清理过平台字段的 RBG,可直接用于回滚更新
- `cw-platform.yaml`:`kmctl get cw -o yaml` 的平台清单(`Spec.Manifest` 是旧值,见下方漂移)
- 本说明
## 生效配置摘要
P:
- `tp4 / dp4 / ep4`、`megamoe`、`flashmla_mega`、`ep-redundant 32`
- DSPARK block 3、decoder SWA bounded replay、`swa-full-tokens-ratio 0.12`
- `chunked-prefill-size 131072`、`max-prefill-tokens 131072`
- `max-running-requests 512`、`tokenizer-worker-num 16`
- `hicache-ratio 4`、`hicache-mem-layout page_first_direct`、`hicache-io-backend direct`
- `hicache-write-policy write_through`、`hicache-storage-backend mooncake`
- **默认 host memory mode(cache),没有 `--hicache-host-memory-mode buffer_only`**
- **没有 `--optimistic-prefill-attempts`**(`#optimistic-req` 为 0)
- L3 SSD:4 TB `mooncake-ssd`,master `--offload_on_evict=false`
D:
- 同并行 tp4/dp4/ep4、`ep-redundant 64`、DSPARK block 3
- `max-running-requests 512`、`cuda-graph-max-bs-decode 512`
- `chunked-prefill-size 32768`、`max-prefill-tokens 8192`
- **没有 `--tokenizer-worker-num`**(默认 1)
- `mem-fraction-static 0.80`
Gateway:
- 实际 live:`--g9i-p-queue-reqs-cap 64 --g9i-p-queue-token-cap 32000000 --g9i-p-qps 40 --g9i-p-refresh-ms 250 --g9i-window-fixed 256`
- `--g9i-regular-p-queue-reject-secs 0`、`--g9i-decode-max-running 512`
- 网关二进制 hostPath:`/data/ymk/gw-ns-est-td/sgl-model-gateway`
> ⚠️ 平台 `cw-platform.yaml` 的 `Spec.Manifest` 里 gateway 仍是旧的
> `p-queue-reqs-cap 32 / token-cap 16000000 / p-qps 20`;**live RBG 是 64 / 32M / 40**。
> 本次快照与回滚文件都以 live RBG 为准,避免回滚时把网关参数打回旧值。
## 切换前状态
- 所有 Pod `1/1 Running`;P/D/gateway `/health` 200。
- 为切 `buffer_only` 做 L3 SSD 清理前,`/data/ymk/mooncake-ssd` 占用 **2.3T**;清理后 4.0K,`/data` 7.5T available。
- 启动基线(切换后复核):P `max_total_num_tokens=16,158,208` / 128 per-rank / `available_gpu_mem=23.00 GB`;
D `58,859,520` / 128 per-rank / `available_gpu_mem=79.27 GB`。
## 回滚
```bash
MSYS_NO_PATHCONV=1 wsl -e /home/ubuntu/.local/bin/kmctl stop cw ds41-gist-exp --wait --timeout 10m --request-timeout 180s
ssh 24.125.93.200.207 'bash /data/ymk/deploy/b300-k8s/clean_l3ssd_cache.sh'
MSYS_NO_PATHCONV=1 wsl -e /home/ubuntu/.local/bin/kmctl update cw ds41-gist-exp \
-f /mnt/d/B300/deploy/b300-k8s/snapshots/20261116-pre-optimistic/cw-ds41-gist-exp-11-16-pre-optimistic.yaml
```
回滚前确认 `kmctl get cw ds41-gist-exp` 为 `Terminated 0/0`、GPU 0-7 为 0 MiB。

Some files were not shown because too many files have changed in this diff Show More