Compare commits

...
69 Commits
Author SHA1 Message Date
minke.yu aa23c29d52 deploy(b300/dynamo): publish default thinking mode from the workers
The AgentX baseline runs with thinking=true/high by default; the Dynamo
frontend only applies a default when the worker publishes
default_thinking_mode. Pass --dyn-default-thinking-mode enabled on both
P and D so the SGLang chat processor matches the sglang baseline.
2026-11-17 22:55:08 +08:00
minke.yu 99603451b9 docs(b300/dynamo): record tokenizer/chat-processor fixes and first-run results
Document --dyn-chat-processor sglang + --dyn-preprocess-workers +
--tokenizer fastokens, the encoding_dsv4 shim, the fastokens propagation
gap in sglang_processor._load_tokenizer, the v41 thinking opt-in fix, and
the warm/cold smoke numbers.
2026-11-17 22:42:19 +08:00
minke.yu 81428f230c deploy(b300/dynamo): restore disaggregation lanes and mount frontend patches
- SGLANG_DISAGGREGATION_QUEUE_SIZE=16 / THREAD_POOL_SIZE=32 (cw baseline
  values; service defaults are 4).
- NCCL all-gather sync-batch env on P; batch spans off until the next
  image build carries the TraceNullContext guard.
- FE mounts the encoding_dsv4 shim and the v41 thinking patch.
2026-11-17 22:29:38 +08:00
minke.yu 2372edd73d observability: skip batch-span links for TraceNullContext requests
TraceNullContext forwards every attribute to itself, so the previous
root_span.get_span_context() probe returned the null context itself and
otel's exporter failed with 'expected bytes, TraceNullContext found'.
Gate on tracing_enable before probing.
2026-11-17 22:27:55 +08:00
minke.yu 34d2224b7f dynamo: add encoding_dsv4 shim for the frontend SGLang chat processor
Dynamo's sglang_post imports upstream sglang.srt.entrypoints.openai.
encoding_dsv4.encode_messages; our fork uses the deepseek-recipe renderer.
Expose the upstream API on top of the recipe (V4.1 spec), returning the
prompt string; tool schemas follow Dynamo's system-message convention.
2026-11-17 22:22:18 +08:00
minke.yu 36f6a9cdb3 fix(b300/dynamo): restore FE_PORT definition 2026-11-17 22:11:31 +08:00
minke.yu 2b75283daa deploy(b300/dynamo): run frontend chat processing on SGLang processor
Use the 1.6.2 frontend tokenizer controls instead of per-worker gateways:
--dyn-chat-processor sglang + --dyn-preprocess-workers N + --tokenizer
fastokens, with explicit deepseekv41 / deepseek-v41 parsers (sglang's
'auto' is not resolved by the frontend parser factory). Also start the
L3 mooncake master from the same image.
2026-11-17 22:10:57 +08:00
minke.yu 0fbd87d739 docs(b300/dynamo): Dynamo does not remove SGLang's bootstrap/prealloc tail
Bootstrap completion is discovered at scheduler pass boundaries via
poll_and_all_reduce_attn_cp_tp_group; Dynamo's frontend still waits on it.
Record the measured prefill_bootstrap vs mooncake-send gap and point at
the sglang-side levers.
2026-11-17 21:51:15 +08:00
minke.yu bcfdd8a1f7 docs(b300/dynamo): spell out Dynamo PD control plane vs SGLang KV data plane
Dynamo's frontend/PrefillRouter owns P/D selection and bootstrap-info
propagation; the SGLang backend still delegates the KV transfer to
sglang's native disaggregation (mooncake + our NVLink patches).
--connector nixl / --kv-transfer-config apply to vLLM/TRT-LLM only.
2026-11-17 21:42:19 +08:00
minke.yu 6f546386ca deploy(b300/dynamo): add Dynamo 1.6.2 image recipe and 4P4D PD startup
- Dockerfile/build script: current B300 baked image (runtime tree identical to
  b300/dynamo) + ai-dynamo/ai-dynamo-runtime 1.6.2 installed with --no-deps.
- start_dynamo_pd.sh: dynamo.frontend :33000 + 4P(dp4)/4D(dp4) workers on
  GPU 0-3/4-7, mooncake NVLink, HiCache L2 + L3 SSD.
- README: compatibility audit (dynamo 1.6.2 unit tests match the nightly
  control run), version matrix, known constraints, open items.
2026-11-17 21:01:52 +08:00
minke.yu 8b67636010 Merge b300/integration into main: 2026-11-17 c580 baseline
CI Model Inventory / build-inventory (push) Failing after 41s
PR Benchmark (SMG Components) / Benchmark - Manual Policy (push) Skipped
PR Benchmark (SMG Components) / Benchmark - Request Processing (push) Skipped
PR Test (Arm64) / check-changes (push) Successful in 14s
PR Test (Xeon) / pr-gate (push) Successful in 2s
PR Test (SMG) / k8s-integration (push) Failing after 12m21s
PR Test (Rust extensions) / sglang-mm-unit (push) Failing after 12s
PR Benchmark (SMG Components) / Benchmark Compilation Check (push) Failing after 1m45s
PR Test (Arm64) / pr-gate (push) Successful in 2s
PR Test (sglang-processor) / ${{ github.event.action == 'labeled' && github.event.label.name != 'run-ci' && 'ignored-label-sgl-processor-gate' || 'sgl-processor-gate' }} (push) Successful in 2s
Seed Rust Ext Cache / seed (push) Failing after 1m53s
PR Benchmark (SMG Components) / Benchmark Summary (push) Skipped
PR Test (SMG) / docker-build-test (push) Failing after 18m37s
pr-test-xpu.yml / pr-gate (push) Successful in 2s
PR Test (sglang-processor) / ${{ github.event.action == 'labeled' && github.event.label.name != 'run-ci' && 'ignored-label-sgl-processor-lint' || 'sgl-processor-lint' }} (push) Failing after 10s
PR Test (SMG) / finish (push) Skipped
Lint / lint (push) Failing after 2m40s
pr-test-arm64.yml / pr-gate (push) Successful in 2s
PR Test (XPU) / multimodal-gen-test-1-gpu-xpu (push) Canceled after 0s
PR Test (XPU) / stage-a-test-1-gpu-xpu (push) Canceled after 0s
Release SGLang Kernels / build-rocm-matrix (3.10, ${{ fromJson(inputs.target == 'rocm720' && '["720"]' || inputs.target == 'rocm1000' && '["1000"]' || '["720", "1000"]') }}) (push) Canceled after 0s
Release SGLang Kernels / build-musa43 (43, 3.10) (push) Canceled after 0s
PR Test (Xeon) / build-test (spr1, 1, 0, 4, spr, xeon-spr, stage-a-test-cpu-intel,stage-b-test-cpu-intel) (push) Canceled after 0s
PR Test (Xeon) / build-test (spr2, 3, 2, 4, spr, xeon-spr, stage-a-test-cpu-intel,stage-b-test-cpu-intel) (push) Canceled after 0s
Seed Rust Ext Cache / seed-aarch64 (push) Canceled after 27h16m58s
PR Test (Arm64) / build-test (push) Canceled after 0s
Release SGLang Kernels / release-musa43 (push) Canceled after 0s
PR Test (XPU) / finish (push) Canceled after 0s
Release SGLang Kernels / release-cu130 (push) Canceled after 0s
Release SGLang Kernels / release-rocm720 (push) Canceled after 0s
Release SGLang Kernels / release-rocm1000 (push) Canceled after 0s
PR Test - Build Rust Extensions / Collect Rust Ext (push) Canceled after 0s
PR Test (Rust workspace) / cargo-workspace (push) Failing after 2m27s
PR Test (Rust workspace) / sglang-radix-tree (push) Successful in 2m45s
PR Test (SMG) / unit-tests (push) Failing after 1m47s
PR Test (sglang-processor) / ${{ github.event.action == 'labeled' && github.event.label.name != 'run-ci' && 'ignored-label-sgl-processor-changes' || 'sgl-processor-changes' }} (push) Successful in 14s
PR Test (sgl-router) / changes (push) Successful in 13s
PR Test (sgl-router) / gate (push) Successful in 2s
PR Test (XPU) / pr-gate (push) Successful in 2s
PR Test (Xeon) / build-test (gnr, gnr, xeon-gnr, stage-a-tp-test-cpu-intel) (push) Canceled after 0s
Release SGLang Kernels / build-cu130-matrix (aarch64, 13.0, 3.11, arm-kernel-build-node) (push) Canceled after 0s
Release SGLang Kernels / build-cu130-matrix (x86_64, 13.0, 3.11, x64-kernel-build-node) (push) Canceled after 0s
PR Test - Build Rust Extensions / Restore Rust Ext (push) Successful in 5s
PR Test - Build Rust Extensions / Build Rust Ext (3.12) (push) Canceled after 0s
PR Test (SMG) / build-wheel (push) Failing after 1m38s
pr-test-sgl-processor.yml / pr-gate (push) Successful in 2s
pr-test-sgl-router.yml / pr-gate (push) Successful in 2s
pr-test-xeon.yml / pr-gate (push) Successful in 2s
PR Test (sglang-processor) / ${{ github.event.action == 'labeled' && github.event.label.name != 'run-ci' && 'ignored-label-sgl-processor-test' || 'sgl-processor-test' }} (push) Skipped
PR Test (sgl-router) / tier-1 — lint (push) Failing after 55s
PR Test (sgl-router) / tier-3 — e2e (push) Skipped
PR Test (sglang-processor) / ${{ github.event.action == 'labeled' && github.event.label.name != 'run-ci' && 'ignored-label-sgl-processor-finish' || 'sgl-processor-finish' }} (push) Failing after 1s
PR Test (SMG) / python-unit-tests (push) Skipped
PR Test (SMG) / benchmarks (push) Skipped
PR Test (SMG) / chat-completions (push) Skipped
PR Test (SMG) / chat-completions-4gpu (push) Skipped
PR Test (SMG) / e2e (push) Skipped
PR Test (SMG) / summarize-benchmarks (push) Skipped
PR Test (Xeon) / check-changes (push) Successful in 14s
PR Test (XPU) / check-changes (push) Successful in 20s
PR Test (sgl-router) / tier-2 — build + test (push) Skipped
PR Test (sgl-router) / tier-3 — docker (placeholder) (push) Skipped
PR Test (sgl-router) / tier-3 — k8s integration (push) Skipped
PR Test (sgl-router) / finish (push) Failing after 2s
Promote the validated ds41-gist-exp baseline:
- image recipe-ns-l3ssd-433257c7b-baked-20261115 (SGLang + Mooncake 0.3.14 + L3 SSD)
- P: cache / no-optimistic / fastokens / tokenizer-worker 16 / L3 SSD 4T
- D: fastokens / tokenizer-worker 16
- gateway: normal health/circuit, no P admission caps
- c580 baseline 24.045 rps; c620 and buffer_only results recorded
- Feishu baseline doc and deploy manifests included

This is the base for the next Dynamo layer.
2026-11-17 20:08:08 +08:00
minke.yu 310ad35430 deploy: record 2026-11-17 c580 baseline (D16+fastokens) and experiments 2026-11-17 20:05:11 +08:00
minke.yu bdbd8dffb8 deploy(ds41-gist-exp): P buffer_only + optimistic, D tokenizer 16 2026-11-16 22:03:20 +08:00
minke.yu f363a6ab7b deploy: record 11/16 pre-optimistic stable config 2026-11-16 22:03:11 +08:00
minke.yu e4c7e540be deploy(g9i): T26 qps20 + P queue32 production candidate 2026-11-16 00:42:44 +08:00
minke.yu b8b205b367 deploy(g9i): merged ns+td gateway, 32-request P queue cap (p95 5.70s) 2026-11-15 22:25:33 +08:00
minke.yu b3d6d19c7d deploy(g9i): set regular P queue reject 32s (experiment) 2026-11-15 21:45:51 +08:00
minke.yu 8f53c1fb73 deploy(l3ssd): offload_on_evict=false, 4TB cap, L3 reset script 2026-11-15 20:33:38 +08:00
minke.yu 42f228bfb8 docs(deploy): fresh-instance multi-round tokenizer A/B (B2 TTFT p95 < 5s) 2026-11-15 18:20:53 +08:00
minke.yu 242a7ad23a docs(deploy): record L3 SSD tokenizer A/B (fastokens vs huggingface) 2026-11-15 15:49:56 +08:00
minke.yu 2eb163ad73 fix(deploy): point sglang cache at the shared warm cache 2026-11-15 14:23:52 +08:00
minke.yu d3240c745e deploy(b300): L3 Mooncake SSD baseline with mooncake 0.3.14 2026-11-15 14:10:30 +08:00
minke.yu 433257c7b0 fix(recipe): drop the pre-48928 _fast_prompt_tokenizer fallback stub 2026-11-15 10:53:15 +08:00
leon 7980a09c0a [Perf] DeepSeek-V4.1-Flash: Use segmented fastokens for prompt encoding (#48928)
(cherry picked from commit 219b7e7a688363bac9ef09eacf80e44eef9efe4b)
2026-11-15 10:51:40 +08:00
minke.yu a7c668a936 fix(chat): move namespace-tool flattening validator into ChatCompletionRequest 2026-11-14 14:10:07 +08:00
minke.yu 0c6849e378 chat: flatten namespace tools to ns__function for Chat Completions 2026-11-14 14:09:36 +08:00
minke.yu 85605b8c72 fix(protocol): ResponseTool accepts extra fields (tool_search execution) 2026-11-13 23:25:50 +08:00
minke.yu b6a5d70cba fix(responses): tolerate tools without model_extra in native validation 2026-11-13 23:24:00 +08:00
minke.yu cac60f7c2e fix(protocol): effective_tool_choice preserves namespace/type 2026-11-13 23:22:21 +08:00
minke.yu 68c2af7957 fix(protocol): Function.strict is Optional[bool] for recipe tool schemas 2026-11-13 23:21:04 +08:00
minke.yu 497aeff5f6 fix(recipe): define _fast_prompt_tokenizer fallback on this base 2026-11-13 22:44:35 +08:00
minke.yu 5fa8c23227 vendor: DeepSeek-recipe namespace tools (#46370 + #49625)
Backport the upstream namespace-tool stack onto the B300 recipe base:
- #46370: Function.namespace / ChatMessage.action protocol fields.
- #49625: Responses/Messages native recipe rendering (convert_native),
  tool_search/defer_loading, namespace identity in/out/history replay.

Adapted to this base: skips serving_kimi_k3.py (not present), drops the
Minimax/Beam/wire-only dependencies added between 10-23 and the PR base,
and keeps the existing tool-parsing fallback where the newer
`parses_without_tools()` helper is unavailable.

Also includes the PR's Anthropic tool-reference plumbing and unit tests.
2026-11-13 22:43:37 +08:00
minke.yu f4de9f2141 observability: ignore sub-tolerance CPU>wall skew in accounting anomaly
A 2 ms phase whose CPU clock reads 0.1 us over its wall clock is clock
granularity, not an accounting bug; the anomaly flag fired on nearly every
short phase in the live verification run.

Only flag (and record non_cpu_raw_ms) when the deficit exceeds both 100 us and
1% of the phase wall. Larger real anomalies keep the previous behaviour.
2026-11-12 12:30:58 +08:00
minke.yu 7f42a0806a observability: per-rank request counter for P/D RPS
The frontend sglang:num_requests_total has no dp_rank label (verified from the
series registry), so per-rank RPS was not derivable from any existing metric.

Add sglang:batch_requests_total{phase,dp_rank}, incremented by the scheduler
process that actually completed the request:

  phase=prefill  prefill compute finished and the request entered the PD
                 transfer queue (not a guarantee that the transfer succeeded)
  phase=decode   request finished decoding

Only the canonical scheduler rank reports (attn_tp_rank==0, attn_cp_rank==0,
last PP stage) — every TP/CP rank runs a scheduler and sees the same request,
so an ungated counter would be multiplied by TP*CP in the multiprocess
registry. Both hooks count each request exactly once: the decode path counts
after _handle_finish_state_updated_req and on the sampling-mask abort branch,
skipping rows that finished on an earlier step; the prefill path counts after
the request enters the disagg inflight queue (aborted/retried requests never
reach it). The per-request finished() check is skipped entirely when
SGLANG_BATCH_TELEMETRY is off.
2026-11-12 10:30:56 +08:00
minke.yu 17b922c84f test: prefill_batch_init detail and eviction semantics
Covers the design's unit-level acceptance points:
* disabled/unsampled loops never read the thread clock;
* wall/CPU/non-CPU semantics, unmeasured != zero, partial measurement omits
  non-CPU, busy vs sleeping windows are distinguishable;
* CPU > wall flags an accounting anomaly with a raw negative delta;
* child spans carry the CPU split;
* repeated phases (prefix indices) stay attribute-only, inclusive phases are
  not summed, and eviction demand/shortfall/plan plus Mamba plan/zero results
  are recorded as separate counters.
2026-11-12 10:30:47 +08:00
minke.yu 5ad9ff6a5f observability: prefill_batch_init wall+CPU, sub-phases and eviction coverage
Parent span scheduler.loop.prefill_batch_init records wall + the scheduler
thread's CPU at both existing close points (empty admission and normal
return); the CPU clock is only read when loop detail is enabled and sampled,
through the safe batch_telemetry.thread_cpu_ns() wrapper so a missing clock
degrades to 'not measured' instead of failing the scheduling path.

Fixed children, in call order:
  schedule_batch_init, ready_to_load_host_cache, prepare_for_extend,
  alloc_for_extend (+ prefix_device_indices attribute-only, sampling_batch_info),
  evict_to_free_tokens, evict_for_alloc, prefill_stats (+ pending_tokens),
  mixed_chunk_tail.

* prefill_stats wraps the whole assignment so _get_num_pending_tokens() is
  included; mixed_chunk_tail only exists when that branch is taken.
* prefill-only shape counters (n_reqs/n_new/n_cached) are snapshotted before
  mixed-chunk folding; a mixed batch additionally publishes decode_reqs and
  merged totals under explicit *_merged_* names.
* eviction is timed at the call site (outside the allocator wrapper) so
  rank-consensus waits are included, and covers base, SWA (x2), UnifiedSWA
  (x3) and the extend Mamba reclaim through evict_for_alloc_observed().
  Demand, shortfall and plan are separate counters; Mamba plan and realized
  counts are both recorded (including zero).

All of it is gated by detail on a sampled loop.
2026-11-12 10:30:38 +08:00
minke.yu fec10023e0 observability: optional thread-CPU recording for detail phases
Adds an optional thread CPU clock next to the existing wall clock:

* detail_begin/end record time.thread_time_ns() only when detail is on and the
  loop is sampled; detail_add() accepts an explicit thread_cpu_ns for callers
  that already hold absolute start/end points (the prefill_batch_init parent).
* phase records carry [wall, count, start_wall, end_wall, cpu_sum, cpu_count].
  thread_cpu_ms and thread_cpu_measured_count are emitted whenever any
  occurrence was measured; non_cpu_ms only when every occurrence was, so a
  partial measurement never attributes unmeasured wall to non-CPU time. A
  phase whose CPU exceeds its wall keeps the clamped value, adds
  non_cpu_raw_ms and accounting_anomaly, and bumps a per-loop anomaly count.
* new detail_attr_sum()/detail_attr_setdefault() accumulate low-cardinality
  numeric loop attributes (eviction tokens, free-token snapshots).
* prefix_device_indices joins the attribute-only prefixes so per-request
  repeats never create spans.
* child spans carry phase_thread_cpu_ms / phase_cpu_measured_count /
  phase_non_cpu_ms, so the CPU split is visible without the loop attributes.

Default-off path is unchanged: no thread clock, no new dicts, no spans.
2026-11-12 10:30:29 +08:00
minke.yu 9f9dce1ff6 test: restore _emit_pass_span residual default
The detail-phase refactor made residual a required positional argument while
test_loop_span_accounting.py still calls _emit_pass_span with four arguments,
so that test failed on the base commit. Default it to 0.0.

Split out of the prefill_batch_init observability series to keep that review
scope focused.
2026-11-12 10:30:17 +08:00
minke.yu c401c614fa fix(dp): rank uncached candidates by published backlog only
Deriving each candidate's uncached estimate from its own recent hit rate is
self-reinforcing: a rank that falls behind sees its hit rate drop (fewer
chances to hit), gets charged more for every new request, and falls further
behind. Measured on c580: one rank ended 16% below the others, 30 s cross-rank
dispersion max 952x, and 9.6% lower rps than the total_tokens arm.

Charge the fleet-average uncached size instead and rank by the published
num_waiting_uncached_tokens, which already carries each rank's exact prefix
match.
2026-11-11 02:06:59 +08:00
minke.yu 24b284d4e7 feat(dp): uncached_tokens load-balance method
DP balancing on total_tokens charges each request's whole prompt, but with
prefix caching only the unmatched part costs prefill compute. On a workload
with a 95% hit rate the prompt is ~95% noise: per request the uncached part
is a few percent of the prompt yet varies by more than an order of magnitude
(p50 1.9k / p99 59k tokens against a 95k prompt), so one cold request
outweighs many hot ones and per-rank prefill work drifts apart.

Add --load-balance-method uncached_tokens: DPBudget tracks the published
LoadSnapshot.num_waiting_uncached_tokens (prefix match minus accepted storage
hits) plus the rank's recent cache_hit_rate for a newly dispatched request.
2026-11-11 00:03:55 +08:00
QNan0513 fef023b64f [Fix] scheduler: Count all pending work in DP load snapshots
total_tokens DP balancing replaces its per-rank estimate with the rank's
latest load snapshot, so a snapshot that misses work already sent to the
rank lets the controller keep sending to it. Close three gaps:

- count grammar-queue requests in num_waiting_reqs and num_total_tokens
- count the in-progress chunked request's unprefilled tokens
- publish a snapshot once newly received requests are queued, since the
  one published after a forward predates the requests sent during it
2026-11-10 23:58:23 +08:00
ymk 15218b2f65 deploy: bake the live hostPath overrides into the tree
Copies the three files that the live cw mounts over the image:
- managers/load_snapshot.py: queued_total_input_tokens + drain_rate_tokens_per_s
- managers/scheduler_components/load_inquirer.py: fills the two new fields
- models/deepseek_v4.py: env-gated MM trace (SGLANG_MM_TRACE, off by default)

After this commit every runtime file the P/D pods need is in the tree, so the
hostPath file mounts can be dropped once the image is rebuilt.
2026-11-09 06:31:48 +00:00
minke.yu 883f7d08e1 fix(mooncake): use enable_client_http_server (not enable_http_server) 2026-11-09 11:04:01 +08:00
minke.yu e40c6dfe83 observability(mooncake): opt-in client metrics HTTP server per rank
SGLANG_MOONCAKE_CLIENT_METRICS_PORT_BASE=<base> makes MooncakeStore.setup()
pass enable_http_server=True and client_http_port=base+dp_rank, so each DP
rank's embedded Mooncake client exposes /metrics (needs MC_STORE_CLIENT_METRIC=1).
Off by default.
2026-11-09 10:55:17 +08:00
minke.yu 32e5e0f30f observability: GPU-window probe + forward/prefill sub-phase timing
Phase 1/3 of the async-backup handoff instrumentation plan:

* new fwd_timeline probe (SGLANG_FWD_TIMELINE=1): one CUDA event pair around
  the model execute call per forward, resolved without blocking and written as
  wall-clock JSONL per DP rank. Joined with loop spans it separates host time
  spent while the GPU is busy (overlapped) from GPU-idle host stalls.
* detail phases: fwd_init_new, fwd_prepare, fwd_execute, fwd_graph_load,
  fwd_graph_replay, fwd_post, fwd_runner_tail,
  prefill_policy, prefill_adder_init, prefill_scan, prefill_batch_init.

All spans ride the existing SGLANG_TRACE_LOOP_DETAIL path; the GPU probe is
off by default.
2026-11-08 01:19:09 +08:00
minke.yu a5fbe834e8 observability: split get_next residual into its four phases
Adds the Phase A instrumentation from the async-backup handoff (§7.1):

* PassTimings: pending_abort / chunk / new_batch / set_sched (+ chunk_count)
* prefill.get_next_disagg_prefill_batch_to_run: time
  process_pending_chunked_abort, process_prefill_chunk,
  get_new_batch_prefill and set_schedule_time_batch
* scheduler.prefill.pass gains sglang.{pending_abort,chunk,new_batch,set_sched}_ms,
  sglang.get_next_residual_ms (= other - these four) and sglang.chunk_count;
  the existing sglang.other_ms is unchanged for dashboard compatibility.

~5 monotonic calls per pass; enabled by SGLANG_BATCH_STAGE_TIMING (already on).
2026-11-08 00:20:55 +08:00
minke.yu d6f1e1efbd rebase(hicache): per-op async backup on top of #46884 + #48782
Net async change (from exp-async-backup-perop-20261107) applied onto
b300-pick-46884-20261107. Runtime files applied cleanly; the new tests and
the timeline probe are added as-is.

Interaction to validate on hardware: #46884 publishes prefill read-done before
the replay, so the scheduler-stream fence that the index copy waits on is
shorter; the async worker's wait should shrink and the copy may overlap more of
the in-flight forward. Re-baseline the 4-arm A/B on this branch.
2026-11-07 22:04:33 +08:00
alphabetc1 520ca548ea [Fix] hicache: Wait for overlapping write-through acknowledgements
(cherry picked from commit 032725770c33c61c236f2884ce7e7ff1fa99cef9)
2026-11-07 14:37:24 +08:00
metamergebot 5f7be7fab4 [Perf] hicache: Publish prefill read-done when offload copies are fenced on the forward stream
The prefill shared-read-done fastpath lets the schedule stream run ahead of
the forward that writes a stashed chunk's KV instead of waiting for the whole
forward. It was disabled whenever hierarchical cache or a storage backend was
enabled, because host and storage offload copies used to be ordered only
behind the schedule stream and could have read unfinished KV.

The cache controller now orders its device-to-host copies behind the forward
stream through write_fence_stream, and its load-back H2D copies through
load_fence_stream, so that reason no longer holds when every KV offload copy
carries that fence. The scheduler records on the model runner whether that is
the case: hierarchical cache with a cache controller and no decode offload
manager, which has no such fence. The publish helper keeps the coarse
whole-forward barrier for LMCache and for unfenced offload, and publishes
read-done otherwise.

With the coarse barrier, a synchronous page-index copy issued while preparing
the next batch inherited the wait on the entire in-flight forward, leaving the
GPU idle between batches. With read-done published, that copy only waits for
the shared-table reads to finish; KV ordering is unchanged because the backup
copies still wait on the forward stream.

Tests: the CPU runner test covers the fenced hierarchical-cache and storage
cases publishing, and LMCache keeping the barrier even when hierarchical cache
is fenced. The GPU ordering test checks that a write-through backup of a chunk
stashed before its forward finished still copies the final KV.

Co-authored-by: ZYHowell <ZYHowell@users.noreply.github.com>
(cherry picked from commit bb1cb2dded6fb9352d98076280342b10684e061b)
2026-11-07 14:36:38 +08:00
minke.yu 97c087778b observability: harden flush span attributes
* Normalize the host-pool layout to a string (a None attribute would make
  the OTel SDK drop the whole flush span).
* Drop any remaining None-valued flush attributes for the same reason.
2026-11-07 11:35:03 +08:00
minke.yu 53267ee18f observability: time loop-other inflight poll and release/insert
Adds loop-detail phases outside run_batch/process_result/get_next:

* inflight_poll: process_disagg_prefill_inflight_queue() (overlap loop)
* inflight_release: release_kv_cache(checkpoint=True) + tree_cache.finish()
  inside it, so the checkpoint/insert cost is separated from the poll
* launch_sample: launch_batch_sample_if_needed()

The get_next side already publishes match/adder/hicache/bootstrap/mlp_sync
and a pass residual (sglang.other_ms) on scheduler.prefill.pass.
2026-11-07 09:26:46 +08:00
minke.yu aa53670fcd observability: split flush_pending_backups host chain (off by default)
Adds the phase-3/phase-4 instrumentation from
handoff/prefill-flush-backups-validation-20261107.md:

* process_batch_result outer phases: flush_trace, load_snapshot,
  flush_backups, step_counters, metrics_report. process_residual now
  subtracts them, so the 37.5% residual is attributable.
* SGLANG_TRACE_BACKUP_DETAIL=1 (default 0) traces one flush:
  merge_ops / move_write (anchor + per-sidecar indices) / indices_cpu /
  index_sort / index_select / write_fence / build_l2 / submit_d2h
  (start-event wait, prepare, per-pool launch, ack events) / ack_enqueue,
  plus token/op/pool/device dimensions and ack queue depth.
* Per-flush resolution: one scheduler.cache_flush_backup span per
  non-empty flush, child of the loop span. Sub-phases never enter the
  loop aggregate, so the residual identity cannot double count.
* Empty vs non-empty flushes are counted on the loop.

All timings are scheduler host wall, not device transfer time.
2026-11-07 09:23:44 +08:00
ymk 7b3f34cbc0 test: cover async SWA state-payload call chain; clarify CP gate
Review follow-up for the v2 async state payload:

- P2: new tests use CustomTestCase (with a local fallback when sglang is not
  importable), matching the disaggregation/Mooncake test convention.
- P3: CP gate docstring now says all-CP-rank transfer *without DSA cache
  layer split* keeps the synchronous path.
- Add real call-chain coverage:
  * _send_kv_chunk -> sender.send -> add_transfer_request -> transfer_worker
    -> materialize_state_indices -> maybe_send_extra, with DSV4's four
    components (DSV4_REQUEST_STATE, REQUEST_ROW, SWA, SWA);
  * the worker materializes pinned tensors only after indices_event and
    fails loudly when a tensor has no event;
  * a second materialization without an event (staging retry) is a no-op;
  * real CUDA ordering: the pinned helper returns before a long kernel and
    the event-protected values match kv_to_page_indices.
- Verified on b300 (P container, overlay source): 9 passed in
  test_async_state_payload.py, 2 passed in the new
  TestMooncakeAsyncStateMaterialization class; the old batching suite has
  11 pre-existing failures that also fail on the image source.

The test-only batching suite requires the detail envs to be off when run in
the live pod (the inherited SGLANG_TRACE_MOONCAKE_DETAIL=1 exposes an existing
test fixture omission); noted, not changed here.
2026-11-07 00:22:57 +08:00
ymk 3b0baa3455 prefill: async SWA state-payload D2H (first version)
Implements the v2 plan in handoff/prefill-state-payload-async-review-20261106.md.

- mem_cache/common.py: add kv_to_page_indices_pinned() (pinned CPU buffer,
  non-blocking copy; no synchronous .cpu()).
- prefill.py: gate via should_async_state_payload() (last chunk, independent
  supports_async_state_indices, kv_shard_size==1, CP guard); only the
  _swa_payload request_window-is-None hot branch uses the pinned helper.
  Final segment asserts async state has an indices_event. Other state
  builders (mamba/full/DSA) stay synchronous.
- mooncake/conn.py: MooncakeKVSender advertises
  supports_async_state_indices; the worker materializes pinned state tensors
  to numpy only after indices_event.synchronize(), and fails loudly if a
  tensor arrives without an event.
- tests: guard/event/materialize/pinned-body unit tests + CUDA ordering test
  (skipped unless torch+CUDA+sglang are available).

No scheduler/transfer semantics change; rollback is the helper switch or
reverting this commit. 49 tests pass, 1 skipped locally.
2026-11-06 23:00:22 +08:00
ymk fe9c6574fe observability: trace KV transfer chunks end-to-end (M2)
Implements M2 of handoff/prefill-cpu-kv-overlap-design-20261106.md, behind
SGLANG_TRACE_MOONCAKE_DETAIL=1 (default off; reuses the existing trace stack).

- TransferKVChunk gains optional observation fields: chunk_seq, loop_seq,
  send_kind, enqueue/last_enqueue monotonic+wall stamps, queue_wait_accum_ns,
  queue_id, requeue_count, num_pages, approx_bytes, observe
- chunk_seq and enqueue timestamps are assigned at the real queue.put()
  point (after every early return), so skipped paths cannot leave holes
- Mooncake transfer worker measures queue wait, source-event wait, indices
  wait, KV/state/aux transfer, worker busy and residual; emits one
  mooncake.kv_chunk span per attempt (+ queue/source/indices wait children)
  and updates a per-room join summary
- request completion emits kv.request_join with chunk_count,
  last_chunk_queue_wait/worker, final_exposed_ms, completion_poll_ms,
  request_join_ms and first-enqueue-to-completion. First enqueue is stamped
  by the Mooncake manager at the real queue.put(); no ReqTimeStats field is
  added and the legacy queue_entry semantics are unchanged.
- nixl/mori add_transfer_request accept the observation kwarg (no-op there)

No scheduling/transfer semantics change; queue wait is excluded from worker
busy, and worker_busy excludes source/indices event waits. Unit tests: test_mooncake_chunk_detail.py (15 cases, including the
abort/worker interleaving and attempt-vs-total queue wait windows).
2026-11-06 20:01:56 +08:00
ymk 72e556ec81 observability: split prefill run_batch/process_result into host phases (M1)
Implements M1 of handoff/prefill-cpu-kv-overlap-design-20261106.md:
per-loop detail phases behind SGLANG_TRACE_LOOP_DETAIL=1 (requires
SGLANG_BATCH_STAGE_TIMING=1 + SGLANG_TRACE_LOOP_SPAN=1, only sampled loops).

- run_batch: early_send_host, resolve_inputs, model_forward_host,
  copy_result_submit (+ run_residual_ms)
- process_result: copy_done_wait, auxiliary_and_logprob, per_request_result,
  kv_state_payload/page_indices/sender_send by send_kind, report_stats
  (+ process_residual_ms)
- war_barrier is measured as its own loop phase (outside run_batch's span)
- repeated/per-request phases are summed attributes; single phases are child
  spans, so span count never grows with n_reqs/n_segments
- DP-skew join fields: pass_end_wall_ns, arrival_est_wall_ns = pass_end -
  mlp_sync, previous-loop phase summary (sglang.prev_*, published only for
  consecutive loop_seq; prev_detail_missing marks a gap)

No CUDA syncs added; when disabled every hook is a module-level bool check.
Unit tests: test_loop_detail_phases.py (10 cases).
2026-11-06 20:01:56 +08:00
minke.yu 23053730fc observability: count SWA gate rejections per prefill admission
Review follow-up. match_kv_cache also runs on waiting-queue probes
(calc_priority), PD decode _match_prefix_and_lock and HiCache rematches.
Emitting the counter there counted the same gap once per scheduling round
and attributed "re-prefilled" tokens to decode engines.

- match_kv_cache now only records req.full_kv_hit_length (plus the match-time
  gap); it no longer emits.
- the metric is emitted once in PrefillAdder._commit_prefill_admission, where
  the accepted prefix is final after any host load-back.
- rename the request counter to swa_gate_reject_admissions_total and document
  that a retracted and re-admitted request is counted again (it is re-prefilled).
- pre-create zero-valued children so gate-off deployments expose the series
  before the first rejection.
- tests: repeated match does not emit, admission emits once, a retry counts
  again, and host load-back closes the gap.

Not runtime verified (no GPU run on this build).
2026-11-06 13:18:07 +08:00
ymk bf8cf7f1dc hicache: pipeline prefetch/backup IO lanes + stage timing
- prefetch_io_aux_func: allow up to SGLANG_HICACHE_PREFETCH_IO_THREADS
  transfers in flight (default 1 = legacy behaviour); PrefetchAck is still
  emitted strictly FIFO so the per-rank ack stream (and the TP all_reduce in
  prefetch_sync_thread_func) is unchanged.
- backup_thread_func: same pipelining for L2->L3 writes via
  SGLANG_HICACHE_BACKUP_IO_THREADS.
- SGLANG_HICACHE_TIMING=1: emit [HC_TIMING] prefetch/backup queue, inflight,
  p50/p95 per 100 ops (the prefetch/backup lanes were previously single
  threaded; this is what caps L3<->L2 at ~1 op at a time).
2026-11-06 13:18:07 +08:00
minke.yu c911d1b572 observability: expose SWA prefix-match gate rejections
The unified-cache SWA component gates prefix matching on a live sliding
window: when the window is missing, the match stops before an otherwise
available Full-KV prefix. The existing hit counters do not show the tokens
lost to that gate, which makes the gate-on/gate-off trade-off invisible.

Record the gap between the longest Full-KV match and the accepted prefix
(full_kv_hit_length - (device_prefix_len + host_hit_length)) on the Req and
export it as:
  sglang:swa_gate_reject_tokens_total
  sglang:swa_gate_reject_reqs_total

This is the token volume that will be re-prefilled because a component
validator (SWA window; Mamba where present) rejected the match. With the
gate enabled it measures the hit length lost to a missing window; with the
gate disabled it only captures the host-only / load-back-gap fallback.

Includes unit tests for the gap and the host-hit case. Not runtime verified
yet (no GPU run on this build).
2026-11-06 10:33:52 +08:00
minke.yu 145e05c44d observability: fix scheduler no-batch accounting 2026-11-05 16:42:39 +08:00
minke.yu 5d5870f3ea observability: add scheduler loop span and no-batch accounting
Adds an opt-in scheduler.loop span (SGLANG_TRACE_LOOP_SPAN) with recv /
get_next / run_batch / process_result / kv_send children. The span is only
created for sampled loops that have pending work; fully idle loops stay in
the counters so the 714 pass/s idle spin never floods Jaeger.

No-batch accounting is always on with SGLANG_BATCH_STAGE_TIMING:
- sglang:scheduler_no_batch_{seconds,total}{dp_rank,reason,forward_overlap}
- sglang:scheduler_dp_lockstep_idle_seconds_total{dp_rank,forward_overlap}
with the design's reason priority (result_backpressure, deferred_input,
waiting_bootstrap, waiting_prefetch, admission_blocked, waiting_pd,
fully_idle, no_ready_other) and local_batch_kind/global_outcome from the DP
sync. DP lockstep idle is counted separately from pure no-batch wait.

loop gpu_ms reuses the existing DeviceTimer intervals (no new CUDA events);
the forward.gpu lane is optional and approximate. Pass/recv spans become
children of the loop span when it is active.

Tests: test_loop_span_accounting (5) and the existing stage-timing suite (8)
pass.
2026-11-05 15:00:23 +08:00
minke.yu 309af6e6e1 observability: record prefetch ready latency on the L3 hit success path
_handle_prefetch_result() retires the operation with a direct
del ongoing_prefetch[request] instead of _retire_ongoing_prefetch(), so
the ready-latency hook only ever saw miss/revoke paths: with a cache-hit
workload storage_prefetch_ready_seconds{result="hit"} stayed at 0 even
though storage_prefetch_hit_tokens_total was moving ~312k tok/s.

Observe right after the del, classifying by loaded_from_storage.
2026-11-04 19:14:02 +08:00
minke.yu 1a274df27e observability: fix prefetch ready latency to cover fast lookups
The first version stamped the request on its first in-flight
check_prefetch_progress() call. A fast L3 query can be drained by
_process_hicache_events before that check runs (the op is already gone
from ongoing_prefetch), so those requests produced no sample -- exactly
the common case.

Stamp at issue time in prefetch_from_storage() and observe once at
_retire_ongoing_prefetch(), classifying by operation.storage_hit_count.
Retire is the single exit for hit completion, below-threshold revoke,
anchor loss and host-capacity fallback, so hit and miss are both covered.

No behavior change; the hook is still behind SGLANG_BATCH_STAGE_TIMING.
2026-11-04 18:04:34 +08:00
minke.yu cb7988f849 observability: measure storage prefetch ready latency (hit/miss)
Under --hicache-storage-prefetch-policy wait_complete a request is not
admitted until its L3 lookup resolves, and a miss pays the same query round
trip as a hit. The scheduler only exposed a skip counter, so the actual wait
was unknown.

Add sglang:storage_prefetch_ready_seconds{dp_rank, result=hit|miss}:
  * stamp cache_request_handle on its first in-flight check,
  * observe at the moment check_prefetch_progress() turns usable, split by
    whether prefetch_loaded_tokens_by_reqid ended up > 0,
  * clear the stamp on revoke / abort / accounting discard.

Buckets cover 1 ms .. 30 s. Gated by SGLANG_BATCH_STAGE_TIMING (off by
default). The measurement is ~2 dict ops + one observe per request, not per
pass.

Tests: test/registered/unit/observability/test_batch_telemetry_stage_timing.py
adds the hit/miss split assertion (9 tests, all passing).
2026-11-04 17:26:41 +08:00
minke.yu 0866012b80 observability: split scheduler recv_requests into pull/broadcast/unwrap
Runtime accounting on the prefill service (64 x 14.6k-token requests,
8-way concurrency) showed the non-forward wall time is dominated by the
scheduler receive path, not by radix match:

  forward (extend+idle) 58.2% | get_next 6.9% | recv_requests 25.6%
  process_batch_result 1.8% | run_batch overhead 0.9% | other ~6%

The existing pass span only covers get_next, so it cannot explain that
25.6%.  Split it with the same opt-in gate:

  * sglang:recv_requests_seconds{dp_rank, part} (total/pull/broadcast/
    unwrap/mm), 5 new series per rank
  * sglang:recv_requests_msgs_total, sglang:recv_requests_prompt_tokens_total
  * OTLP span scheduler.recv_requests with the same breakdown, sampled
    deterministically by a per-process counter (SGLANG_RECV_SPAN_SAMPLE,
    default 0.05; SGLANG_RECV_SPAN=0 disables the span only)

Note on scope: tokenization itself does NOT run in the scheduler. The
tokenizer workers (--tokenizer-worker-num) produce TokenizedGenerateReqInput
and the scheduler receives the already-tokenized ids. What this measures is
ZMQ pull + decode, the cross-rank broadcast of the pickled objects, and
unwrap_pickle_fields() of the token-id buffers -- which is why large prompts
make recv_requests expensive.

Tests: test/registered/unit/observability/test_batch_telemetry_stage_timing.py
now also covers the recv histogram parts, counters, zero-message passes and
deterministic recv-span sampling (8 tests total, all passing).
2026-11-04 16:11:13 +08:00
minke.yu 85f4f9576c scheduler: add opt-in per-pass prefill stage timing (SGLANG_BATCH_STAGE_TIMING)
Motivation: on the PD prefill node the wall-clock budget is dominated by
get_next_batch_to_run (~50% of the hour on 2026-11-02 16:00-17:00), and
forward_overlap=partial is only 21-87 s/h, i.e. it does not overlap the GPU
forward. We know the total but not which segment inside it costs: radix
match, PrefillAdder admission, the HiCache event collective, the bootstrap
all-reduce, or the DP metadata gather.

Measurement design (one sample per scheduling pass, including passes that
produce no batch; see handoff/prefill-batch-stage-timing-20261103.md v2/v3):

  L1  7 per-pass histograms:
      sglang:batch_prefill_{get_next,match,adder,hicache_events,
      bootstrap,mlp_sync,other}_seconds
      other = total - (match + adder + hicache + bootstrap + mlp_sync),
      clamped to >= 0. Nested stages are not double counted.

  L3  5 per-pass counters:
      candidates / skipped_prefetch / admitted / resumed_chunk / new_chunk

  L2  a dedicated per-pass span scheduler.prefill.pass (not the batch span),
      emitted from end_pass() on every pass. Its sampling decision is a pure
      function of mlp_sync_seq, so a sampled gather round is sampled on all
      DP ranks. Idle ranks and ranks without a local batch still emit, which
      is exactly what the cross-rank skew analysis needs. No links.

Hooks are installed in both scheduling loops:
  * get_next_disagg_prefill_batch_to_run() for --disaggregation-mode prefill
    (hicache / bootstrap / mlp_sync segments), and
  * get_next_batch_to_run() for standalone prefill (hicache / mlp_sync),
so the instrumentation can be validated on ds41-prefill-dp4 without a decode
peer. match/adder/counters live in the shared get_new_batch_prefill path.

Semantics:
  * match times only init_next_round_input; check_prefetch_progress counts as
    skipped_prefetch, never as match.
  * bootstrap and hicache_events are cross-rank sync points measured at the
    outer call, not attributed to local CPU work.
  * dp_gather_ms comes from the existing MLPSyncBatchInfo.sync_wait_seconds;
    sglang:dp_attention_sync_wait_seconds is reused, no duplicate metric.
    The value is cleared before each outer call and mlp_sync_seq only
    increments when a gather actually ran, so stale values cannot leak.
  * resumed vs new chunked requests are captured separately: had_chunked is
    recorded at pass start (before process_prefill_chunk/admission can
    consume the last chunk) and new_chunk from adder.new_chunked_req. This
    fixes both the first-chunk and last-chunk mislabels.

Opt-in:
  * SGLANG_BATCH_STAGE_TIMING=1 (default 0);
    SGLANG_BATCH_PASS_SPAN=1 / SGLANG_BATCH_PASS_SPAN_SAMPLE=0.05.
  * When off every hook is one module-level bool check; no logging, no CUDA
    calls, no new collectives.

Tests: test/registered/unit/observability/test_batch_telemetry_stage_timing.py
covers first/middle/last chunk labelling, all-prefetch-skip passes without a
batch, stale-gather clearing, rank-deterministic pass-span sampling, and the
disabled no-op path. py_compile passes on all touched files.
2026-11-04 14:53:29 +08:00
minke.yu 2434d09848 observability: commit the in-image per-batch telemetry baseline (btelem4-6)
These three files were already running in the gist production image
(gist-nvlink-ee9170e7bb-pzcfix) but were never committed: the image was
produced by docker-commit with the btelem4/5/6 files overlaid on top of
the built source tree.

Port them so the tree matches the image before layering new instrumentation:

  * observability/batch_telemetry.py: per-batch histograms, per-DP-step
    shape histograms and the sampled OTLP span scheduler.batch.{prefill,decode}.
  * managers/scheduler_components/metrics_reporter.py: observe_prefill/
    observe_decode call sites and the batch-trace log suffix.
  * managers/scheduler_components/dp_attn.py: observe_dp_step().

Content is identical to the image; only line endings were normalised to LF
(the image copy was mixed CRLF/LF). No behavior change when
SGLANG_BATCH_TELEMETRY is unset (default off).
2026-11-04 14:48:38 +08:00
Florian Wunderlich d47a99b2ad test: import SWA eviction helper directly
(cherry picked from commit 5729f61e175bde3ef2b67dc3a95abb1042f4bf1b)
2026-11-04 10:41:52 +08:00
Florian Wunderlich 2ba9b8431a Fix SWA eviction behind the replay-trimmed prefix frontier
(cherry picked from commit f399d1e9a2bf7c33c7244ecf405e756292ddaf2b)
2026-11-04 10:41:52 +08:00
111 changed files with 24626 additions and 1519 deletions
@@ -0,0 +1,25 @@
# L3 SSD image: recipe-ns engine + official mooncake 0.3.14 (CUDA 13, cp312).
#
# Build context: this directory plus the 0.3.14 wheel
# mooncake_transfer_engine_cuda13-0.3.14-cp312-cp312-manylinux_2_28_x86_64.whl
# Use build-l3ssd-image.sh, which downloads and verifies the wheel.
#
# Base tag recipe-ns-433257c7b-baked-20261114 = b300/integration @ 433257c7b
# (recipe namespace/chat-ns + gateway binary baked). Rebuild it from that commit
# if the local tag is missing.
FROM sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-433257c7b-baked-20261114
COPY mooncake_transfer_engine_cuda13-0.3.14-cp312-cp312-manylinux_2_28_x86_64.whl /tmp/
RUN python3 -m pip install --no-deps --force-reinstall \
/tmp/mooncake_transfer_engine_cuda13-0.3.14-cp312-cp312-manylinux_2_28_x86_64.whl \
&& rm -f /tmp/mooncake_transfer_engine_cuda13-0.3.14-cp312-cp312-manylinux_2_28_x86_64.whl \
# The wheel's master binary uses $ORIGIN/../mooncake_transfer_engine_cuda13.libs;
# keep it in place and expose a wrapper at the legacy path used by the manifests.
&& printf '%s\n' \
'#!/bin/sh' \
'export LD_LIBRARY_PATH=/opt/sglang/lib/python3.12/site-packages/mooncake_transfer_engine_cuda13.libs:$LD_LIBRARY_PATH' \
'exec /opt/sglang/lib/python3.12/site-packages/mooncake/mooncake_master "$@"' \
> /opt/sglang/bin/mooncake_master \
&& chmod +x /opt/sglang/bin/mooncake_master \
&& python3 -c "import importlib.metadata as m; print('mooncake:', m.version('mooncake-transfer-engine-cuda13'))" \
&& /opt/sglang/bin/mooncake_master --version | head -1
+139
View File
@@ -0,0 +1,139 @@
# B300 L3 Mooncake SSD 基线部署(2026-11-15)
> 以同事在 `ds41-gist-exp` 上验证的 **L3 Mooncake SSD** 配置为基础,使用我们的
> `b300/integration` 引擎代码 + 官方 **mooncake 0.3.14**,不再依赖 hostPath
> `/opt/mooncake-latest`。
## 1. 版本要求(重要)
- **mooncake `0.3.14`**(`mooncake-transfer-engine-cuda13`,CUDA 13 / cp312 wheel)。
- 镜像内原为 `0.3.13.post1`;0.3.14 含上游 SSD offload 相关修复/增强。
- wheel 与 sha256 见 `requirements-mooncake.txt`,安装必须 `--no-deps`(基底已有运行时依赖)。
- 引擎:`b300/integration` @ `433257c7b`(recipe namespace/chat-ns + 新 gateway)。
- 镜像:`sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115`
(40.1GB,已 `ctr -n k8s.io images import`;本地 docker tag `ymkymx/sglang:...` 同名)。
- 构建/装载:`./build-l3ssd-image.sh`(下载校验 wheel → docker build → ctr import)。
镜像内 `/opt/sglang/bin/mooncake_master` 是 wrapper:设置 wheel `.libs` 的
`LD_LIBRARY_PATH` 后 exec 包内 master(已验证 `0.3.14 (git e1ea5d3)`)。
## 2. L3 SSD 配置
`mooncake-store-l3ssd.json`:
```json
{"master_server_address":"127.0.0.1:50051","metadata_server":"P2PHANDSHAKE",
"global_segment_size":"200gb","protocol":"tcp",
"enable_ssd_offload":true,"ssd_offload_path":"/data/ymk/mooncake-ssd"}
```
- 部署时把该文件放到节点 `/data/ymk/ds41/mooncake-store-l3ssd.json`,SSD 目录
`/data/ymk/mooncake-ssd` 需存在。
- `MOONCAKE_OFFLOAD_*`:`bucket_storage_backend` / 512GiB bucket 上限 / `fifo` /
`flush_partial=0`(见 manifest)。
- master 参数必须带
`--enable_offload=true --offload_on_evict=true --promotion_on_hit=false --offload_force_evict=false`。
## 3. manifest
| 文件 | 用途 |
|---|---|
| `ds41-l3ssd-test-baked-20261115.yaml` | 新建 cw `ds41-l3ssd-test`(推荐先做基线验证) |
| `ds41-gist-exp-l3ssd-baked-20261115.yaml` | 原名 `ds41-gist-exp`,用于直接更新原栈 |
已通过 `kmctl create/update --dry-run` 校验。启动前先停掉占用 8 卡的 stack
(`ds41-recipe-ns-test` 或旧 `ds41-gist-exp`),确认 GPU 归零:
```bash
kmctl create cw ds41-l3ssd-test --node-group B300 \
-f deploy/b300-k8s/ds41-l3ssd-test-baked-20261115.yaml
# 或替换原栈:
# kmctl update cw ds41-gist-exp -f deploy/b300-k8s/ds41-gist-exp-l3ssd-baked-20261115.yaml
```
端口/角色与旧 `ds41-gist-exp` 一致:master 50051、P 30000、D 30300、gateway 30400。
启动约 10-15 分钟;核对 P/D `max_total_num_tokens` 和 master 日志中的 SSD offload 初始化。
## 4. 与上一版配置的差异
P/D 的 `sglang serve` 参数**完全一致**;差异只在 mooncake:
| 项 | 旧(11-09) | 本基线 |
|---|---|---|
| mooncake | 镜像内 0.3.13.post1 | 镜像内 **0.3.14** |
| master | `/opt/sglang/bin/mooncake_master`,无 offload | 同路径(0.3.14 wrapper),带 offload 参数 |
| P/D PYTHONPATH | 无 | 无(0.3.14 已烘入) |
| L3 配置 | 纯 TCP,无 SSD | `enable_ssd_offload` + `ssd_offload_path` |
| `MOONCAKE_OFFLOAD_*` | 无 | bucket backend / 512GiB / fifo / flush_partial=0 |
| TMPDIR/XDG/HF | 默认/`/data/ymk` | `/data/ymk/l3ssd/{master,p,d}-*` |
## 5. 注意
- 同事原配置使用 `SGLANG_TRACE_LOOP_SAMPLE=1.0`(全量 detail tracing,约 2% 吞吐 +
p95 开销)。本基线为对齐其配置保留 1.0;正式上线建议改回 `0.05`。
- `global_segment_size=200gb` + 512GiB SSD offload:上线前按宿主内存总账复核。
- SSD 目录增长需监控(`/data` 当前约 7.6T 可用)。
- 旧镜像 `recipe-ns-433257c7b-baked-20261114` 与 `recipe-ns-a7c668a93-baked-20261114`
保留作回滚/对照。
## 6. Tokenizer A/B(2026-11-15,c580 / 1200s)
同一 `ds41-gist-exp`(L3 SSD + mooncake 0.3.14)整栈重启,只切 P/D
`--tokenizer-backend`;AgentX c580 / 1200s / warmup 256 / seed 42,经 30400。
| 臂 | tokenizer | rps | in tok/s | out tok/s | TTFT p50/p95/p99 | ITL p50/p95/p99 | requests |
|---|---|---:|---:|---:|---|---|---:|
| A | `huggingface`(默认) | 23.789 | 2,257,661 | 22,224 | 2.043 / 4.176 / 7.579 s | 8.02 / 14.48 / 20.91 ms | 29,498 |
| B | **`fastokens`** | **24.045** | **2,289,885** | **22,475** | **1.746 / 3.676 / 6.820 s** | 8.13 / 11.65 / 14.87 ms | 29,816 |
- 结论:fastokens 臂 **rps +1.1%**、TTFT p50/p95/p99 **−14.5% / −12.0% / −10.0%**;
ITL p95 的差异大概率是 run-to-run 噪声(decode 不走 tokenizer),不作为判据。
- #48928 的分段编码已生效:容器内 `DeepSeekRecipe.from_hf_tokenizer(..., fastokens tokenizer)`
的 `_fast_prompt_tokenizer` 非 None(`fastokens.Tokenizer`);服务端未打印
“fastokens segmented” 日志,属日志路径差异,不影响结论。
- **当前线上实验栈保留 fastokens 臂**;要回默认用
`ds41-gist-exp-l3ssd-baked-20261115.yaml` 更新并整栈重启。
- 单轮对比,c580 本身有方差;如需正式定论建议每臂 ≥3 轮。
- SSD offload 已配置(master 报 `SSD Storage: 0 B / 2.00 TB`),本次 c580 工作集未触发
SSD 逐出(`/data/ymk/mooncake-ssd` 约 20K),不是配置未生效。
## 7. 多轮验证(2026-11-15,每轮整栈重启)
> 之前 B2/B3 在同一实例上连跑、未重启,数据受实例状态影响;下面只列**整栈重启后**的轮次。
> 整栈重启 = `kmctl stop` + `update`(换臂),或同臂时删除 p/d/gateway/master 四个 Pod。
| 臂 | 轮次 | rps | TTFT p50/p95/p99 (s) | ITL p95 (ms) | requests |
|---|---|---:|---|---:|---:|
| A(huggingface) | A1 | 23.789 | 2.043 / 4.176 / 7.579 | 14.48 | 29,498 |
| A | A2 | 23.552 | 2.305 / 5.894 / 8.600 | 15.05 | 29,205 |
| B(fastokens) | B1 | 24.045 | 1.746 / 3.676 / 6.820 | 11.65 | 29,816 |
| B | **B2(fresh)** | **24.040** | **1.708 / 3.451 / 7.105** | 11.71 | 29,810 |
- **B2 满足判据:TTFT p95 = 3.451s < 5s**,且与 B1(3.676s)一致;
A 臂 fresh 轮 p95 为 4.176 / 5.894s。
- fresh 轮均值:A rps 23.67 / p95 5.04s;B rps 24.04 / p95 3.56s
(B +1.6% rps、p95 −29%)。
- 结论:保留 fastokens;之前未重启的 B2/B3(p95 5.99/6.69s)仅作历史参考、不纳入。
## 8. L3 SSD 4TB + offload_on_evict=false(2026-11-15)
- 启动清单 `ds41-gist-exp-l3ssd-fastokens-20261115.yaml` 已改:master
`--offload_on_evict=false`;P 的 `MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE` /
`MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES` 改为 1 TiB/rank,4 rank 合计
master 报 `SSD Storage: 0 B / 4.00 TB`。
- 每次实验前必须重置 L3:stop cw → 确认 Terminated 0/0 + GPU 0 MiB →
`bash clean_l3ssd_cache.sh`(删除 `/data/ymk/mooncake-ssd` 下的 rank_*)→ update/start。
本次清理释放 1.2T,`/data` 可用 6.4T→7.5T。
- c580(1200s,warmup 256,seed 42,fastokens,干净 L3):
rps 24.003、in 2.285M tok/s、out 22,447、TTFT p50/p95/p99 1.768/3.500/7.549s、
ITL 8.15/11.74/15.31ms、29,764 请求、0 错误、valid。
对比 B1/B2:24.045/1.746/3.676/6.820 与 24.040/1.708/3.451/7.105——
吞吐与尾延迟均在方差内,**没有可归因于 offload_on_evict/4TB/清理的收益或回归**。
- **重要:`offload_on_evict=false` 并没有停止 SSD 写入**。run 后 master
`SSD Storage: 533.81 GB / 4.00 TB`,`du /data/ymk/mooncake-ssd = 535G`;
这是 `--hicache-write-policy write_through` 的 backup 写入,和内存 eviction 无关。
若要真正避免 SSD 写入,需要改 `write_back` 或关闭 `--enable_offload`,但那会改变
L3 语义,需另做 A/B。
- run 后 P cgroup `memory.current=2,198,904,565,760` ≈ 2047.5 GiB,已顶到 2 TiB
limit;宿主 MemAvailable 1.2 TiB。下一次实验前必须先停/清/重启;不要在同一实例
上连续跑,否则 L3 与 cgroup 都处于顶格状态。
@@ -0,0 +1,10 @@
# 2026-11-17 c580 基线:D tokenizer 16 + fastokens
- manifest: `manifest.yaml`(= `ds41-gist-exp-20261117-dtok16-fastokens.yaml`)
- P: cache / 无 optimistic / tokenizer-worker 16 / fastokens / hicache ratio 4 / L3 SSD 4T
- D: tokenizer-worker 16 / fastokens
- Gateway: 正常(无 P admission cap,health/circuit 开启,decode-max-running 512)
- L3:每轮前清空;完整重启
- c580 结果:rps 24.045,in 2.290M tok/s,out 22.449k tok/s,
TTFT 1.743/3.772/6.580 s,ITL p95 10.74 ms,29,816 req,valid=true。
- 原始结果:`experiments/agentx-align/results-c580-dtok16-fastokens-20261117/`
@@ -0,0 +1,488 @@
apiVersion: workloads.x-k8s.io/v1alpha2
kind: RoleBasedGroup
metadata:
annotations:
rbg.workloads.x-k8s.io/discovery-config-mode: refine
labels:
app: ds41-gist-exp
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
sglang-model: ds-v41-flash
name: ds41-gist-exp
namespace: prj-d7te67bj0sq2fgdrmbig
spec:
roles:
- minReadySeconds: 0
name: mooncake-master
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '19003'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-mooncake
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "# 18081: 宿主机 18080 被 AgentX fake upstream(./fakeup)长期占用;\n# 我们的 P/D 用 metadata_server=P2PHANDSHAKE,此 HTTP 端口只是 master 可选项。\nexec /opt/sglang/bin/mooncake_master \\\n --port 50051 --enable_http_metadata_server=true \\\n --http_metadata_server_port=18081 --metrics_port=19003 --eviction_high_watermark_ratio=0.90 --enable_offload=true --offload_on_evict=false --promotion_on_hit=false --offload_force_evict=false\n"
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: mooncake-master
ports:
- containerPort: 19003
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 60
initialDelaySeconds: 5
periodSeconds: 5
tcpSocket:
port: 50051
resources:
limits:
cpu: '4'
memory: 64Gi
requests:
cpu: '1'
memory: 8Gi
env:
- name: TMPDIR
value: /data/ymk/l3ssd/master-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/master-xdg
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
- dependencies:
- mooncake-master
minReadySeconds: 0
name: p
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '30000'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-p
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 32 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r32_ep4.json \\\n --mem-fraction-static 0.80 \\\n --chunked-prefill-size 131072 --max-prefill-tokens 131072 \\\n --max-running-requests 512 --tokenizer-worker-num 16 \\\n --load-balance-method total_tokens \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --enable-decoder-swa-bounded-replay --swa-full-tokens-ratio 0.12 \\\n --cuda-graph-max-bs-decode 1 --disable-decode-cuda-graph \\\n --cuda-graph-backend-prefill breakable \\\n --cuda-graph-bs-prefill 1024 2048 4096 8192 16384 32768 \\\n --cuda-graph-prefill-max-context 262144 --cuda-graph-max-seq-len-prefill 262144 \\\n --enable-hierarchical-cache --hicache-ratio 4 \\\n --hicache-mem-layout page_first_direct --hicache-io-backend direct \\\n --hicache-write-policy write_through --hicache-storage-backend mooncake \\\n --hicache-storage-prefetch-policy wait_complete \\\n --disaggregation-mode prefill --disaggregation-transfer-backend mooncake \\\n --disaggregation-bootstrap-port 8998 \\\n --expert-balancedness-report-mode prometheus \\\n --expert-distribution-recorder-buffer-size 64 \\\n --host 0.0.0.0 --port 30000 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-p \\\n --watchdog-timeout 3600 --enable-dp-attention-local-control-broadcast\n"
env:
- name: CUDA_VISIBLE_DEVICES
value: 0,1,2,3
- name: SGLANG_LOG_MS
value: '1'
- name: SGLANG_OPT_DSV41_ENGRAM_PEER_SHARDS
value: '1'
- name: SGLANG_OPT_DSV41_MEGA_MHC_PREFILL
value: '1'
- name: SGLANG_OPT_DSV41_BF16_INDEX_LOGITS
value: '1'
- name: SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK
value: '65536'
- name: SGLANG_DSV4_KV_LAYOUT
value: v41_new
- name: SGLANG_DP_USE_GATHERV
value: '1'
- name: SGLANG_RAGGED_VERIFY_MODE
value: static
- name: SGLANG_ENABLE_DP_SPEC_PREFILL_COORDINATION
value: '0'
- name: SGLANG_NUMA_BIND_V2
value: '0'
- name: SGLANG_DEFAULT_THINKING
value: '1'
- name: SGLANG_DSV41_REASONING_EFFORT
value: high
- name: SGLANG_TIMEOUT_KEEP_ALIVE
value: '900'
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
value: '1'
- name: MC_INTRANODE_NVLINK
value: 'true'
- name: MC_INTRA_NVLINK
value: 'true'
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
value: 'true'
- name: SGLANG_HICACHE_MOONCAKE_CONFIG_PATH
value: /data/ymk/ds41/mooncake-store-l3ssd.json
- name: SGLANG_HICACHE_PREFETCH_IO_THREADS
value: '4'
- name: SGLANG_HICACHE_BACKUP_IO_THREADS
value: '4'
- name: SGLANG_HICACHE_TIMING
value: '1'
- name: SGLANG_BATCH_TELEMETRY
value: '1'
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
value: '1'
- name: SGLANG_ENABLE_METRICS_DEVICE_TIMER
value: '1'
- name: SGLANG_EPLB_HEATMAP_COLLECTION_INTERVAL
value: '100'
- name: SGLANG_MM_TRACE
value: '1'
- name: SGLANG_BATCH_SPAN_MIN_MS
value: '200'
- name: SGLANG_BATCH_SPAN_SAMPLE
value: '0.05'
- name: SGLANG_BATCH_SPAN_MAX_LINKS
value: '8'
- name: SGLANG_BATCH_STAGE_TIMING
value: '1'
- name: SGLANG_BATCH_PASS_SPAN
value: '1'
- name: SGLANG_BATCH_PASS_SPAN_SAMPLE
value: '1.0'
- name: SGLANG_RECV_SPAN
value: '1'
- name: SGLANG_RECV_SPAN_SAMPLE
value: '1.0'
- name: SGLANG_MM_TRACE_SYNC
value: '0'
- name: SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK
value: '1'
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
value: '600'
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
value: '16'
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
value: '32'
- name: TZ
value: Asia/Shanghai
- name: SGLANG_TRACE_LOOP_SPAN
value: '1'
- name: SGLANG_TRACE_LOOP_SAMPLE
value: '1.0'
- name: SGLANG_FWD_TIMELINE
value: '0'
- name: SGLANG_TRACE_LOOP_DETAIL
value: '1'
- name: SGLANG_TRACE_MOONCAKE_DETAIL
value: '0'
- name: SGLANG_TRACE_BACKUP_DETAIL
value: '1'
- name: SGLANG_ASYNC_BACKUP_SUBMIT
value: '1'
- name: SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH
value: '1'
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
value: '1024'
- name: SGLANG_MOONCAKE_CLIENT_METRICS_PORT_BASE
value: '9400'
- name: MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR
value: bucket_storage_backend
- name: MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE
value: '1099511627776'
- name: MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES
value: '1099511627776'
- name: MOONCAKE_OFFLOAD_BUCKET_EVICTION_POLICY
value: fifo
- name: TMPDIR
value: /data/ymk/l3ssd/p-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/p-xdg
- name: HF_HOME
value: /data/ymk/l3ssd/p-hf
- name: MOONCAKE_OFFLOAD_FLUSH_PARTIAL_BUCKETS
value: '0'
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: sglang
ports:
- containerPort: 30000
name: metrics
protocol: TCP
- containerPort: 8998
name: bootstrap
protocol: TCP
- name: mc-0
containerPort: 9400
protocol: TCP
- name: mc-1
containerPort: 9401
protocol: TCP
- name: mc-2
containerPort: 9402
protocol: TCP
- name: mc-3
containerPort: 9403
protocol: TCP
readinessProbe:
failureThreshold: 120
httpGet:
path: /health
port: metrics
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '96'
memory: 2Ti
nvidia.com/gpu: '4'
requests:
cpu: '32'
memory: 512Gi
nvidia.com/gpu: '4'
securityContext:
privileged: true
volumeMounts:
- mountPath: /data
name: data
- mountPath: /root/.cache/sglang
name: sglang-cache
- mountPath: /dev/shm
name: dshm
- mountPath: /root/.cargo/bin/cargo
name: fake-cargo
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
hostPID: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/cache/sglang
type: Directory
name: sglang-cache
- emptyDir:
medium: Memory
sizeLimit: 64Gi
name: dshm
- hostPath:
path: /data/ymk/ds41/fake-cargo
type: File
name: fake-cargo
- dependencies:
- mooncake-master
minReadySeconds: 0
name: d
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '30300'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-d
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --enable-fp32-lm-head --enable-dp-attention-local-control-broadcast \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 64 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r64_ep4.json \\\n --mem-fraction-static 0.80 \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --stream-interval 32 --swa-prefix-tails 2048 --enable-decoder-swa-bounded-replay \\\n --max-running-requests 512 --cuda-graph-max-bs-decode 512 --tokenizer-worker-num 16 \\\n --chunked-prefill-size 32768 --max-prefill-tokens 8192 \\\n --disaggregation-mode decode --disaggregation-transfer-backend mooncake \\\n --host 0.0.0.0 --port 30300 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-d \\\n --watchdog-timeout 3600\n"
env:
- name: CUDA_VISIBLE_DEVICES
value: 4,5,6,7
- name: SGLANG_RAGGED_VERIFY_MODE
value: static
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
value: '1'
- name: SGLANG_DSV4_KV_LAYOUT
value: v41_new
- name: SGLANG_DEFAULT_THINKING
value: '1'
- name: SGLANG_DSV41_REASONING_EFFORT
value: high
- name: MC_INTRANODE_NVLINK
value: 'true'
- name: MC_INTRA_NVLINK
value: 'true'
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
value: 'true'
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
value: '1'
- name: SGLANG_BATCH_TELEMETRY
value: '1'
- name: SGLANG_BATCH_SPAN_SAMPLE
value: '0.1'
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
value: '600'
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
value: '16'
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
value: '32'
- name: TZ
value: Asia/Shanghai
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
value: '1024'
- name: TMPDIR
value: /data/ymk/l3ssd/d-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/d-xdg
- name: HF_HOME
value: /data/ymk/l3ssd/d-hf
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: sglang
ports:
- containerPort: 30300
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 150
httpGet:
path: /health
port: metrics
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '96'
memory: 550Gi
nvidia.com/gpu: '4'
requests:
cpu: '32'
memory: 256Gi
nvidia.com/gpu: '4'
securityContext:
privileged: true
volumeMounts:
- mountPath: /data
name: data
- mountPath: /root/.cache/sglang
name: sglang-cache
- mountPath: /dev/shm
name: dshm
- mountPath: /root/.cargo/bin/cargo
name: fake-cargo
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
hostPID: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/cache/sglang
type: Directory
name: sglang-cache
- emptyDir:
medium: Memory
sizeLimit: 64Gi
name: dshm
- hostPath:
path: /data/ymk/ds41/fake-cargo
type: File
name: fake-cargo
- dependencies:
- p
- d
minReadySeconds: 0
name: gateway
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '29000'
prometheus.io/scheme: http
prometheus.io/scrape: 'true'
labels:
app: ds41-l3ssd-gw
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec /opt/sglang/bin/sgl-model-gateway launch \\\n --pd-disaggregation \\\n --prefill http://127.0.0.1:30000 8998 \\\n --decode http://127.0.0.1:30300 \\\n --prefill-policy cache_aware --decode-policy cache_aware \\\n --g9i-enabled --g9i-max-new-tokens 4096 \\\n \\\n \\\n --g9i-regular-p-queue-reject-secs 0 \\\n --g9i-decode-max-running 512 \\\n --health-check-timeout-secs 30 --health-check-interval-secs 30 \\\n --cb-failure-threshold 50 \\\n --model-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --tokenizer-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --host 0.0.0.0 --port 30400\n"
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: gateway
ports:
- containerPort: 30400
name: http
protocol: TCP
- containerPort: 29000
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 60
httpGet:
path: /health
port: http
initialDelaySeconds: 20
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '16'
memory: 32Gi
requests:
cpu: '2'
memory: 8Gi
volumeMounts:
- mountPath: /data
name: data
- mountPath: /opt/sglang/bin/sgl-model-gateway
name: gw-bin
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- name: gw-bin
hostPath:
path: /data/ymk/cache/gw-est-td/sgl-model-gateway
type: File
@@ -0,0 +1 @@
{"rps": 24.04512015569562, "in_tput": 2290117.7612668574, "out_tput": 22449.32127276801, "ttft_p50": 1743.335193, "ttft_p95": 3771.88552025, "ttft_p99": 6579.86958935, "itl_p50": 8.162268671611512, "itl_p95": 10.741317697249674, "itl_p99": 11.806237803568294, "lat_p95": 31884.86991075, "requests": 29816.0, "isl_p50": 79909.0, "osl_p50": 339.0, "cache_pct": NaN, "valid": true}
+20
View File
@@ -0,0 +1,20 @@
#!/usr/bin/env bash
# Build and load the L3 SSD image (mooncake 0.3.14) on b300-01.
set -euo pipefail
cd "$(dirname "$0")"
WHEEL=mooncake_transfer_engine_cuda13-0.3.14-cp312-cp312-manylinux_2_28_x86_64.whl
URL=https://files.pythonhosted.org/packages/b9/bf/a117c9c9f892e86893caf0dd8b5bfae1d11edbfb0173dcd30108a7168fc3/$WHEEL
SHA=74fa5f0684b143167503e33f85cccb9ca687d3f9843d4b594dbd5342c7df3e17
TAG=recipe-ns-l3ssd-433257c7b-baked-20261115
REPO=sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang
if [ ! -f "$WHEEL" ]; then
curl -fL -o "$WHEEL" "$URL"
fi
echo "$SHA $WHEEL" | sha256sum -c -
docker build -t "ymkymx/sglang:$TAG" -f Dockerfile.mooncake-0.3.14 .
docker tag "ymkymx/sglang:$TAG" "$REPO:$TAG"
docker save "$REPO:$TAG" | ctr -n k8s.io images import -
echo "loaded $REPO:$TAG"
+30
View File
@@ -0,0 +1,30 @@
#!/usr/bin/env bash
# Reset the mooncake L3 SSD KV cache before an experiment.
#
# MUST run on b300-01 only after the ds41-gist-exp workload is fully stopped:
# 1) kmctl stop cw ds41-gist-exp --wait ...
# 2) kubectl get cw ... -> Terminated 0/0, nvidia-smi -> 0 MiB on GPU0-7
# 3) bash clean_l3ssd_cache.sh
# 4) kmctl update/start the new manifest
#
# The script refuses to delete anything while a mooncake master or sglang
# server process is still visible on the host.
set -euo pipefail
SSD_DIR="${SSD_DIR:-/data/ymk/mooncake-ssd}"
if pgrep -af '[m]ooncake_master|[s]glang serve' >/dev/null 2>&1; then
echo "refusing: mooncake_master or sglang serve is still running" >&2
pgrep -af '[m]ooncake_master|[s]glang serve' >&2 || true
exit 1
fi
if [ ! -d "$SSD_DIR" ]; then
echo "SSD cache dir does not exist: $SSD_DIR"
exit 0
fi
echo "before: $(du -sh "$SSD_DIR" | awk '{print $1}') ($SSD_DIR)"
find "$SSD_DIR" -mindepth 1 -maxdepth 1 -exec rm -rf {} +
echo "after: $(du -sh "$SSD_DIR" | awk '{print $1}') ($SSD_DIR)"
df -h /data | tail -1
@@ -0,0 +1,542 @@
apiVersion: workloads.x-k8s.io/v1alpha2
kind: RoleBasedGroup
metadata:
annotations:
rbg.workloads.x-k8s.io/discovery-config-mode: refine
labels:
app: ds41-gist-exp
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
sglang-model: ds-v41-flash
name: ds41-gist-exp
namespace: prj-d7te67bj0sq2fgdrmbig
spec:
roles:
- minReadySeconds: 0
name: mooncake-master
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '19003'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-mooncake
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "# 18081: 宿主机 18080 被 AgentX fake upstream(./fakeup)长期占用;\n# 我们的 P/D\
\ 用 metadata_server=P2PHANDSHAKE,此 HTTP 端口只是 master 可选项。\nexec /opt/sglang/bin/mooncake_master\
\ \\\n --port 50051 --enable_http_metadata_server=true \\\n --http_metadata_server_port=18081\
\ --metrics_port=19003 --eviction_high_watermark_ratio=0.90 --enable_offload=true\
\ --offload_on_evict=false --promotion_on_hit=false --offload_force_evict=false\n"
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: mooncake-master
ports:
- containerPort: 19003
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 60
initialDelaySeconds: 5
periodSeconds: 5
tcpSocket:
port: 50051
resources:
limits:
cpu: '4'
memory: 64Gi
requests:
cpu: '1'
memory: 8Gi
env:
- name: TMPDIR
value: /data/ymk/l3ssd/master-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/master-xdg
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
- dependencies:
- mooncake-master
minReadySeconds: 0
name: p
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '30000'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-p
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens\
\ --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name\
\ /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser\
\ auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"\
reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4\
\ --enable-dp-attention --enable-dp-lm-head \\\n --moe-a2a-backend\
\ megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts\
\ 32 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r32_ep4.json\
\ \\\n --mem-fraction-static 0.80 \\\n --chunked-prefill-size 131072\
\ --max-prefill-tokens 131072 \\\n --max-running-requests 512 --tokenizer-worker-num\
\ 16 \\\n --load-balance-method total_tokens \\\n --speculative-algorithm\
\ DSPARK --speculative-dspark-block-size 3 \\\n --enable-decoder-swa-bounded-replay\
\ --swa-full-tokens-ratio 0.12 \\\n --cuda-graph-max-bs-decode 1 --disable-decode-cuda-graph\
\ \\\n --cuda-graph-backend-prefill breakable \\\n --cuda-graph-bs-prefill\
\ 1024 2048 4096 8192 16384 32768 \\\n --cuda-graph-prefill-max-context\
\ 262144 --cuda-graph-max-seq-len-prefill 262144 \\\n --enable-hierarchical-cache\
\ --hicache-ratio 4 \\\n --hicache-mem-layout page_first_direct --hicache-io-backend\
\ direct \\\n --hicache-write-policy write_through --hicache-storage-backend\
\ mooncake \\\n --hicache-storage-prefetch-policy wait_complete \\\n\
\ --disaggregation-mode prefill --disaggregation-transfer-backend mooncake\
\ \\\n --disaggregation-bootstrap-port 8998 \\\n --expert-balancedness-report-mode\
\ prometheus \\\n --expert-distribution-recorder-buffer-size 64 \\\n\
\ --host 0.0.0.0 --port 30000 --enable-metrics --enable-cache-report\
\ \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint\
\ 127.0.0.1:4317 --otlp-service-name gist-exp-p \\\n --watchdog-timeout\
\ 3600 --enable-dp-attention-local-control-broadcast\n"
env:
- name: CUDA_VISIBLE_DEVICES
value: 0,1,2,3
- name: SGLANG_LOG_MS
value: '1'
- name: SGLANG_OPT_DSV41_ENGRAM_PEER_SHARDS
value: '1'
- name: SGLANG_OPT_DSV41_MEGA_MHC_PREFILL
value: '1'
- name: SGLANG_OPT_DSV41_BF16_INDEX_LOGITS
value: '1'
- name: SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK
value: '65536'
- name: SGLANG_DSV4_KV_LAYOUT
value: v41_new
- name: SGLANG_DP_USE_GATHERV
value: '1'
- name: SGLANG_RAGGED_VERIFY_MODE
value: static
- name: SGLANG_ENABLE_DP_SPEC_PREFILL_COORDINATION
value: '0'
- name: SGLANG_NUMA_BIND_V2
value: '0'
- name: SGLANG_DEFAULT_THINKING
value: '1'
- name: SGLANG_DSV41_REASONING_EFFORT
value: high
- name: SGLANG_TIMEOUT_KEEP_ALIVE
value: '900'
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
value: '1'
- name: MC_INTRANODE_NVLINK
value: 'true'
- name: MC_INTRA_NVLINK
value: 'true'
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
value: 'true'
- name: SGLANG_HICACHE_MOONCAKE_CONFIG_PATH
value: /data/ymk/ds41/mooncake-store-l3ssd.json
- name: SGLANG_HICACHE_PREFETCH_IO_THREADS
value: '4'
- name: SGLANG_HICACHE_BACKUP_IO_THREADS
value: '4'
- name: SGLANG_HICACHE_TIMING
value: '1'
- name: SGLANG_BATCH_TELEMETRY
value: '1'
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
value: '1'
- name: SGLANG_ENABLE_METRICS_DEVICE_TIMER
value: '1'
- name: SGLANG_EPLB_HEATMAP_COLLECTION_INTERVAL
value: '100'
- name: SGLANG_MM_TRACE
value: '1'
- name: SGLANG_BATCH_SPAN_MIN_MS
value: '200'
- name: SGLANG_BATCH_SPAN_SAMPLE
value: '0.05'
- name: SGLANG_BATCH_SPAN_MAX_LINKS
value: '8'
- name: SGLANG_BATCH_STAGE_TIMING
value: '1'
- name: SGLANG_BATCH_PASS_SPAN
value: '1'
- name: SGLANG_BATCH_PASS_SPAN_SAMPLE
value: '1.0'
- name: SGLANG_RECV_SPAN
value: '1'
- name: SGLANG_RECV_SPAN_SAMPLE
value: '1.0'
- name: SGLANG_MM_TRACE_SYNC
value: '0'
- name: SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK
value: '1'
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
value: '600'
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
value: '16'
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
value: '32'
- name: TZ
value: Asia/Shanghai
- name: SGLANG_TRACE_LOOP_SPAN
value: '1'
- name: SGLANG_TRACE_LOOP_SAMPLE
value: '1.0'
- name: SGLANG_FWD_TIMELINE
value: '0'
- name: SGLANG_TRACE_LOOP_DETAIL
value: '1'
- name: SGLANG_TRACE_MOONCAKE_DETAIL
value: '0'
- name: SGLANG_TRACE_BACKUP_DETAIL
value: '1'
- name: SGLANG_ASYNC_BACKUP_SUBMIT
value: '1'
- name: SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH
value: '1'
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
value: '1024'
- name: SGLANG_MOONCAKE_CLIENT_METRICS_PORT_BASE
value: '9400'
- name: MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR
value: bucket_storage_backend
- name: MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE
value: '1099511627776'
- name: MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES
value: '1099511627776'
- name: MOONCAKE_OFFLOAD_BUCKET_EVICTION_POLICY
value: fifo
- name: TMPDIR
value: /data/ymk/l3ssd/p-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/p-xdg
- name: HF_HOME
value: /data/ymk/l3ssd/p-hf
- name: MOONCAKE_OFFLOAD_FLUSH_PARTIAL_BUCKETS
value: '0'
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: sglang
ports:
- containerPort: 30000
name: metrics
protocol: TCP
- containerPort: 8998
name: bootstrap
protocol: TCP
- name: mc-0
containerPort: 9400
protocol: TCP
- name: mc-1
containerPort: 9401
protocol: TCP
- name: mc-2
containerPort: 9402
protocol: TCP
- name: mc-3
containerPort: 9403
protocol: TCP
readinessProbe:
failureThreshold: 120
httpGet:
path: /health
port: metrics
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '96'
memory: 2Ti
nvidia.com/gpu: '4'
requests:
cpu: '32'
memory: 512Gi
nvidia.com/gpu: '4'
securityContext:
privileged: true
volumeMounts:
- mountPath: /data
name: data
- mountPath: /root/.cache/sglang
name: sglang-cache
- mountPath: /dev/shm
name: dshm
- mountPath: /root/.cargo/bin/cargo
name: fake-cargo
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
hostPID: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/cache/sglang
type: Directory
name: sglang-cache
- emptyDir:
medium: Memory
sizeLimit: 64Gi
name: dshm
- hostPath:
path: /data/ymk/ds41/fake-cargo
type: File
name: fake-cargo
- dependencies:
- mooncake-master
minReadySeconds: 0
name: d
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '30300'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-d
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens\
\ --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name\
\ /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser\
\ auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"\
reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4\
\ --enable-dp-attention --enable-dp-lm-head \\\n --enable-fp32-lm-head\
\ --enable-dp-attention-local-control-broadcast \\\n --moe-a2a-backend\
\ megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts\
\ 64 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r64_ep4.json\
\ \\\n --mem-fraction-static 0.80 \\\n --speculative-algorithm DSPARK\
\ --speculative-dspark-block-size 3 \\\n --stream-interval 32 --swa-prefix-tails\
\ 2048 --enable-decoder-swa-bounded-replay \\\n --max-running-requests\
\ 512 --cuda-graph-max-bs-decode 512 \\\n --chunked-prefill-size 32768\
\ --max-prefill-tokens 8192 \\\n --disaggregation-mode decode --disaggregation-transfer-backend\
\ mooncake \\\n --host 0.0.0.0 --port 30300 --enable-metrics --enable-cache-report\
\ \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint\
\ 127.0.0.1:4317 --otlp-service-name gist-exp-d \\\n --watchdog-timeout\
\ 3600\n"
env:
- name: CUDA_VISIBLE_DEVICES
value: 4,5,6,7
- name: SGLANG_RAGGED_VERIFY_MODE
value: static
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
value: '1'
- name: SGLANG_DSV4_KV_LAYOUT
value: v41_new
- name: SGLANG_DEFAULT_THINKING
value: '1'
- name: SGLANG_DSV41_REASONING_EFFORT
value: high
- name: MC_INTRANODE_NVLINK
value: 'true'
- name: MC_INTRA_NVLINK
value: 'true'
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
value: 'true'
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
value: '1'
- name: SGLANG_BATCH_TELEMETRY
value: '1'
- name: SGLANG_BATCH_SPAN_SAMPLE
value: '0.1'
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
value: '600'
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
value: '16'
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
value: '32'
- name: TZ
value: Asia/Shanghai
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
value: '1024'
- name: TMPDIR
value: /data/ymk/l3ssd/d-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/d-xdg
- name: HF_HOME
value: /data/ymk/l3ssd/d-hf
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: sglang
ports:
- containerPort: 30300
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 150
httpGet:
path: /health
port: metrics
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '96'
memory: 550Gi
nvidia.com/gpu: '4'
requests:
cpu: '32'
memory: 256Gi
nvidia.com/gpu: '4'
securityContext:
privileged: true
volumeMounts:
- mountPath: /data
name: data
- mountPath: /root/.cache/sglang
name: sglang-cache
- mountPath: /dev/shm
name: dshm
- mountPath: /root/.cargo/bin/cargo
name: fake-cargo
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
hostPID: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/cache/sglang
type: Directory
name: sglang-cache
- emptyDir:
medium: Memory
sizeLimit: 64Gi
name: dshm
- hostPath:
path: /data/ymk/ds41/fake-cargo
type: File
name: fake-cargo
- dependencies:
- p
- d
minReadySeconds: 0
name: gateway
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '29000'
prometheus.io/scheme: http
prometheus.io/scrape: 'true'
labels:
app: ds41-l3ssd-gw
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec /opt/sglang/bin/sgl-model-gateway launch \\\n --pd-disaggregation\
\ \\\n --prefill http://127.0.0.1:30000 8998 \\\n --decode http://127.0.0.1:30300\
\ \\\n --prefill-policy cache_aware --decode-policy cache_aware \\\n\
\ --g9i-enabled --g9i-max-new-tokens 4096 \\\n \\\n \\\n --g9i-regular-p-queue-reject-secs\
\ 32 \\\n --g9i-decode-max-running 512 \\\n --health-check-timeout-secs\
\ 30 --health-check-interval-secs 30 \\\n --cb-failure-threshold 50\
\ \\\n --model-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --tokenizer-path\
\ /data/models/DeepSeek-V4.1-Flash-1017 \\\n --host 0.0.0.0 --port\
\ 30400\n"
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: gateway
ports:
- containerPort: 30400
name: http
protocol: TCP
- containerPort: 29000
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 60
httpGet:
path: /health
port: http
initialDelaySeconds: 20
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '16'
memory: 32Gi
requests:
cpu: '2'
memory: 8Gi
volumeMounts:
- mountPath: /data
name: data
- mountPath: /opt/sglang/bin/sgl-model-gateway
name: gw-bin
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- name: gw-bin
hostPath:
path: /data/ymk/cache/gw-est-td/sgl-model-gateway
type: File
@@ -0,0 +1,544 @@
apiVersion: workloads.x-k8s.io/v1alpha2
kind: RoleBasedGroup
metadata:
annotations:
rbg.workloads.x-k8s.io/discovery-config-mode: refine
labels:
app: ds41-gist-exp
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
sglang-model: ds-v41-flash
name: ds41-gist-exp
namespace: prj-d7te67bj0sq2fgdrmbig
spec:
roles:
- minReadySeconds: 0
name: mooncake-master
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '19003'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-mooncake
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "# 18081: 宿主机 18080 被 AgentX fake upstream(./fakeup)长期占用;\n# 我们的 P/D\
\ 用 metadata_server=P2PHANDSHAKE,此 HTTP 端口只是 master 可选项。\nexec /opt/sglang/bin/mooncake_master\
\ \\\n --port 50051 --enable_http_metadata_server=true \\\n --http_metadata_server_port=18081\
\ --metrics_port=19003 --eviction_high_watermark_ratio=0.90 --enable_offload=true\
\ --offload_on_evict=false --promotion_on_hit=false --offload_force_evict=false\n"
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: mooncake-master
ports:
- containerPort: 19003
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 60
initialDelaySeconds: 5
periodSeconds: 5
tcpSocket:
port: 50051
resources:
limits:
cpu: '4'
memory: 64Gi
requests:
cpu: '1'
memory: 8Gi
env:
- name: TMPDIR
value: /data/ymk/l3ssd/master-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/master-xdg
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
- dependencies:
- mooncake-master
minReadySeconds: 0
name: p
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '30000'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-p
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens\
\ --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name\
\ /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser\
\ auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"\
reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4\
\ --enable-dp-attention --enable-dp-lm-head \\\n --moe-a2a-backend\
\ megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts\
\ 32 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r32_ep4.json\
\ \\\n --mem-fraction-static 0.80 \\\n --chunked-prefill-size 131072\
\ --max-prefill-tokens 131072 \\\n --max-running-requests 512 --tokenizer-worker-num\
\ 16 \\\n --load-balance-method total_tokens \\\n --speculative-algorithm\
\ DSPARK --speculative-dspark-block-size 3 \\\n --enable-decoder-swa-bounded-replay\
\ --swa-full-tokens-ratio 0.12 \\\n --cuda-graph-max-bs-decode 1 --disable-decode-cuda-graph\
\ \\\n --cuda-graph-backend-prefill breakable \\\n --cuda-graph-bs-prefill\
\ 1024 2048 4096 8192 16384 32768 \\\n --cuda-graph-prefill-max-context\
\ 262144 --cuda-graph-max-seq-len-prefill 262144 \\\n --enable-hierarchical-cache\
\ --hicache-ratio 4 \\\n --hicache-mem-layout page_first_direct --hicache-io-backend\
\ direct \\\n --hicache-write-policy write_through --hicache-storage-backend\
\ mooncake \\\n --hicache-storage-prefetch-policy wait_complete \\\n\
\ --disaggregation-mode prefill --disaggregation-transfer-backend mooncake\
\ \\\n --disaggregation-bootstrap-port 8998 \\\n --expert-balancedness-report-mode\
\ prometheus \\\n --expert-distribution-recorder-buffer-size 64 \\\n\
\ --host 0.0.0.0 --port 30000 --enable-metrics --enable-cache-report\
\ \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint\
\ 127.0.0.1:4317 --otlp-service-name gist-exp-p \\\n --watchdog-timeout\
\ 3600 --enable-dp-attention-local-control-broadcast\n"
env:
- name: CUDA_VISIBLE_DEVICES
value: 0,1,2,3
- name: SGLANG_LOG_MS
value: '1'
- name: SGLANG_OPT_DSV41_ENGRAM_PEER_SHARDS
value: '1'
- name: SGLANG_OPT_DSV41_MEGA_MHC_PREFILL
value: '1'
- name: SGLANG_OPT_DSV41_BF16_INDEX_LOGITS
value: '1'
- name: SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK
value: '65536'
- name: SGLANG_DSV4_KV_LAYOUT
value: v41_new
- name: SGLANG_DP_USE_GATHERV
value: '1'
- name: SGLANG_RAGGED_VERIFY_MODE
value: static
- name: SGLANG_ENABLE_DP_SPEC_PREFILL_COORDINATION
value: '0'
- name: SGLANG_NUMA_BIND_V2
value: '0'
- name: SGLANG_DEFAULT_THINKING
value: '1'
- name: SGLANG_DSV41_REASONING_EFFORT
value: high
- name: SGLANG_TIMEOUT_KEEP_ALIVE
value: '900'
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
value: '1'
- name: MC_INTRANODE_NVLINK
value: 'true'
- name: MC_INTRA_NVLINK
value: 'true'
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
value: 'true'
- name: SGLANG_HICACHE_MOONCAKE_CONFIG_PATH
value: /data/ymk/ds41/mooncake-store-l3ssd.json
- name: SGLANG_HICACHE_PREFETCH_IO_THREADS
value: '4'
- name: SGLANG_HICACHE_BACKUP_IO_THREADS
value: '4'
- name: SGLANG_HICACHE_TIMING
value: '1'
- name: SGLANG_BATCH_TELEMETRY
value: '1'
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
value: '1'
- name: SGLANG_ENABLE_METRICS_DEVICE_TIMER
value: '1'
- name: SGLANG_EPLB_HEATMAP_COLLECTION_INTERVAL
value: '100'
- name: SGLANG_MM_TRACE
value: '1'
- name: SGLANG_BATCH_SPAN_MIN_MS
value: '200'
- name: SGLANG_BATCH_SPAN_SAMPLE
value: '0.05'
- name: SGLANG_BATCH_SPAN_MAX_LINKS
value: '8'
- name: SGLANG_BATCH_STAGE_TIMING
value: '1'
- name: SGLANG_BATCH_PASS_SPAN
value: '1'
- name: SGLANG_BATCH_PASS_SPAN_SAMPLE
value: '1.0'
- name: SGLANG_RECV_SPAN
value: '1'
- name: SGLANG_RECV_SPAN_SAMPLE
value: '1.0'
- name: SGLANG_MM_TRACE_SYNC
value: '0'
- name: SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK
value: '1'
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
value: '600'
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
value: '16'
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
value: '32'
- name: TZ
value: Asia/Shanghai
- name: SGLANG_TRACE_LOOP_SPAN
value: '1'
- name: SGLANG_TRACE_LOOP_SAMPLE
value: '1.0'
- name: SGLANG_FWD_TIMELINE
value: '0'
- name: SGLANG_TRACE_LOOP_DETAIL
value: '1'
- name: SGLANG_TRACE_MOONCAKE_DETAIL
value: '0'
- name: SGLANG_TRACE_BACKUP_DETAIL
value: '1'
- name: SGLANG_ASYNC_BACKUP_SUBMIT
value: '1'
- name: SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH
value: '1'
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
value: '1024'
- name: SGLANG_MOONCAKE_CLIENT_METRICS_PORT_BASE
value: '9400'
- name: MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR
value: bucket_storage_backend
- name: MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE
value: '1099511627776'
- name: MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES
value: '1099511627776'
- name: MOONCAKE_OFFLOAD_BUCKET_EVICTION_POLICY
value: fifo
- name: TMPDIR
value: /data/ymk/l3ssd/p-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/p-xdg
- name: HF_HOME
value: /data/ymk/l3ssd/p-hf
- name: MOONCAKE_OFFLOAD_FLUSH_PARTIAL_BUCKETS
value: '0'
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: sglang
ports:
- containerPort: 30000
name: metrics
protocol: TCP
- containerPort: 8998
name: bootstrap
protocol: TCP
- name: mc-0
containerPort: 9400
protocol: TCP
- name: mc-1
containerPort: 9401
protocol: TCP
- name: mc-2
containerPort: 9402
protocol: TCP
- name: mc-3
containerPort: 9403
protocol: TCP
readinessProbe:
failureThreshold: 120
httpGet:
path: /health
port: metrics
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '96'
memory: 2Ti
nvidia.com/gpu: '4'
requests:
cpu: '32'
memory: 512Gi
nvidia.com/gpu: '4'
securityContext:
privileged: true
volumeMounts:
- mountPath: /data
name: data
- mountPath: /root/.cache/sglang
name: sglang-cache
- mountPath: /dev/shm
name: dshm
- mountPath: /root/.cargo/bin/cargo
name: fake-cargo
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
hostPID: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/cache/sglang
type: Directory
name: sglang-cache
- emptyDir:
medium: Memory
sizeLimit: 64Gi
name: dshm
- hostPath:
path: /data/ymk/ds41/fake-cargo
type: File
name: fake-cargo
- dependencies:
- mooncake-master
minReadySeconds: 0
name: d
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '30300'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-d
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens\
\ --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name\
\ /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser\
\ auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"\
reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4\
\ --enable-dp-attention --enable-dp-lm-head \\\n --enable-fp32-lm-head\
\ --enable-dp-attention-local-control-broadcast \\\n --moe-a2a-backend\
\ megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts\
\ 64 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r64_ep4.json\
\ \\\n --mem-fraction-static 0.80 \\\n --speculative-algorithm DSPARK\
\ --speculative-dspark-block-size 3 \\\n --stream-interval 32 --swa-prefix-tails\
\ 2048 --enable-decoder-swa-bounded-replay \\\n --max-running-requests\
\ 512 --cuda-graph-max-bs-decode 512 \\\n --chunked-prefill-size 32768\
\ --max-prefill-tokens 8192 \\\n --disaggregation-mode decode --disaggregation-transfer-backend\
\ mooncake \\\n --host 0.0.0.0 --port 30300 --enable-metrics --enable-cache-report\
\ \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint\
\ 127.0.0.1:4317 --otlp-service-name gist-exp-d \\\n --watchdog-timeout\
\ 3600\n"
env:
- name: CUDA_VISIBLE_DEVICES
value: 4,5,6,7
- name: SGLANG_RAGGED_VERIFY_MODE
value: static
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
value: '1'
- name: SGLANG_DSV4_KV_LAYOUT
value: v41_new
- name: SGLANG_DEFAULT_THINKING
value: '1'
- name: SGLANG_DSV41_REASONING_EFFORT
value: high
- name: MC_INTRANODE_NVLINK
value: 'true'
- name: MC_INTRA_NVLINK
value: 'true'
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
value: 'true'
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
value: '1'
- name: SGLANG_BATCH_TELEMETRY
value: '1'
- name: SGLANG_BATCH_SPAN_SAMPLE
value: '0.1'
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
value: '600'
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
value: '16'
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
value: '32'
- name: TZ
value: Asia/Shanghai
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
value: '1024'
- name: TMPDIR
value: /data/ymk/l3ssd/d-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/d-xdg
- name: HF_HOME
value: /data/ymk/l3ssd/d-hf
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: sglang
ports:
- containerPort: 30300
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 150
httpGet:
path: /health
port: metrics
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '96'
memory: 550Gi
nvidia.com/gpu: '4'
requests:
cpu: '32'
memory: 256Gi
nvidia.com/gpu: '4'
securityContext:
privileged: true
volumeMounts:
- mountPath: /data
name: data
- mountPath: /root/.cache/sglang
name: sglang-cache
- mountPath: /dev/shm
name: dshm
- mountPath: /root/.cargo/bin/cargo
name: fake-cargo
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
hostPID: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/cache/sglang
type: Directory
name: sglang-cache
- emptyDir:
medium: Memory
sizeLimit: 64Gi
name: dshm
- hostPath:
path: /data/ymk/ds41/fake-cargo
type: File
name: fake-cargo
- dependencies:
- p
- d
minReadySeconds: 0
name: gateway
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '29000'
prometheus.io/scheme: http
prometheus.io/scrape: 'true'
labels:
app: ds41-l3ssd-gw
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec /opt/sglang/bin/sgl-model-gateway launch \\\n --pd-disaggregation\
\ \\\n --prefill http://127.0.0.1:30000 8998 \\\n --decode http://127.0.0.1:30300\
\ \\\n --prefill-policy cache_aware --decode-policy cache_aware \\\n\
\ --g9i-enabled --g9i-max-new-tokens 4096 --g9i-est-td-enabled --g9i-shadow-enabled\
\ --g9i-td-admit enforce --g9i-p-queue-reqs-cap 32 --g9i-p-queue-token-cap\
\ 16000000 \\\n \\\n \\\n --g9i-regular-p-queue-reject-secs 0 \\\n\
\ --g9i-decode-max-running 512 \\\n --health-check-timeout-secs 30\
\ --health-check-interval-secs 30 \\\n --cb-failure-threshold 50 \\\
\n --model-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --tokenizer-path\
\ /data/models/DeepSeek-V4.1-Flash-1017 \\\n --host 0.0.0.0 --port\
\ 30400\n"
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: gateway
ports:
- containerPort: 30400
name: http
protocol: TCP
- containerPort: 29000
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 60
httpGet:
path: /health
port: http
initialDelaySeconds: 20
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '16'
memory: 32Gi
requests:
cpu: '2'
memory: 8Gi
volumeMounts:
- mountPath: /data
name: data
- mountPath: /opt/sglang/bin/sgl-model-gateway
name: gw-bin
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- name: gw-bin
hostPath:
path: /data/ymk/gw-ns-est-td/sgl-model-gateway
type: File
@@ -0,0 +1,544 @@
apiVersion: workloads.x-k8s.io/v1alpha2
kind: RoleBasedGroup
metadata:
annotations:
rbg.workloads.x-k8s.io/discovery-config-mode: refine
labels:
app: ds41-gist-exp
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
sglang-model: ds-v41-flash
name: ds41-gist-exp
namespace: prj-d7te67bj0sq2fgdrmbig
spec:
roles:
- minReadySeconds: 0
name: mooncake-master
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '19003'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-mooncake
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "# 18081: 宿主机 18080 被 AgentX fake upstream(./fakeup)长期占用;\n# 我们的 P/D\
\ 用 metadata_server=P2PHANDSHAKE,此 HTTP 端口只是 master 可选项。\nexec /opt/sglang/bin/mooncake_master\
\ \\\n --port 50051 --enable_http_metadata_server=true \\\n --http_metadata_server_port=18081\
\ --metrics_port=19003 --eviction_high_watermark_ratio=0.90 --enable_offload=true\
\ --offload_on_evict=false --promotion_on_hit=false --offload_force_evict=false\n"
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: mooncake-master
ports:
- containerPort: 19003
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 60
initialDelaySeconds: 5
periodSeconds: 5
tcpSocket:
port: 50051
resources:
limits:
cpu: '4'
memory: 64Gi
requests:
cpu: '1'
memory: 8Gi
env:
- name: TMPDIR
value: /data/ymk/l3ssd/master-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/master-xdg
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
- dependencies:
- mooncake-master
minReadySeconds: 0
name: p
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '30000'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-p
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens\
\ --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name\
\ /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser\
\ auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"\
reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4\
\ --enable-dp-attention --enable-dp-lm-head \\\n --moe-a2a-backend\
\ megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts\
\ 32 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r32_ep4.json\
\ \\\n --mem-fraction-static 0.80 \\\n --chunked-prefill-size 131072\
\ --max-prefill-tokens 131072 \\\n --max-running-requests 512 --tokenizer-worker-num\
\ 16 \\\n --load-balance-method total_tokens \\\n --speculative-algorithm\
\ DSPARK --speculative-dspark-block-size 3 \\\n --enable-decoder-swa-bounded-replay\
\ --swa-full-tokens-ratio 0.12 \\\n --cuda-graph-max-bs-decode 1 --disable-decode-cuda-graph\
\ \\\n --cuda-graph-backend-prefill breakable \\\n --cuda-graph-bs-prefill\
\ 1024 2048 4096 8192 16384 32768 \\\n --cuda-graph-prefill-max-context\
\ 262144 --cuda-graph-max-seq-len-prefill 262144 \\\n --enable-hierarchical-cache\
\ --hicache-ratio 4 \\\n --hicache-mem-layout page_first_direct --hicache-io-backend\
\ direct \\\n --hicache-write-policy write_through --hicache-storage-backend\
\ mooncake \\\n --hicache-storage-prefetch-policy wait_complete \\\n\
\ --disaggregation-mode prefill --disaggregation-transfer-backend mooncake\
\ \\\n --disaggregation-bootstrap-port 8998 \\\n --expert-balancedness-report-mode\
\ prometheus \\\n --expert-distribution-recorder-buffer-size 64 \\\n\
\ --host 0.0.0.0 --port 30000 --enable-metrics --enable-cache-report\
\ \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint\
\ 127.0.0.1:4317 --otlp-service-name gist-exp-p \\\n --watchdog-timeout\
\ 3600 --enable-dp-attention-local-control-broadcast\n"
env:
- name: CUDA_VISIBLE_DEVICES
value: 0,1,2,3
- name: SGLANG_LOG_MS
value: '1'
- name: SGLANG_OPT_DSV41_ENGRAM_PEER_SHARDS
value: '1'
- name: SGLANG_OPT_DSV41_MEGA_MHC_PREFILL
value: '1'
- name: SGLANG_OPT_DSV41_BF16_INDEX_LOGITS
value: '1'
- name: SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK
value: '65536'
- name: SGLANG_DSV4_KV_LAYOUT
value: v41_new
- name: SGLANG_DP_USE_GATHERV
value: '1'
- name: SGLANG_RAGGED_VERIFY_MODE
value: static
- name: SGLANG_ENABLE_DP_SPEC_PREFILL_COORDINATION
value: '0'
- name: SGLANG_NUMA_BIND_V2
value: '0'
- name: SGLANG_DEFAULT_THINKING
value: '1'
- name: SGLANG_DSV41_REASONING_EFFORT
value: high
- name: SGLANG_TIMEOUT_KEEP_ALIVE
value: '900'
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
value: '1'
- name: MC_INTRANODE_NVLINK
value: 'true'
- name: MC_INTRA_NVLINK
value: 'true'
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
value: 'true'
- name: SGLANG_HICACHE_MOONCAKE_CONFIG_PATH
value: /data/ymk/ds41/mooncake-store-l3ssd.json
- name: SGLANG_HICACHE_PREFETCH_IO_THREADS
value: '4'
- name: SGLANG_HICACHE_BACKUP_IO_THREADS
value: '4'
- name: SGLANG_HICACHE_TIMING
value: '1'
- name: SGLANG_BATCH_TELEMETRY
value: '1'
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
value: '1'
- name: SGLANG_ENABLE_METRICS_DEVICE_TIMER
value: '1'
- name: SGLANG_EPLB_HEATMAP_COLLECTION_INTERVAL
value: '100'
- name: SGLANG_MM_TRACE
value: '1'
- name: SGLANG_BATCH_SPAN_MIN_MS
value: '200'
- name: SGLANG_BATCH_SPAN_SAMPLE
value: '0.05'
- name: SGLANG_BATCH_SPAN_MAX_LINKS
value: '8'
- name: SGLANG_BATCH_STAGE_TIMING
value: '1'
- name: SGLANG_BATCH_PASS_SPAN
value: '1'
- name: SGLANG_BATCH_PASS_SPAN_SAMPLE
value: '1.0'
- name: SGLANG_RECV_SPAN
value: '1'
- name: SGLANG_RECV_SPAN_SAMPLE
value: '1.0'
- name: SGLANG_MM_TRACE_SYNC
value: '0'
- name: SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK
value: '1'
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
value: '600'
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
value: '16'
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
value: '32'
- name: TZ
value: Asia/Shanghai
- name: SGLANG_TRACE_LOOP_SPAN
value: '1'
- name: SGLANG_TRACE_LOOP_SAMPLE
value: '1.0'
- name: SGLANG_FWD_TIMELINE
value: '0'
- name: SGLANG_TRACE_LOOP_DETAIL
value: '1'
- name: SGLANG_TRACE_MOONCAKE_DETAIL
value: '0'
- name: SGLANG_TRACE_BACKUP_DETAIL
value: '1'
- name: SGLANG_ASYNC_BACKUP_SUBMIT
value: '1'
- name: SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH
value: '1'
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
value: '1024'
- name: SGLANG_MOONCAKE_CLIENT_METRICS_PORT_BASE
value: '9400'
- name: MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR
value: bucket_storage_backend
- name: MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE
value: '1099511627776'
- name: MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES
value: '1099511627776'
- name: MOONCAKE_OFFLOAD_BUCKET_EVICTION_POLICY
value: fifo
- name: TMPDIR
value: /data/ymk/l3ssd/p-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/p-xdg
- name: HF_HOME
value: /data/ymk/l3ssd/p-hf
- name: MOONCAKE_OFFLOAD_FLUSH_PARTIAL_BUCKETS
value: '0'
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: sglang
ports:
- containerPort: 30000
name: metrics
protocol: TCP
- containerPort: 8998
name: bootstrap
protocol: TCP
- name: mc-0
containerPort: 9400
protocol: TCP
- name: mc-1
containerPort: 9401
protocol: TCP
- name: mc-2
containerPort: 9402
protocol: TCP
- name: mc-3
containerPort: 9403
protocol: TCP
readinessProbe:
failureThreshold: 120
httpGet:
path: /health
port: metrics
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '96'
memory: 2Ti
nvidia.com/gpu: '4'
requests:
cpu: '32'
memory: 512Gi
nvidia.com/gpu: '4'
securityContext:
privileged: true
volumeMounts:
- mountPath: /data
name: data
- mountPath: /root/.cache/sglang
name: sglang-cache
- mountPath: /dev/shm
name: dshm
- mountPath: /root/.cargo/bin/cargo
name: fake-cargo
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
hostPID: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/cache/sglang
type: Directory
name: sglang-cache
- emptyDir:
medium: Memory
sizeLimit: 64Gi
name: dshm
- hostPath:
path: /data/ymk/ds41/fake-cargo
type: File
name: fake-cargo
- dependencies:
- mooncake-master
minReadySeconds: 0
name: d
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '30300'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-d
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens\
\ --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name\
\ /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser\
\ auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"\
reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4\
\ --enable-dp-attention --enable-dp-lm-head \\\n --enable-fp32-lm-head\
\ --enable-dp-attention-local-control-broadcast \\\n --moe-a2a-backend\
\ megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts\
\ 64 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r64_ep4.json\
\ \\\n --mem-fraction-static 0.80 \\\n --speculative-algorithm DSPARK\
\ --speculative-dspark-block-size 3 \\\n --stream-interval 32 --swa-prefix-tails\
\ 2048 --enable-decoder-swa-bounded-replay \\\n --max-running-requests\
\ 512 --cuda-graph-max-bs-decode 512 \\\n --chunked-prefill-size 32768\
\ --max-prefill-tokens 8192 \\\n --disaggregation-mode decode --disaggregation-transfer-backend\
\ mooncake \\\n --host 0.0.0.0 --port 30300 --enable-metrics --enable-cache-report\
\ \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint\
\ 127.0.0.1:4317 --otlp-service-name gist-exp-d \\\n --watchdog-timeout\
\ 3600\n"
env:
- name: CUDA_VISIBLE_DEVICES
value: 4,5,6,7
- name: SGLANG_RAGGED_VERIFY_MODE
value: static
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
value: '1'
- name: SGLANG_DSV4_KV_LAYOUT
value: v41_new
- name: SGLANG_DEFAULT_THINKING
value: '1'
- name: SGLANG_DSV41_REASONING_EFFORT
value: high
- name: MC_INTRANODE_NVLINK
value: 'true'
- name: MC_INTRA_NVLINK
value: 'true'
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
value: 'true'
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
value: '1'
- name: SGLANG_BATCH_TELEMETRY
value: '1'
- name: SGLANG_BATCH_SPAN_SAMPLE
value: '0.1'
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
value: '600'
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
value: '16'
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
value: '32'
- name: TZ
value: Asia/Shanghai
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
value: '1024'
- name: TMPDIR
value: /data/ymk/l3ssd/d-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/d-xdg
- name: HF_HOME
value: /data/ymk/l3ssd/d-hf
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: sglang
ports:
- containerPort: 30300
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 150
httpGet:
path: /health
port: metrics
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '96'
memory: 550Gi
nvidia.com/gpu: '4'
requests:
cpu: '32'
memory: 256Gi
nvidia.com/gpu: '4'
securityContext:
privileged: true
volumeMounts:
- mountPath: /data
name: data
- mountPath: /root/.cache/sglang
name: sglang-cache
- mountPath: /dev/shm
name: dshm
- mountPath: /root/.cargo/bin/cargo
name: fake-cargo
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
hostPID: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/cache/sglang
type: Directory
name: sglang-cache
- emptyDir:
medium: Memory
sizeLimit: 64Gi
name: dshm
- hostPath:
path: /data/ymk/ds41/fake-cargo
type: File
name: fake-cargo
- dependencies:
- p
- d
minReadySeconds: 0
name: gateway
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '29000'
prometheus.io/scheme: http
prometheus.io/scrape: 'true'
labels:
app: ds41-l3ssd-gw
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec /opt/sglang/bin/sgl-model-gateway launch \\\n --pd-disaggregation\
\ \\\n --prefill http://127.0.0.1:30000 8998 \\\n --decode http://127.0.0.1:30300\
\ \\\n --prefill-policy cache_aware --decode-policy cache_aware \\\n\
\ --g9i-enabled --g9i-max-new-tokens 4096 --g9i-est-td-enabled --g9i-shadow-enabled\
\ --g9i-td-admit enforce --g9i-p-queue-reqs-cap 32 --g9i-p-queue-token-cap\
\ 16000000 --g9i-p-qps 20 --g9i-p-refresh-ms 250 \\\n \\\n \\\n --g9i-regular-p-queue-reject-secs\
\ 0 \\\n --g9i-decode-max-running 512 \\\n --health-check-timeout-secs\
\ 30 --health-check-interval-secs 30 \\\n --cb-failure-threshold 50\
\ \\\n --model-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --tokenizer-path\
\ /data/models/DeepSeek-V4.1-Flash-1017 \\\n --host 0.0.0.0 --port\
\ 30400\n"
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: gateway
ports:
- containerPort: 30400
name: http
protocol: TCP
- containerPort: 29000
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 60
httpGet:
path: /health
port: http
initialDelaySeconds: 20
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '16'
memory: 32Gi
requests:
cpu: '2'
memory: 8Gi
volumeMounts:
- mountPath: /data
name: data
- mountPath: /opt/sglang/bin/sgl-model-gateway
name: gw-bin
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- name: gw-bin
hostPath:
path: /data/ymk/gw-ns-est-td/sgl-model-gateway
type: File
@@ -0,0 +1,488 @@
apiVersion: workloads.x-k8s.io/v1alpha2
kind: RoleBasedGroup
metadata:
annotations:
rbg.workloads.x-k8s.io/discovery-config-mode: refine
labels:
app: ds41-gist-exp
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
sglang-model: ds-v41-flash
name: ds41-gist-exp
namespace: prj-d7te67bj0sq2fgdrmbig
spec:
roles:
- minReadySeconds: 0
name: mooncake-master
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '19003'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-mooncake
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "# 18081: 宿主机 18080 被 AgentX fake upstream(./fakeup)长期占用;\n# 我们的 P/D 用 metadata_server=P2PHANDSHAKE,此 HTTP 端口只是 master 可选项。\nexec /opt/sglang/bin/mooncake_master \\\n --port 50051 --enable_http_metadata_server=true \\\n --http_metadata_server_port=18081 --metrics_port=19003 --eviction_high_watermark_ratio=0.90 --enable_offload=true --offload_on_evict=false --promotion_on_hit=false --offload_force_evict=false\n"
env:
- name: TMPDIR
value: /data/ymk/l3ssd/master-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/master-xdg
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: mooncake-master
ports:
- containerPort: 19003
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 60
initialDelaySeconds: 5
periodSeconds: 5
tcpSocket:
port: 50051
resources:
limits:
cpu: '4'
memory: 64Gi
requests:
cpu: '1'
memory: 8Gi
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
- dependencies:
- mooncake-master
minReadySeconds: 0
name: p
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '30000'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-p
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 32 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r32_ep4.json \\\n --mem-fraction-static 0.80 \\\n --chunked-prefill-size 131072 --max-prefill-tokens 131072 \\\n --max-running-requests 512 --tokenizer-worker-num 16 \\\n --load-balance-method total_tokens \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --enable-decoder-swa-bounded-replay --swa-full-tokens-ratio 0.12 \\\n --cuda-graph-max-bs-decode 1 --disable-decode-cuda-graph \\\n --cuda-graph-backend-prefill breakable \\\n --cuda-graph-bs-prefill 1024 2048 4096 8192 16384 32768 \\\n --cuda-graph-prefill-max-context 262144 --cuda-graph-max-seq-len-prefill 262144 \\\n --enable-hierarchical-cache --hicache-ratio 4 --hicache-host-memory-mode buffer_only \\\n --hicache-mem-layout page_first_direct --hicache-io-backend direct \\\n --hicache-write-policy write_through --hicache-storage-backend mooncake \\\n --hicache-storage-prefetch-policy wait_complete --optimistic-prefill-attempts 4 \\\n --disaggregation-mode prefill --disaggregation-transfer-backend mooncake \\\n --disaggregation-bootstrap-port 8998 \\\n --expert-balancedness-report-mode prometheus \\\n --expert-distribution-recorder-buffer-size 64 \\\n --host 0.0.0.0 --port 30000 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-p \\\n --watchdog-timeout 3600 --enable-dp-attention-local-control-broadcast\n"
env:
- name: CUDA_VISIBLE_DEVICES
value: 0,1,2,3
- name: SGLANG_LOG_MS
value: '1'
- name: SGLANG_OPT_DSV41_ENGRAM_PEER_SHARDS
value: '1'
- name: SGLANG_OPT_DSV41_MEGA_MHC_PREFILL
value: '1'
- name: SGLANG_OPT_DSV41_BF16_INDEX_LOGITS
value: '1'
- name: SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK
value: '65536'
- name: SGLANG_DSV4_KV_LAYOUT
value: v41_new
- name: SGLANG_DP_USE_GATHERV
value: '1'
- name: SGLANG_RAGGED_VERIFY_MODE
value: static
- name: SGLANG_ENABLE_DP_SPEC_PREFILL_COORDINATION
value: '0'
- name: SGLANG_NUMA_BIND_V2
value: '0'
- name: SGLANG_DEFAULT_THINKING
value: '1'
- name: SGLANG_DSV41_REASONING_EFFORT
value: high
- name: SGLANG_TIMEOUT_KEEP_ALIVE
value: '900'
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
value: '1'
- name: MC_INTRANODE_NVLINK
value: 'true'
- name: MC_INTRA_NVLINK
value: 'true'
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
value: 'true'
- name: SGLANG_HICACHE_MOONCAKE_CONFIG_PATH
value: /data/ymk/ds41/mooncake-store-l3ssd.json
- name: SGLANG_HICACHE_PREFETCH_IO_THREADS
value: '4'
- name: SGLANG_HICACHE_BACKUP_IO_THREADS
value: '4'
- name: SGLANG_HICACHE_TIMING
value: '1'
- name: SGLANG_BATCH_TELEMETRY
value: '1'
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
value: '1'
- name: SGLANG_ENABLE_METRICS_DEVICE_TIMER
value: '1'
- name: SGLANG_EPLB_HEATMAP_COLLECTION_INTERVAL
value: '100'
- name: SGLANG_MM_TRACE
value: '1'
- name: SGLANG_BATCH_SPAN_MIN_MS
value: '200'
- name: SGLANG_BATCH_SPAN_SAMPLE
value: '0.05'
- name: SGLANG_BATCH_SPAN_MAX_LINKS
value: '8'
- name: SGLANG_BATCH_STAGE_TIMING
value: '1'
- name: SGLANG_BATCH_PASS_SPAN
value: '1'
- name: SGLANG_BATCH_PASS_SPAN_SAMPLE
value: '1.0'
- name: SGLANG_RECV_SPAN
value: '1'
- name: SGLANG_RECV_SPAN_SAMPLE
value: '1.0'
- name: SGLANG_MM_TRACE_SYNC
value: '0'
- name: SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK
value: '1'
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
value: '600'
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
value: '16'
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
value: '32'
- name: TZ
value: Asia/Shanghai
- name: SGLANG_TRACE_LOOP_SPAN
value: '1'
- name: SGLANG_TRACE_LOOP_SAMPLE
value: '1.0'
- name: SGLANG_FWD_TIMELINE
value: '0'
- name: SGLANG_TRACE_LOOP_DETAIL
value: '1'
- name: SGLANG_TRACE_MOONCAKE_DETAIL
value: '0'
- name: SGLANG_TRACE_BACKUP_DETAIL
value: '1'
- name: SGLANG_ASYNC_BACKUP_SUBMIT
value: '1'
- name: SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH
value: '1'
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
value: '1024'
- name: SGLANG_MOONCAKE_CLIENT_METRICS_PORT_BASE
value: '9400'
- name: MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR
value: bucket_storage_backend
- name: MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE
value: '1099511627776'
- name: MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES
value: '1099511627776'
- name: MOONCAKE_OFFLOAD_BUCKET_EVICTION_POLICY
value: fifo
- name: TMPDIR
value: /data/ymk/l3ssd/p-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/p-xdg
- name: HF_HOME
value: /data/ymk/l3ssd/p-hf
- name: MOONCAKE_OFFLOAD_FLUSH_PARTIAL_BUCKETS
value: '0'
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: sglang
ports:
- containerPort: 30000
name: metrics
protocol: TCP
- containerPort: 8998
name: bootstrap
protocol: TCP
- containerPort: 9400
name: mc-0
protocol: TCP
- containerPort: 9401
name: mc-1
protocol: TCP
- containerPort: 9402
name: mc-2
protocol: TCP
- containerPort: 9403
name: mc-3
protocol: TCP
readinessProbe:
failureThreshold: 120
httpGet:
path: /health
port: metrics
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '96'
memory: 2Ti
nvidia.com/gpu: '4'
requests:
cpu: '32'
memory: 512Gi
nvidia.com/gpu: '4'
securityContext:
privileged: true
volumeMounts:
- mountPath: /data
name: data
- mountPath: /root/.cache/sglang
name: sglang-cache
- mountPath: /dev/shm
name: dshm
- mountPath: /root/.cargo/bin/cargo
name: fake-cargo
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
hostPID: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/cache/sglang
type: Directory
name: sglang-cache
- emptyDir:
medium: Memory
sizeLimit: 64Gi
name: dshm
- hostPath:
path: /data/ymk/ds41/fake-cargo
type: File
name: fake-cargo
- dependencies:
- mooncake-master
minReadySeconds: 0
name: d
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '30300'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-d
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --enable-fp32-lm-head --enable-dp-attention-local-control-broadcast \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 64 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r64_ep4.json \\\n --mem-fraction-static 0.80 \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --stream-interval 32 --swa-prefix-tails 2048 --enable-decoder-swa-bounded-replay \\\n --max-running-requests 512 --cuda-graph-max-bs-decode 512 --tokenizer-worker-num 16 \\\n --chunked-prefill-size 32768 --max-prefill-tokens 8192 \\\n --disaggregation-mode decode --disaggregation-transfer-backend mooncake \\\n --host 0.0.0.0 --port 30300 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-d \\\n --watchdog-timeout 3600\n"
env:
- name: CUDA_VISIBLE_DEVICES
value: 4,5,6,7
- name: SGLANG_RAGGED_VERIFY_MODE
value: static
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
value: '1'
- name: SGLANG_DSV4_KV_LAYOUT
value: v41_new
- name: SGLANG_DEFAULT_THINKING
value: '1'
- name: SGLANG_DSV41_REASONING_EFFORT
value: high
- name: MC_INTRANODE_NVLINK
value: 'true'
- name: MC_INTRA_NVLINK
value: 'true'
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
value: 'true'
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
value: '1'
- name: SGLANG_BATCH_TELEMETRY
value: '1'
- name: SGLANG_BATCH_SPAN_SAMPLE
value: '0.1'
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
value: '600'
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
value: '16'
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
value: '32'
- name: TZ
value: Asia/Shanghai
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
value: '1024'
- name: TMPDIR
value: /data/ymk/l3ssd/d-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/d-xdg
- name: HF_HOME
value: /data/ymk/l3ssd/d-hf
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: sglang
ports:
- containerPort: 30300
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 150
httpGet:
path: /health
port: metrics
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '96'
memory: 550Gi
nvidia.com/gpu: '4'
requests:
cpu: '32'
memory: 256Gi
nvidia.com/gpu: '4'
securityContext:
privileged: true
volumeMounts:
- mountPath: /data
name: data
- mountPath: /root/.cache/sglang
name: sglang-cache
- mountPath: /dev/shm
name: dshm
- mountPath: /root/.cargo/bin/cargo
name: fake-cargo
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
hostPID: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/cache/sglang
type: Directory
name: sglang-cache
- emptyDir:
medium: Memory
sizeLimit: 64Gi
name: dshm
- hostPath:
path: /data/ymk/ds41/fake-cargo
type: File
name: fake-cargo
- dependencies:
- p
- d
minReadySeconds: 0
name: gateway
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '29000'
prometheus.io/scheme: http
prometheus.io/scrape: 'true'
labels:
app: ds41-l3ssd-gw
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec /opt/sglang/bin/sgl-model-gateway launch \\\n --pd-disaggregation \\\n --prefill http://127.0.0.1:30000 8998 \\\n --decode http://127.0.0.1:30300 \\\n --prefill-policy cache_aware --decode-policy cache_aware \\\n --g9i-enabled --g9i-max-new-tokens 4096 --g9i-est-td-enabled --g9i-shadow-enabled --g9i-td-admit enforce --g9i-p-queue-reqs-cap 64 --g9i-p-queue-token-cap 32000000 --g9i-p-qps 40 --g9i-p-refresh-ms 250 --g9i-window-fixed 256 \\\n \\\n \\\n --g9i-regular-p-queue-reject-secs 0 \\\n --g9i-decode-max-running 512 \\\n --pool-idle-timeout-secs 2 --pool-max-idle-per-host 0 \\\n --health-check-timeout-secs 30 --health-check-interval-secs 30 \\\n --cb-failure-threshold 50 \\\n --model-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --tokenizer-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --host 0.0.0.0 --port 30400\n"
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: gateway
ports:
- containerPort: 30400
name: http
protocol: TCP
- containerPort: 29000
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 60
httpGet:
path: /health
port: http
initialDelaySeconds: 20
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '16'
memory: 32Gi
requests:
cpu: '2'
memory: 8Gi
volumeMounts:
- mountPath: /data
name: data
- mountPath: /opt/sglang/bin/sgl-model-gateway
name: gw-bin
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/gw-ns-est-td/sgl-model-gateway
type: File
name: gw-bin
@@ -0,0 +1,488 @@
apiVersion: workloads.x-k8s.io/v1alpha2
kind: RoleBasedGroup
metadata:
annotations:
rbg.workloads.x-k8s.io/discovery-config-mode: refine
labels:
app: ds41-gist-exp
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
sglang-model: ds-v41-flash
name: ds41-gist-exp
namespace: prj-d7te67bj0sq2fgdrmbig
spec:
roles:
- minReadySeconds: 0
name: mooncake-master
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '19003'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-mooncake
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "# 18081: 宿主机 18080 被 AgentX fake upstream(./fakeup)长期占用;\n# 我们的 P/D 用 metadata_server=P2PHANDSHAKE,此 HTTP 端口只是 master 可选项。\nexec /opt/sglang/bin/mooncake_master \\\n --port 50051 --enable_http_metadata_server=true \\\n --http_metadata_server_port=18081 --metrics_port=19003 --eviction_high_watermark_ratio=0.90 --enable_offload=true --offload_on_evict=false --promotion_on_hit=false --offload_force_evict=false\n"
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: mooncake-master
ports:
- containerPort: 19003
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 60
initialDelaySeconds: 5
periodSeconds: 5
tcpSocket:
port: 50051
resources:
limits:
cpu: '4'
memory: 64Gi
requests:
cpu: '1'
memory: 8Gi
env:
- name: TMPDIR
value: /data/ymk/l3ssd/master-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/master-xdg
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
- dependencies:
- mooncake-master
minReadySeconds: 0
name: p
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '30000'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-p
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 32 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r32_ep4.json \\\n --mem-fraction-static 0.80 \\\n --chunked-prefill-size 131072 --max-prefill-tokens 131072 \\\n --max-running-requests 512 --tokenizer-worker-num 16 \\\n --load-balance-method total_tokens \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --enable-decoder-swa-bounded-replay --swa-full-tokens-ratio 0.12 \\\n --cuda-graph-max-bs-decode 1 --disable-decode-cuda-graph \\\n --cuda-graph-backend-prefill breakable \\\n --cuda-graph-bs-prefill 1024 2048 4096 8192 16384 32768 \\\n --cuda-graph-prefill-max-context 262144 --cuda-graph-max-seq-len-prefill 262144 \\\n --enable-hierarchical-cache --hicache-ratio 4 --hicache-host-memory-mode buffer_only \\\n --hicache-mem-layout page_first_direct --hicache-io-backend direct \\\n --hicache-write-policy write_through --hicache-storage-backend mooncake \\\n --hicache-storage-prefetch-policy wait_complete \\\n --disaggregation-mode prefill --disaggregation-transfer-backend mooncake \\\n --disaggregation-bootstrap-port 8998 \\\n --expert-balancedness-report-mode prometheus \\\n --expert-distribution-recorder-buffer-size 64 \\\n --host 0.0.0.0 --port 30000 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-p \\\n --watchdog-timeout 3600 --enable-dp-attention-local-control-broadcast\n"
env:
- name: CUDA_VISIBLE_DEVICES
value: 0,1,2,3
- name: SGLANG_LOG_MS
value: '1'
- name: SGLANG_OPT_DSV41_ENGRAM_PEER_SHARDS
value: '1'
- name: SGLANG_OPT_DSV41_MEGA_MHC_PREFILL
value: '1'
- name: SGLANG_OPT_DSV41_BF16_INDEX_LOGITS
value: '1'
- name: SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK
value: '65536'
- name: SGLANG_DSV4_KV_LAYOUT
value: v41_new
- name: SGLANG_DP_USE_GATHERV
value: '1'
- name: SGLANG_RAGGED_VERIFY_MODE
value: static
- name: SGLANG_ENABLE_DP_SPEC_PREFILL_COORDINATION
value: '0'
- name: SGLANG_NUMA_BIND_V2
value: '0'
- name: SGLANG_DEFAULT_THINKING
value: '1'
- name: SGLANG_DSV41_REASONING_EFFORT
value: high
- name: SGLANG_TIMEOUT_KEEP_ALIVE
value: '900'
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
value: '1'
- name: MC_INTRANODE_NVLINK
value: 'true'
- name: MC_INTRA_NVLINK
value: 'true'
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
value: 'true'
- name: SGLANG_HICACHE_MOONCAKE_CONFIG_PATH
value: /data/ymk/ds41/mooncake-store-l3ssd.json
- name: SGLANG_HICACHE_PREFETCH_IO_THREADS
value: '4'
- name: SGLANG_HICACHE_BACKUP_IO_THREADS
value: '4'
- name: SGLANG_HICACHE_TIMING
value: '1'
- name: SGLANG_BATCH_TELEMETRY
value: '1'
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
value: '1'
- name: SGLANG_ENABLE_METRICS_DEVICE_TIMER
value: '1'
- name: SGLANG_EPLB_HEATMAP_COLLECTION_INTERVAL
value: '100'
- name: SGLANG_MM_TRACE
value: '1'
- name: SGLANG_BATCH_SPAN_MIN_MS
value: '200'
- name: SGLANG_BATCH_SPAN_SAMPLE
value: '0.05'
- name: SGLANG_BATCH_SPAN_MAX_LINKS
value: '8'
- name: SGLANG_BATCH_STAGE_TIMING
value: '1'
- name: SGLANG_BATCH_PASS_SPAN
value: '1'
- name: SGLANG_BATCH_PASS_SPAN_SAMPLE
value: '1.0'
- name: SGLANG_RECV_SPAN
value: '1'
- name: SGLANG_RECV_SPAN_SAMPLE
value: '1.0'
- name: SGLANG_MM_TRACE_SYNC
value: '0'
- name: SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK
value: '1'
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
value: '600'
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
value: '16'
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
value: '32'
- name: TZ
value: Asia/Shanghai
- name: SGLANG_TRACE_LOOP_SPAN
value: '1'
- name: SGLANG_TRACE_LOOP_SAMPLE
value: '1.0'
- name: SGLANG_FWD_TIMELINE
value: '0'
- name: SGLANG_TRACE_LOOP_DETAIL
value: '1'
- name: SGLANG_TRACE_MOONCAKE_DETAIL
value: '0'
- name: SGLANG_TRACE_BACKUP_DETAIL
value: '1'
- name: SGLANG_ASYNC_BACKUP_SUBMIT
value: '1'
- name: SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH
value: '1'
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
value: '1024'
- name: SGLANG_MOONCAKE_CLIENT_METRICS_PORT_BASE
value: '9400'
- name: MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR
value: bucket_storage_backend
- name: MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE
value: '1099511627776'
- name: MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES
value: '1099511627776'
- name: MOONCAKE_OFFLOAD_BUCKET_EVICTION_POLICY
value: fifo
- name: TMPDIR
value: /data/ymk/l3ssd/p-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/p-xdg
- name: HF_HOME
value: /data/ymk/l3ssd/p-hf
- name: MOONCAKE_OFFLOAD_FLUSH_PARTIAL_BUCKETS
value: '0'
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: sglang
ports:
- containerPort: 30000
name: metrics
protocol: TCP
- containerPort: 8998
name: bootstrap
protocol: TCP
- name: mc-0
containerPort: 9400
protocol: TCP
- name: mc-1
containerPort: 9401
protocol: TCP
- name: mc-2
containerPort: 9402
protocol: TCP
- name: mc-3
containerPort: 9403
protocol: TCP
readinessProbe:
failureThreshold: 120
httpGet:
path: /health
port: metrics
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '96'
memory: 2Ti
nvidia.com/gpu: '4'
requests:
cpu: '32'
memory: 512Gi
nvidia.com/gpu: '4'
securityContext:
privileged: true
volumeMounts:
- mountPath: /data
name: data
- mountPath: /root/.cache/sglang
name: sglang-cache
- mountPath: /dev/shm
name: dshm
- mountPath: /root/.cargo/bin/cargo
name: fake-cargo
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
hostPID: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/cache/sglang
type: Directory
name: sglang-cache
- emptyDir:
medium: Memory
sizeLimit: 64Gi
name: dshm
- hostPath:
path: /data/ymk/ds41/fake-cargo
type: File
name: fake-cargo
- dependencies:
- mooncake-master
minReadySeconds: 0
name: d
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '30300'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-d
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --enable-fp32-lm-head --enable-dp-attention-local-control-broadcast \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 64 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r64_ep4.json \\\n --mem-fraction-static 0.80 \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --stream-interval 32 --swa-prefix-tails 2048 --enable-decoder-swa-bounded-replay \\\n --max-running-requests 512 --cuda-graph-max-bs-decode 512 --tokenizer-worker-num 16 \\\n --chunked-prefill-size 32768 --max-prefill-tokens 8192 \\\n --disaggregation-mode decode --disaggregation-transfer-backend mooncake \\\n --host 0.0.0.0 --port 30300 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-d \\\n --watchdog-timeout 3600\n"
env:
- name: CUDA_VISIBLE_DEVICES
value: 4,5,6,7
- name: SGLANG_RAGGED_VERIFY_MODE
value: static
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
value: '1'
- name: SGLANG_DSV4_KV_LAYOUT
value: v41_new
- name: SGLANG_DEFAULT_THINKING
value: '1'
- name: SGLANG_DSV41_REASONING_EFFORT
value: high
- name: MC_INTRANODE_NVLINK
value: 'true'
- name: MC_INTRA_NVLINK
value: 'true'
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
value: 'true'
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
value: '1'
- name: SGLANG_BATCH_TELEMETRY
value: '1'
- name: SGLANG_BATCH_SPAN_SAMPLE
value: '0.1'
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
value: '600'
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
value: '16'
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
value: '32'
- name: TZ
value: Asia/Shanghai
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
value: '1024'
- name: TMPDIR
value: /data/ymk/l3ssd/d-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/d-xdg
- name: HF_HOME
value: /data/ymk/l3ssd/d-hf
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: sglang
ports:
- containerPort: 30300
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 150
httpGet:
path: /health
port: metrics
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '96'
memory: 550Gi
nvidia.com/gpu: '4'
requests:
cpu: '32'
memory: 256Gi
nvidia.com/gpu: '4'
securityContext:
privileged: true
volumeMounts:
- mountPath: /data
name: data
- mountPath: /root/.cache/sglang
name: sglang-cache
- mountPath: /dev/shm
name: dshm
- mountPath: /root/.cargo/bin/cargo
name: fake-cargo
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
hostPID: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/cache/sglang
type: Directory
name: sglang-cache
- emptyDir:
medium: Memory
sizeLimit: 64Gi
name: dshm
- hostPath:
path: /data/ymk/ds41/fake-cargo
type: File
name: fake-cargo
- dependencies:
- p
- d
minReadySeconds: 0
name: gateway
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '29000'
prometheus.io/scheme: http
prometheus.io/scrape: 'true'
labels:
app: ds41-l3ssd-gw
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec /opt/sglang/bin/sgl-model-gateway launch \\\n --pd-disaggregation \\\n --prefill http://127.0.0.1:30000 8998 \\\n --decode http://127.0.0.1:30300 \\\n --prefill-policy cache_aware --decode-policy cache_aware \\\n --g9i-enabled --g9i-max-new-tokens 4096 \\\n \\\n \\\n --g9i-regular-p-queue-reject-secs 0 \\\n --g9i-decode-max-running 512 \\\n --health-check-timeout-secs 30 --health-check-interval-secs 30 \\\n --cb-failure-threshold 50 \\\n --model-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --tokenizer-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --host 0.0.0.0 --port 30400\n"
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: gateway
ports:
- containerPort: 30400
name: http
protocol: TCP
- containerPort: 29000
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 60
httpGet:
path: /health
port: http
initialDelaySeconds: 20
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '16'
memory: 32Gi
requests:
cpu: '2'
memory: 8Gi
volumeMounts:
- mountPath: /data
name: data
- mountPath: /opt/sglang/bin/sgl-model-gateway
name: gw-bin
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- name: gw-bin
hostPath:
path: /data/ymk/cache/gw-est-td/sgl-model-gateway
type: File
@@ -0,0 +1,488 @@
apiVersion: workloads.x-k8s.io/v1alpha2
kind: RoleBasedGroup
metadata:
annotations:
rbg.workloads.x-k8s.io/discovery-config-mode: refine
labels:
app: ds41-gist-exp
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
sglang-model: ds-v41-flash
name: ds41-gist-exp
namespace: prj-d7te67bj0sq2fgdrmbig
spec:
roles:
- minReadySeconds: 0
name: mooncake-master
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '19003'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-mooncake
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "# 18081: 宿主机 18080 被 AgentX fake upstream(./fakeup)长期占用;\n# 我们的 P/D 用 metadata_server=P2PHANDSHAKE,此 HTTP 端口只是 master 可选项。\nexec /opt/sglang/bin/mooncake_master \\\n --port 50051 --enable_http_metadata_server=true \\\n --http_metadata_server_port=18081 --metrics_port=19003 --eviction_high_watermark_ratio=0.90 --enable_offload=true --offload_on_evict=false --promotion_on_hit=false --offload_force_evict=false\n"
env:
- name: TMPDIR
value: /data/ymk/l3ssd/master-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/master-xdg
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: mooncake-master
ports:
- containerPort: 19003
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 60
initialDelaySeconds: 5
periodSeconds: 5
tcpSocket:
port: 50051
resources:
limits:
cpu: '4'
memory: 64Gi
requests:
cpu: '1'
memory: 8Gi
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
- dependencies:
- mooncake-master
minReadySeconds: 0
name: p
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '30000'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-p
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 32 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r32_ep4.json \\\n --mem-fraction-static 0.80 \\\n --chunked-prefill-size 131072 --max-prefill-tokens 131072 \\\n --max-running-requests 512 --tokenizer-worker-num 16 \\\n --load-balance-method total_tokens \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --enable-decoder-swa-bounded-replay --swa-full-tokens-ratio 0.12 \\\n --cuda-graph-max-bs-decode 1 --disable-decode-cuda-graph \\\n --cuda-graph-backend-prefill breakable \\\n --cuda-graph-bs-prefill 1024 2048 4096 8192 16384 32768 \\\n --cuda-graph-prefill-max-context 262144 --cuda-graph-max-seq-len-prefill 262144 \\\n --enable-hierarchical-cache --hicache-ratio 4 \\\n --hicache-mem-layout page_first_direct --hicache-io-backend direct \\\n --hicache-write-policy write_through --hicache-storage-backend mooncake \\\n --hicache-storage-prefetch-policy wait_complete \\\n --disaggregation-mode prefill --disaggregation-transfer-backend mooncake \\\n --disaggregation-bootstrap-port 8998 \\\n --expert-balancedness-report-mode prometheus \\\n --expert-distribution-recorder-buffer-size 64 \\\n --host 0.0.0.0 --port 30000 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-p \\\n --watchdog-timeout 3600 --enable-dp-attention-local-control-broadcast\n"
env:
- name: CUDA_VISIBLE_DEVICES
value: 0,1,2,3
- name: SGLANG_LOG_MS
value: '1'
- name: SGLANG_OPT_DSV41_ENGRAM_PEER_SHARDS
value: '1'
- name: SGLANG_OPT_DSV41_MEGA_MHC_PREFILL
value: '1'
- name: SGLANG_OPT_DSV41_BF16_INDEX_LOGITS
value: '1'
- name: SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK
value: '65536'
- name: SGLANG_DSV4_KV_LAYOUT
value: v41_new
- name: SGLANG_DP_USE_GATHERV
value: '1'
- name: SGLANG_RAGGED_VERIFY_MODE
value: static
- name: SGLANG_ENABLE_DP_SPEC_PREFILL_COORDINATION
value: '0'
- name: SGLANG_NUMA_BIND_V2
value: '0'
- name: SGLANG_DEFAULT_THINKING
value: '1'
- name: SGLANG_DSV41_REASONING_EFFORT
value: high
- name: SGLANG_TIMEOUT_KEEP_ALIVE
value: '900'
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
value: '1'
- name: MC_INTRANODE_NVLINK
value: 'true'
- name: MC_INTRA_NVLINK
value: 'true'
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
value: 'true'
- name: SGLANG_HICACHE_MOONCAKE_CONFIG_PATH
value: /data/ymk/ds41/mooncake-store-l3ssd.json
- name: SGLANG_HICACHE_PREFETCH_IO_THREADS
value: '4'
- name: SGLANG_HICACHE_BACKUP_IO_THREADS
value: '4'
- name: SGLANG_HICACHE_TIMING
value: '1'
- name: SGLANG_BATCH_TELEMETRY
value: '1'
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
value: '1'
- name: SGLANG_ENABLE_METRICS_DEVICE_TIMER
value: '1'
- name: SGLANG_EPLB_HEATMAP_COLLECTION_INTERVAL
value: '100'
- name: SGLANG_MM_TRACE
value: '1'
- name: SGLANG_BATCH_SPAN_MIN_MS
value: '200'
- name: SGLANG_BATCH_SPAN_SAMPLE
value: '0.05'
- name: SGLANG_BATCH_SPAN_MAX_LINKS
value: '8'
- name: SGLANG_BATCH_STAGE_TIMING
value: '1'
- name: SGLANG_BATCH_PASS_SPAN
value: '1'
- name: SGLANG_BATCH_PASS_SPAN_SAMPLE
value: '1.0'
- name: SGLANG_RECV_SPAN
value: '1'
- name: SGLANG_RECV_SPAN_SAMPLE
value: '1.0'
- name: SGLANG_MM_TRACE_SYNC
value: '0'
- name: SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK
value: '1'
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
value: '600'
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
value: '16'
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
value: '32'
- name: TZ
value: Asia/Shanghai
- name: SGLANG_TRACE_LOOP_SPAN
value: '1'
- name: SGLANG_TRACE_LOOP_SAMPLE
value: '1.0'
- name: SGLANG_FWD_TIMELINE
value: '0'
- name: SGLANG_TRACE_LOOP_DETAIL
value: '1'
- name: SGLANG_TRACE_MOONCAKE_DETAIL
value: '0'
- name: SGLANG_TRACE_BACKUP_DETAIL
value: '1'
- name: SGLANG_ASYNC_BACKUP_SUBMIT
value: '1'
- name: SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH
value: '1'
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
value: '1024'
- name: SGLANG_MOONCAKE_CLIENT_METRICS_PORT_BASE
value: '9400'
- name: MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR
value: bucket_storage_backend
- name: MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE
value: '1099511627776'
- name: MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES
value: '1099511627776'
- name: MOONCAKE_OFFLOAD_BUCKET_EVICTION_POLICY
value: fifo
- name: TMPDIR
value: /data/ymk/l3ssd/p-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/p-xdg
- name: HF_HOME
value: /data/ymk/l3ssd/p-hf
- name: MOONCAKE_OFFLOAD_FLUSH_PARTIAL_BUCKETS
value: '0'
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: sglang
ports:
- containerPort: 30000
name: metrics
protocol: TCP
- containerPort: 8998
name: bootstrap
protocol: TCP
- containerPort: 9400
name: mc-0
protocol: TCP
- containerPort: 9401
name: mc-1
protocol: TCP
- containerPort: 9402
name: mc-2
protocol: TCP
- containerPort: 9403
name: mc-3
protocol: TCP
readinessProbe:
failureThreshold: 120
httpGet:
path: /health
port: metrics
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '96'
memory: 2Ti
nvidia.com/gpu: '4'
requests:
cpu: '32'
memory: 512Gi
nvidia.com/gpu: '4'
securityContext:
privileged: true
volumeMounts:
- mountPath: /data
name: data
- mountPath: /root/.cache/sglang
name: sglang-cache
- mountPath: /dev/shm
name: dshm
- mountPath: /root/.cargo/bin/cargo
name: fake-cargo
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
hostPID: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/cache/sglang
type: Directory
name: sglang-cache
- emptyDir:
medium: Memory
sizeLimit: 64Gi
name: dshm
- hostPath:
path: /data/ymk/ds41/fake-cargo
type: File
name: fake-cargo
- dependencies:
- mooncake-master
minReadySeconds: 0
name: d
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '30300'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-d
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --enable-fp32-lm-head --enable-dp-attention-local-control-broadcast \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 64 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r64_ep4.json \\\n --mem-fraction-static 0.80 \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --stream-interval 32 --swa-prefix-tails 2048 --enable-decoder-swa-bounded-replay \\\n --max-running-requests 512 --cuda-graph-max-bs-decode 512 --tokenizer-worker-num 16 \\\n --chunked-prefill-size 32768 --max-prefill-tokens 8192 \\\n --disaggregation-mode decode --disaggregation-transfer-backend mooncake \\\n --host 0.0.0.0 --port 30300 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-d \\\n --watchdog-timeout 3600\n"
env:
- name: CUDA_VISIBLE_DEVICES
value: 4,5,6,7
- name: SGLANG_RAGGED_VERIFY_MODE
value: static
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
value: '1'
- name: SGLANG_DSV4_KV_LAYOUT
value: v41_new
- name: SGLANG_DEFAULT_THINKING
value: '1'
- name: SGLANG_DSV41_REASONING_EFFORT
value: high
- name: MC_INTRANODE_NVLINK
value: 'true'
- name: MC_INTRA_NVLINK
value: 'true'
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
value: 'true'
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
value: '1'
- name: SGLANG_BATCH_TELEMETRY
value: '1'
- name: SGLANG_BATCH_SPAN_SAMPLE
value: '0.1'
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
value: '600'
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
value: '16'
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
value: '32'
- name: TZ
value: Asia/Shanghai
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
value: '1024'
- name: TMPDIR
value: /data/ymk/l3ssd/d-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/d-xdg
- name: HF_HOME
value: /data/ymk/l3ssd/d-hf
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: sglang
ports:
- containerPort: 30300
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 150
httpGet:
path: /health
port: metrics
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '96'
memory: 550Gi
nvidia.com/gpu: '4'
requests:
cpu: '32'
memory: 256Gi
nvidia.com/gpu: '4'
securityContext:
privileged: true
volumeMounts:
- mountPath: /data
name: data
- mountPath: /root/.cache/sglang
name: sglang-cache
- mountPath: /dev/shm
name: dshm
- mountPath: /root/.cargo/bin/cargo
name: fake-cargo
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
hostPID: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/cache/sglang
type: Directory
name: sglang-cache
- emptyDir:
medium: Memory
sizeLimit: 64Gi
name: dshm
- hostPath:
path: /data/ymk/ds41/fake-cargo
type: File
name: fake-cargo
- dependencies:
- p
- d
minReadySeconds: 0
name: gateway
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '29000'
prometheus.io/scheme: http
prometheus.io/scrape: 'true'
labels:
app: ds41-l3ssd-gw
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec /opt/sglang/bin/sgl-model-gateway launch \\\n --pd-disaggregation \\\n --prefill http://127.0.0.1:30000 8998 \\\n --decode http://127.0.0.1:30300 \\\n --prefill-policy cache_aware --decode-policy cache_aware \\\n --g9i-enabled --g9i-max-new-tokens 4096 --g9i-est-td-enabled --g9i-shadow-enabled --g9i-td-admit off --g9i-p-queue-reqs-cap 0 --g9i-p-queue-token-cap 0 --g9i-p-qps 0 --g9i-p-refresh-ms 250 --g9i-window-fixed 256 \\\n \\\n \\\n --g9i-regular-p-queue-reject-secs 0 \\\n --g9i-decode-max-running 1000000 \\\n --pool-idle-timeout-secs 2 --pool-max-idle-per-host 0 \\\n --health-check-timeout-secs 30 --health-check-interval-secs 30 \\\n --cb-failure-threshold 50 --disable-circuit-breaker --disable-health-check \\\n --model-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --tokenizer-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --host 0.0.0.0 --port 30400\n"
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: gateway
ports:
- containerPort: 30400
name: http
protocol: TCP
- containerPort: 29000
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 60
httpGet:
path: /health
port: http
initialDelaySeconds: 20
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '16'
memory: 32Gi
requests:
cpu: '2'
memory: 8Gi
volumeMounts:
- mountPath: /data
name: data
- mountPath: /opt/sglang/bin/sgl-model-gateway
name: gw-bin
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/gw-ns-est-td/sgl-model-gateway
type: File
name: gw-bin
@@ -0,0 +1,488 @@
apiVersion: workloads.x-k8s.io/v1alpha2
kind: RoleBasedGroup
metadata:
annotations:
rbg.workloads.x-k8s.io/discovery-config-mode: refine
labels:
app: ds41-gist-exp
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
sglang-model: ds-v41-flash
name: ds41-gist-exp
namespace: prj-d7te67bj0sq2fgdrmbig
spec:
roles:
- minReadySeconds: 0
name: mooncake-master
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '19003'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-mooncake
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "# 18081: 宿主机 18080 被 AgentX fake upstream(./fakeup)长期占用;\n# 我们的 P/D 用 metadata_server=P2PHANDSHAKE,此 HTTP 端口只是 master 可选项。\nexec /opt/sglang/bin/mooncake_master \\\n --port 50051 --enable_http_metadata_server=true \\\n --http_metadata_server_port=18081 --metrics_port=19003 --eviction_high_watermark_ratio=0.90 --enable_offload=true --offload_on_evict=false --promotion_on_hit=false --offload_force_evict=false\n"
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: mooncake-master
ports:
- containerPort: 19003
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 60
initialDelaySeconds: 5
periodSeconds: 5
tcpSocket:
port: 50051
resources:
limits:
cpu: '4'
memory: 64Gi
requests:
cpu: '1'
memory: 8Gi
env:
- name: TMPDIR
value: /data/ymk/l3ssd/master-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/master-xdg
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
- dependencies:
- mooncake-master
minReadySeconds: 0
name: p
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '30000'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-p
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 32 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r32_ep4.json \\\n --mem-fraction-static 0.80 \\\n --chunked-prefill-size 131072 --max-prefill-tokens 131072 \\\n --max-running-requests 512 --tokenizer-worker-num 16 \\\n --load-balance-method total_tokens \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --enable-decoder-swa-bounded-replay --swa-full-tokens-ratio 0.12 \\\n --cuda-graph-max-bs-decode 1 --disable-decode-cuda-graph \\\n --cuda-graph-backend-prefill breakable \\\n --cuda-graph-bs-prefill 1024 2048 4096 8192 16384 32768 \\\n --cuda-graph-prefill-max-context 262144 --cuda-graph-max-seq-len-prefill 262144 \\\n --enable-hierarchical-cache --hicache-ratio 4 \\\n --hicache-mem-layout page_first_direct --hicache-io-backend direct \\\n --hicache-write-policy write_through --hicache-storage-backend mooncake \\\n --hicache-storage-prefetch-policy wait_complete \\\n --disaggregation-mode prefill --disaggregation-transfer-backend mooncake \\\n --disaggregation-bootstrap-port 8998 \\\n --expert-balancedness-report-mode prometheus \\\n --expert-distribution-recorder-buffer-size 64 \\\n --host 0.0.0.0 --port 30000 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-p \\\n --watchdog-timeout 3600 --enable-dp-attention-local-control-broadcast\n"
env:
- name: CUDA_VISIBLE_DEVICES
value: 0,1,2,3
- name: SGLANG_LOG_MS
value: '1'
- name: SGLANG_OPT_DSV41_ENGRAM_PEER_SHARDS
value: '1'
- name: SGLANG_OPT_DSV41_MEGA_MHC_PREFILL
value: '1'
- name: SGLANG_OPT_DSV41_BF16_INDEX_LOGITS
value: '1'
- name: SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK
value: '65536'
- name: SGLANG_DSV4_KV_LAYOUT
value: v41_new
- name: SGLANG_DP_USE_GATHERV
value: '1'
- name: SGLANG_RAGGED_VERIFY_MODE
value: static
- name: SGLANG_ENABLE_DP_SPEC_PREFILL_COORDINATION
value: '0'
- name: SGLANG_NUMA_BIND_V2
value: '0'
- name: SGLANG_DEFAULT_THINKING
value: '1'
- name: SGLANG_DSV41_REASONING_EFFORT
value: high
- name: SGLANG_TIMEOUT_KEEP_ALIVE
value: '900'
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
value: '1'
- name: MC_INTRANODE_NVLINK
value: 'true'
- name: MC_INTRA_NVLINK
value: 'true'
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
value: 'true'
- name: SGLANG_HICACHE_MOONCAKE_CONFIG_PATH
value: /data/ymk/ds41/mooncake-store-l3ssd.json
- name: SGLANG_HICACHE_PREFETCH_IO_THREADS
value: '4'
- name: SGLANG_HICACHE_BACKUP_IO_THREADS
value: '4'
- name: SGLANG_HICACHE_TIMING
value: '1'
- name: SGLANG_BATCH_TELEMETRY
value: '1'
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
value: '1'
- name: SGLANG_ENABLE_METRICS_DEVICE_TIMER
value: '1'
- name: SGLANG_EPLB_HEATMAP_COLLECTION_INTERVAL
value: '100'
- name: SGLANG_MM_TRACE
value: '1'
- name: SGLANG_BATCH_SPAN_MIN_MS
value: '200'
- name: SGLANG_BATCH_SPAN_SAMPLE
value: '0.05'
- name: SGLANG_BATCH_SPAN_MAX_LINKS
value: '8'
- name: SGLANG_BATCH_STAGE_TIMING
value: '1'
- name: SGLANG_BATCH_PASS_SPAN
value: '1'
- name: SGLANG_BATCH_PASS_SPAN_SAMPLE
value: '1.0'
- name: SGLANG_RECV_SPAN
value: '1'
- name: SGLANG_RECV_SPAN_SAMPLE
value: '1.0'
- name: SGLANG_MM_TRACE_SYNC
value: '0'
- name: SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK
value: '1'
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
value: '600'
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
value: '16'
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
value: '32'
- name: TZ
value: Asia/Shanghai
- name: SGLANG_TRACE_LOOP_SPAN
value: '1'
- name: SGLANG_TRACE_LOOP_SAMPLE
value: '1.0'
- name: SGLANG_FWD_TIMELINE
value: '0'
- name: SGLANG_TRACE_LOOP_DETAIL
value: '1'
- name: SGLANG_TRACE_MOONCAKE_DETAIL
value: '0'
- name: SGLANG_TRACE_BACKUP_DETAIL
value: '1'
- name: SGLANG_ASYNC_BACKUP_SUBMIT
value: '1'
- name: SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH
value: '1'
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
value: '1024'
- name: SGLANG_MOONCAKE_CLIENT_METRICS_PORT_BASE
value: '9400'
- name: MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR
value: bucket_storage_backend
- name: MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE
value: '1099511627776'
- name: MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES
value: '1099511627776'
- name: MOONCAKE_OFFLOAD_BUCKET_EVICTION_POLICY
value: fifo
- name: TMPDIR
value: /data/ymk/l3ssd/p-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/p-xdg
- name: HF_HOME
value: /data/ymk/l3ssd/p-hf
- name: MOONCAKE_OFFLOAD_FLUSH_PARTIAL_BUCKETS
value: '0'
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: sglang
ports:
- containerPort: 30000
name: metrics
protocol: TCP
- containerPort: 8998
name: bootstrap
protocol: TCP
- name: mc-0
containerPort: 9400
protocol: TCP
- name: mc-1
containerPort: 9401
protocol: TCP
- name: mc-2
containerPort: 9402
protocol: TCP
- name: mc-3
containerPort: 9403
protocol: TCP
readinessProbe:
failureThreshold: 120
httpGet:
path: /health
port: metrics
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '96'
memory: 2Ti
nvidia.com/gpu: '4'
requests:
cpu: '32'
memory: 512Gi
nvidia.com/gpu: '4'
securityContext:
privileged: true
volumeMounts:
- mountPath: /data
name: data
- mountPath: /root/.cache/sglang
name: sglang-cache
- mountPath: /dev/shm
name: dshm
- mountPath: /root/.cargo/bin/cargo
name: fake-cargo
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
hostPID: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/cache/sglang
type: Directory
name: sglang-cache
- emptyDir:
medium: Memory
sizeLimit: 64Gi
name: dshm
- hostPath:
path: /data/ymk/ds41/fake-cargo
type: File
name: fake-cargo
- dependencies:
- mooncake-master
minReadySeconds: 0
name: d
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '30300'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-d
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --enable-fp32-lm-head --enable-dp-attention-local-control-broadcast \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 64 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r64_ep4.json \\\n --mem-fraction-static 0.80 \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --stream-interval 32 --swa-prefix-tails 2048 --enable-decoder-swa-bounded-replay \\\n --max-running-requests 512 --cuda-graph-max-bs-decode 512 --tokenizer-worker-num 16 \\\n --chunked-prefill-size 32768 --max-prefill-tokens 8192 \\\n --disaggregation-mode decode --disaggregation-transfer-backend mooncake \\\n --host 0.0.0.0 --port 30300 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-d \\\n --watchdog-timeout 3600\n"
env:
- name: CUDA_VISIBLE_DEVICES
value: 4,5,6,7
- name: SGLANG_RAGGED_VERIFY_MODE
value: static
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
value: '1'
- name: SGLANG_DSV4_KV_LAYOUT
value: v41_new
- name: SGLANG_DEFAULT_THINKING
value: '1'
- name: SGLANG_DSV41_REASONING_EFFORT
value: high
- name: MC_INTRANODE_NVLINK
value: 'true'
- name: MC_INTRA_NVLINK
value: 'true'
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
value: 'true'
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
value: '1'
- name: SGLANG_BATCH_TELEMETRY
value: '1'
- name: SGLANG_BATCH_SPAN_SAMPLE
value: '0.1'
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
value: '600'
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
value: '16'
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
value: '32'
- name: TZ
value: Asia/Shanghai
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
value: '1024'
- name: TMPDIR
value: /data/ymk/l3ssd/d-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/d-xdg
- name: HF_HOME
value: /data/ymk/l3ssd/d-hf
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: sglang
ports:
- containerPort: 30300
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 150
httpGet:
path: /health
port: metrics
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '96'
memory: 550Gi
nvidia.com/gpu: '4'
requests:
cpu: '32'
memory: 256Gi
nvidia.com/gpu: '4'
securityContext:
privileged: true
volumeMounts:
- mountPath: /data
name: data
- mountPath: /root/.cache/sglang
name: sglang-cache
- mountPath: /dev/shm
name: dshm
- mountPath: /root/.cargo/bin/cargo
name: fake-cargo
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
hostPID: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/cache/sglang
type: Directory
name: sglang-cache
- emptyDir:
medium: Memory
sizeLimit: 64Gi
name: dshm
- hostPath:
path: /data/ymk/ds41/fake-cargo
type: File
name: fake-cargo
- dependencies:
- p
- d
minReadySeconds: 0
name: gateway
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '29000'
prometheus.io/scheme: http
prometheus.io/scrape: 'true'
labels:
app: ds41-l3ssd-gw
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec /opt/sglang/bin/sgl-model-gateway launch \\\n --pd-disaggregation \\\n --prefill http://127.0.0.1:30000 8998 \\\n --decode http://127.0.0.1:30300 \\\n --prefill-policy cache_aware --decode-policy cache_aware \\\n --g9i-enabled --g9i-max-new-tokens 4096 \\\n \\\n \\\n --g9i-regular-p-queue-reject-secs 0 \\\n --g9i-decode-max-running 512 \\\n --health-check-timeout-secs 30 --health-check-interval-secs 30 \\\n --cb-failure-threshold 50 \\\n --model-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --tokenizer-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --host 0.0.0.0 --port 30400\n"
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: gateway
ports:
- containerPort: 30400
name: http
protocol: TCP
- containerPort: 29000
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 60
httpGet:
path: /health
port: http
initialDelaySeconds: 20
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '16'
memory: 32Gi
requests:
cpu: '2'
memory: 8Gi
volumeMounts:
- mountPath: /data
name: data
- mountPath: /opt/sglang/bin/sgl-model-gateway
name: gw-bin
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- name: gw-bin
hostPath:
path: /data/ymk/cache/gw-est-td/sgl-model-gateway
type: File
@@ -0,0 +1,488 @@
apiVersion: workloads.x-k8s.io/v1alpha2
kind: RoleBasedGroup
metadata:
annotations:
rbg.workloads.x-k8s.io/discovery-config-mode: refine
labels:
app: ds41-gist-exp
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
sglang-model: ds-v41-flash
name: ds41-gist-exp
namespace: prj-d7te67bj0sq2fgdrmbig
spec:
roles:
- minReadySeconds: 0
name: mooncake-master
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '19003'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-mooncake
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "# 18081: 宿主机 18080 被 AgentX fake upstream(./fakeup)长期占用;\n# 我们的 P/D 用 metadata_server=P2PHANDSHAKE,此 HTTP 端口只是 master 可选项。\nexec /opt/sglang/bin/mooncake_master \\\n --port 50051 --enable_http_metadata_server=true \\\n --http_metadata_server_port=18081 --metrics_port=19003 --eviction_high_watermark_ratio=0.90 --enable_offload=true --offload_on_evict=false --promotion_on_hit=false --offload_force_evict=false\n"
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: mooncake-master
ports:
- containerPort: 19003
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 60
initialDelaySeconds: 5
periodSeconds: 5
tcpSocket:
port: 50051
resources:
limits:
cpu: '4'
memory: 64Gi
requests:
cpu: '1'
memory: 8Gi
env:
- name: TMPDIR
value: /data/ymk/l3ssd/master-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/master-xdg
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
- dependencies:
- mooncake-master
minReadySeconds: 0
name: p
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '30000'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-p
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend huggingface --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 32 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r32_ep4.json \\\n --mem-fraction-static 0.80 \\\n --chunked-prefill-size 131072 --max-prefill-tokens 131072 \\\n --max-running-requests 512 --tokenizer-worker-num 16 \\\n --load-balance-method total_tokens \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --enable-decoder-swa-bounded-replay --swa-full-tokens-ratio 0.12 \\\n --cuda-graph-max-bs-decode 1 --disable-decode-cuda-graph \\\n --cuda-graph-backend-prefill breakable \\\n --cuda-graph-bs-prefill 1024 2048 4096 8192 16384 32768 \\\n --cuda-graph-prefill-max-context 262144 --cuda-graph-max-seq-len-prefill 262144 \\\n --enable-hierarchical-cache --hicache-ratio 4 \\\n --hicache-mem-layout page_first_direct --hicache-io-backend direct \\\n --hicache-write-policy write_through --hicache-storage-backend mooncake \\\n --hicache-storage-prefetch-policy wait_complete \\\n --disaggregation-mode prefill --disaggregation-transfer-backend mooncake \\\n --disaggregation-bootstrap-port 8998 \\\n --expert-balancedness-report-mode prometheus \\\n --expert-distribution-recorder-buffer-size 64 \\\n --host 0.0.0.0 --port 30000 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-p \\\n --watchdog-timeout 3600 --enable-dp-attention-local-control-broadcast\n"
env:
- name: CUDA_VISIBLE_DEVICES
value: 0,1,2,3
- name: SGLANG_LOG_MS
value: '1'
- name: SGLANG_OPT_DSV41_ENGRAM_PEER_SHARDS
value: '1'
- name: SGLANG_OPT_DSV41_MEGA_MHC_PREFILL
value: '1'
- name: SGLANG_OPT_DSV41_BF16_INDEX_LOGITS
value: '1'
- name: SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK
value: '65536'
- name: SGLANG_DSV4_KV_LAYOUT
value: v41_new
- name: SGLANG_DP_USE_GATHERV
value: '1'
- name: SGLANG_RAGGED_VERIFY_MODE
value: static
- name: SGLANG_ENABLE_DP_SPEC_PREFILL_COORDINATION
value: '0'
- name: SGLANG_NUMA_BIND_V2
value: '0'
- name: SGLANG_DEFAULT_THINKING
value: '1'
- name: SGLANG_DSV41_REASONING_EFFORT
value: high
- name: SGLANG_TIMEOUT_KEEP_ALIVE
value: '900'
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
value: '1'
- name: MC_INTRANODE_NVLINK
value: 'true'
- name: MC_INTRA_NVLINK
value: 'true'
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
value: 'true'
- name: SGLANG_HICACHE_MOONCAKE_CONFIG_PATH
value: /data/ymk/ds41/mooncake-store-l3ssd.json
- name: SGLANG_HICACHE_PREFETCH_IO_THREADS
value: '4'
- name: SGLANG_HICACHE_BACKUP_IO_THREADS
value: '4'
- name: SGLANG_HICACHE_TIMING
value: '1'
- name: SGLANG_BATCH_TELEMETRY
value: '1'
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
value: '1'
- name: SGLANG_ENABLE_METRICS_DEVICE_TIMER
value: '1'
- name: SGLANG_EPLB_HEATMAP_COLLECTION_INTERVAL
value: '100'
- name: SGLANG_MM_TRACE
value: '1'
- name: SGLANG_BATCH_SPAN_MIN_MS
value: '200'
- name: SGLANG_BATCH_SPAN_SAMPLE
value: '0.05'
- name: SGLANG_BATCH_SPAN_MAX_LINKS
value: '8'
- name: SGLANG_BATCH_STAGE_TIMING
value: '1'
- name: SGLANG_BATCH_PASS_SPAN
value: '1'
- name: SGLANG_BATCH_PASS_SPAN_SAMPLE
value: '1.0'
- name: SGLANG_RECV_SPAN
value: '1'
- name: SGLANG_RECV_SPAN_SAMPLE
value: '1.0'
- name: SGLANG_MM_TRACE_SYNC
value: '0'
- name: SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK
value: '1'
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
value: '600'
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
value: '16'
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
value: '32'
- name: TZ
value: Asia/Shanghai
- name: SGLANG_TRACE_LOOP_SPAN
value: '1'
- name: SGLANG_TRACE_LOOP_SAMPLE
value: '1.0'
- name: SGLANG_FWD_TIMELINE
value: '0'
- name: SGLANG_TRACE_LOOP_DETAIL
value: '1'
- name: SGLANG_TRACE_MOONCAKE_DETAIL
value: '0'
- name: SGLANG_TRACE_BACKUP_DETAIL
value: '1'
- name: SGLANG_ASYNC_BACKUP_SUBMIT
value: '1'
- name: SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH
value: '1'
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
value: '1024'
- name: SGLANG_MOONCAKE_CLIENT_METRICS_PORT_BASE
value: '9400'
- name: MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR
value: bucket_storage_backend
- name: MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE
value: '1099511627776'
- name: MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES
value: '1099511627776'
- name: MOONCAKE_OFFLOAD_BUCKET_EVICTION_POLICY
value: fifo
- name: TMPDIR
value: /data/ymk/l3ssd/p-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/p-xdg
- name: HF_HOME
value: /data/ymk/l3ssd/p-hf
- name: MOONCAKE_OFFLOAD_FLUSH_PARTIAL_BUCKETS
value: '0'
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: sglang
ports:
- containerPort: 30000
name: metrics
protocol: TCP
- containerPort: 8998
name: bootstrap
protocol: TCP
- name: mc-0
containerPort: 9400
protocol: TCP
- name: mc-1
containerPort: 9401
protocol: TCP
- name: mc-2
containerPort: 9402
protocol: TCP
- name: mc-3
containerPort: 9403
protocol: TCP
readinessProbe:
failureThreshold: 120
httpGet:
path: /health
port: metrics
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '96'
memory: 2Ti
nvidia.com/gpu: '4'
requests:
cpu: '32'
memory: 512Gi
nvidia.com/gpu: '4'
securityContext:
privileged: true
volumeMounts:
- mountPath: /data
name: data
- mountPath: /root/.cache/sglang
name: sglang-cache
- mountPath: /dev/shm
name: dshm
- mountPath: /root/.cargo/bin/cargo
name: fake-cargo
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
hostPID: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/cache/sglang
type: Directory
name: sglang-cache
- emptyDir:
medium: Memory
sizeLimit: 64Gi
name: dshm
- hostPath:
path: /data/ymk/ds41/fake-cargo
type: File
name: fake-cargo
- dependencies:
- mooncake-master
minReadySeconds: 0
name: d
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '30300'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-d
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend huggingface --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --enable-fp32-lm-head --enable-dp-attention-local-control-broadcast \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 64 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r64_ep4.json \\\n --mem-fraction-static 0.80 \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --stream-interval 32 --swa-prefix-tails 2048 --enable-decoder-swa-bounded-replay \\\n --max-running-requests 512 --cuda-graph-max-bs-decode 512 --tokenizer-worker-num 16 \\\n --chunked-prefill-size 32768 --max-prefill-tokens 8192 \\\n --disaggregation-mode decode --disaggregation-transfer-backend mooncake \\\n --host 0.0.0.0 --port 30300 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-d \\\n --watchdog-timeout 3600\n"
env:
- name: CUDA_VISIBLE_DEVICES
value: 4,5,6,7
- name: SGLANG_RAGGED_VERIFY_MODE
value: static
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
value: '1'
- name: SGLANG_DSV4_KV_LAYOUT
value: v41_new
- name: SGLANG_DEFAULT_THINKING
value: '1'
- name: SGLANG_DSV41_REASONING_EFFORT
value: high
- name: MC_INTRANODE_NVLINK
value: 'true'
- name: MC_INTRA_NVLINK
value: 'true'
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
value: 'true'
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
value: '1'
- name: SGLANG_BATCH_TELEMETRY
value: '1'
- name: SGLANG_BATCH_SPAN_SAMPLE
value: '0.1'
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
value: '600'
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
value: '16'
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
value: '32'
- name: TZ
value: Asia/Shanghai
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
value: '1024'
- name: TMPDIR
value: /data/ymk/l3ssd/d-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/d-xdg
- name: HF_HOME
value: /data/ymk/l3ssd/d-hf
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: sglang
ports:
- containerPort: 30300
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 150
httpGet:
path: /health
port: metrics
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '96'
memory: 550Gi
nvidia.com/gpu: '4'
requests:
cpu: '32'
memory: 256Gi
nvidia.com/gpu: '4'
securityContext:
privileged: true
volumeMounts:
- mountPath: /data
name: data
- mountPath: /root/.cache/sglang
name: sglang-cache
- mountPath: /dev/shm
name: dshm
- mountPath: /root/.cargo/bin/cargo
name: fake-cargo
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
hostPID: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/cache/sglang
type: Directory
name: sglang-cache
- emptyDir:
medium: Memory
sizeLimit: 64Gi
name: dshm
- hostPath:
path: /data/ymk/ds41/fake-cargo
type: File
name: fake-cargo
- dependencies:
- p
- d
minReadySeconds: 0
name: gateway
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '29000'
prometheus.io/scheme: http
prometheus.io/scrape: 'true'
labels:
app: ds41-l3ssd-gw
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec /opt/sglang/bin/sgl-model-gateway launch \\\n --pd-disaggregation \\\n --prefill http://127.0.0.1:30000 8998 \\\n --decode http://127.0.0.1:30300 \\\n --prefill-policy cache_aware --decode-policy cache_aware \\\n --g9i-enabled --g9i-max-new-tokens 4096 \\\n \\\n \\\n --g9i-regular-p-queue-reject-secs 0 \\\n --g9i-decode-max-running 512 \\\n --health-check-timeout-secs 30 --health-check-interval-secs 30 \\\n --cb-failure-threshold 50 \\\n --model-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --tokenizer-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --host 0.0.0.0 --port 30400\n"
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: gateway
ports:
- containerPort: 30400
name: http
protocol: TCP
- containerPort: 29000
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 60
httpGet:
path: /health
port: http
initialDelaySeconds: 20
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '16'
memory: 32Gi
requests:
cpu: '2'
memory: 8Gi
volumeMounts:
- mountPath: /data
name: data
- mountPath: /opt/sglang/bin/sgl-model-gateway
name: gw-bin
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- name: gw-bin
hostPath:
path: /data/ymk/cache/gw-est-td/sgl-model-gateway
type: File
@@ -0,0 +1,488 @@
apiVersion: workloads.x-k8s.io/v1alpha2
kind: RoleBasedGroup
metadata:
annotations:
rbg.workloads.x-k8s.io/discovery-config-mode: refine
labels:
app: ds41-gist-exp
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
sglang-model: ds-v41-flash
name: ds41-gist-exp
namespace: prj-d7te67bj0sq2fgdrmbig
spec:
roles:
- minReadySeconds: 0
name: mooncake-master
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '19003'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-mooncake
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "# 18081: 宿主机 18080 被 AgentX fake upstream(./fakeup)长期占用;\n# 我们的 P/D 用 metadata_server=P2PHANDSHAKE,此 HTTP 端口只是 master 可选项。\nexec /opt/sglang/bin/mooncake_master \\\n --port 50051 --enable_http_metadata_server=true \\\n --http_metadata_server_port=18081 --metrics_port=19003 --eviction_high_watermark_ratio=0.90 --enable_offload=true --offload_on_evict=true --promotion_on_hit=false --offload_force_evict=false\n"
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: mooncake-master
ports:
- containerPort: 19003
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 60
initialDelaySeconds: 5
periodSeconds: 5
tcpSocket:
port: 50051
resources:
limits:
cpu: '4'
memory: 64Gi
requests:
cpu: '1'
memory: 8Gi
env:
- name: TMPDIR
value: /data/ymk/l3ssd/master-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/master-xdg
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
- dependencies:
- mooncake-master
minReadySeconds: 0
name: p
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '30000'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-p
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec sglang serve \\\n --trust-remote-code --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 32 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r32_ep4.json \\\n --mem-fraction-static 0.80 \\\n --chunked-prefill-size 131072 --max-prefill-tokens 131072 \\\n --max-running-requests 512 --tokenizer-worker-num 16 \\\n --load-balance-method total_tokens \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --enable-decoder-swa-bounded-replay --swa-full-tokens-ratio 0.12 \\\n --cuda-graph-max-bs-decode 1 --disable-decode-cuda-graph \\\n --cuda-graph-backend-prefill breakable \\\n --cuda-graph-bs-prefill 1024 2048 4096 8192 16384 32768 \\\n --cuda-graph-prefill-max-context 262144 --cuda-graph-max-seq-len-prefill 262144 \\\n --enable-hierarchical-cache --hicache-ratio 4 \\\n --hicache-mem-layout page_first_direct --hicache-io-backend direct \\\n --hicache-write-policy write_through --hicache-storage-backend mooncake \\\n --hicache-storage-prefetch-policy wait_complete \\\n --disaggregation-mode prefill --disaggregation-transfer-backend mooncake \\\n --disaggregation-bootstrap-port 8998 \\\n --expert-balancedness-report-mode prometheus \\\n --expert-distribution-recorder-buffer-size 64 \\\n --host 0.0.0.0 --port 30000 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-p \\\n --watchdog-timeout 3600 --enable-dp-attention-local-control-broadcast\n"
env:
- name: CUDA_VISIBLE_DEVICES
value: 0,1,2,3
- name: SGLANG_LOG_MS
value: '1'
- name: SGLANG_OPT_DSV41_ENGRAM_PEER_SHARDS
value: '1'
- name: SGLANG_OPT_DSV41_MEGA_MHC_PREFILL
value: '1'
- name: SGLANG_OPT_DSV41_BF16_INDEX_LOGITS
value: '1'
- name: SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK
value: '65536'
- name: SGLANG_DSV4_KV_LAYOUT
value: v41_new
- name: SGLANG_DP_USE_GATHERV
value: '1'
- name: SGLANG_RAGGED_VERIFY_MODE
value: static
- name: SGLANG_ENABLE_DP_SPEC_PREFILL_COORDINATION
value: '0'
- name: SGLANG_NUMA_BIND_V2
value: '0'
- name: SGLANG_DEFAULT_THINKING
value: '1'
- name: SGLANG_DSV41_REASONING_EFFORT
value: high
- name: SGLANG_TIMEOUT_KEEP_ALIVE
value: '900'
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
value: '1'
- name: MC_INTRANODE_NVLINK
value: 'true'
- name: MC_INTRA_NVLINK
value: 'true'
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
value: 'true'
- name: SGLANG_HICACHE_MOONCAKE_CONFIG_PATH
value: /data/ymk/ds41/mooncake-store-l3ssd.json
- name: SGLANG_HICACHE_PREFETCH_IO_THREADS
value: '4'
- name: SGLANG_HICACHE_BACKUP_IO_THREADS
value: '4'
- name: SGLANG_HICACHE_TIMING
value: '1'
- name: SGLANG_BATCH_TELEMETRY
value: '1'
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
value: '1'
- name: SGLANG_ENABLE_METRICS_DEVICE_TIMER
value: '1'
- name: SGLANG_EPLB_HEATMAP_COLLECTION_INTERVAL
value: '100'
- name: SGLANG_MM_TRACE
value: '1'
- name: SGLANG_BATCH_SPAN_MIN_MS
value: '200'
- name: SGLANG_BATCH_SPAN_SAMPLE
value: '0.05'
- name: SGLANG_BATCH_SPAN_MAX_LINKS
value: '8'
- name: SGLANG_BATCH_STAGE_TIMING
value: '1'
- name: SGLANG_BATCH_PASS_SPAN
value: '1'
- name: SGLANG_BATCH_PASS_SPAN_SAMPLE
value: '1.0'
- name: SGLANG_RECV_SPAN
value: '1'
- name: SGLANG_RECV_SPAN_SAMPLE
value: '1.0'
- name: SGLANG_MM_TRACE_SYNC
value: '0'
- name: SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK
value: '1'
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
value: '600'
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
value: '16'
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
value: '32'
- name: TZ
value: Asia/Shanghai
- name: SGLANG_TRACE_LOOP_SPAN
value: '1'
- name: SGLANG_TRACE_LOOP_SAMPLE
value: '1.0'
- name: SGLANG_FWD_TIMELINE
value: '0'
- name: SGLANG_TRACE_LOOP_DETAIL
value: '1'
- name: SGLANG_TRACE_MOONCAKE_DETAIL
value: '0'
- name: SGLANG_TRACE_BACKUP_DETAIL
value: '1'
- name: SGLANG_ASYNC_BACKUP_SUBMIT
value: '1'
- name: SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH
value: '1'
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
value: '1024'
- name: SGLANG_MOONCAKE_CLIENT_METRICS_PORT_BASE
value: '9400'
- name: MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR
value: bucket_storage_backend
- name: MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE
value: '549755813888'
- name: MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES
value: '549755813888'
- name: MOONCAKE_OFFLOAD_BUCKET_EVICTION_POLICY
value: fifo
- name: TMPDIR
value: /data/ymk/l3ssd/p-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/p-xdg
- name: HF_HOME
value: /data/ymk/l3ssd/p-hf
- name: MOONCAKE_OFFLOAD_FLUSH_PARTIAL_BUCKETS
value: '0'
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: sglang
ports:
- containerPort: 30000
name: metrics
protocol: TCP
- containerPort: 8998
name: bootstrap
protocol: TCP
- name: mc-0
containerPort: 9400
protocol: TCP
- name: mc-1
containerPort: 9401
protocol: TCP
- name: mc-2
containerPort: 9402
protocol: TCP
- name: mc-3
containerPort: 9403
protocol: TCP
readinessProbe:
failureThreshold: 120
httpGet:
path: /health
port: metrics
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '96'
memory: 2Ti
nvidia.com/gpu: '4'
requests:
cpu: '32'
memory: 512Gi
nvidia.com/gpu: '4'
securityContext:
privileged: true
volumeMounts:
- mountPath: /data
name: data
- mountPath: /root/.cache/sglang
name: sglang-cache
- mountPath: /dev/shm
name: dshm
- mountPath: /root/.cargo/bin/cargo
name: fake-cargo
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
hostPID: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/cache/sglang
type: Directory
name: sglang-cache
- emptyDir:
medium: Memory
sizeLimit: 64Gi
name: dshm
- hostPath:
path: /data/ymk/ds41/fake-cargo
type: File
name: fake-cargo
- dependencies:
- mooncake-master
minReadySeconds: 0
name: d
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '30300'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-d
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec sglang serve \\\n --trust-remote-code --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --enable-fp32-lm-head --enable-dp-attention-local-control-broadcast \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 64 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r64_ep4.json \\\n --mem-fraction-static 0.80 \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --stream-interval 32 --swa-prefix-tails 2048 --enable-decoder-swa-bounded-replay \\\n --max-running-requests 512 --cuda-graph-max-bs-decode 512 \\\n --chunked-prefill-size 32768 --max-prefill-tokens 8192 \\\n --disaggregation-mode decode --disaggregation-transfer-backend mooncake \\\n --host 0.0.0.0 --port 30300 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-d \\\n --watchdog-timeout 3600\n"
env:
- name: CUDA_VISIBLE_DEVICES
value: 4,5,6,7
- name: SGLANG_RAGGED_VERIFY_MODE
value: static
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
value: '1'
- name: SGLANG_DSV4_KV_LAYOUT
value: v41_new
- name: SGLANG_DEFAULT_THINKING
value: '1'
- name: SGLANG_DSV41_REASONING_EFFORT
value: high
- name: MC_INTRANODE_NVLINK
value: 'true'
- name: MC_INTRA_NVLINK
value: 'true'
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
value: 'true'
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
value: '1'
- name: SGLANG_BATCH_TELEMETRY
value: '1'
- name: SGLANG_BATCH_SPAN_SAMPLE
value: '0.1'
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
value: '600'
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
value: '16'
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
value: '32'
- name: TZ
value: Asia/Shanghai
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
value: '1024'
- name: TMPDIR
value: /data/ymk/l3ssd/d-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/d-xdg
- name: HF_HOME
value: /data/ymk/l3ssd/d-hf
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: sglang
ports:
- containerPort: 30300
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 150
httpGet:
path: /health
port: metrics
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '96'
memory: 550Gi
nvidia.com/gpu: '4'
requests:
cpu: '32'
memory: 256Gi
nvidia.com/gpu: '4'
securityContext:
privileged: true
volumeMounts:
- mountPath: /data
name: data
- mountPath: /root/.cache/sglang
name: sglang-cache
- mountPath: /dev/shm
name: dshm
- mountPath: /root/.cargo/bin/cargo
name: fake-cargo
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
hostPID: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/cache/sglang
type: Directory
name: sglang-cache
- emptyDir:
medium: Memory
sizeLimit: 64Gi
name: dshm
- hostPath:
path: /data/ymk/ds41/fake-cargo
type: File
name: fake-cargo
- dependencies:
- p
- d
minReadySeconds: 0
name: gateway
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '29000'
prometheus.io/scheme: http
prometheus.io/scrape: 'true'
labels:
app: ds41-l3ssd-gw
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec /opt/sglang/bin/sgl-model-gateway launch \\\n --pd-disaggregation \\\n --prefill http://127.0.0.1:30000 8998 \\\n --decode http://127.0.0.1:30300 \\\n --prefill-policy cache_aware --decode-policy cache_aware \\\n --g9i-enabled --g9i-max-new-tokens 4096 \\\n \\\n \\\n --g9i-regular-p-queue-reject-secs 0 \\\n --g9i-decode-max-running 512 \\\n --health-check-timeout-secs 30 --health-check-interval-secs 30 \\\n --cb-failure-threshold 50 \\\n --model-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --tokenizer-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --host 0.0.0.0 --port 30400\n"
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: gateway
ports:
- containerPort: 30400
name: http
protocol: TCP
- containerPort: 29000
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 60
httpGet:
path: /health
port: http
initialDelaySeconds: 20
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '16'
memory: 32Gi
requests:
cpu: '2'
memory: 8Gi
volumeMounts:
- mountPath: /data
name: data
- mountPath: /opt/sglang/bin/sgl-model-gateway
name: gw-bin
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- name: gw-bin
hostPath:
path: /data/ymk/cache/gw-est-td/sgl-model-gateway
type: File
@@ -0,0 +1,488 @@
apiVersion: workloads.x-k8s.io/v1alpha2
kind: RoleBasedGroup
metadata:
annotations:
rbg.workloads.x-k8s.io/discovery-config-mode: refine
labels:
app: ds41-gist-exp
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
sglang-model: ds-v41-flash
name: ds41-gist-exp
namespace: prj-d7te67bj0sq2fgdrmbig
spec:
roles:
- minReadySeconds: 0
name: mooncake-master
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '19003'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-mooncake
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "# 18081: 宿主机 18080 被 AgentX fake upstream(./fakeup)长期占用;\n# 我们的 P/D 用 metadata_server=P2PHANDSHAKE,此 HTTP 端口只是 master 可选项。\nexec /opt/sglang/bin/mooncake_master \\\n --port 50051 --enable_http_metadata_server=true \\\n --http_metadata_server_port=18081 --metrics_port=19003 --eviction_high_watermark_ratio=0.90 --enable_offload=true --offload_on_evict=false --promotion_on_hit=false --offload_force_evict=false\n"
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: mooncake-master
ports:
- containerPort: 19003
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 60
initialDelaySeconds: 5
periodSeconds: 5
tcpSocket:
port: 50051
resources:
limits:
cpu: '4'
memory: 64Gi
requests:
cpu: '1'
memory: 8Gi
env:
- name: TMPDIR
value: /data/ymk/l3ssd/master-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/master-xdg
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
- dependencies:
- mooncake-master
minReadySeconds: 0
name: p
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '30000'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-p
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 32 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r32_ep4.json \\\n --mem-fraction-static 0.80 \\\n --chunked-prefill-size 131072 --max-prefill-tokens 131072 \\\n --max-running-requests 512 --tokenizer-worker-num 16 \\\n --load-balance-method total_tokens \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --enable-decoder-swa-bounded-replay --swa-full-tokens-ratio 0.12 \\\n --cuda-graph-max-bs-decode 1 --disable-decode-cuda-graph \\\n --cuda-graph-backend-prefill breakable \\\n --cuda-graph-bs-prefill 1024 2048 4096 8192 16384 32768 \\\n --cuda-graph-prefill-max-context 262144 --cuda-graph-max-seq-len-prefill 262144 \\\n --enable-hierarchical-cache --hicache-ratio 4 \\\n --hicache-mem-layout page_first_direct --hicache-io-backend direct \\\n --hicache-write-policy write_through --hicache-storage-backend mooncake \\\n --hicache-storage-prefetch-policy wait_complete \\\n --disaggregation-mode prefill --disaggregation-transfer-backend mooncake \\\n --disaggregation-bootstrap-port 8998 \\\n --expert-balancedness-report-mode prometheus \\\n --expert-distribution-recorder-buffer-size 64 \\\n --host 0.0.0.0 --port 30000 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-p \\\n --watchdog-timeout 3600 --enable-dp-attention-local-control-broadcast\n"
env:
- name: CUDA_VISIBLE_DEVICES
value: 0,1,2,3
- name: SGLANG_LOG_MS
value: '1'
- name: SGLANG_OPT_DSV41_ENGRAM_PEER_SHARDS
value: '1'
- name: SGLANG_OPT_DSV41_MEGA_MHC_PREFILL
value: '1'
- name: SGLANG_OPT_DSV41_BF16_INDEX_LOGITS
value: '1'
- name: SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK
value: '65536'
- name: SGLANG_DSV4_KV_LAYOUT
value: v41_new
- name: SGLANG_DP_USE_GATHERV
value: '1'
- name: SGLANG_RAGGED_VERIFY_MODE
value: static
- name: SGLANG_ENABLE_DP_SPEC_PREFILL_COORDINATION
value: '0'
- name: SGLANG_NUMA_BIND_V2
value: '0'
- name: SGLANG_DEFAULT_THINKING
value: '1'
- name: SGLANG_DSV41_REASONING_EFFORT
value: high
- name: SGLANG_TIMEOUT_KEEP_ALIVE
value: '900'
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
value: '1'
- name: MC_INTRANODE_NVLINK
value: 'true'
- name: MC_INTRA_NVLINK
value: 'true'
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
value: 'true'
- name: SGLANG_HICACHE_MOONCAKE_CONFIG_PATH
value: /data/ymk/ds41/mooncake-store-l3ssd.json
- name: SGLANG_HICACHE_PREFETCH_IO_THREADS
value: '4'
- name: SGLANG_HICACHE_BACKUP_IO_THREADS
value: '4'
- name: SGLANG_HICACHE_TIMING
value: '1'
- name: SGLANG_BATCH_TELEMETRY
value: '1'
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
value: '1'
- name: SGLANG_ENABLE_METRICS_DEVICE_TIMER
value: '1'
- name: SGLANG_EPLB_HEATMAP_COLLECTION_INTERVAL
value: '100'
- name: SGLANG_MM_TRACE
value: '1'
- name: SGLANG_BATCH_SPAN_MIN_MS
value: '200'
- name: SGLANG_BATCH_SPAN_SAMPLE
value: '0.05'
- name: SGLANG_BATCH_SPAN_MAX_LINKS
value: '8'
- name: SGLANG_BATCH_STAGE_TIMING
value: '1'
- name: SGLANG_BATCH_PASS_SPAN
value: '1'
- name: SGLANG_BATCH_PASS_SPAN_SAMPLE
value: '1.0'
- name: SGLANG_RECV_SPAN
value: '1'
- name: SGLANG_RECV_SPAN_SAMPLE
value: '1.0'
- name: SGLANG_MM_TRACE_SYNC
value: '0'
- name: SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK
value: '1'
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
value: '600'
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
value: '16'
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
value: '32'
- name: TZ
value: Asia/Shanghai
- name: SGLANG_TRACE_LOOP_SPAN
value: '1'
- name: SGLANG_TRACE_LOOP_SAMPLE
value: '1.0'
- name: SGLANG_FWD_TIMELINE
value: '0'
- name: SGLANG_TRACE_LOOP_DETAIL
value: '1'
- name: SGLANG_TRACE_MOONCAKE_DETAIL
value: '0'
- name: SGLANG_TRACE_BACKUP_DETAIL
value: '1'
- name: SGLANG_ASYNC_BACKUP_SUBMIT
value: '1'
- name: SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH
value: '1'
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
value: '1024'
- name: SGLANG_MOONCAKE_CLIENT_METRICS_PORT_BASE
value: '9400'
- name: MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR
value: bucket_storage_backend
- name: MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE
value: '1099511627776'
- name: MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES
value: '1099511627776'
- name: MOONCAKE_OFFLOAD_BUCKET_EVICTION_POLICY
value: fifo
- name: TMPDIR
value: /data/ymk/l3ssd/p-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/p-xdg
- name: HF_HOME
value: /data/ymk/l3ssd/p-hf
- name: MOONCAKE_OFFLOAD_FLUSH_PARTIAL_BUCKETS
value: '0'
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: sglang
ports:
- containerPort: 30000
name: metrics
protocol: TCP
- containerPort: 8998
name: bootstrap
protocol: TCP
- name: mc-0
containerPort: 9400
protocol: TCP
- name: mc-1
containerPort: 9401
protocol: TCP
- name: mc-2
containerPort: 9402
protocol: TCP
- name: mc-3
containerPort: 9403
protocol: TCP
readinessProbe:
failureThreshold: 120
httpGet:
path: /health
port: metrics
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '96'
memory: 2Ti
nvidia.com/gpu: '4'
requests:
cpu: '32'
memory: 512Gi
nvidia.com/gpu: '4'
securityContext:
privileged: true
volumeMounts:
- mountPath: /data
name: data
- mountPath: /root/.cache/sglang
name: sglang-cache
- mountPath: /dev/shm
name: dshm
- mountPath: /root/.cargo/bin/cargo
name: fake-cargo
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
hostPID: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/cache/sglang
type: Directory
name: sglang-cache
- emptyDir:
medium: Memory
sizeLimit: 64Gi
name: dshm
- hostPath:
path: /data/ymk/ds41/fake-cargo
type: File
name: fake-cargo
- dependencies:
- mooncake-master
minReadySeconds: 0
name: d
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '30300'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-d
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --enable-fp32-lm-head --enable-dp-attention-local-control-broadcast \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 64 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r64_ep4.json \\\n --mem-fraction-static 0.80 \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --stream-interval 32 --swa-prefix-tails 2048 --enable-decoder-swa-bounded-replay \\\n --max-running-requests 512 --cuda-graph-max-bs-decode 512 \\\n --chunked-prefill-size 32768 --max-prefill-tokens 8192 \\\n --disaggregation-mode decode --disaggregation-transfer-backend mooncake \\\n --host 0.0.0.0 --port 30300 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-d \\\n --watchdog-timeout 3600\n"
env:
- name: CUDA_VISIBLE_DEVICES
value: 4,5,6,7
- name: SGLANG_RAGGED_VERIFY_MODE
value: static
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
value: '1'
- name: SGLANG_DSV4_KV_LAYOUT
value: v41_new
- name: SGLANG_DEFAULT_THINKING
value: '1'
- name: SGLANG_DSV41_REASONING_EFFORT
value: high
- name: MC_INTRANODE_NVLINK
value: 'true'
- name: MC_INTRA_NVLINK
value: 'true'
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
value: 'true'
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
value: '1'
- name: SGLANG_BATCH_TELEMETRY
value: '1'
- name: SGLANG_BATCH_SPAN_SAMPLE
value: '0.1'
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
value: '600'
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
value: '16'
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
value: '32'
- name: TZ
value: Asia/Shanghai
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
value: '1024'
- name: TMPDIR
value: /data/ymk/l3ssd/d-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/d-xdg
- name: HF_HOME
value: /data/ymk/l3ssd/d-hf
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: sglang
ports:
- containerPort: 30300
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 150
httpGet:
path: /health
port: metrics
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '96'
memory: 550Gi
nvidia.com/gpu: '4'
requests:
cpu: '32'
memory: 256Gi
nvidia.com/gpu: '4'
securityContext:
privileged: true
volumeMounts:
- mountPath: /data
name: data
- mountPath: /root/.cache/sglang
name: sglang-cache
- mountPath: /dev/shm
name: dshm
- mountPath: /root/.cargo/bin/cargo
name: fake-cargo
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
hostPID: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/cache/sglang
type: Directory
name: sglang-cache
- emptyDir:
medium: Memory
sizeLimit: 64Gi
name: dshm
- hostPath:
path: /data/ymk/ds41/fake-cargo
type: File
name: fake-cargo
- dependencies:
- p
- d
minReadySeconds: 0
name: gateway
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '29000'
prometheus.io/scheme: http
prometheus.io/scrape: 'true'
labels:
app: ds41-l3ssd-gw
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec /opt/sglang/bin/sgl-model-gateway launch \\\n --pd-disaggregation \\\n --prefill http://127.0.0.1:30000 8998 \\\n --decode http://127.0.0.1:30300 \\\n --prefill-policy cache_aware --decode-policy cache_aware \\\n --g9i-enabled --g9i-max-new-tokens 4096 \\\n \\\n \\\n --g9i-regular-p-queue-reject-secs 0 \\\n --g9i-decode-max-running 512 \\\n --health-check-timeout-secs 30 --health-check-interval-secs 30 \\\n --cb-failure-threshold 50 \\\n --model-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --tokenizer-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --host 0.0.0.0 --port 30400\n"
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: gateway
ports:
- containerPort: 30400
name: http
protocol: TCP
- containerPort: 29000
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 60
httpGet:
path: /health
port: http
initialDelaySeconds: 20
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '16'
memory: 32Gi
requests:
cpu: '2'
memory: 8Gi
volumeMounts:
- mountPath: /data
name: data
- mountPath: /opt/sglang/bin/sgl-model-gateway
name: gw-bin
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- name: gw-bin
hostPath:
path: /data/ymk/cache/gw-est-td/sgl-model-gateway
type: File
@@ -0,0 +1,488 @@
apiVersion: workloads.x-k8s.io/v1alpha2
kind: RoleBasedGroup
metadata:
annotations:
rbg.workloads.x-k8s.io/discovery-config-mode: refine
labels:
app: ds41-gist-exp
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
sglang-model: ds-v41-flash
name: ds41-l3ssd-test
namespace: prj-d7te67bj0sq2fgdrmbig
spec:
roles:
- minReadySeconds: 0
name: mooncake-master
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '19003'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-mooncake
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "# 18081: 宿主机 18080 被 AgentX fake upstream(./fakeup)长期占用;\n# 我们的 P/D 用 metadata_server=P2PHANDSHAKE,此 HTTP 端口只是 master 可选项。\nexec /opt/sglang/bin/mooncake_master \\\n --port 50051 --enable_http_metadata_server=true \\\n --http_metadata_server_port=18081 --metrics_port=19003 --eviction_high_watermark_ratio=0.90 --enable_offload=true --offload_on_evict=true --promotion_on_hit=false --offload_force_evict=false\n"
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: mooncake-master
ports:
- containerPort: 19003
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 60
initialDelaySeconds: 5
periodSeconds: 5
tcpSocket:
port: 50051
resources:
limits:
cpu: '4'
memory: 64Gi
requests:
cpu: '1'
memory: 8Gi
env:
- name: TMPDIR
value: /data/ymk/l3ssd/master-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/master-xdg
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
- dependencies:
- mooncake-master
minReadySeconds: 0
name: p
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '30000'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-p
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec sglang serve \\\n --trust-remote-code --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 32 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r32_ep4.json \\\n --mem-fraction-static 0.80 \\\n --chunked-prefill-size 131072 --max-prefill-tokens 131072 \\\n --max-running-requests 512 --tokenizer-worker-num 16 \\\n --load-balance-method total_tokens \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --enable-decoder-swa-bounded-replay --swa-full-tokens-ratio 0.12 \\\n --cuda-graph-max-bs-decode 1 --disable-decode-cuda-graph \\\n --cuda-graph-backend-prefill breakable \\\n --cuda-graph-bs-prefill 1024 2048 4096 8192 16384 32768 \\\n --cuda-graph-prefill-max-context 262144 --cuda-graph-max-seq-len-prefill 262144 \\\n --enable-hierarchical-cache --hicache-ratio 4 \\\n --hicache-mem-layout page_first_direct --hicache-io-backend direct \\\n --hicache-write-policy write_through --hicache-storage-backend mooncake \\\n --hicache-storage-prefetch-policy wait_complete \\\n --disaggregation-mode prefill --disaggregation-transfer-backend mooncake \\\n --disaggregation-bootstrap-port 8998 \\\n --expert-balancedness-report-mode prometheus \\\n --expert-distribution-recorder-buffer-size 64 \\\n --host 0.0.0.0 --port 30000 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-p \\\n --watchdog-timeout 3600 --enable-dp-attention-local-control-broadcast\n"
env:
- name: CUDA_VISIBLE_DEVICES
value: 0,1,2,3
- name: SGLANG_LOG_MS
value: '1'
- name: SGLANG_OPT_DSV41_ENGRAM_PEER_SHARDS
value: '1'
- name: SGLANG_OPT_DSV41_MEGA_MHC_PREFILL
value: '1'
- name: SGLANG_OPT_DSV41_BF16_INDEX_LOGITS
value: '1'
- name: SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK
value: '65536'
- name: SGLANG_DSV4_KV_LAYOUT
value: v41_new
- name: SGLANG_DP_USE_GATHERV
value: '1'
- name: SGLANG_RAGGED_VERIFY_MODE
value: static
- name: SGLANG_ENABLE_DP_SPEC_PREFILL_COORDINATION
value: '0'
- name: SGLANG_NUMA_BIND_V2
value: '0'
- name: SGLANG_DEFAULT_THINKING
value: '1'
- name: SGLANG_DSV41_REASONING_EFFORT
value: high
- name: SGLANG_TIMEOUT_KEEP_ALIVE
value: '900'
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
value: '1'
- name: MC_INTRANODE_NVLINK
value: 'true'
- name: MC_INTRA_NVLINK
value: 'true'
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
value: 'true'
- name: SGLANG_HICACHE_MOONCAKE_CONFIG_PATH
value: /data/ymk/ds41/mooncake-store-l3ssd.json
- name: SGLANG_HICACHE_PREFETCH_IO_THREADS
value: '4'
- name: SGLANG_HICACHE_BACKUP_IO_THREADS
value: '4'
- name: SGLANG_HICACHE_TIMING
value: '1'
- name: SGLANG_BATCH_TELEMETRY
value: '1'
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
value: '1'
- name: SGLANG_ENABLE_METRICS_DEVICE_TIMER
value: '1'
- name: SGLANG_EPLB_HEATMAP_COLLECTION_INTERVAL
value: '100'
- name: SGLANG_MM_TRACE
value: '1'
- name: SGLANG_BATCH_SPAN_MIN_MS
value: '200'
- name: SGLANG_BATCH_SPAN_SAMPLE
value: '0.05'
- name: SGLANG_BATCH_SPAN_MAX_LINKS
value: '8'
- name: SGLANG_BATCH_STAGE_TIMING
value: '1'
- name: SGLANG_BATCH_PASS_SPAN
value: '1'
- name: SGLANG_BATCH_PASS_SPAN_SAMPLE
value: '1.0'
- name: SGLANG_RECV_SPAN
value: '1'
- name: SGLANG_RECV_SPAN_SAMPLE
value: '1.0'
- name: SGLANG_MM_TRACE_SYNC
value: '0'
- name: SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK
value: '1'
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
value: '600'
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
value: '16'
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
value: '32'
- name: TZ
value: Asia/Shanghai
- name: SGLANG_TRACE_LOOP_SPAN
value: '1'
- name: SGLANG_TRACE_LOOP_SAMPLE
value: '1.0'
- name: SGLANG_FWD_TIMELINE
value: '0'
- name: SGLANG_TRACE_LOOP_DETAIL
value: '1'
- name: SGLANG_TRACE_MOONCAKE_DETAIL
value: '0'
- name: SGLANG_TRACE_BACKUP_DETAIL
value: '1'
- name: SGLANG_ASYNC_BACKUP_SUBMIT
value: '1'
- name: SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH
value: '1'
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
value: '1024'
- name: SGLANG_MOONCAKE_CLIENT_METRICS_PORT_BASE
value: '9400'
- name: MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR
value: bucket_storage_backend
- name: MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE
value: '549755813888'
- name: MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES
value: '549755813888'
- name: MOONCAKE_OFFLOAD_BUCKET_EVICTION_POLICY
value: fifo
- name: TMPDIR
value: /data/ymk/l3ssd/p-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/p-xdg
- name: HF_HOME
value: /data/ymk/l3ssd/p-hf
- name: MOONCAKE_OFFLOAD_FLUSH_PARTIAL_BUCKETS
value: '0'
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: sglang
ports:
- containerPort: 30000
name: metrics
protocol: TCP
- containerPort: 8998
name: bootstrap
protocol: TCP
- name: mc-0
containerPort: 9400
protocol: TCP
- name: mc-1
containerPort: 9401
protocol: TCP
- name: mc-2
containerPort: 9402
protocol: TCP
- name: mc-3
containerPort: 9403
protocol: TCP
readinessProbe:
failureThreshold: 120
httpGet:
path: /health
port: metrics
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '96'
memory: 2Ti
nvidia.com/gpu: '4'
requests:
cpu: '32'
memory: 512Gi
nvidia.com/gpu: '4'
securityContext:
privileged: true
volumeMounts:
- mountPath: /data
name: data
- mountPath: /root/.cache/sglang
name: sglang-cache
- mountPath: /dev/shm
name: dshm
- mountPath: /root/.cargo/bin/cargo
name: fake-cargo
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
hostPID: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/cache/sglang
type: Directory
name: sglang-cache
- emptyDir:
medium: Memory
sizeLimit: 64Gi
name: dshm
- hostPath:
path: /data/ymk/ds41/fake-cargo
type: File
name: fake-cargo
- dependencies:
- mooncake-master
minReadySeconds: 0
name: d
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '30300'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-d
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec sglang serve \\\n --trust-remote-code --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --enable-fp32-lm-head --enable-dp-attention-local-control-broadcast \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 64 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r64_ep4.json \\\n --mem-fraction-static 0.80 \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --stream-interval 32 --swa-prefix-tails 2048 --enable-decoder-swa-bounded-replay \\\n --max-running-requests 512 --cuda-graph-max-bs-decode 512 \\\n --chunked-prefill-size 32768 --max-prefill-tokens 8192 \\\n --disaggregation-mode decode --disaggregation-transfer-backend mooncake \\\n --host 0.0.0.0 --port 30300 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-d \\\n --watchdog-timeout 3600\n"
env:
- name: CUDA_VISIBLE_DEVICES
value: 4,5,6,7
- name: SGLANG_RAGGED_VERIFY_MODE
value: static
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
value: '1'
- name: SGLANG_DSV4_KV_LAYOUT
value: v41_new
- name: SGLANG_DEFAULT_THINKING
value: '1'
- name: SGLANG_DSV41_REASONING_EFFORT
value: high
- name: MC_INTRANODE_NVLINK
value: 'true'
- name: MC_INTRA_NVLINK
value: 'true'
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
value: 'true'
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
value: '1'
- name: SGLANG_BATCH_TELEMETRY
value: '1'
- name: SGLANG_BATCH_SPAN_SAMPLE
value: '0.1'
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
value: '600'
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
value: '16'
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
value: '32'
- name: TZ
value: Asia/Shanghai
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
value: '1024'
- name: TMPDIR
value: /data/ymk/l3ssd/d-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/d-xdg
- name: HF_HOME
value: /data/ymk/l3ssd/d-hf
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: sglang
ports:
- containerPort: 30300
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 150
httpGet:
path: /health
port: metrics
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '96'
memory: 550Gi
nvidia.com/gpu: '4'
requests:
cpu: '32'
memory: 256Gi
nvidia.com/gpu: '4'
securityContext:
privileged: true
volumeMounts:
- mountPath: /data
name: data
- mountPath: /root/.cache/sglang
name: sglang-cache
- mountPath: /dev/shm
name: dshm
- mountPath: /root/.cargo/bin/cargo
name: fake-cargo
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
hostPID: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/cache/sglang
type: Directory
name: sglang-cache
- emptyDir:
medium: Memory
sizeLimit: 64Gi
name: dshm
- hostPath:
path: /data/ymk/ds41/fake-cargo
type: File
name: fake-cargo
- dependencies:
- p
- d
minReadySeconds: 0
name: gateway
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '29000'
prometheus.io/scheme: http
prometheus.io/scrape: 'true'
labels:
app: ds41-l3ssd-gw
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec /opt/sglang/bin/sgl-model-gateway launch \\\n --pd-disaggregation \\\n --prefill http://127.0.0.1:30000 8998 \\\n --decode http://127.0.0.1:30300 \\\n --prefill-policy cache_aware --decode-policy cache_aware \\\n --g9i-enabled --g9i-max-new-tokens 4096 \\\n \\\n \\\n --g9i-regular-p-queue-reject-secs 0 \\\n --g9i-decode-max-running 512 \\\n --health-check-timeout-secs 30 --health-check-interval-secs 30 \\\n --cb-failure-threshold 50 \\\n --model-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --tokenizer-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --host 0.0.0.0 --port 30400\n"
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: gateway
ports:
- containerPort: 30400
name: http
protocol: TCP
- containerPort: 29000
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 60
httpGet:
path: /health
port: http
initialDelaySeconds: 20
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '16'
memory: 32Gi
requests:
cpu: '2'
memory: 8Gi
volumeMounts:
- mountPath: /data
name: data
- mountPath: /opt/sglang/bin/sgl-model-gateway
name: gw-bin
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- name: gw-bin
hostPath:
path: /data/ymk/cache/gw-est-td/sgl-model-gateway
type: File
+41
View File
@@ -0,0 +1,41 @@
# Dynamo-enabled B300 serving image.
#
# The sglang runtime tree inside the base image is byte-identical to
# b300/dynamo @ 8b67636010 (the only commits between the baked tree
# 433257c7b0 and 8b67636010 are deploy/docs), so this layer only swaps the
# preinstalled Dynamo 1.6.0.dev snapshot for the released 1.6.2 runtime.
#
# Build:
# docker build -t ymkymx/sglang:dynamo-8b6763601-baked-20261117 .
# or use ./build-dynamo-image.sh (also imports into containerd).
ARG BASE=ymkymx/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
FROM ${BASE}
ARG DYNAMO_VERSION=1.6.2
ARG PIP_INDEX=https://pypi.tuna.tsinghua.edu.cn/simple
LABEL ai.sglang.build.commit="8b67636010e54fc96e5aeba9b0139b38aa18b000"
LABEL ai.dynamo.version="${DYNAMO_VERSION}"
LABEL org.opencontainers.image.description="SGLang B300 (b300/dynamo) + NVIDIA Dynamo ${DYNAMO_VERSION}"
# --no-deps: the base image already carries the full runtime dependency set
# (torch 2.14.1, ai-dynamo deps, nixl 1.5.0). Resolving would try to pull
# sglang==0.5.19 from the `sglang` extra and downgrade torch.
RUN pip uninstall -y ai-dynamo ai-dynamo-runtime >/dev/null 2>&1 || true \
&& pip install --no-cache-dir --no-deps -i "${PIP_INDEX}" \
"ai-dynamo==${DYNAMO_VERSION}" "ai-dynamo-runtime==${DYNAMO_VERSION}"
# CPU-only smoke: both CLIs must import the sglang args we build from source.
RUN python - <<'PY'
import importlib.metadata as m
import sglang
import dynamo.sglang # noqa: F401
import dynamo.frontend # noqa: F401
print("dynamo", m.version("ai-dynamo"), "runtime", m.version("ai-dynamo-runtime"))
print("sglang", sglang.__file__)
PY
RUN python -m dynamo.sglang --help >/dev/null && python -m dynamo.frontend --help >/dev/null
ENTRYPOINT ["/opt/nvidia/nvidia_entrypoint.sh"]
CMD []
+216
View File
@@ -0,0 +1,216 @@
# Dynamo 支持(b300/dynamo)
> 分支:`b300/dynamo`(基点 `8b67636010`,= `main` = `b300/integration` 的 2026-11-17 c580 基线)
> 状态:**代码/镜像/启动脚本已就绪,静态兼容性已验证;GPU 运行时验证待窗口**
> (2026-11-17 时 `ds41-gist-exp` 占满 8 卡)。
本目录是 B300 线切换到 **NVIDIA Dynamo** 的入口:Dynamo frontend + 我们的 sglang
worker(P/D 分离)。它与现有 `ds41-gist-exp` 的区别:**入口 + PD 编排从 g9i gateway
换成 Dynamo frontend(`:33000`)**;**KV 数据面仍是 sglang 原生 disaggregation(mooncake)**。
## 1. 兼容性结论(2026-11-17)
- **不需要 sglang 代码改动**。Dynamo 1.6.2 的 `dynamo.sglang` 后端通过 `sgl.Engine`
公开接口工作,`_compat.py` 对 0.5.18/0.5.19 的兼容层在当前树(main-era)全部命中
新版分支。
- 用 Dynamo 1.6.2 自带单测跑了当前树(`PYTHONPATH=<tree>/python`,镜像内先装
`pytest-asyncio`):**985 passed / 20 failed / 1 skipped**。同一镜像用自带 sglang
nightly 跑对照也是 **985 passed / 20 failed / 1 skipped**,`diff` 后失败用例集合
**逐条相同**(均为视频生成等既有环境相关用例)。→ 无 API / 行为漂移。
- `python -m dynamo.sglang --help`(1398 行,sglang 参数已合并)与
`python -m dynamo.frontend --help` 均 rc=0。
- Dynamo 依赖的 `ServerArgs` 字段(`enable_forward_pass_metrics`、
`forward_pass_metrics_ipc_name`、`elastic_ep_backend`、`dllm_algorithm`、
`incremental_streaming_output`、`kv_events_config`、`allow_auto_truncate` 等)
在当前树全部存在。
复现命令(CPU,容器内):
```bash
docker run --rm --entrypoint bash \
-e PYTHONPATH=/data/ymk/wt-dynamo/python \
-v /data/ymk/wt-dynamo:/data/ymk/wt-dynamo:ro \
<dynamo-image> -c 'python -m dynamo.sglang --help >/dev/null && echo ok'
```
## 2. 版本矩阵
| 组件 | 版本 | 备注 |
|---|---|---|
| sglang | `b300/dynamo` @ `8b67636010` | 与镜像内已烘源码 `433257c7b0` 的 `python/sglang` **逐字节一致**(中间只有 deploy/docs 提交) |
| Dynamo | **1.6.2**(`ai-dynamo` + `ai-dynamo-runtime`) | 基底镜像原本带 `1.6.0.dev20261019`,Dockerfile 覆盖 |
| nixl | 1.5.0(`nixl-cu13` 1.5.0) | 基底已带;Dynamo 的 SGLang 后端**不用**它做 KV 传输 |
| torch | 2.14.1+cu130 | 基底已带,不因 Dynamo 变化 |
| sglang-kernel | 0.4.10 | 基底已带,与当前树 pin 一致 |
| mooncake | 0.3.14(L3 SSD) | 基底已带(见 `../README.md`) |
- KV 传输仍是 **sglang 原生 disaggregation + mooncake**;`--connector nixl` 只对
vLLM/TRT-LLM 生效,对 SGLang 是 no-op(10-26 已确认,本脚本不带它)。
- NVLink:`MC_INTRANODE_NVLINK=true` + `MC_INTRA_NVLINK=true` + `--pid host` +
宿主已烘的 NVLink host 修复。
### 2.1 PD 的两层要分开看(常被混淆)
| 层 | 由谁实现 | 在本部署里的表现 |
|---|---|---|
| **PD 编排 / 路由**:选 P、选 D、把 `bootstrap_info` 从 P 传给 D、流式回传 | **Dynamo 自己的 Rust runtime + frontend PrefillRouter**(不是 sglang-router/mini_lb) | `dynamo.frontend :33000`;Dynamo 的 `sglang` handler 只负责把 bootstrap 信息塞进/取出 `Engine.async_generate` |
| **KV 数据面**:bootstrap 握手后的 KV 传输 | **sglang 原生 disaggregation**(`--disaggregation-mode` + `--disaggregation-transfer-backend mooncake`) | P/D 日志的 `Using Intra-Node NVLink transport`;10-26 的 NVLink host 补丁仍然生效 |
Dynamo 的 `--connector nixl` / `--kv-transfer-config` 只对 vLLM/TRT-LLM 生效(vLLM 需要
`--kv-transfer-config '{"kv_connector":"NixlConnector",...}'`);SGLang 后端把
`bootstrap_host/port/room` 直接传给 `Engine.async_generate`,传输由 sglang 的 bootstrap
server + mooncake 完成(`prefill_handler.py` 产出 `disaggregated_params`,
`decode_handler.py` 消费 `bootstrap_info`)。所以 P/D 两侧仍然必须写
`--disaggregation-mode`,且 `--disaggregation-transfer-backend` 要与 sglang 侧一致。
> 结论:**编排是 Dynamo 的,KV 线是 sglang 的**。两者不是同一套,但也不是各走各的——
> Dynamo 控制面负责“谁和谁配对”,sglang 数据面负责真正搬 KV。
### 2.3 Dynamo 1.6.2 的 tokenizer / chat processor(实测)
DeepSeek-V4.1 没有 HF `chat_template`,默认的 Rust processor 无法渲染;需要用
SGLang 的 pre/post processor:
```
--dyn-chat-processor sglang \
--dyn-preprocess-workers 8 \
--tokenizer fastokens --trust-remote-code \
--tool-call-parser deepseekv41 \
--reasoning-parser deepseek-v41
```
本次踩通/修掉的四个点:
1. **缺 `enencoding_dsv4`**:`--dyn-chat-processor sglang` 会 import
`sglang.srt.entrypoints.openai.enencoding_dsv4`;我们的 fork 用 deepseek-recipe,没有该模块,
chat 直接 500。已在 `b300/dynamo` 加 shim `enencoding_dsv4.py`(基于 recipe 的
`render_conversation`),token ids 与生产 `DeepSeekRecipe.encode` 逐 token 相同。
2. **`--tokenizer fastokens` 不会自动传给 SGLang processor**:该 flag 只作用于 Dynamo 的
Rust model-card tokenizer;`sglang_processor._load_tokenizer()` 调 `get_tokenizer()`
时默认 `huggingface`。需要把 `FrontendConfig.tokenizer_backend` 传进去(FE hostPath 补丁),
日志出现 `fastokens backend enabled - transformers patched successfully` 才算生效。
3. **v41 thinking 解析**:Dynamo 的 `_THINKING_OPT_IN` 缺 `deepseek-v41`,显式
`chat_template_kwargs.thinking=true` 时 `reasoning_content` 不会拆出来。已补上;
默认请求仍是 chat(worker 没有发布 `default_thinking_mode`;生产默认 thinking 需要
register 侧补丁,TODO)。
4. **`--gateway-workers N`(worker 侧)** 是另一条路:给默认 Rust processor 用,启动 N 个
tokenizer/gateway 进程并设置 SGLang `--tokenizer-worker-num`;`--use-sglang-tokenizer`
已 deprecated。本次走 frontend processor,不需要它。
另:`encoding_dsv4`/`thinking` 的 FE 补丁目前以 hostPath 挂载在容器里,尚未烘进镜像;
重建镜像时要一起带上(见 `start_dynamo_pd.sh` 的 `-v` 列表)。
### 2.4 2026-11-17 首次运行结果(4P(dp4)+4D(dp4))
- 启动约 7 分钟;P `max_total_num_tokens=16,157,696`、D `58,859,520`(与 11-17
`ds41-gist-exp` 基线一致);P/D 均出现 `Using Intra-Node NVLink transport`。
- chat / tool_calls / reasoning_content 均可;prefix cache 正常(16k prompt 第二轮
cached 15,872;80k 第二轮 cached 79,872)。
- 32 并发 × ~900 token prompt(85% 命中,warm):**wall 0.62 s,32/32 成功**;
16k 冷 0.47 s / 热 0.24 s;80k 热 0.25 s(冷 80k ~1.04 s,HF tokenizer 时的数字)。
**重启后前 1–2 波会明显慢(8–9 s)是 warmup 伪影,benchmark 必须先热身。**
- P 的 OTel batch-span link 会报 `expected bytes, TraceNullContext found`(旧镜像);
已在分支修 `batch_telemetry.py`,本次运行用 `SGLANG_BATCH_SPAN=0` 规避。
### 2.2 Dynamo 不会消除 sglang 的 bootstrap / 握手尾巴
Dynamo 换掉的是控制面,sglang 内部的 bootstrap 路径原样保留:
- P 侧 `PrefillBootstrapQueue.pop_bootstrapped()` 与 `resolve_waiting_queue_bootstrap()`
仍然在 scheduler 每个 pass 里做 `poll_and_all_reduce_attn_cp_tp_group(...)`
(`disaggregation/prefill.py:495-556`、`638-675`);bootstrap 完成只在 pass 边界被发现。
- D 侧 `DecodePreallocQueue` / `_ensure_prefill_info` / 传输接收队列也一样。
- 因此 `prefill_bootstrap`(P queue entry → bootstrap done)与 `decode_transferred`
(等 P 完成 + 收 KV)的口径不变,Dynamo frontend 必须等它们完成才会开始 decode;
长 pass / DP skew / 大 batch 会按 pass 放大这个延迟。
实测参考(当前栈):c620 下 `prefill_bootstrap` p50 0.23–0.36 s、p95 1.66 s(fresh)/ 5.05 s
(不重启),而 `mooncake_worker_send_session` p50 只有 2.6 ms —— 卡的是握手/轮询,不是搬 KV。
Dynamo 的收益在入口/路由/可观测性;要压 bootstrap 尾部必须改 sglang scheduler 的轮询/预分配路径
(或先用 optimistic prefill + `hicache-write-policy write_back` 把 prefill 与握手重叠)。
## 3. 构建镜像
```bash
cd deploy/b300-k8s/dynamo
./build-dynamo-image.sh # 构建 + ctr import
# IMPORT=0 只构建;TAG=... 自定义 tag
```
默认产物:`ymkymx/sglang:dynamo-8b6763601-baked-20261117`
(= `sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:...`,同一 image ID)。
要点:Dynamo 必须 `pip install --no-deps`。基底已带全部运行时依赖;一旦让 pip 解析,
`ai-dynamo` 的 `sglang` extra 会试图装 `sglang==0.5.19` 并降级 torch。
## 4. 启动(docker,4P(dp4)+4D(dp4))
先停掉占卡的工作负载并确认 GPU 0-7 为 0 MiB,再:
```bash
./start_dynamo_pd.sh
./smoke.sh # /v1/models + /v1/chat/completions
tail -f /data/ymk/gist-dsv41/dyn-{fe,p,d}.log
```
端口:frontend **33000**、P **30000**、D **30300**;P/D 的 sglang 指标仍在原端口,
Dynamo 自己的指标在 frontend。
> ⚠️ 启动耗时与普通 P/D 栈相同(P/D 各 ~6-10 分钟;`/root/.cache/sglang` 挂载可复用
> autotune/JIT 缓存)。**先等 P/D ready 再开压测**:Dynamo 有冷启动首波长尾
> (空闲后首波 3–22s,暖机后消失)。
## 5. 与 2026-10-26 Dynamo 试验的差异
| 项 | 10-26 试验 | 本目录 |
|---|---|---|
| Dynamo | runtime 镜像 `sglang-runtime:1.6.0` | PyPI **1.6.2**(烘进 B300 镜像) |
| 模型 | `DeepSeek-V4.1-Flash` | `DeepSeek-V4.1-Flash-1017`(served name 保持原路径) |
| 并行 | tp4 + tp4(`--moe-a2a-backend none`) | **dp4 + dp4**(dp-attention + dp-lm-head + megamoe + flashmla_mega) |
| 入口 | `dynamo.frontend :33000` | 同上 |
| 缓存 | L2/L3 32GB | 当前生产 L3 SSD 4TB + `write_through` |
| 门控 | `SGLANG_DSV4_SWA_GATES_PREFIX_MATCH=0` | 同(依赖树内 `7f6fe22445`/`ee9170e7bb` fail-soft) |
dp4 是 DSpark PD 的硬要求:两侧 `attn_tp_size` 必须相等。dp4 下 attn_tp = tp/dp = 1,
P/D 一致;若一侧 tp4、另一侧 dp4,`disaggregation/common/conn.py:1355` 会直接拒绝
(`DSpark PD requires the same TP size on both servers`)。
## 6. 已知约束 / 坑
1. **`--host 0.0.0.0` 必须**:sglang bootstrap server 跟随 `--host`;写 127.0.0.1 时
D 侧拿不到 prefill parallel info → `HTTP 500: Decode handshake failed`。
2. **`SGLANG_UNIFIED_RADIX_TREE_CORE_BACKEND=python`**:baked 镜像没有
`rust_tree_core` `.so`(fake-cargo 阻止现场编译);1M 上下文下 unified cache 会选
Rust TreeCore 并 `ModuleNotFoundError`。本脚本已设为 `python`。
3. **Dynamo 1.6.2 安装用 `--no-deps`**:见 §3。`aisimulate` 在清华源只有
`0.13.0.dev2`(Dynamo 1.6.2 元数据要 `0.13.0`),实测 import 正常。
4. **`--dyn-preprocess-workers`** 取代了旧报告的 “gateway workers”:1.6.2 默认
auto(每个 model 最多 2),可显式设 4–8 提升高并发 tokenize 吞吐;仅在
`--dyn-chat-processor sglang` 下生效。
5. **Dynamo frontend 没有 g9i 的准入闸门**(`--g9i-p-queue-reqs-cap` 等)。Dynamo 自己的
flow control(`--router-queue-threshold` / `--load-aware`)在 10-26 实测无效,
有效闸门仍是客户端/网关并发上限。
6. **P/D 的 health / readiness**:`dynamo.sglang` worker 通过 discovery 注册,不使用
sglang 的 `/health` 作为就绪判据;k8s 化时探针应指向 frontend,或改用 dynamo 的
health payload(`dynamo.sglang.health_check`)。
7. `--enable-trace` 的 OTLP 服务名分别为 `dyn-p` / `dyn-d`(与旧试验一致);
Jaeger `:4317` 可用。
8. 端口分配遵守 `30000-32767` 固定端口纪律;frontend 33000 与现有栈不冲突。
## 7. 待办
- [ ] **GPU 冒烟**:等 `ds41-gist-exp` 停/窗口;验证 PD 全链路、NVLink transport 日志
(`Using Intra-Node NVLink transport`)、DSpark accept len、L2/L3 命中。
- [ ] **性能基线**:c580/1200s(同 `baselines/20261117-dtok16-fastokens` 口径)对比
g9i gateway 栈;关注 Dynamo frontend 的 tokenize 开销与冷启动长尾。
- [ ] **k8s cw manifest**:`ds41-dynamo`(frontend + p + d + mooncake-master),
探针/服务注册按 Dynamo discovery 设计;本目录先提供 docker 形态。
- [ ] 视需要把 namespace / Chat-Completions 工具展平(`b300/feat-recipe-ns` 已进
`b300/integration`)在 Dynamo frontend 侧的行为纳入 `preflight/issue-set`。
## 8. 参考
- 旧运行记录:`D:\B300\experiments\dsv41-gist-deepinfer\REPORT-DYNAMO-20261026.md`
- 启动脚本(旧,tp4):`D:\B300\experiments\dsv41-gist-deepinfer\start_dynamo_pd.sh`
- Dynamo 仓库(本地):`D:\B300\dynamo`(1.6.0 源码树,仅作 API 参考;实际运行用 1.6.2 wheel)
- L3 SSD 基线:`deploy/b300-k8s/README.md`
+39
View File
@@ -0,0 +1,39 @@
#!/usr/bin/env bash
# Build (and optionally load into containerd) the Dynamo-enabled B300 image.
#
# ./build-dynamo-image.sh # build + ctr import
# IMPORT=0 ./build-dynamo-image.sh # build only
# TAG=my-tag ./build-dynamo-image.sh # custom tag
set -euo pipefail
cd "$(dirname "$0")"
BASE=${BASE:-ymkymx/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115}
TAG=${TAG:-dynamo-8b6763601-baked-20261117}
REPO=${REPO:-sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang}
IMPORT=${IMPORT:-1}
DYNAMO_VERSION=${DYNAMO_VERSION:-1.6.2}
echo "base=$BASE tag=$TAG dynamo=$DYNAMO_VERSION"
docker build \
--build-arg BASE="$BASE" \
--build-arg DYNAMO_VERSION="$DYNAMO_VERSION" \
-t "ymkymx/sglang:$TAG" \
-f Dockerfile .
docker tag "ymkymx/sglang:$TAG" "$REPO:$TAG"
docker run --rm --entrypoint bash "ymkymx/sglang:$TAG" -c '
python - <<PY
import importlib.metadata as m, importlib.util
import dynamo.sglang, dynamo.frontend # noqa: F401
print("ok: ai-dynamo", m.version("ai-dynamo"), "| sglang", importlib.util.find_spec("sglang").origin)
PY
python -m dynamo.sglang --help >/dev/null
python -m dynamo.frontend --help >/dev/null
'
if [ "$IMPORT" = "1" ]; then
echo "importing $REPO:$TAG into containerd (k8s.io) ..."
docker save "$REPO:$TAG" | ctr -n k8s.io images import -
fi
echo "done: $REPO:$TAG"
+15
View File
@@ -0,0 +1,15 @@
#!/usr/bin/env bash
# Dynamo frontend smoke test.
# ./smoke.sh [base_url] [model_name]
set -euo pipefail
BASE=${1:-http://127.0.0.1:33000}
MODEL=${2:-/data/models/DeepSeek-V4.1-Flash}
echo "== /v1/models"
curl -fsS "$BASE/v1/models" | head -c 600; echo
echo "== chat"
curl -fsS "$BASE/v1/chat/completions" \
-H 'Content-Type: application/json' \
-d "{\"model\":\"$MODEL\",\"messages\":[{\"role\":\"user\",\"content\":\"2+3=? Answer with the number only.\"}],\"max_tokens\":64,\"temperature\":0}" \
| head -c 1200; echo
+183
View File
@@ -0,0 +1,183 @@
#!/usr/bin/env bash
# 4P(dp4)+4D(dp4) PD on NVIDIA Dynamo 1.6.2 + our B300 sglang (b300/dynamo @ 8b67636010).
#
# frontend :33000 (Dynamo router; replaces the g9i gateway)
# prefill :30000 GPU 0-3, mooncake NVLink, HiCache L2 + L3 SSD
# decode :30300 GPU 4-7
#
# Differences vs the 2026-10-26 Dynamo bring-up (REPORT-DYNAMO-20261026.md):
# * dp4 (tp4 + dp-attention + dp-lm-head) on both sides instead of tp4, matching
# the current production stack. Both sides must expose attn_tp_size==1 or the
# DSpark PD handshake rejects the pair ("same TP size on both servers").
# * image carries Dynamo 1.6.2 (the 10-26 run used the 1.6.0 runtime image).
# * L3 SSD mooncake 0.3.14 + the 1017 model / fastokens tokenizer.
#
# NOT validated on GPU yet (2026-11-17: ds41-gist-exp owns all 8 GPUs).
set -uo pipefail
IMG=${IMG:-ymkymx/sglang:dynamo-8b6763601-baked-20261117}
MODEL=${MODEL:-/data/models/DeepSeek-V4.1-Flash-1017}
SERVED=${SERVED:-/data/models/DeepSeek-V4.1-Flash}
D=${D:-/data/ymk/gist-dsv41}
STORE=${STORE:-$D/dynamo-store}
L3_CONF=${L3_CONF:-/data/ymk/ds41/mooncake-store-l3ssd.json}
JITCACHE=${JITCACHE:-/data/ymk/cache/sglang}
FE_PORT=${FE_PORT:-33000}
# Frontend tokenizer/chat processing (1.6.2):
# --dyn-chat-processor sglang frontend does SGLang-native template rendering,
# tokenization, tool/reasoning parsing
# --dyn-preprocess-workers N ProcessPool of N tokenizer/template workers
# --tokenizer fastokens fastokens BPE encoder (falls back to HF)
# Parser names must be explicit: sglang's "auto" is resolved at server startup
# and is not understood by the frontend's parser factory.
PREPROCESS_WORKERS=${PREPROCESS_WORKERS:-8}
TOOL_PARSER=${TOOL_PARSER:-deepseekv41}
REASONING_PARSER=${REASONING_PARSER:-deepseek-v41}
FE_ARGS=${FE_ARGS:---router-mode round-robin \
--dyn-chat-processor sglang \
--dyn-preprocess-workers $PREPROCESS_WORKERS \
--tokenizer fastokens --trust-remote-code \
--tool-call-parser $TOOL_PARSER \
--reasoning-parser $REASONING_PARSER}
mkdir -p "$STORE"; chmod 777 "$STORE" 2>/dev/null || true
docker rm -f dyn-fe dyn-p dyn-d >/dev/null 2>&1
sleep 3
# Dynamo/sglang children can outlive the container when --pid host is used.
for p in $(ps -eo pid,comm | awk '$2 ~ /^sglang::/ {print $1}'); do kill -9 "$p" 2>/dev/null; done
sleep 2
# L3 store master (not part of the Dynamo graph). Same pod/port as the cw master.
if ! docker ps --format '{{.Names}}' | grep -qx dyn-mooncake; then
docker rm -f dyn-mooncake >/dev/null 2>&1
docker run -d --name dyn-mooncake --network host "$IMG" \
/opt/sglang/bin/mooncake_master \
--port 50051 --enable_http_metadata_server=true \
--http_metadata_server_port=18081 --metrics_port=19003 \
--eviction_high_watermark_ratio=0.90 \
--enable_offload=true --offload_on_evict=false \
--promotion_on_hit=false --offload_force_evict=false \
> "$D/dyn-mooncake.log" 2>&1
sleep 3
fi
COMMON_ENVS=(
-e SGLANG_RAGGED_VERIFY_MODE=static
-e SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE=1
-e SGLANG_DSV4_KV_LAYOUT=v41_new
-e MC_INTRANODE_NVLINK=true
-e MC_INTRA_NVLINK=true
-e SGLANG_MOONCAKE_SEND_AUX_TCP=true
# The baked image has no rust_tree_core .so; force the Python radix tree.
-e SGLANG_UNIFIED_RADIX_TREE_CORE_BACKEND=python
-e SGLANG_DISAGGREGATION_WAITING_TIMEOUT=600
-e SGLANG_DISAGGREGATION_QUEUE_SIZE=16
-e SGLANG_DISAGGREGATION_THREAD_POOL_SIZE=32
-e SGLANG_BATCH_TELEMETRY=1
)
P_ENVS=(
-e SGLANG_OPT_DSV41_ENGRAM_PEER_SHARDS=1
-e SGLANG_OPT_DSV41_MEGA_MHC_PREFILL=1
-e SGLANG_OPT_DSV41_BF16_INDEX_LOGITS=1
-e SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=65536
-e SGLANG_DSV4_SWA_GATES_PREFIX_MATCH=0
-e SGLANG_HICACHE_MOONCAKE_CONFIG_PATH=$L3_CONF
-e SGLANG_HICACHE_PREFETCH_IO_THREADS=4
-e SGLANG_HICACHE_BACKUP_IO_THREADS=4
-e SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK=1
-e SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH=1
# The baked image predates the TraceNullContext guard; keep batch spans off
# until the next image build (histograms are unaffected).
-e SGLANG_BATCH_SPAN=0
)
D_ENVS=(
-e SGLANG_ENABLE_METRICS_DP_ATTENTION=1
)
# ---------------------------------------------------------------- frontend
docker run -d --name dyn-fe --network host --entrypoint bash \
-v /data:/data -v "$STORE":/tmp/dynamo_store_kv \
-v /data/ymk/wt-dynamo/python/sglang/srt/entrypoints/openai/encoding_dsv4.py:/sgl-workspace/sglang/python/sglang/srt/entrypoints/openai/encoding_dsv4.py:ro \
-v /data/ymk/dynamo-patch-sglang_prepost.py:/opt/sglang/lib/python3.12/site-packages/dynamo/frontend/sglang_prepost.py:ro \
-v /data/ymk/dynamo-sglang_processor.py:/opt/sglang/lib/python3.12/site-packages/dynamo/frontend/sglang_processor.py:ro \
"$IMG" \
-c "python -m dynamo.frontend --discovery-backend file --http-port $FE_PORT $FE_ARGS \
> $D/dyn-fe.log 2>&1"
# ---------------------------------------------------------------- prefill
docker run -d --name dyn-p --gpus all -e CUDA_VISIBLE_DEVICES=0,1,2,3 \
--network host --pid host --ipc host --shm-size 64g --cap-add SYS_NICE \
"${COMMON_ENVS[@]}" "${P_ENVS[@]}" \
-v /data:/data -v "$JITCACHE":/root/.cache/sglang \
-v "$STORE":/tmp/dynamo_store_kv \
-v /data/ymk/fake-cargo/cargo:/root/.cargo/bin/cargo:ro \
"$IMG" python -m dynamo.sglang \
--endpoint dyn://dynamo.prefill.generate --discovery-backend file \
--disaggregation-mode prefill \
--trust-remote-code --tokenizer-backend fastokens \
--model-path "$MODEL" --served-model-name "$SERVED" \
--reasoning-parser auto --tool-call-parser auto \
--dyn-default-thinking-mode enabled \
--default-chat-template-kwargs '{"thinking": true, "reasoning_effort": "high"}' \
--tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \
--moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \
--ep-num-redundant-experts 32 --ep-dispatch-algorithm dynamic \
--init-expert-location /data/ymk/gist-dsv41/placement_r32_ep4.json \
--mem-fraction-static 0.80 \
--chunked-prefill-size 131072 --max-prefill-tokens 131072 \
--max-running-requests 512 --tokenizer-worker-num 16 \
--load-balance-method total_tokens \
--speculative-algorithm DSPARK --speculative-dspark-block-size 3 \
--enable-decoder-swa-bounded-replay --swa-full-tokens-ratio 0.12 \
--cuda-graph-max-bs-decode 1 --disable-decode-cuda-graph \
--cuda-graph-backend-prefill breakable \
--cuda-graph-bs-prefill 1024 2048 4096 8192 16384 32768 \
--cuda-graph-prefill-max-context 262144 --cuda-graph-max-seq-len-prefill 262144 \
--enable-hierarchical-cache --hicache-ratio 4 \
--hicache-mem-layout page_first_direct --hicache-io-backend direct \
--hicache-write-policy write_through --hicache-storage-backend mooncake \
--hicache-storage-prefetch-policy wait_complete \
--host 0.0.0.0 --port 30000 --enable-metrics --enable-cache-report \
--enable-dp-attention-local-control-broadcast \
--enable-trace --trace-modules request,mooncake \
--otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name dyn-p \
--watchdog-timeout 3600 \
> "$D/dyn-p.log" 2>&1
# ---------------------------------------------------------------- decode
docker run -d --name dyn-d --gpus all -e CUDA_VISIBLE_DEVICES=4,5,6,7 \
--network host --pid host --ipc host --shm-size 64g --cap-add SYS_NICE \
"${COMMON_ENVS[@]}" "${D_ENVS[@]}" \
-v /data:/data -v "$JITCACHE":/root/.cache/sglang \
-v "$STORE":/tmp/dynamo_store_kv \
-v /data/ymk/fake-cargo/cargo:/root/.cargo/bin/cargo:ro \
"$IMG" python -m dynamo.sglang \
--endpoint dyn://dynamo.backend.generate --discovery-backend file \
--disaggregation-mode decode \
--trust-remote-code --tokenizer-backend fastokens \
--model-path "$MODEL" --served-model-name "$SERVED" \
--reasoning-parser auto --tool-call-parser auto \
--dyn-default-thinking-mode enabled \
--tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \
--enable-fp32-lm-head --enable-dp-attention-local-control-broadcast \
--moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \
--ep-num-redundant-experts 64 --ep-dispatch-algorithm dynamic \
--init-expert-location /data/ymk/gist-dsv41/placement_r64_ep4.json \
--mem-fraction-static 0.80 \
--speculative-algorithm DSPARK --speculative-dspark-block-size 3 \
--stream-interval 32 --swa-prefix-tails 2048 --enable-decoder-swa-bounded-replay \
--max-running-requests 512 --cuda-graph-max-bs-decode 512 --tokenizer-worker-num 16 \
--chunked-prefill-size 32768 --max-prefill-tokens 8192 \
--host 0.0.0.0 --port 30300 --enable-metrics --enable-cache-report \
--enable-trace --trace-modules request,mooncake \
--otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name dyn-d \
--watchdog-timeout 3600 \
> "$D/dyn-d.log" 2>&1
sleep 12
docker ps --format '{{.Names}}\t{{.Status}}' | grep -E '^dyn-' || true
echo "---- frontend log"; tail -3 "$D/dyn-fe.log" 2>/dev/null
echo "---- prefill log"; tail -3 "$D/dyn-p.log" 2>/dev/null
echo "---- decode log"; tail -3 "$D/dyn-d.log" 2>/dev/null
echo "[$(date +%H:%M:%S)] launched: FE=$FE_PORT P=30000 D=30300 (image $IMG)"
@@ -0,0 +1,62 @@
<h1>当前基线:ds41-gist-exp(4T L3 SSD,2026-11-17)</h1>
<callout emoji="📌" background-color="light-blue"><p>本节是 2026-11-17 定稿的 c580/c620 基线,后续 Dynamo 叠加以此为底座。旧的 ds41-gist-1p1d 章节已归档,不再作为当前部署。</p></callout>
<h2>1. 基线身份与镜像</h2>
<ul>
<li><b>工作负载</b>:<code>ds41-gist-exp</code> / <code>cw-dblft6sgrhagmpdt8ef0</code>,namespace <code>prj-d7te67bj0sq2fgdrmbig</code>,b300-01。</li>
<li><b>镜像</b>:<code>sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115</code>(40.1GB,含 Mooncake 0.3.14 + L3 SSD 支持)。</li>
<li><b>manifest</b>:<code>deploy/b300-k8s/ds41-gist-exp-20261117-dtok16-fastokens.yaml</code>;基线副本 <code>deploy/b300-k8s/baselines/20261117-dtok16-fastokens/manifest.yaml</code>;b300 副本在 <code>/data/ymk/deploy/b300-k8s/</code>。</li>
<li><b>形态</b>:RBG 三角色 <code>p</code> / <code>d</code> / <code>gateway</code> + <code>mooncake-master</code>;全 hostNetwork / hostPID / privileged;P=GPU0-3,D=GPU4-7。</li>
</ul>
<h2>2. 部署方式与端口</h2>
<table>
<thead><tr><th><p>角色</p></th><th><p>端口</p></th><th><p>说明</p></th></tr></thead>
<tbody>
<tr><td><p>P</p></td><td><p>30000 / 8998 / 9400-9403</p></td><td><p>HTTP / bootstrap / Mooncake client metrics</p></td></tr>
<tr><td><p>D</p></td><td><p>30300</p></td><td><p>HTTP</p></td></tr>
<tr><td><p>gateway</p></td><td><p>30400 / 29000</p></td><td><p>业务 / metrics</p></td></tr>
<tr><td><p>mooncake-master</p></td><td><p>50051 / 18081 / 19003</p></td><td><p>store / metadata / metrics</p></td></tr>
</tbody>
</table>
<p>默认入口:<code>http://127.0.0.1:30400</code>;当前 new-api/AgentX 直接使用该入口。</p>
<h2>3. 关键参数</h2>
<p><b>P(prefill)</b>:tp4 / dp4 / ep4,megamoe + flashmla_mega,ep-redundant 32,DSpark block 3,decoder SWA bounded replay;chunked-prefill 131072 / max-prefill 131072,max-running 512;fastokens + tokenizer-worker 16;HiCache ratio 4、page_first_direct、direct、write_through、Mooncake L3;<b>cache 模式,无 buffer_only、无 optimistic</b>;mem-fraction 0.80。</p>
<p><b>D(decode)</b>:tp4 / dp4 / ep4,ep-redundant 64,DSpark block 3,stream-interval 32,swa-prefix-tails 2048;max-running 512,cuda-graph-max-bs-decode 512;chunked 32768 / max-prefill 8192;fastokens + tokenizer-worker 16;mem-fraction 0.80。</p>
<p><b>gateway</b>:镜像内 <code>sgl-model-gateway</code>;<b>无 P admission cap</b>(无 p-qps / p-queue-reqs-cap / p-queue-token-cap,td-admit 默认 off);保留 health check + circuit breaker;<code>--g9i-decode-max-running 512</code>。这是 c580 能跑通的关键。</p>
<p><b>L3</b>:Mooncake master 4TB SSD offload;<code>offload_on_evict=false</code>;L3 配置见 <code>/data/ymk/ds41/mooncake-store-l3ssd.json</code>。</p>
<h2>4. 启动 / 重启流程</h2>
<pre lang="bash"><code># 1) 停完整工作负载并确认 GPU 0
MSYS_NO_PATHCONV=1 wsl -e /home/ubuntu/.local/bin/kmctl stop cw ds41-gist-exp --wait --timeout 10m --request-timeout 180s
ssh 24.125.93.200.207 "nvidia-smi --query-gpu=index,memory.used --format=csv,noheader"
# 2) 清 L3 SSD(实验/基线前必须)
ssh 24.125.93.200.207 'bash /data/ymk/deploy/b300-k8s/clean_l3ssd_cache.sh'
# 3) 用完整平台 manifest 启动(不要只 kubectl patch gateway)
MSYS_NO_PATHCONV=1 wsl -e /home/ubuntu/.local/bin/kmctl update cw ds41-gist-exp \
-f /mnt/d/B300/deploy/b300-k8s/ds41-gist-exp-20261117-dtok16-fastokens.yaml
# 4) 等 4 Pod 1/1,检查 P/D/gateway health 200,再 smoke
curl -sS http://127.0.0.1:30400/v1/chat/completions -H 'Content-Type: application/json' \
-d '{"model":"/data/models/DeepSeek-V4.1-Flash","messages":[{"role":"user","content":"1+1=?"}],"max_tokens":8}'
</code></pre>
<p>启动基线:P <code>max_total_num_tokens≈16,158,208</code> / 128 per-rank / avail ≈23GB;D <code>58,859,520</code> / 128 per-rank / avail ≈79GB。正常 warmup 6,385 条约 6.5 分钟。</p>
<h2>5. 基线结果</h2>
<table>
<thead><tr><th><p>场景</p></th><th><p>RPS</p></th><th><p>TTFT p50/p95/p99</p></th><th><p>ITL p95</p></th><th><p>结论</p></th></tr></thead>
<tbody>
<tr><td><p>c580 基线</p></td><td><p>24.045</p></td><td><p>1.743 / 3.772 / 6.580 s</p></td><td><p>10.74 ms</p></td><td><p>推荐工作点</p></td></tr>
<tr><td><p>c620 首轮(清 L3)</p></td><td><p>25.998</p></td><td><p>1.814 / 4.041 / 8.257 s</p></td><td><p>11.69 ms</p></td><td><p>吞吐 +8%,尾部变差</p></td></tr>
<tr><td><p>c620 不重启 r2/r3</p></td><td><p>25.85 / 25.61</p></td><td><p>p95 5.39 / 6.44 s,p99 9.56 / 10.79 s</p></td><td><p>10.96 / 10.68 ms</p></td><td><p>D KV 接近/打满,TTFT 尾部持续劣化,不建议连测</p></td></tr>
<tr><td><p>buffer_only</p></td><td><p>23.749</p></td><td><p>2.153 / 4.904 / 10.483 s</p></td><td><p>10.32 ms</p></td><td><p>吞吐 -1.2%、TTFT 更差,不采用</p></td></tr>
</tbody>
</table>
<p>结论:<b>c580 是稳定基线</b>;c620 只适合短时冲吞吐,且每轮之间必须重启/清池。buffer_only 不作为基线。</p>
<h2>6. 作为 Dynamo 底座</h2>
<p>当前镜像是 <b>SGLang + Mooncake 0.3.14 + L3 SSD</b> 基线,<b>不包含 Dynamo/nixl</b>。后续套 Dynamo 需要以本镜像为 base 另装 Dynamo 1.6.x / nixl,旧 <code>gist-nvlink-7f6fe2244-dynamo-local-20261026</code> 是更早的 10-23 代码,不是本基线。</p>
<p>Dynamo 叠加时保留本页的 P/D 参数、L3 SSD 和 gateway 入口;先做单机 4P4D 冒烟,再评估 Dynamo 的 P/D 角色、discovery 和 KV 传输配置。</p>
@@ -0,0 +1,8 @@
{
"master_server_address": "127.0.0.1:50051",
"metadata_server": "P2PHANDSHAKE",
"global_segment_size": "200gb",
"protocol": "tcp",
"enable_ssd_offload": true,
"ssd_offload_path": "/data/ymk/mooncake-ssd"
}
@@ -0,0 +1,6 @@
# L3 Mooncake SSD baseline requires the official 0.3.14 wheel.
# Previous image shipped 0.3.13.post1; 0.3.14 contains the upstream SSD-offload fixes.
# Wheel: https://files.pythonhosted.org/packages/b9/bf/a117c9c9f892e86893caf0dd8b5bfae1d11edbfb0173dcd30108a7168fc3/mooncake_transfer_engine_cuda13-0.3.14-cp312-cp312-manylinux_2_28_x86_64.whl
# sha256: 74fa5f0684b143167503e33f85cccb9ca687d3f9843d4b594dbd5342c7df3e17
# Installed with --no-deps; the base image already provides the runtime deps.
mooncake-transfer-engine-cuda13==0.3.14
@@ -0,0 +1,44 @@
# 2026-11-16 buffer_only + optimistic 首启记录
## 变更(相对 `stable-20261116-pre-optimistic`)
P:
- 增加 `--hicache-host-memory-mode buffer_only`(保留 `--hicache-write-policy write_through`、
`--hicache-ratio 4`、mooncake L3)
- 增加 `--optimistic-prefill-attempts 4`
D:
- 增加 `--tokenizer-worker-num 16`(与 P 相同)
其余角色/参数不变;网关保持 live 的 `p-queue-reqs-cap 64 / token-cap 32M / p-qps 40`。
## 启动前清理
- `kmctl stop cw ds41-gist-exp --wait --timeout 10m --request-timeout 180s` → `Terminated 0/0`,GPU 0-7 全 0 MiB
- `/data/ymk/mooncake-ssd`:2.3T → 4.0K;`/data` 7.5T available
- `kmctl update cw ds41-gist-exp -f ds41-gist-exp-20261116-bufferonly-optimistic.yaml`
## 首启验证(2026-11-16 22:00 CST)
- 4 个 Pod 全 `1/1 Running`;P/D/gateway `/health` 全 200
- P `server_args`:`hicache_host_memory_mode='buffer_only'`、`optimistic_prefill_attempts=4`;
没有 "Optimistic prefill supports ..." 的禁用 warning
- D `server_args`:`tokenizer_worker_num=16`;日志有 16 个 multi-tokenizer worker
- 启动基线:P `max_total_num_tokens=16,158,208` / 128 per-rank / `available_gpu_mem=23.00 GB`;
D `58,859,520` / 128 per-rank / `available_gpu_mem=79.27 GB`
- 冒烟:chat `/v1/chat/completions` 200,回答正确;16 并发 4k 字符请求全 200
- **optimistic 已生效**:P 日志出现 `#optimistic-req: 1`(其余为 0)
- P/D 日志无 Traceback / ERROR / assert
## 回滚
```bash
MSYS_NO_PATHCONV=1 wsl -e /home/ubuntu/.local/bin/kmctl stop cw ds41-gist-exp --wait --timeout 10m --request-timeout 180s
ssh 24.125.93.200.207 'bash /data/ymk/deploy/b300-k8s/clean_l3ssd_cache.sh'
MSYS_NO_PATHCONV=1 wsl -e /home/ubuntu/.local/bin/kmctl update cw ds41-gist-exp \
-f /mnt/d/B300/sglang/deploy/b300-k8s/snapshots/20261116-pre-optimistic/cw-ds41-gist-exp-11-16-pre-optimistic.yaml
```
代码仓库标签:`stable-20261116-pre-optimistic`。
@@ -0,0 +1,608 @@
apiVersion: workloads.x-k8s.io/v1alpha2
kind: RoleBasedGroup
metadata:
annotations:
rbg.workloads.x-k8s.io/discovery-config-mode: refine
creationTimestamp: "2026-11-16T13:54:16Z"
generation: 1
labels:
app: ds41-gist-exp
kongming.io/history-revision: hrev-a416985ca3a8cd59f808d12dbcd3426b833d85b0
kongming.io/history-run: hrun-4ca5f278e80f5e20e03b98351874ad7cd859d5e7
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-id: cw-dblft6sgrhagmpdt8ef0
kongming.io/resource-type: custom-workload
kongming.io/user-id: usr-d7upbr3j0sq2fgdrmcr0
sglang-model: ds-v41-flash
name: ds41-gist-exp
namespace: prj-d7te67bj0sq2fgdrmbig
resourceVersion: "30768532"
uid: 242c2504-0c97-4414-ae60-8376f7d63de0
spec:
roles:
- minReadySeconds: 0
name: mooncake-master
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: "19003"
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-mooncake
kongming.io/history-revision: hrev-a416985ca3a8cd59f808d12dbcd3426b833d85b0
kongming.io/history-run: hrun-4ca5f278e80f5e20e03b98351874ad7cd859d5e7
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-id: cw-dblft6sgrhagmpdt8ef0
kongming.io/resource-type: custom-workload
kongming.io/user-id: usr-d7upbr3j0sq2fgdrmcr0
spec:
containers:
- command:
- /bin/bash
- -c
- |
# 18081: 宿主机 18080 被 AgentX fake upstream(./fakeup)长期占用;
# 我们的 P/D 用 metadata_server=P2PHANDSHAKE,此 HTTP 端口只是 master 可选项。
exec /opt/sglang/bin/mooncake_master \
--port 50051 --enable_http_metadata_server=true \
--http_metadata_server_port=18081 --metrics_port=19003 --eviction_high_watermark_ratio=0.90 --enable_offload=true --offload_on_evict=false --promotion_on_hit=false --offload_force_evict=false
env:
- name: TMPDIR
value: /data/ymk/l3ssd/master-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/master-xdg
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: mooncake-master
ports:
- containerPort: 19003
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 60
initialDelaySeconds: 5
periodSeconds: 5
tcpSocket:
port: 50051
resources:
limits:
cpu: "4"
memory: 64Gi
requests:
cpu: "1"
memory: 8Gi
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
- dependencies:
- mooncake-master
minReadySeconds: 0
name: p
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: "30000"
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-p
kongming.io/history-revision: hrev-a416985ca3a8cd59f808d12dbcd3426b833d85b0
kongming.io/history-run: hrun-4ca5f278e80f5e20e03b98351874ad7cd859d5e7
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-id: cw-dblft6sgrhagmpdt8ef0
kongming.io/resource-type: custom-workload
kongming.io/user-id: usr-d7upbr3j0sq2fgdrmcr0
spec:
containers:
- command:
- /bin/bash
- -c
- |
exec sglang serve \
--trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \
--reasoning-parser auto --tool-call-parser auto \
--default-chat-template-kwargs '{"thinking": true, "reasoning_effort": "high"}' \
--tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \
--moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \
--ep-num-redundant-experts 32 --ep-dispatch-algorithm dynamic \
--init-expert-location /data/ymk/gist-dsv41/placement_r32_ep4.json \
--mem-fraction-static 0.80 \
--chunked-prefill-size 131072 --max-prefill-tokens 131072 \
--max-running-requests 512 --tokenizer-worker-num 16 \
--load-balance-method total_tokens \
--speculative-algorithm DSPARK --speculative-dspark-block-size 3 \
--enable-decoder-swa-bounded-replay --swa-full-tokens-ratio 0.12 \
--cuda-graph-max-bs-decode 1 --disable-decode-cuda-graph \
--cuda-graph-backend-prefill breakable \
--cuda-graph-bs-prefill 1024 2048 4096 8192 16384 32768 \
--cuda-graph-prefill-max-context 262144 --cuda-graph-max-seq-len-prefill 262144 \
--enable-hierarchical-cache --hicache-ratio 4 --hicache-host-memory-mode buffer_only \
--hicache-mem-layout page_first_direct --hicache-io-backend direct \
--hicache-write-policy write_through --hicache-storage-backend mooncake \
--hicache-storage-prefetch-policy wait_complete --optimistic-prefill-attempts 4 \
--disaggregation-mode prefill --disaggregation-transfer-backend mooncake \
--disaggregation-bootstrap-port 8998 \
--expert-balancedness-report-mode prometheus \
--expert-distribution-recorder-buffer-size 64 \
--host 0.0.0.0 --port 30000 --enable-metrics --enable-cache-report \
--enable-trace --trace-modules request,mooncake \
--otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-p \
--watchdog-timeout 3600 --enable-dp-attention-local-control-broadcast
env:
- name: CUDA_VISIBLE_DEVICES
value: 0,1,2,3
- name: SGLANG_LOG_MS
value: "1"
- name: SGLANG_OPT_DSV41_ENGRAM_PEER_SHARDS
value: "1"
- name: SGLANG_OPT_DSV41_MEGA_MHC_PREFILL
value: "1"
- name: SGLANG_OPT_DSV41_BF16_INDEX_LOGITS
value: "1"
- name: SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK
value: "65536"
- name: SGLANG_DSV4_KV_LAYOUT
value: v41_new
- name: SGLANG_DP_USE_GATHERV
value: "1"
- name: SGLANG_RAGGED_VERIFY_MODE
value: static
- name: SGLANG_ENABLE_DP_SPEC_PREFILL_COORDINATION
value: "0"
- name: SGLANG_NUMA_BIND_V2
value: "0"
- name: SGLANG_DEFAULT_THINKING
value: "1"
- name: SGLANG_DSV41_REASONING_EFFORT
value: high
- name: SGLANG_TIMEOUT_KEEP_ALIVE
value: "900"
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
value: "1"
- name: MC_INTRANODE_NVLINK
value: "true"
- name: MC_INTRA_NVLINK
value: "true"
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
value: "true"
- name: SGLANG_HICACHE_MOONCAKE_CONFIG_PATH
value: /data/ymk/ds41/mooncake-store-l3ssd.json
- name: SGLANG_HICACHE_PREFETCH_IO_THREADS
value: "4"
- name: SGLANG_HICACHE_BACKUP_IO_THREADS
value: "4"
- name: SGLANG_HICACHE_TIMING
value: "1"
- name: SGLANG_BATCH_TELEMETRY
value: "1"
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
value: "1"
- name: SGLANG_ENABLE_METRICS_DEVICE_TIMER
value: "1"
- name: SGLANG_EPLB_HEATMAP_COLLECTION_INTERVAL
value: "100"
- name: SGLANG_MM_TRACE
value: "1"
- name: SGLANG_BATCH_SPAN_MIN_MS
value: "200"
- name: SGLANG_BATCH_SPAN_SAMPLE
value: "0.05"
- name: SGLANG_BATCH_SPAN_MAX_LINKS
value: "8"
- name: SGLANG_BATCH_STAGE_TIMING
value: "1"
- name: SGLANG_BATCH_PASS_SPAN
value: "1"
- name: SGLANG_BATCH_PASS_SPAN_SAMPLE
value: "1.0"
- name: SGLANG_RECV_SPAN
value: "1"
- name: SGLANG_RECV_SPAN_SAMPLE
value: "1.0"
- name: SGLANG_MM_TRACE_SYNC
value: "0"
- name: SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK
value: "1"
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
value: "600"
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
value: "16"
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
value: "32"
- name: TZ
value: Asia/Shanghai
- name: SGLANG_TRACE_LOOP_SPAN
value: "1"
- name: SGLANG_TRACE_LOOP_SAMPLE
value: "1.0"
- name: SGLANG_FWD_TIMELINE
value: "0"
- name: SGLANG_TRACE_LOOP_DETAIL
value: "1"
- name: SGLANG_TRACE_MOONCAKE_DETAIL
value: "0"
- name: SGLANG_TRACE_BACKUP_DETAIL
value: "1"
- name: SGLANG_ASYNC_BACKUP_SUBMIT
value: "1"
- name: SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH
value: "1"
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
value: "1024"
- name: SGLANG_MOONCAKE_CLIENT_METRICS_PORT_BASE
value: "9400"
- name: MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR
value: bucket_storage_backend
- name: MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE
value: "1099511627776"
- name: MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES
value: "1099511627776"
- name: MOONCAKE_OFFLOAD_BUCKET_EVICTION_POLICY
value: fifo
- name: TMPDIR
value: /data/ymk/l3ssd/p-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/p-xdg
- name: HF_HOME
value: /data/ymk/l3ssd/p-hf
- name: MOONCAKE_OFFLOAD_FLUSH_PARTIAL_BUCKETS
value: "0"
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: sglang
ports:
- containerPort: 30000
name: metrics
protocol: TCP
- containerPort: 8998
name: bootstrap
protocol: TCP
- containerPort: 9400
name: mc-0
protocol: TCP
- containerPort: 9401
name: mc-1
protocol: TCP
- containerPort: 9402
name: mc-2
protocol: TCP
- containerPort: 9403
name: mc-3
protocol: TCP
readinessProbe:
failureThreshold: 120
httpGet:
path: /health
port: metrics
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: "96"
memory: 2Ti
nvidia.com/gpu: "4"
requests:
cpu: "32"
memory: 512Gi
nvidia.com/gpu: "4"
securityContext:
privileged: true
volumeMounts:
- mountPath: /data
name: data
- mountPath: /root/.cache/sglang
name: sglang-cache
- mountPath: /dev/shm
name: dshm
- mountPath: /root/.cargo/bin/cargo
name: fake-cargo
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
hostPID: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/cache/sglang
type: Directory
name: sglang-cache
- emptyDir:
medium: Memory
sizeLimit: 64Gi
name: dshm
- hostPath:
path: /data/ymk/ds41/fake-cargo
type: File
name: fake-cargo
- dependencies:
- mooncake-master
minReadySeconds: 0
name: d
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: "30300"
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-d
kongming.io/history-revision: hrev-a416985ca3a8cd59f808d12dbcd3426b833d85b0
kongming.io/history-run: hrun-4ca5f278e80f5e20e03b98351874ad7cd859d5e7
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-id: cw-dblft6sgrhagmpdt8ef0
kongming.io/resource-type: custom-workload
kongming.io/user-id: usr-d7upbr3j0sq2fgdrmcr0
spec:
containers:
- command:
- /bin/bash
- -c
- |
exec sglang serve \
--trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \
--reasoning-parser auto --tool-call-parser auto \
--default-chat-template-kwargs '{"thinking": true, "reasoning_effort": "high"}' \
--tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \
--enable-fp32-lm-head --enable-dp-attention-local-control-broadcast \
--moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \
--ep-num-redundant-experts 64 --ep-dispatch-algorithm dynamic \
--init-expert-location /data/ymk/gist-dsv41/placement_r64_ep4.json \
--mem-fraction-static 0.80 \
--speculative-algorithm DSPARK --speculative-dspark-block-size 3 \
--stream-interval 32 --swa-prefix-tails 2048 --enable-decoder-swa-bounded-replay \
--max-running-requests 512 --cuda-graph-max-bs-decode 512 --tokenizer-worker-num 16 \
--chunked-prefill-size 32768 --max-prefill-tokens 8192 \
--disaggregation-mode decode --disaggregation-transfer-backend mooncake \
--host 0.0.0.0 --port 30300 --enable-metrics --enable-cache-report \
--enable-trace --trace-modules request,mooncake \
--otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-d \
--watchdog-timeout 3600
env:
- name: CUDA_VISIBLE_DEVICES
value: 4,5,6,7
- name: SGLANG_RAGGED_VERIFY_MODE
value: static
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
value: "1"
- name: SGLANG_DSV4_KV_LAYOUT
value: v41_new
- name: SGLANG_DEFAULT_THINKING
value: "1"
- name: SGLANG_DSV41_REASONING_EFFORT
value: high
- name: MC_INTRANODE_NVLINK
value: "true"
- name: MC_INTRA_NVLINK
value: "true"
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
value: "true"
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
value: "1"
- name: SGLANG_BATCH_TELEMETRY
value: "1"
- name: SGLANG_BATCH_SPAN_SAMPLE
value: "0.1"
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
value: "600"
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
value: "16"
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
value: "32"
- name: TZ
value: Asia/Shanghai
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
value: "1024"
- name: TMPDIR
value: /data/ymk/l3ssd/d-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/d-xdg
- name: HF_HOME
value: /data/ymk/l3ssd/d-hf
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: sglang
ports:
- containerPort: 30300
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 150
httpGet:
path: /health
port: metrics
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: "96"
memory: 550Gi
nvidia.com/gpu: "4"
requests:
cpu: "32"
memory: 256Gi
nvidia.com/gpu: "4"
securityContext:
privileged: true
volumeMounts:
- mountPath: /data
name: data
- mountPath: /root/.cache/sglang
name: sglang-cache
- mountPath: /dev/shm
name: dshm
- mountPath: /root/.cargo/bin/cargo
name: fake-cargo
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
hostPID: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/cache/sglang
type: Directory
name: sglang-cache
- emptyDir:
medium: Memory
sizeLimit: 64Gi
name: dshm
- hostPath:
path: /data/ymk/ds41/fake-cargo
type: File
name: fake-cargo
- dependencies:
- p
- d
minReadySeconds: 0
name: gateway
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: "29000"
prometheus.io/scheme: http
prometheus.io/scrape: "true"
labels:
app: ds41-l3ssd-gw
kongming.io/history-revision: hrev-a416985ca3a8cd59f808d12dbcd3426b833d85b0
kongming.io/history-run: hrun-4ca5f278e80f5e20e03b98351874ad7cd859d5e7
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-id: cw-dblft6sgrhagmpdt8ef0
kongming.io/resource-type: custom-workload
kongming.io/user-id: usr-d7upbr3j0sq2fgdrmcr0
spec:
containers:
- command:
- /bin/bash
- -c
- |
exec /opt/sglang/bin/sgl-model-gateway launch \
--pd-disaggregation \
--prefill http://127.0.0.1:30000 8998 \
--decode http://127.0.0.1:30300 \
--prefill-policy cache_aware --decode-policy cache_aware \
--g9i-enabled --g9i-max-new-tokens 4096 --g9i-est-td-enabled --g9i-shadow-enabled --g9i-td-admit enforce --g9i-p-queue-reqs-cap 64 --g9i-p-queue-token-cap 32000000 --g9i-p-qps 40 --g9i-p-refresh-ms 250 --g9i-window-fixed 256 \
\
\
--g9i-regular-p-queue-reject-secs 0 \
--g9i-decode-max-running 512 \
--pool-idle-timeout-secs 2 --pool-max-idle-per-host 0 \
--health-check-timeout-secs 30 --health-check-interval-secs 30 \
--cb-failure-threshold 50 \
--model-path /data/models/DeepSeek-V4.1-Flash-1017 \
--tokenizer-path /data/models/DeepSeek-V4.1-Flash-1017 \
--host 0.0.0.0 --port 30400
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: gateway
ports:
- containerPort: 30400
name: http
protocol: TCP
- containerPort: 29000
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 60
httpGet:
path: /health
port: http
initialDelaySeconds: 20
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: "16"
memory: 32Gi
requests:
cpu: "2"
memory: 8Gi
volumeMounts:
- mountPath: /data
name: data
- mountPath: /opt/sglang/bin/sgl-model-gateway
name: gw-bin
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/gw-ns-est-td/sgl-model-gateway
type: File
name: gw-bin
status:
conditions:
- lastTransitionTime: "2026-11-16T14:01:52Z"
message: All roles are ready
observedGeneration: 1
reason: AllRolesReady
status: "True"
type: Ready
observedGeneration: 1
roleStatuses:
- name: mooncake-master
readyReplicas: 1
replicas: 1
updatedReplicas: 1
- name: p
readyReplicas: 1
replicas: 1
updatedReplicas: 1
- name: d
readyReplicas: 1
replicas: 1
updatedReplicas: 1
- name: gateway
readyReplicas: 1
replicas: 1
updatedReplicas: 1
@@ -0,0 +1,68 @@
# 11/16 上线前稳定配置(ds41-gist-exp)
记录时间:2026-11-16 21:49 CST(13:49Z),在切换 `buffer_only + optimistic` 之前。
## 身份
- 平台 cw:`ds41-gist-exp` / `cw-dblft6sgrhagmpdt8ef0`
- RBG:`ds41-gist-exp`(namespace `prj-d7te67bj0sq2fgdrmbig`)
- 角色:`mooncake-master` / `p` / `d` / `gateway`
- 镜像:`sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115`
- P:GPU 0-3;D:GPU 4-7;全 hostNetwork / hostPID / privileged,钉 b300-01
## 文件
- `rbg-live-raw.yaml`:`kubectl get rbg -o yaml` 原始快照(含 status,仅存档)
- `cw-ds41-gist-exp-11-16-pre-optimistic.yaml`:清理过平台字段的 RBG,可直接用于回滚更新
- `cw-platform.yaml`:`kmctl get cw -o yaml` 的平台清单(`Spec.Manifest` 是旧值,见下方漂移)
- 本说明
## 生效配置摘要
P:
- `tp4 / dp4 / ep4`、`megamoe`、`flashmla_mega`、`ep-redundant 32`
- DSPARK block 3、decoder SWA bounded replay、`swa-full-tokens-ratio 0.12`
- `chunked-prefill-size 131072`、`max-prefill-tokens 131072`
- `max-running-requests 512`、`tokenizer-worker-num 16`
- `hicache-ratio 4`、`hicache-mem-layout page_first_direct`、`hicache-io-backend direct`
- `hicache-write-policy write_through`、`hicache-storage-backend mooncake`
- **默认 host memory mode(cache),没有 `--hicache-host-memory-mode buffer_only`**
- **没有 `--optimistic-prefill-attempts`**(`#optimistic-req` 为 0)
- L3 SSD:4 TB `mooncake-ssd`,master `--offload_on_evict=false`
D:
- 同并行 tp4/dp4/ep4、`ep-redundant 64`、DSPARK block 3
- `max-running-requests 512`、`cuda-graph-max-bs-decode 512`
- `chunked-prefill-size 32768`、`max-prefill-tokens 8192`
- **没有 `--tokenizer-worker-num`**(默认 1)
- `mem-fraction-static 0.80`
Gateway:
- 实际 live:`--g9i-p-queue-reqs-cap 64 --g9i-p-queue-token-cap 32000000 --g9i-p-qps 40 --g9i-p-refresh-ms 250 --g9i-window-fixed 256`
- `--g9i-regular-p-queue-reject-secs 0`、`--g9i-decode-max-running 512`
- 网关二进制 hostPath:`/data/ymk/gw-ns-est-td/sgl-model-gateway`
> ⚠️ 平台 `cw-platform.yaml` 的 `Spec.Manifest` 里 gateway 仍是旧的
> `p-queue-reqs-cap 32 / token-cap 16000000 / p-qps 20`;**live RBG 是 64 / 32M / 40**。
> 本次快照与回滚文件都以 live RBG 为准,避免回滚时把网关参数打回旧值。
## 切换前状态
- 所有 Pod `1/1 Running`;P/D/gateway `/health` 200。
- 为切 `buffer_only` 做 L3 SSD 清理前,`/data/ymk/mooncake-ssd` 占用 **2.3T**;清理后 4.0K,`/data` 7.5T available。
- 启动基线(切换后复核):P `max_total_num_tokens=16,158,208` / 128 per-rank / `available_gpu_mem=23.00 GB`;
D `58,859,520` / 128 per-rank / `available_gpu_mem=79.27 GB`。
## 回滚
```bash
MSYS_NO_PATHCONV=1 wsl -e /home/ubuntu/.local/bin/kmctl stop cw ds41-gist-exp --wait --timeout 10m --request-timeout 180s
ssh 24.125.93.200.207 'bash /data/ymk/deploy/b300-k8s/clean_l3ssd_cache.sh'
MSYS_NO_PATHCONV=1 wsl -e /home/ubuntu/.local/bin/kmctl update cw ds41-gist-exp \
-f /mnt/d/B300/deploy/b300-k8s/snapshots/20261116-pre-optimistic/cw-ds41-gist-exp-11-16-pre-optimistic.yaml
```
回滚前确认 `kmctl get cw ds41-gist-exp` 为 `Terminated 0/0`、GPU 0-7 为 0 MiB。
@@ -0,0 +1,488 @@
apiVersion: workloads.x-k8s.io/v1alpha2
kind: RoleBasedGroup
metadata:
annotations:
rbg.workloads.x-k8s.io/discovery-config-mode: refine
labels:
app: ds41-gist-exp
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
sglang-model: ds-v41-flash
name: ds41-gist-exp
namespace: prj-d7te67bj0sq2fgdrmbig
spec:
roles:
- minReadySeconds: 0
name: mooncake-master
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '19003'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-mooncake
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "# 18081: 宿主机 18080 被 AgentX fake upstream(./fakeup)长期占用;\n# 我们的 P/D 用 metadata_server=P2PHANDSHAKE,此 HTTP 端口只是 master 可选项。\nexec /opt/sglang/bin/mooncake_master \\\n --port 50051 --enable_http_metadata_server=true \\\n --http_metadata_server_port=18081 --metrics_port=19003 --eviction_high_watermark_ratio=0.90 --enable_offload=true --offload_on_evict=false --promotion_on_hit=false --offload_force_evict=false\n"
env:
- name: TMPDIR
value: /data/ymk/l3ssd/master-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/master-xdg
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: mooncake-master
ports:
- containerPort: 19003
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 60
initialDelaySeconds: 5
periodSeconds: 5
tcpSocket:
port: 50051
resources:
limits:
cpu: '4'
memory: 64Gi
requests:
cpu: '1'
memory: 8Gi
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
- dependencies:
- mooncake-master
minReadySeconds: 0
name: p
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '30000'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-p
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 32 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r32_ep4.json \\\n --mem-fraction-static 0.80 \\\n --chunked-prefill-size 131072 --max-prefill-tokens 131072 \\\n --max-running-requests 512 --tokenizer-worker-num 16 \\\n --load-balance-method total_tokens \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --enable-decoder-swa-bounded-replay --swa-full-tokens-ratio 0.12 \\\n --cuda-graph-max-bs-decode 1 --disable-decode-cuda-graph \\\n --cuda-graph-backend-prefill breakable \\\n --cuda-graph-bs-prefill 1024 2048 4096 8192 16384 32768 \\\n --cuda-graph-prefill-max-context 262144 --cuda-graph-max-seq-len-prefill 262144 \\\n --enable-hierarchical-cache --hicache-ratio 4 \\\n --hicache-mem-layout page_first_direct --hicache-io-backend direct \\\n --hicache-write-policy write_through --hicache-storage-backend mooncake \\\n --hicache-storage-prefetch-policy wait_complete \\\n --disaggregation-mode prefill --disaggregation-transfer-backend mooncake \\\n --disaggregation-bootstrap-port 8998 \\\n --expert-balancedness-report-mode prometheus \\\n --expert-distribution-recorder-buffer-size 64 \\\n --host 0.0.0.0 --port 30000 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-p \\\n --watchdog-timeout 3600 --enable-dp-attention-local-control-broadcast\n"
env:
- name: CUDA_VISIBLE_DEVICES
value: 0,1,2,3
- name: SGLANG_LOG_MS
value: '1'
- name: SGLANG_OPT_DSV41_ENGRAM_PEER_SHARDS
value: '1'
- name: SGLANG_OPT_DSV41_MEGA_MHC_PREFILL
value: '1'
- name: SGLANG_OPT_DSV41_BF16_INDEX_LOGITS
value: '1'
- name: SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK
value: '65536'
- name: SGLANG_DSV4_KV_LAYOUT
value: v41_new
- name: SGLANG_DP_USE_GATHERV
value: '1'
- name: SGLANG_RAGGED_VERIFY_MODE
value: static
- name: SGLANG_ENABLE_DP_SPEC_PREFILL_COORDINATION
value: '0'
- name: SGLANG_NUMA_BIND_V2
value: '0'
- name: SGLANG_DEFAULT_THINKING
value: '1'
- name: SGLANG_DSV41_REASONING_EFFORT
value: high
- name: SGLANG_TIMEOUT_KEEP_ALIVE
value: '900'
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
value: '1'
- name: MC_INTRANODE_NVLINK
value: 'true'
- name: MC_INTRA_NVLINK
value: 'true'
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
value: 'true'
- name: SGLANG_HICACHE_MOONCAKE_CONFIG_PATH
value: /data/ymk/ds41/mooncake-store-l3ssd.json
- name: SGLANG_HICACHE_PREFETCH_IO_THREADS
value: '4'
- name: SGLANG_HICACHE_BACKUP_IO_THREADS
value: '4'
- name: SGLANG_HICACHE_TIMING
value: '1'
- name: SGLANG_BATCH_TELEMETRY
value: '1'
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
value: '1'
- name: SGLANG_ENABLE_METRICS_DEVICE_TIMER
value: '1'
- name: SGLANG_EPLB_HEATMAP_COLLECTION_INTERVAL
value: '100'
- name: SGLANG_MM_TRACE
value: '1'
- name: SGLANG_BATCH_SPAN_MIN_MS
value: '200'
- name: SGLANG_BATCH_SPAN_SAMPLE
value: '0.05'
- name: SGLANG_BATCH_SPAN_MAX_LINKS
value: '8'
- name: SGLANG_BATCH_STAGE_TIMING
value: '1'
- name: SGLANG_BATCH_PASS_SPAN
value: '1'
- name: SGLANG_BATCH_PASS_SPAN_SAMPLE
value: '1.0'
- name: SGLANG_RECV_SPAN
value: '1'
- name: SGLANG_RECV_SPAN_SAMPLE
value: '1.0'
- name: SGLANG_MM_TRACE_SYNC
value: '0'
- name: SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK
value: '1'
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
value: '600'
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
value: '16'
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
value: '32'
- name: TZ
value: Asia/Shanghai
- name: SGLANG_TRACE_LOOP_SPAN
value: '1'
- name: SGLANG_TRACE_LOOP_SAMPLE
value: '1.0'
- name: SGLANG_FWD_TIMELINE
value: '0'
- name: SGLANG_TRACE_LOOP_DETAIL
value: '1'
- name: SGLANG_TRACE_MOONCAKE_DETAIL
value: '0'
- name: SGLANG_TRACE_BACKUP_DETAIL
value: '1'
- name: SGLANG_ASYNC_BACKUP_SUBMIT
value: '1'
- name: SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH
value: '1'
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
value: '1024'
- name: SGLANG_MOONCAKE_CLIENT_METRICS_PORT_BASE
value: '9400'
- name: MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR
value: bucket_storage_backend
- name: MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE
value: '1099511627776'
- name: MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES
value: '1099511627776'
- name: MOONCAKE_OFFLOAD_BUCKET_EVICTION_POLICY
value: fifo
- name: TMPDIR
value: /data/ymk/l3ssd/p-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/p-xdg
- name: HF_HOME
value: /data/ymk/l3ssd/p-hf
- name: MOONCAKE_OFFLOAD_FLUSH_PARTIAL_BUCKETS
value: '0'
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: sglang
ports:
- containerPort: 30000
name: metrics
protocol: TCP
- containerPort: 8998
name: bootstrap
protocol: TCP
- containerPort: 9400
name: mc-0
protocol: TCP
- containerPort: 9401
name: mc-1
protocol: TCP
- containerPort: 9402
name: mc-2
protocol: TCP
- containerPort: 9403
name: mc-3
protocol: TCP
readinessProbe:
failureThreshold: 120
httpGet:
path: /health
port: metrics
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '96'
memory: 2Ti
nvidia.com/gpu: '4'
requests:
cpu: '32'
memory: 512Gi
nvidia.com/gpu: '4'
securityContext:
privileged: true
volumeMounts:
- mountPath: /data
name: data
- mountPath: /root/.cache/sglang
name: sglang-cache
- mountPath: /dev/shm
name: dshm
- mountPath: /root/.cargo/bin/cargo
name: fake-cargo
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
hostPID: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/cache/sglang
type: Directory
name: sglang-cache
- emptyDir:
medium: Memory
sizeLimit: 64Gi
name: dshm
- hostPath:
path: /data/ymk/ds41/fake-cargo
type: File
name: fake-cargo
- dependencies:
- mooncake-master
minReadySeconds: 0
name: d
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '30300'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-d
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --enable-fp32-lm-head --enable-dp-attention-local-control-broadcast \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 64 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r64_ep4.json \\\n --mem-fraction-static 0.80 \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --stream-interval 32 --swa-prefix-tails 2048 --enable-decoder-swa-bounded-replay \\\n --max-running-requests 512 --cuda-graph-max-bs-decode 512 \\\n --chunked-prefill-size 32768 --max-prefill-tokens 8192 \\\n --disaggregation-mode decode --disaggregation-transfer-backend mooncake \\\n --host 0.0.0.0 --port 30300 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-d \\\n --watchdog-timeout 3600\n"
env:
- name: CUDA_VISIBLE_DEVICES
value: 4,5,6,7
- name: SGLANG_RAGGED_VERIFY_MODE
value: static
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
value: '1'
- name: SGLANG_DSV4_KV_LAYOUT
value: v41_new
- name: SGLANG_DEFAULT_THINKING
value: '1'
- name: SGLANG_DSV41_REASONING_EFFORT
value: high
- name: MC_INTRANODE_NVLINK
value: 'true'
- name: MC_INTRA_NVLINK
value: 'true'
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
value: 'true'
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
value: '1'
- name: SGLANG_BATCH_TELEMETRY
value: '1'
- name: SGLANG_BATCH_SPAN_SAMPLE
value: '0.1'
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
value: '600'
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
value: '16'
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
value: '32'
- name: TZ
value: Asia/Shanghai
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
value: '1024'
- name: TMPDIR
value: /data/ymk/l3ssd/d-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/d-xdg
- name: HF_HOME
value: /data/ymk/l3ssd/d-hf
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: sglang
ports:
- containerPort: 30300
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 150
httpGet:
path: /health
port: metrics
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '96'
memory: 550Gi
nvidia.com/gpu: '4'
requests:
cpu: '32'
memory: 256Gi
nvidia.com/gpu: '4'
securityContext:
privileged: true
volumeMounts:
- mountPath: /data
name: data
- mountPath: /root/.cache/sglang
name: sglang-cache
- mountPath: /dev/shm
name: dshm
- mountPath: /root/.cargo/bin/cargo
name: fake-cargo
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
hostPID: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/cache/sglang
type: Directory
name: sglang-cache
- emptyDir:
medium: Memory
sizeLimit: 64Gi
name: dshm
- hostPath:
path: /data/ymk/ds41/fake-cargo
type: File
name: fake-cargo
- dependencies:
- p
- d
minReadySeconds: 0
name: gateway
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '29000'
prometheus.io/scheme: http
prometheus.io/scrape: 'true'
labels:
app: ds41-l3ssd-gw
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec /opt/sglang/bin/sgl-model-gateway launch \\\n --pd-disaggregation \\\n --prefill http://127.0.0.1:30000 8998 \\\n --decode http://127.0.0.1:30300 \\\n --prefill-policy cache_aware --decode-policy cache_aware \\\n --g9i-enabled --g9i-max-new-tokens 4096 --g9i-est-td-enabled --g9i-shadow-enabled --g9i-td-admit enforce --g9i-p-queue-reqs-cap 64 --g9i-p-queue-token-cap 32000000 --g9i-p-qps 40 --g9i-p-refresh-ms 250 --g9i-window-fixed 256 \\\n \\\n \\\n --g9i-regular-p-queue-reject-secs 0 \\\n --g9i-decode-max-running 512 \\\n --pool-idle-timeout-secs 2 --pool-max-idle-per-host 0 \\\n --health-check-timeout-secs 30 --health-check-interval-secs 30 \\\n --cb-failure-threshold 50 \\\n --model-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --tokenizer-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --host 0.0.0.0 --port 30400\n"
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: gateway
ports:
- containerPort: 30400
name: http
protocol: TCP
- containerPort: 29000
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 60
httpGet:
path: /health
port: http
initialDelaySeconds: 20
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '16'
memory: 32Gi
requests:
cpu: '2'
memory: 8Gi
volumeMounts:
- mountPath: /data
name: data
- mountPath: /opt/sglang/bin/sgl-model-gateway
name: gw-bin
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/gw-ns-est-td/sgl-model-gateway
type: File
name: gw-bin
File diff suppressed because one or more lines are too long
@@ -0,0 +1,608 @@
apiVersion: workloads.x-k8s.io/v1alpha2
kind: RoleBasedGroup
metadata:
annotations:
rbg.workloads.x-k8s.io/discovery-config-mode: refine
creationTimestamp: "2026-11-16T00:16:46Z"
generation: 9
labels:
app: ds41-gist-exp
kongming.io/history-revision: hrev-0d055a5880fbf263aac7967a2d38378d47209571
kongming.io/history-run: hrun-4ca5f278e80f5e20e03b98351874ad7cd859d5e7
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-id: cw-dblft6sgrhagmpdt8ef0
kongming.io/resource-type: custom-workload
kongming.io/user-id: usr-d7upbr3j0sq2fgdrmcr0
sglang-model: ds-v41-flash
name: ds41-gist-exp
namespace: prj-d7te67bj0sq2fgdrmbig
resourceVersion: "30735810"
uid: cc94a410-abf2-41ec-b711-52f7c87598e1
spec:
roles:
- minReadySeconds: 0
name: mooncake-master
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: "19003"
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-mooncake
kongming.io/history-revision: hrev-0d055a5880fbf263aac7967a2d38378d47209571
kongming.io/history-run: hrun-4ca5f278e80f5e20e03b98351874ad7cd859d5e7
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-id: cw-dblft6sgrhagmpdt8ef0
kongming.io/resource-type: custom-workload
kongming.io/user-id: usr-d7upbr3j0sq2fgdrmcr0
spec:
containers:
- command:
- /bin/bash
- -c
- |
# 18081: 宿主机 18080 被 AgentX fake upstream(./fakeup)长期占用;
# 我们的 P/D 用 metadata_server=P2PHANDSHAKE,此 HTTP 端口只是 master 可选项。
exec /opt/sglang/bin/mooncake_master \
--port 50051 --enable_http_metadata_server=true \
--http_metadata_server_port=18081 --metrics_port=19003 --eviction_high_watermark_ratio=0.90 --enable_offload=true --offload_on_evict=false --promotion_on_hit=false --offload_force_evict=false
env:
- name: TMPDIR
value: /data/ymk/l3ssd/master-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/master-xdg
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: mooncake-master
ports:
- containerPort: 19003
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 60
initialDelaySeconds: 5
periodSeconds: 5
tcpSocket:
port: 50051
resources:
limits:
cpu: "4"
memory: 64Gi
requests:
cpu: "1"
memory: 8Gi
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
- dependencies:
- mooncake-master
minReadySeconds: 0
name: p
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: "30000"
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-p
kongming.io/history-revision: hrev-0d055a5880fbf263aac7967a2d38378d47209571
kongming.io/history-run: hrun-4ca5f278e80f5e20e03b98351874ad7cd859d5e7
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-id: cw-dblft6sgrhagmpdt8ef0
kongming.io/resource-type: custom-workload
kongming.io/user-id: usr-d7upbr3j0sq2fgdrmcr0
spec:
containers:
- command:
- /bin/bash
- -c
- |
exec sglang serve \
--trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \
--reasoning-parser auto --tool-call-parser auto \
--default-chat-template-kwargs '{"thinking": true, "reasoning_effort": "high"}' \
--tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \
--moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \
--ep-num-redundant-experts 32 --ep-dispatch-algorithm dynamic \
--init-expert-location /data/ymk/gist-dsv41/placement_r32_ep4.json \
--mem-fraction-static 0.80 \
--chunked-prefill-size 131072 --max-prefill-tokens 131072 \
--max-running-requests 512 --tokenizer-worker-num 16 \
--load-balance-method total_tokens \
--speculative-algorithm DSPARK --speculative-dspark-block-size 3 \
--enable-decoder-swa-bounded-replay --swa-full-tokens-ratio 0.12 \
--cuda-graph-max-bs-decode 1 --disable-decode-cuda-graph \
--cuda-graph-backend-prefill breakable \
--cuda-graph-bs-prefill 1024 2048 4096 8192 16384 32768 \
--cuda-graph-prefill-max-context 262144 --cuda-graph-max-seq-len-prefill 262144 \
--enable-hierarchical-cache --hicache-ratio 4 \
--hicache-mem-layout page_first_direct --hicache-io-backend direct \
--hicache-write-policy write_through --hicache-storage-backend mooncake \
--hicache-storage-prefetch-policy wait_complete \
--disaggregation-mode prefill --disaggregation-transfer-backend mooncake \
--disaggregation-bootstrap-port 8998 \
--expert-balancedness-report-mode prometheus \
--expert-distribution-recorder-buffer-size 64 \
--host 0.0.0.0 --port 30000 --enable-metrics --enable-cache-report \
--enable-trace --trace-modules request,mooncake \
--otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-p \
--watchdog-timeout 3600 --enable-dp-attention-local-control-broadcast
env:
- name: CUDA_VISIBLE_DEVICES
value: 0,1,2,3
- name: SGLANG_LOG_MS
value: "1"
- name: SGLANG_OPT_DSV41_ENGRAM_PEER_SHARDS
value: "1"
- name: SGLANG_OPT_DSV41_MEGA_MHC_PREFILL
value: "1"
- name: SGLANG_OPT_DSV41_BF16_INDEX_LOGITS
value: "1"
- name: SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK
value: "65536"
- name: SGLANG_DSV4_KV_LAYOUT
value: v41_new
- name: SGLANG_DP_USE_GATHERV
value: "1"
- name: SGLANG_RAGGED_VERIFY_MODE
value: static
- name: SGLANG_ENABLE_DP_SPEC_PREFILL_COORDINATION
value: "0"
- name: SGLANG_NUMA_BIND_V2
value: "0"
- name: SGLANG_DEFAULT_THINKING
value: "1"
- name: SGLANG_DSV41_REASONING_EFFORT
value: high
- name: SGLANG_TIMEOUT_KEEP_ALIVE
value: "900"
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
value: "1"
- name: MC_INTRANODE_NVLINK
value: "true"
- name: MC_INTRA_NVLINK
value: "true"
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
value: "true"
- name: SGLANG_HICACHE_MOONCAKE_CONFIG_PATH
value: /data/ymk/ds41/mooncake-store-l3ssd.json
- name: SGLANG_HICACHE_PREFETCH_IO_THREADS
value: "4"
- name: SGLANG_HICACHE_BACKUP_IO_THREADS
value: "4"
- name: SGLANG_HICACHE_TIMING
value: "1"
- name: SGLANG_BATCH_TELEMETRY
value: "1"
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
value: "1"
- name: SGLANG_ENABLE_METRICS_DEVICE_TIMER
value: "1"
- name: SGLANG_EPLB_HEATMAP_COLLECTION_INTERVAL
value: "100"
- name: SGLANG_MM_TRACE
value: "1"
- name: SGLANG_BATCH_SPAN_MIN_MS
value: "200"
- name: SGLANG_BATCH_SPAN_SAMPLE
value: "0.05"
- name: SGLANG_BATCH_SPAN_MAX_LINKS
value: "8"
- name: SGLANG_BATCH_STAGE_TIMING
value: "1"
- name: SGLANG_BATCH_PASS_SPAN
value: "1"
- name: SGLANG_BATCH_PASS_SPAN_SAMPLE
value: "1.0"
- name: SGLANG_RECV_SPAN
value: "1"
- name: SGLANG_RECV_SPAN_SAMPLE
value: "1.0"
- name: SGLANG_MM_TRACE_SYNC
value: "0"
- name: SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK
value: "1"
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
value: "600"
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
value: "16"
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
value: "32"
- name: TZ
value: Asia/Shanghai
- name: SGLANG_TRACE_LOOP_SPAN
value: "1"
- name: SGLANG_TRACE_LOOP_SAMPLE
value: "1.0"
- name: SGLANG_FWD_TIMELINE
value: "0"
- name: SGLANG_TRACE_LOOP_DETAIL
value: "1"
- name: SGLANG_TRACE_MOONCAKE_DETAIL
value: "0"
- name: SGLANG_TRACE_BACKUP_DETAIL
value: "1"
- name: SGLANG_ASYNC_BACKUP_SUBMIT
value: "1"
- name: SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH
value: "1"
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
value: "1024"
- name: SGLANG_MOONCAKE_CLIENT_METRICS_PORT_BASE
value: "9400"
- name: MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR
value: bucket_storage_backend
- name: MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE
value: "1099511627776"
- name: MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES
value: "1099511627776"
- name: MOONCAKE_OFFLOAD_BUCKET_EVICTION_POLICY
value: fifo
- name: TMPDIR
value: /data/ymk/l3ssd/p-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/p-xdg
- name: HF_HOME
value: /data/ymk/l3ssd/p-hf
- name: MOONCAKE_OFFLOAD_FLUSH_PARTIAL_BUCKETS
value: "0"
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: sglang
ports:
- containerPort: 30000
name: metrics
protocol: TCP
- containerPort: 8998
name: bootstrap
protocol: TCP
- containerPort: 9400
name: mc-0
protocol: TCP
- containerPort: 9401
name: mc-1
protocol: TCP
- containerPort: 9402
name: mc-2
protocol: TCP
- containerPort: 9403
name: mc-3
protocol: TCP
readinessProbe:
failureThreshold: 120
httpGet:
path: /health
port: metrics
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: "96"
memory: 2Ti
nvidia.com/gpu: "4"
requests:
cpu: "32"
memory: 512Gi
nvidia.com/gpu: "4"
securityContext:
privileged: true
volumeMounts:
- mountPath: /data
name: data
- mountPath: /root/.cache/sglang
name: sglang-cache
- mountPath: /dev/shm
name: dshm
- mountPath: /root/.cargo/bin/cargo
name: fake-cargo
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
hostPID: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/cache/sglang
type: Directory
name: sglang-cache
- emptyDir:
medium: Memory
sizeLimit: 64Gi
name: dshm
- hostPath:
path: /data/ymk/ds41/fake-cargo
type: File
name: fake-cargo
- dependencies:
- mooncake-master
minReadySeconds: 0
name: d
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: "30300"
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-d
kongming.io/history-revision: hrev-0d055a5880fbf263aac7967a2d38378d47209571
kongming.io/history-run: hrun-4ca5f278e80f5e20e03b98351874ad7cd859d5e7
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-id: cw-dblft6sgrhagmpdt8ef0
kongming.io/resource-type: custom-workload
kongming.io/user-id: usr-d7upbr3j0sq2fgdrmcr0
spec:
containers:
- command:
- /bin/bash
- -c
- |
exec sglang serve \
--trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \
--reasoning-parser auto --tool-call-parser auto \
--default-chat-template-kwargs '{"thinking": true, "reasoning_effort": "high"}' \
--tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \
--enable-fp32-lm-head --enable-dp-attention-local-control-broadcast \
--moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \
--ep-num-redundant-experts 64 --ep-dispatch-algorithm dynamic \
--init-expert-location /data/ymk/gist-dsv41/placement_r64_ep4.json \
--mem-fraction-static 0.80 \
--speculative-algorithm DSPARK --speculative-dspark-block-size 3 \
--stream-interval 32 --swa-prefix-tails 2048 --enable-decoder-swa-bounded-replay \
--max-running-requests 512 --cuda-graph-max-bs-decode 512 \
--chunked-prefill-size 32768 --max-prefill-tokens 8192 \
--disaggregation-mode decode --disaggregation-transfer-backend mooncake \
--host 0.0.0.0 --port 30300 --enable-metrics --enable-cache-report \
--enable-trace --trace-modules request,mooncake \
--otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-d \
--watchdog-timeout 3600
env:
- name: CUDA_VISIBLE_DEVICES
value: 4,5,6,7
- name: SGLANG_RAGGED_VERIFY_MODE
value: static
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
value: "1"
- name: SGLANG_DSV4_KV_LAYOUT
value: v41_new
- name: SGLANG_DEFAULT_THINKING
value: "1"
- name: SGLANG_DSV41_REASONING_EFFORT
value: high
- name: MC_INTRANODE_NVLINK
value: "true"
- name: MC_INTRA_NVLINK
value: "true"
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
value: "true"
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
value: "1"
- name: SGLANG_BATCH_TELEMETRY
value: "1"
- name: SGLANG_BATCH_SPAN_SAMPLE
value: "0.1"
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
value: "600"
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
value: "16"
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
value: "32"
- name: TZ
value: Asia/Shanghai
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
value: "1024"
- name: TMPDIR
value: /data/ymk/l3ssd/d-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/d-xdg
- name: HF_HOME
value: /data/ymk/l3ssd/d-hf
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: sglang
ports:
- containerPort: 30300
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 150
httpGet:
path: /health
port: metrics
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: "96"
memory: 550Gi
nvidia.com/gpu: "4"
requests:
cpu: "32"
memory: 256Gi
nvidia.com/gpu: "4"
securityContext:
privileged: true
volumeMounts:
- mountPath: /data
name: data
- mountPath: /root/.cache/sglang
name: sglang-cache
- mountPath: /dev/shm
name: dshm
- mountPath: /root/.cargo/bin/cargo
name: fake-cargo
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
hostPID: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/cache/sglang
type: Directory
name: sglang-cache
- emptyDir:
medium: Memory
sizeLimit: 64Gi
name: dshm
- hostPath:
path: /data/ymk/ds41/fake-cargo
type: File
name: fake-cargo
- dependencies:
- p
- d
minReadySeconds: 0
name: gateway
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: "29000"
prometheus.io/scheme: http
prometheus.io/scrape: "true"
labels:
app: ds41-l3ssd-gw
kongming.io/history-revision: hrev-0d055a5880fbf263aac7967a2d38378d47209571
kongming.io/history-run: hrun-4ca5f278e80f5e20e03b98351874ad7cd859d5e7
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-id: cw-dblft6sgrhagmpdt8ef0
kongming.io/resource-type: custom-workload
kongming.io/user-id: usr-d7upbr3j0sq2fgdrmcr0
spec:
containers:
- command:
- /bin/bash
- -c
- |
exec /opt/sglang/bin/sgl-model-gateway launch \
--pd-disaggregation \
--prefill http://127.0.0.1:30000 8998 \
--decode http://127.0.0.1:30300 \
--prefill-policy cache_aware --decode-policy cache_aware \
--g9i-enabled --g9i-max-new-tokens 4096 --g9i-est-td-enabled --g9i-shadow-enabled --g9i-td-admit enforce --g9i-p-queue-reqs-cap 64 --g9i-p-queue-token-cap 32000000 --g9i-p-qps 40 --g9i-p-refresh-ms 250 --g9i-window-fixed 256 \
\
\
--g9i-regular-p-queue-reject-secs 0 \
--g9i-decode-max-running 512 \
--pool-idle-timeout-secs 2 --pool-max-idle-per-host 0 \
--health-check-timeout-secs 30 --health-check-interval-secs 30 \
--cb-failure-threshold 50 \
--model-path /data/models/DeepSeek-V4.1-Flash-1017 \
--tokenizer-path /data/models/DeepSeek-V4.1-Flash-1017 \
--host 0.0.0.0 --port 30400
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: gateway
ports:
- containerPort: 30400
name: http
protocol: TCP
- containerPort: 29000
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 60
httpGet:
path: /health
port: http
initialDelaySeconds: 20
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: "16"
memory: 32Gi
requests:
cpu: "2"
memory: 8Gi
volumeMounts:
- mountPath: /data
name: data
- mountPath: /opt/sglang/bin/sgl-model-gateway
name: gw-bin
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/gw-ns-est-td/sgl-model-gateway
type: File
name: gw-bin
status:
conditions:
- lastTransitionTime: "2026-11-16T12:36:12Z"
message: All roles are ready
observedGeneration: 9
reason: AllRolesReady
status: "True"
type: Ready
observedGeneration: 9
roleStatuses:
- name: mooncake-master
readyReplicas: 1
replicas: 1
updatedReplicas: 1
- name: p
readyReplicas: 1
replicas: 1
updatedReplicas: 1
- name: d
readyReplicas: 1
replicas: 1
updatedReplicas: 1
- name: gateway
readyReplicas: 1
replicas: 1
updatedReplicas: 1
@@ -0,0 +1,608 @@
apiVersion: workloads.x-k8s.io/v1alpha2
kind: RoleBasedGroup
metadata:
annotations:
rbg.workloads.x-k8s.io/discovery-config-mode: refine
creationTimestamp: "2026-11-16T00:16:46Z"
generation: 9
labels:
app: ds41-gist-exp
kongming.io/history-revision: hrev-0d055a5880fbf263aac7967a2d38378d47209571
kongming.io/history-run: hrun-4ca5f278e80f5e20e03b98351874ad7cd859d5e7
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-id: cw-dblft6sgrhagmpdt8ef0
kongming.io/resource-type: custom-workload
kongming.io/user-id: usr-d7upbr3j0sq2fgdrmcr0
sglang-model: ds-v41-flash
name: ds41-gist-exp
namespace: prj-d7te67bj0sq2fgdrmbig
resourceVersion: "30735810"
uid: cc94a410-abf2-41ec-b711-52f7c87598e1
spec:
roles:
- minReadySeconds: 0
name: mooncake-master
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: "19003"
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-mooncake
kongming.io/history-revision: hrev-0d055a5880fbf263aac7967a2d38378d47209571
kongming.io/history-run: hrun-4ca5f278e80f5e20e03b98351874ad7cd859d5e7
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-id: cw-dblft6sgrhagmpdt8ef0
kongming.io/resource-type: custom-workload
kongming.io/user-id: usr-d7upbr3j0sq2fgdrmcr0
spec:
containers:
- command:
- /bin/bash
- -c
- |
# 18081: 宿主机 18080 被 AgentX fake upstream(./fakeup)长期占用;
# 我们的 P/D 用 metadata_server=P2PHANDSHAKE,此 HTTP 端口只是 master 可选项。
exec /opt/sglang/bin/mooncake_master \
--port 50051 --enable_http_metadata_server=true \
--http_metadata_server_port=18081 --metrics_port=19003 --eviction_high_watermark_ratio=0.90 --enable_offload=true --offload_on_evict=false --promotion_on_hit=false --offload_force_evict=false
env:
- name: TMPDIR
value: /data/ymk/l3ssd/master-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/master-xdg
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: mooncake-master
ports:
- containerPort: 19003
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 60
initialDelaySeconds: 5
periodSeconds: 5
tcpSocket:
port: 50051
resources:
limits:
cpu: "4"
memory: 64Gi
requests:
cpu: "1"
memory: 8Gi
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
- dependencies:
- mooncake-master
minReadySeconds: 0
name: p
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: "30000"
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-p
kongming.io/history-revision: hrev-0d055a5880fbf263aac7967a2d38378d47209571
kongming.io/history-run: hrun-4ca5f278e80f5e20e03b98351874ad7cd859d5e7
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-id: cw-dblft6sgrhagmpdt8ef0
kongming.io/resource-type: custom-workload
kongming.io/user-id: usr-d7upbr3j0sq2fgdrmcr0
spec:
containers:
- command:
- /bin/bash
- -c
- |
exec sglang serve \
--trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \
--reasoning-parser auto --tool-call-parser auto \
--default-chat-template-kwargs '{"thinking": true, "reasoning_effort": "high"}' \
--tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \
--moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \
--ep-num-redundant-experts 32 --ep-dispatch-algorithm dynamic \
--init-expert-location /data/ymk/gist-dsv41/placement_r32_ep4.json \
--mem-fraction-static 0.80 \
--chunked-prefill-size 131072 --max-prefill-tokens 131072 \
--max-running-requests 512 --tokenizer-worker-num 16 \
--load-balance-method total_tokens \
--speculative-algorithm DSPARK --speculative-dspark-block-size 3 \
--enable-decoder-swa-bounded-replay --swa-full-tokens-ratio 0.12 \
--cuda-graph-max-bs-decode 1 --disable-decode-cuda-graph \
--cuda-graph-backend-prefill breakable \
--cuda-graph-bs-prefill 1024 2048 4096 8192 16384 32768 \
--cuda-graph-prefill-max-context 262144 --cuda-graph-max-seq-len-prefill 262144 \
--enable-hierarchical-cache --hicache-ratio 4 \
--hicache-mem-layout page_first_direct --hicache-io-backend direct \
--hicache-write-policy write_through --hicache-storage-backend mooncake \
--hicache-storage-prefetch-policy wait_complete \
--disaggregation-mode prefill --disaggregation-transfer-backend mooncake \
--disaggregation-bootstrap-port 8998 \
--expert-balancedness-report-mode prometheus \
--expert-distribution-recorder-buffer-size 64 \
--host 0.0.0.0 --port 30000 --enable-metrics --enable-cache-report \
--enable-trace --trace-modules request,mooncake \
--otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-p \
--watchdog-timeout 3600 --enable-dp-attention-local-control-broadcast
env:
- name: CUDA_VISIBLE_DEVICES
value: 0,1,2,3
- name: SGLANG_LOG_MS
value: "1"
- name: SGLANG_OPT_DSV41_ENGRAM_PEER_SHARDS
value: "1"
- name: SGLANG_OPT_DSV41_MEGA_MHC_PREFILL
value: "1"
- name: SGLANG_OPT_DSV41_BF16_INDEX_LOGITS
value: "1"
- name: SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK
value: "65536"
- name: SGLANG_DSV4_KV_LAYOUT
value: v41_new
- name: SGLANG_DP_USE_GATHERV
value: "1"
- name: SGLANG_RAGGED_VERIFY_MODE
value: static
- name: SGLANG_ENABLE_DP_SPEC_PREFILL_COORDINATION
value: "0"
- name: SGLANG_NUMA_BIND_V2
value: "0"
- name: SGLANG_DEFAULT_THINKING
value: "1"
- name: SGLANG_DSV41_REASONING_EFFORT
value: high
- name: SGLANG_TIMEOUT_KEEP_ALIVE
value: "900"
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
value: "1"
- name: MC_INTRANODE_NVLINK
value: "true"
- name: MC_INTRA_NVLINK
value: "true"
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
value: "true"
- name: SGLANG_HICACHE_MOONCAKE_CONFIG_PATH
value: /data/ymk/ds41/mooncake-store-l3ssd.json
- name: SGLANG_HICACHE_PREFETCH_IO_THREADS
value: "4"
- name: SGLANG_HICACHE_BACKUP_IO_THREADS
value: "4"
- name: SGLANG_HICACHE_TIMING
value: "1"
- name: SGLANG_BATCH_TELEMETRY
value: "1"
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
value: "1"
- name: SGLANG_ENABLE_METRICS_DEVICE_TIMER
value: "1"
- name: SGLANG_EPLB_HEATMAP_COLLECTION_INTERVAL
value: "100"
- name: SGLANG_MM_TRACE
value: "1"
- name: SGLANG_BATCH_SPAN_MIN_MS
value: "200"
- name: SGLANG_BATCH_SPAN_SAMPLE
value: "0.05"
- name: SGLANG_BATCH_SPAN_MAX_LINKS
value: "8"
- name: SGLANG_BATCH_STAGE_TIMING
value: "1"
- name: SGLANG_BATCH_PASS_SPAN
value: "1"
- name: SGLANG_BATCH_PASS_SPAN_SAMPLE
value: "1.0"
- name: SGLANG_RECV_SPAN
value: "1"
- name: SGLANG_RECV_SPAN_SAMPLE
value: "1.0"
- name: SGLANG_MM_TRACE_SYNC
value: "0"
- name: SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK
value: "1"
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
value: "600"
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
value: "16"
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
value: "32"
- name: TZ
value: Asia/Shanghai
- name: SGLANG_TRACE_LOOP_SPAN
value: "1"
- name: SGLANG_TRACE_LOOP_SAMPLE
value: "1.0"
- name: SGLANG_FWD_TIMELINE
value: "0"
- name: SGLANG_TRACE_LOOP_DETAIL
value: "1"
- name: SGLANG_TRACE_MOONCAKE_DETAIL
value: "0"
- name: SGLANG_TRACE_BACKUP_DETAIL
value: "1"
- name: SGLANG_ASYNC_BACKUP_SUBMIT
value: "1"
- name: SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH
value: "1"
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
value: "1024"
- name: SGLANG_MOONCAKE_CLIENT_METRICS_PORT_BASE
value: "9400"
- name: MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR
value: bucket_storage_backend
- name: MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE
value: "1099511627776"
- name: MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES
value: "1099511627776"
- name: MOONCAKE_OFFLOAD_BUCKET_EVICTION_POLICY
value: fifo
- name: TMPDIR
value: /data/ymk/l3ssd/p-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/p-xdg
- name: HF_HOME
value: /data/ymk/l3ssd/p-hf
- name: MOONCAKE_OFFLOAD_FLUSH_PARTIAL_BUCKETS
value: "0"
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: sglang
ports:
- containerPort: 30000
name: metrics
protocol: TCP
- containerPort: 8998
name: bootstrap
protocol: TCP
- containerPort: 9400
name: mc-0
protocol: TCP
- containerPort: 9401
name: mc-1
protocol: TCP
- containerPort: 9402
name: mc-2
protocol: TCP
- containerPort: 9403
name: mc-3
protocol: TCP
readinessProbe:
failureThreshold: 120
httpGet:
path: /health
port: metrics
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: "96"
memory: 2Ti
nvidia.com/gpu: "4"
requests:
cpu: "32"
memory: 512Gi
nvidia.com/gpu: "4"
securityContext:
privileged: true
volumeMounts:
- mountPath: /data
name: data
- mountPath: /root/.cache/sglang
name: sglang-cache
- mountPath: /dev/shm
name: dshm
- mountPath: /root/.cargo/bin/cargo
name: fake-cargo
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
hostPID: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/cache/sglang
type: Directory
name: sglang-cache
- emptyDir:
medium: Memory
sizeLimit: 64Gi
name: dshm
- hostPath:
path: /data/ymk/ds41/fake-cargo
type: File
name: fake-cargo
- dependencies:
- mooncake-master
minReadySeconds: 0
name: d
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: "30300"
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-d
kongming.io/history-revision: hrev-0d055a5880fbf263aac7967a2d38378d47209571
kongming.io/history-run: hrun-4ca5f278e80f5e20e03b98351874ad7cd859d5e7
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-id: cw-dblft6sgrhagmpdt8ef0
kongming.io/resource-type: custom-workload
kongming.io/user-id: usr-d7upbr3j0sq2fgdrmcr0
spec:
containers:
- command:
- /bin/bash
- -c
- |
exec sglang serve \
--trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \
--reasoning-parser auto --tool-call-parser auto \
--default-chat-template-kwargs '{"thinking": true, "reasoning_effort": "high"}' \
--tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \
--enable-fp32-lm-head --enable-dp-attention-local-control-broadcast \
--moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \
--ep-num-redundant-experts 64 --ep-dispatch-algorithm dynamic \
--init-expert-location /data/ymk/gist-dsv41/placement_r64_ep4.json \
--mem-fraction-static 0.80 \
--speculative-algorithm DSPARK --speculative-dspark-block-size 3 \
--stream-interval 32 --swa-prefix-tails 2048 --enable-decoder-swa-bounded-replay \
--max-running-requests 512 --cuda-graph-max-bs-decode 512 \
--chunked-prefill-size 32768 --max-prefill-tokens 8192 \
--disaggregation-mode decode --disaggregation-transfer-backend mooncake \
--host 0.0.0.0 --port 30300 --enable-metrics --enable-cache-report \
--enable-trace --trace-modules request,mooncake \
--otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-d \
--watchdog-timeout 3600
env:
- name: CUDA_VISIBLE_DEVICES
value: 4,5,6,7
- name: SGLANG_RAGGED_VERIFY_MODE
value: static
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
value: "1"
- name: SGLANG_DSV4_KV_LAYOUT
value: v41_new
- name: SGLANG_DEFAULT_THINKING
value: "1"
- name: SGLANG_DSV41_REASONING_EFFORT
value: high
- name: MC_INTRANODE_NVLINK
value: "true"
- name: MC_INTRA_NVLINK
value: "true"
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
value: "true"
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
value: "1"
- name: SGLANG_BATCH_TELEMETRY
value: "1"
- name: SGLANG_BATCH_SPAN_SAMPLE
value: "0.1"
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
value: "600"
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
value: "16"
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
value: "32"
- name: TZ
value: Asia/Shanghai
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
value: "1024"
- name: TMPDIR
value: /data/ymk/l3ssd/d-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/d-xdg
- name: HF_HOME
value: /data/ymk/l3ssd/d-hf
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: sglang
ports:
- containerPort: 30300
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 150
httpGet:
path: /health
port: metrics
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: "96"
memory: 550Gi
nvidia.com/gpu: "4"
requests:
cpu: "32"
memory: 256Gi
nvidia.com/gpu: "4"
securityContext:
privileged: true
volumeMounts:
- mountPath: /data
name: data
- mountPath: /root/.cache/sglang
name: sglang-cache
- mountPath: /dev/shm
name: dshm
- mountPath: /root/.cargo/bin/cargo
name: fake-cargo
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
hostPID: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/cache/sglang
type: Directory
name: sglang-cache
- emptyDir:
medium: Memory
sizeLimit: 64Gi
name: dshm
- hostPath:
path: /data/ymk/ds41/fake-cargo
type: File
name: fake-cargo
- dependencies:
- p
- d
minReadySeconds: 0
name: gateway
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: "29000"
prometheus.io/scheme: http
prometheus.io/scrape: "true"
labels:
app: ds41-l3ssd-gw
kongming.io/history-revision: hrev-0d055a5880fbf263aac7967a2d38378d47209571
kongming.io/history-run: hrun-4ca5f278e80f5e20e03b98351874ad7cd859d5e7
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-id: cw-dblft6sgrhagmpdt8ef0
kongming.io/resource-type: custom-workload
kongming.io/user-id: usr-d7upbr3j0sq2fgdrmcr0
spec:
containers:
- command:
- /bin/bash
- -c
- |
exec /opt/sglang/bin/sgl-model-gateway launch \
--pd-disaggregation \
--prefill http://127.0.0.1:30000 8998 \
--decode http://127.0.0.1:30300 \
--prefill-policy cache_aware --decode-policy cache_aware \
--g9i-enabled --g9i-max-new-tokens 4096 --g9i-est-td-enabled --g9i-shadow-enabled --g9i-td-admit enforce --g9i-p-queue-reqs-cap 64 --g9i-p-queue-token-cap 32000000 --g9i-p-qps 40 --g9i-p-refresh-ms 250 --g9i-window-fixed 256 \
\
\
--g9i-regular-p-queue-reject-secs 0 \
--g9i-decode-max-running 512 \
--pool-idle-timeout-secs 2 --pool-max-idle-per-host 0 \
--health-check-timeout-secs 30 --health-check-interval-secs 30 \
--cb-failure-threshold 50 \
--model-path /data/models/DeepSeek-V4.1-Flash-1017 \
--tokenizer-path /data/models/DeepSeek-V4.1-Flash-1017 \
--host 0.0.0.0 --port 30400
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: gateway
ports:
- containerPort: 30400
name: http
protocol: TCP
- containerPort: 29000
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 60
httpGet:
path: /health
port: http
initialDelaySeconds: 20
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: "16"
memory: 32Gi
requests:
cpu: "2"
memory: 8Gi
volumeMounts:
- mountPath: /data
name: data
- mountPath: /opt/sglang/bin/sgl-model-gateway
name: gw-bin
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/gw-ns-est-td/sgl-model-gateway
type: File
name: gw-bin
status:
conditions:
- lastTransitionTime: "2026-11-16T12:36:12Z"
message: All roles are ready
observedGeneration: 9
reason: AllRolesReady
status: "True"
type: Ready
observedGeneration: 9
roleStatuses:
- name: mooncake-master
readyReplicas: 1
replicas: 1
updatedReplicas: 1
- name: p
readyReplicas: 1
replicas: 1
updatedReplicas: 1
- name: d
readyReplicas: 1
replicas: 1
updatedReplicas: 1
- name: gateway
readyReplicas: 1
replicas: 1
updatedReplicas: 1
@@ -0,0 +1,13 @@
# 2026-11-17 cache + gateway no-limit 实验参数(失败记录)
- P: hicache_host_memory_mode=cache, optimistic_prefill_attempts=0, tokenizer-worker-num=16,
hicache ratio 4 / write_through / L3 mooncake, L3 SSD 4T.
- D: tokenizer-worker-num=16.
- Gateway: g9i-td-admit off, p-queue-reqs-cap 0, p-queue-token-cap 0, p-qps 0,
decode-max-running 1000000, disable-circuit-breaker, disable-health-check.
- Image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
- c580 run c580-cache-nolimit-20261117: warmup 4,720/6,385 (74%), 02:33:32 失败
(ServerDisconnected/warmup_failure;P/D/gateway 0 restart,无 503)。
- 观察:P 单批 32k/23k chunk 正常,但每 rank 同时仅 1 个请求;gateway
smg_worker_requests_active 峰值 1,840,P queue 峰值 503,gateway 约 1 核。
- 结论:该 no-limit/no-cb 网关配置不适合作为 c580 基线;仅作异常记录。
@@ -0,0 +1,488 @@
apiVersion: workloads.x-k8s.io/v1alpha2
kind: RoleBasedGroup
metadata:
annotations:
rbg.workloads.x-k8s.io/discovery-config-mode: refine
labels:
app: ds41-gist-exp
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
sglang-model: ds-v41-flash
name: ds41-gist-exp
namespace: prj-d7te67bj0sq2fgdrmbig
spec:
roles:
- minReadySeconds: 0
name: mooncake-master
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '19003'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-mooncake
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "# 18081: 宿主机 18080 被 AgentX fake upstream(./fakeup)长期占用;\n# 我们的 P/D 用 metadata_server=P2PHANDSHAKE,此 HTTP 端口只是 master 可选项。\nexec /opt/sglang/bin/mooncake_master \\\n --port 50051 --enable_http_metadata_server=true \\\n --http_metadata_server_port=18081 --metrics_port=19003 --eviction_high_watermark_ratio=0.90 --enable_offload=true --offload_on_evict=false --promotion_on_hit=false --offload_force_evict=false\n"
env:
- name: TMPDIR
value: /data/ymk/l3ssd/master-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/master-xdg
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: mooncake-master
ports:
- containerPort: 19003
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 60
initialDelaySeconds: 5
periodSeconds: 5
tcpSocket:
port: 50051
resources:
limits:
cpu: '4'
memory: 64Gi
requests:
cpu: '1'
memory: 8Gi
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
- dependencies:
- mooncake-master
minReadySeconds: 0
name: p
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '30000'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-p
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 32 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r32_ep4.json \\\n --mem-fraction-static 0.80 \\\n --chunked-prefill-size 131072 --max-prefill-tokens 131072 \\\n --max-running-requests 512 --tokenizer-worker-num 16 \\\n --load-balance-method total_tokens \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --enable-decoder-swa-bounded-replay --swa-full-tokens-ratio 0.12 \\\n --cuda-graph-max-bs-decode 1 --disable-decode-cuda-graph \\\n --cuda-graph-backend-prefill breakable \\\n --cuda-graph-bs-prefill 1024 2048 4096 8192 16384 32768 \\\n --cuda-graph-prefill-max-context 262144 --cuda-graph-max-seq-len-prefill 262144 \\\n --enable-hierarchical-cache --hicache-ratio 4 \\\n --hicache-mem-layout page_first_direct --hicache-io-backend direct \\\n --hicache-write-policy write_through --hicache-storage-backend mooncake \\\n --hicache-storage-prefetch-policy wait_complete \\\n --disaggregation-mode prefill --disaggregation-transfer-backend mooncake \\\n --disaggregation-bootstrap-port 8998 \\\n --expert-balancedness-report-mode prometheus \\\n --expert-distribution-recorder-buffer-size 64 \\\n --host 0.0.0.0 --port 30000 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-p \\\n --watchdog-timeout 3600 --enable-dp-attention-local-control-broadcast\n"
env:
- name: CUDA_VISIBLE_DEVICES
value: 0,1,2,3
- name: SGLANG_LOG_MS
value: '1'
- name: SGLANG_OPT_DSV41_ENGRAM_PEER_SHARDS
value: '1'
- name: SGLANG_OPT_DSV41_MEGA_MHC_PREFILL
value: '1'
- name: SGLANG_OPT_DSV41_BF16_INDEX_LOGITS
value: '1'
- name: SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK
value: '65536'
- name: SGLANG_DSV4_KV_LAYOUT
value: v41_new
- name: SGLANG_DP_USE_GATHERV
value: '1'
- name: SGLANG_RAGGED_VERIFY_MODE
value: static
- name: SGLANG_ENABLE_DP_SPEC_PREFILL_COORDINATION
value: '0'
- name: SGLANG_NUMA_BIND_V2
value: '0'
- name: SGLANG_DEFAULT_THINKING
value: '1'
- name: SGLANG_DSV41_REASONING_EFFORT
value: high
- name: SGLANG_TIMEOUT_KEEP_ALIVE
value: '900'
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
value: '1'
- name: MC_INTRANODE_NVLINK
value: 'true'
- name: MC_INTRA_NVLINK
value: 'true'
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
value: 'true'
- name: SGLANG_HICACHE_MOONCAKE_CONFIG_PATH
value: /data/ymk/ds41/mooncake-store-l3ssd.json
- name: SGLANG_HICACHE_PREFETCH_IO_THREADS
value: '4'
- name: SGLANG_HICACHE_BACKUP_IO_THREADS
value: '4'
- name: SGLANG_HICACHE_TIMING
value: '1'
- name: SGLANG_BATCH_TELEMETRY
value: '1'
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
value: '1'
- name: SGLANG_ENABLE_METRICS_DEVICE_TIMER
value: '1'
- name: SGLANG_EPLB_HEATMAP_COLLECTION_INTERVAL
value: '100'
- name: SGLANG_MM_TRACE
value: '1'
- name: SGLANG_BATCH_SPAN_MIN_MS
value: '200'
- name: SGLANG_BATCH_SPAN_SAMPLE
value: '0.05'
- name: SGLANG_BATCH_SPAN_MAX_LINKS
value: '8'
- name: SGLANG_BATCH_STAGE_TIMING
value: '1'
- name: SGLANG_BATCH_PASS_SPAN
value: '1'
- name: SGLANG_BATCH_PASS_SPAN_SAMPLE
value: '1.0'
- name: SGLANG_RECV_SPAN
value: '1'
- name: SGLANG_RECV_SPAN_SAMPLE
value: '1.0'
- name: SGLANG_MM_TRACE_SYNC
value: '0'
- name: SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK
value: '1'
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
value: '600'
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
value: '16'
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
value: '32'
- name: TZ
value: Asia/Shanghai
- name: SGLANG_TRACE_LOOP_SPAN
value: '1'
- name: SGLANG_TRACE_LOOP_SAMPLE
value: '1.0'
- name: SGLANG_FWD_TIMELINE
value: '0'
- name: SGLANG_TRACE_LOOP_DETAIL
value: '1'
- name: SGLANG_TRACE_MOONCAKE_DETAIL
value: '0'
- name: SGLANG_TRACE_BACKUP_DETAIL
value: '1'
- name: SGLANG_ASYNC_BACKUP_SUBMIT
value: '1'
- name: SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH
value: '1'
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
value: '1024'
- name: SGLANG_MOONCAKE_CLIENT_METRICS_PORT_BASE
value: '9400'
- name: MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR
value: bucket_storage_backend
- name: MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE
value: '1099511627776'
- name: MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES
value: '1099511627776'
- name: MOONCAKE_OFFLOAD_BUCKET_EVICTION_POLICY
value: fifo
- name: TMPDIR
value: /data/ymk/l3ssd/p-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/p-xdg
- name: HF_HOME
value: /data/ymk/l3ssd/p-hf
- name: MOONCAKE_OFFLOAD_FLUSH_PARTIAL_BUCKETS
value: '0'
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: sglang
ports:
- containerPort: 30000
name: metrics
protocol: TCP
- containerPort: 8998
name: bootstrap
protocol: TCP
- containerPort: 9400
name: mc-0
protocol: TCP
- containerPort: 9401
name: mc-1
protocol: TCP
- containerPort: 9402
name: mc-2
protocol: TCP
- containerPort: 9403
name: mc-3
protocol: TCP
readinessProbe:
failureThreshold: 120
httpGet:
path: /health
port: metrics
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '96'
memory: 2Ti
nvidia.com/gpu: '4'
requests:
cpu: '32'
memory: 512Gi
nvidia.com/gpu: '4'
securityContext:
privileged: true
volumeMounts:
- mountPath: /data
name: data
- mountPath: /root/.cache/sglang
name: sglang-cache
- mountPath: /dev/shm
name: dshm
- mountPath: /root/.cargo/bin/cargo
name: fake-cargo
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
hostPID: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/cache/sglang
type: Directory
name: sglang-cache
- emptyDir:
medium: Memory
sizeLimit: 64Gi
name: dshm
- hostPath:
path: /data/ymk/ds41/fake-cargo
type: File
name: fake-cargo
- dependencies:
- mooncake-master
minReadySeconds: 0
name: d
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '30300'
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-d
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec sglang serve \\\n --trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \\\n --reasoning-parser auto --tool-call-parser auto \\\n --default-chat-template-kwargs '{\"thinking\": true, \"reasoning_effort\": \"high\"}' \\\n --tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \\\n --enable-fp32-lm-head --enable-dp-attention-local-control-broadcast \\\n --moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \\\n --ep-num-redundant-experts 64 --ep-dispatch-algorithm dynamic \\\n --init-expert-location /data/ymk/gist-dsv41/placement_r64_ep4.json \\\n --mem-fraction-static 0.80 \\\n --speculative-algorithm DSPARK --speculative-dspark-block-size 3 \\\n --stream-interval 32 --swa-prefix-tails 2048 --enable-decoder-swa-bounded-replay \\\n --max-running-requests 512 --cuda-graph-max-bs-decode 512 --tokenizer-worker-num 16 \\\n --chunked-prefill-size 32768 --max-prefill-tokens 8192 \\\n --disaggregation-mode decode --disaggregation-transfer-backend mooncake \\\n --host 0.0.0.0 --port 30300 --enable-metrics --enable-cache-report \\\n --enable-trace --trace-modules request,mooncake \\\n --otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-d \\\n --watchdog-timeout 3600\n"
env:
- name: CUDA_VISIBLE_DEVICES
value: 4,5,6,7
- name: SGLANG_RAGGED_VERIFY_MODE
value: static
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
value: '1'
- name: SGLANG_DSV4_KV_LAYOUT
value: v41_new
- name: SGLANG_DEFAULT_THINKING
value: '1'
- name: SGLANG_DSV41_REASONING_EFFORT
value: high
- name: MC_INTRANODE_NVLINK
value: 'true'
- name: MC_INTRA_NVLINK
value: 'true'
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
value: 'true'
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
value: '1'
- name: SGLANG_BATCH_TELEMETRY
value: '1'
- name: SGLANG_BATCH_SPAN_SAMPLE
value: '0.1'
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
value: '600'
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
value: '16'
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
value: '32'
- name: TZ
value: Asia/Shanghai
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
value: '1024'
- name: TMPDIR
value: /data/ymk/l3ssd/d-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/d-xdg
- name: HF_HOME
value: /data/ymk/l3ssd/d-hf
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: sglang
ports:
- containerPort: 30300
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 150
httpGet:
path: /health
port: metrics
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '96'
memory: 550Gi
nvidia.com/gpu: '4'
requests:
cpu: '32'
memory: 256Gi
nvidia.com/gpu: '4'
securityContext:
privileged: true
volumeMounts:
- mountPath: /data
name: data
- mountPath: /root/.cache/sglang
name: sglang-cache
- mountPath: /dev/shm
name: dshm
- mountPath: /root/.cargo/bin/cargo
name: fake-cargo
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
hostPID: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/cache/sglang
type: Directory
name: sglang-cache
- emptyDir:
medium: Memory
sizeLimit: 64Gi
name: dshm
- hostPath:
path: /data/ymk/ds41/fake-cargo
type: File
name: fake-cargo
- dependencies:
- p
- d
minReadySeconds: 0
name: gateway
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: '29000'
prometheus.io/scheme: http
prometheus.io/scrape: 'true'
labels:
app: ds41-l3ssd-gw
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-type: custom-workload
spec:
containers:
- command:
- /bin/bash
- -c
- "exec /opt/sglang/bin/sgl-model-gateway launch \\\n --pd-disaggregation \\\n --prefill http://127.0.0.1:30000 8998 \\\n --decode http://127.0.0.1:30300 \\\n --prefill-policy cache_aware --decode-policy cache_aware \\\n --g9i-enabled --g9i-max-new-tokens 4096 --g9i-est-td-enabled --g9i-shadow-enabled --g9i-td-admit off --g9i-p-queue-reqs-cap 0 --g9i-p-queue-token-cap 0 --g9i-p-qps 0 --g9i-p-refresh-ms 250 --g9i-window-fixed 256 \\\n \\\n \\\n --g9i-regular-p-queue-reject-secs 0 \\\n --g9i-decode-max-running 1000000 \\\n --pool-idle-timeout-secs 2 --pool-max-idle-per-host 0 \\\n --health-check-timeout-secs 30 --health-check-interval-secs 30 \\\n --cb-failure-threshold 50 --disable-circuit-breaker --disable-health-check \\\n --model-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --tokenizer-path /data/models/DeepSeek-V4.1-Flash-1017 \\\n --host 0.0.0.0 --port 30400\n"
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: gateway
ports:
- containerPort: 30400
name: http
protocol: TCP
- containerPort: 29000
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 60
httpGet:
path: /health
port: http
initialDelaySeconds: 20
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: '16'
memory: 32Gi
requests:
cpu: '2'
memory: 8Gi
volumeMounts:
- mountPath: /data
name: data
- mountPath: /opt/sglang/bin/sgl-model-gateway
name: gw-bin
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/gw-ns-est-td/sgl-model-gateway
type: File
name: gw-bin
@@ -0,0 +1,608 @@
apiVersion: workloads.x-k8s.io/v1alpha2
kind: RoleBasedGroup
metadata:
annotations:
rbg.workloads.x-k8s.io/discovery-config-mode: refine
creationTimestamp: "2026-11-16T13:54:16Z"
generation: 6
labels:
app: ds41-gist-exp
kongming.io/history-revision: hrev-b90013fa6cc5a2a0e6515115d188135ea41e48b7
kongming.io/history-run: hrun-4ca5f278e80f5e20e03b98351874ad7cd859d5e7
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-id: cw-dblft6sgrhagmpdt8ef0
kongming.io/resource-type: custom-workload
kongming.io/user-id: usr-d7upbr3j0sq2fgdrmcr0
sglang-model: ds-v41-flash
name: ds41-gist-exp
namespace: prj-d7te67bj0sq2fgdrmbig
resourceVersion: "31040141"
uid: 242c2504-0c97-4414-ae60-8376f7d63de0
spec:
roles:
- minReadySeconds: 0
name: mooncake-master
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: "19003"
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-mooncake
kongming.io/history-revision: hrev-b90013fa6cc5a2a0e6515115d188135ea41e48b7
kongming.io/history-run: hrun-4ca5f278e80f5e20e03b98351874ad7cd859d5e7
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-id: cw-dblft6sgrhagmpdt8ef0
kongming.io/resource-type: custom-workload
kongming.io/user-id: usr-d7upbr3j0sq2fgdrmcr0
spec:
containers:
- command:
- /bin/bash
- -c
- |
# 18081: 宿主机 18080 被 AgentX fake upstream(./fakeup)长期占用;
# 我们的 P/D 用 metadata_server=P2PHANDSHAKE,此 HTTP 端口只是 master 可选项。
exec /opt/sglang/bin/mooncake_master \
--port 50051 --enable_http_metadata_server=true \
--http_metadata_server_port=18081 --metrics_port=19003 --eviction_high_watermark_ratio=0.90 --enable_offload=true --offload_on_evict=false --promotion_on_hit=false --offload_force_evict=false
env:
- name: TMPDIR
value: /data/ymk/l3ssd/master-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/master-xdg
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: mooncake-master
ports:
- containerPort: 19003
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 60
initialDelaySeconds: 5
periodSeconds: 5
tcpSocket:
port: 50051
resources:
limits:
cpu: "4"
memory: 64Gi
requests:
cpu: "1"
memory: 8Gi
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
- dependencies:
- mooncake-master
minReadySeconds: 0
name: p
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: "30000"
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-p
kongming.io/history-revision: hrev-b90013fa6cc5a2a0e6515115d188135ea41e48b7
kongming.io/history-run: hrun-4ca5f278e80f5e20e03b98351874ad7cd859d5e7
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-id: cw-dblft6sgrhagmpdt8ef0
kongming.io/resource-type: custom-workload
kongming.io/user-id: usr-d7upbr3j0sq2fgdrmcr0
spec:
containers:
- command:
- /bin/bash
- -c
- |
exec sglang serve \
--trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \
--reasoning-parser auto --tool-call-parser auto \
--default-chat-template-kwargs '{"thinking": true, "reasoning_effort": "high"}' \
--tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \
--moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \
--ep-num-redundant-experts 32 --ep-dispatch-algorithm dynamic \
--init-expert-location /data/ymk/gist-dsv41/placement_r32_ep4.json \
--mem-fraction-static 0.80 \
--chunked-prefill-size 131072 --max-prefill-tokens 131072 \
--max-running-requests 512 --tokenizer-worker-num 16 \
--load-balance-method total_tokens \
--speculative-algorithm DSPARK --speculative-dspark-block-size 3 \
--enable-decoder-swa-bounded-replay --swa-full-tokens-ratio 0.12 \
--cuda-graph-max-bs-decode 1 --disable-decode-cuda-graph \
--cuda-graph-backend-prefill breakable \
--cuda-graph-bs-prefill 1024 2048 4096 8192 16384 32768 \
--cuda-graph-prefill-max-context 262144 --cuda-graph-max-seq-len-prefill 262144 \
--enable-hierarchical-cache --hicache-ratio 4 \
--hicache-mem-layout page_first_direct --hicache-io-backend direct \
--hicache-write-policy write_through --hicache-storage-backend mooncake \
--hicache-storage-prefetch-policy wait_complete \
--disaggregation-mode prefill --disaggregation-transfer-backend mooncake \
--disaggregation-bootstrap-port 8998 \
--expert-balancedness-report-mode prometheus \
--expert-distribution-recorder-buffer-size 64 \
--host 0.0.0.0 --port 30000 --enable-metrics --enable-cache-report \
--enable-trace --trace-modules request,mooncake \
--otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-p \
--watchdog-timeout 3600 --enable-dp-attention-local-control-broadcast
env:
- name: CUDA_VISIBLE_DEVICES
value: 0,1,2,3
- name: SGLANG_LOG_MS
value: "1"
- name: SGLANG_OPT_DSV41_ENGRAM_PEER_SHARDS
value: "1"
- name: SGLANG_OPT_DSV41_MEGA_MHC_PREFILL
value: "1"
- name: SGLANG_OPT_DSV41_BF16_INDEX_LOGITS
value: "1"
- name: SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK
value: "65536"
- name: SGLANG_DSV4_KV_LAYOUT
value: v41_new
- name: SGLANG_DP_USE_GATHERV
value: "1"
- name: SGLANG_RAGGED_VERIFY_MODE
value: static
- name: SGLANG_ENABLE_DP_SPEC_PREFILL_COORDINATION
value: "0"
- name: SGLANG_NUMA_BIND_V2
value: "0"
- name: SGLANG_DEFAULT_THINKING
value: "1"
- name: SGLANG_DSV41_REASONING_EFFORT
value: high
- name: SGLANG_TIMEOUT_KEEP_ALIVE
value: "900"
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
value: "1"
- name: MC_INTRANODE_NVLINK
value: "true"
- name: MC_INTRA_NVLINK
value: "true"
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
value: "true"
- name: SGLANG_HICACHE_MOONCAKE_CONFIG_PATH
value: /data/ymk/ds41/mooncake-store-l3ssd.json
- name: SGLANG_HICACHE_PREFETCH_IO_THREADS
value: "4"
- name: SGLANG_HICACHE_BACKUP_IO_THREADS
value: "4"
- name: SGLANG_HICACHE_TIMING
value: "1"
- name: SGLANG_BATCH_TELEMETRY
value: "1"
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
value: "1"
- name: SGLANG_ENABLE_METRICS_DEVICE_TIMER
value: "1"
- name: SGLANG_EPLB_HEATMAP_COLLECTION_INTERVAL
value: "100"
- name: SGLANG_MM_TRACE
value: "1"
- name: SGLANG_BATCH_SPAN_MIN_MS
value: "200"
- name: SGLANG_BATCH_SPAN_SAMPLE
value: "0.05"
- name: SGLANG_BATCH_SPAN_MAX_LINKS
value: "8"
- name: SGLANG_BATCH_STAGE_TIMING
value: "1"
- name: SGLANG_BATCH_PASS_SPAN
value: "1"
- name: SGLANG_BATCH_PASS_SPAN_SAMPLE
value: "1.0"
- name: SGLANG_RECV_SPAN
value: "1"
- name: SGLANG_RECV_SPAN_SAMPLE
value: "1.0"
- name: SGLANG_MM_TRACE_SYNC
value: "0"
- name: SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK
value: "1"
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
value: "600"
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
value: "16"
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
value: "32"
- name: TZ
value: Asia/Shanghai
- name: SGLANG_TRACE_LOOP_SPAN
value: "1"
- name: SGLANG_TRACE_LOOP_SAMPLE
value: "1.0"
- name: SGLANG_FWD_TIMELINE
value: "0"
- name: SGLANG_TRACE_LOOP_DETAIL
value: "1"
- name: SGLANG_TRACE_MOONCAKE_DETAIL
value: "0"
- name: SGLANG_TRACE_BACKUP_DETAIL
value: "1"
- name: SGLANG_ASYNC_BACKUP_SUBMIT
value: "1"
- name: SGLANG_NCCL_ALL_GATHER_IN_OVERLAP_SCHEDULER_SYNC_BATCH
value: "1"
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
value: "1024"
- name: SGLANG_MOONCAKE_CLIENT_METRICS_PORT_BASE
value: "9400"
- name: MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR
value: bucket_storage_backend
- name: MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE
value: "1099511627776"
- name: MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES
value: "1099511627776"
- name: MOONCAKE_OFFLOAD_BUCKET_EVICTION_POLICY
value: fifo
- name: TMPDIR
value: /data/ymk/l3ssd/p-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/p-xdg
- name: HF_HOME
value: /data/ymk/l3ssd/p-hf
- name: MOONCAKE_OFFLOAD_FLUSH_PARTIAL_BUCKETS
value: "0"
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: sglang
ports:
- containerPort: 30000
name: metrics
protocol: TCP
- containerPort: 8998
name: bootstrap
protocol: TCP
- containerPort: 9400
name: mc-0
protocol: TCP
- containerPort: 9401
name: mc-1
protocol: TCP
- containerPort: 9402
name: mc-2
protocol: TCP
- containerPort: 9403
name: mc-3
protocol: TCP
readinessProbe:
failureThreshold: 120
httpGet:
path: /health
port: metrics
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: "96"
memory: 2Ti
nvidia.com/gpu: "4"
requests:
cpu: "32"
memory: 512Gi
nvidia.com/gpu: "4"
securityContext:
privileged: true
volumeMounts:
- mountPath: /data
name: data
- mountPath: /root/.cache/sglang
name: sglang-cache
- mountPath: /dev/shm
name: dshm
- mountPath: /root/.cargo/bin/cargo
name: fake-cargo
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
hostPID: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/cache/sglang
type: Directory
name: sglang-cache
- emptyDir:
medium: Memory
sizeLimit: 64Gi
name: dshm
- hostPath:
path: /data/ymk/ds41/fake-cargo
type: File
name: fake-cargo
- dependencies:
- mooncake-master
minReadySeconds: 0
name: d
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: "30300"
prometheus.io/scheme: http
labels:
app: ds41-l3ssd-d
kongming.io/history-revision: hrev-b90013fa6cc5a2a0e6515115d188135ea41e48b7
kongming.io/history-run: hrun-4ca5f278e80f5e20e03b98351874ad7cd859d5e7
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-id: cw-dblft6sgrhagmpdt8ef0
kongming.io/resource-type: custom-workload
kongming.io/user-id: usr-d7upbr3j0sq2fgdrmcr0
spec:
containers:
- command:
- /bin/bash
- -c
- |
exec sglang serve \
--trust-remote-code --tokenizer-backend fastokens --model-path /data/models/DeepSeek-V4.1-Flash-1017 --served-model-name /data/models/DeepSeek-V4.1-Flash \
--reasoning-parser auto --tool-call-parser auto \
--default-chat-template-kwargs '{"thinking": true, "reasoning_effort": "high"}' \
--tp 4 --dp-size 4 --ep-size 4 --enable-dp-attention --enable-dp-lm-head \
--enable-fp32-lm-head --enable-dp-attention-local-control-broadcast \
--moe-a2a-backend megamoe --dsv4-attn-backend flashmla_mega \
--ep-num-redundant-experts 64 --ep-dispatch-algorithm dynamic \
--init-expert-location /data/ymk/gist-dsv41/placement_r64_ep4.json \
--mem-fraction-static 0.80 \
--speculative-algorithm DSPARK --speculative-dspark-block-size 3 \
--stream-interval 32 --swa-prefix-tails 2048 --enable-decoder-swa-bounded-replay \
--max-running-requests 512 --cuda-graph-max-bs-decode 512 --tokenizer-worker-num 16 \
--chunked-prefill-size 32768 --max-prefill-tokens 8192 \
--disaggregation-mode decode --disaggregation-transfer-backend mooncake \
--host 0.0.0.0 --port 30300 --enable-metrics --enable-cache-report \
--enable-trace --trace-modules request,mooncake \
--otlp-traces-endpoint 127.0.0.1:4317 --otlp-service-name gist-exp-d \
--watchdog-timeout 3600
env:
- name: CUDA_VISIBLE_DEVICES
value: 4,5,6,7
- name: SGLANG_RAGGED_VERIFY_MODE
value: static
- name: SGLANG_ENABLE_DSV41_ENGRAM_HOST_TABLE
value: "1"
- name: SGLANG_DSV4_KV_LAYOUT
value: v41_new
- name: SGLANG_DEFAULT_THINKING
value: "1"
- name: SGLANG_DSV41_REASONING_EFFORT
value: high
- name: MC_INTRANODE_NVLINK
value: "true"
- name: MC_INTRA_NVLINK
value: "true"
- name: SGLANG_MOONCAKE_SEND_AUX_TCP
value: "true"
- name: SGLANG_ENABLE_METRICS_DP_ATTENTION
value: "1"
- name: SGLANG_BATCH_TELEMETRY
value: "1"
- name: SGLANG_BATCH_SPAN_SAMPLE
value: "0.1"
- name: SGLANG_DISAGGREGATION_WAITING_TIMEOUT
value: "600"
- name: SGLANG_DISAGGREGATION_QUEUE_SIZE
value: "16"
- name: SGLANG_DISAGGREGATION_THREAD_POOL_SIZE
value: "32"
- name: TZ
value: Asia/Shanghai
- name: OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT
value: "1024"
- name: TMPDIR
value: /data/ymk/l3ssd/d-tmp
- name: XDG_CACHE_HOME
value: /data/ymk/l3ssd/d-xdg
- name: HF_HOME
value: /data/ymk/l3ssd/d-hf
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: sglang
ports:
- containerPort: 30300
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 150
httpGet:
path: /health
port: metrics
initialDelaySeconds: 60
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: "96"
memory: 550Gi
nvidia.com/gpu: "4"
requests:
cpu: "32"
memory: 256Gi
nvidia.com/gpu: "4"
securityContext:
privileged: true
volumeMounts:
- mountPath: /data
name: data
- mountPath: /root/.cache/sglang
name: sglang-cache
- mountPath: /dev/shm
name: dshm
- mountPath: /root/.cargo/bin/cargo
name: fake-cargo
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
hostPID: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/cache/sglang
type: Directory
name: sglang-cache
- emptyDir:
medium: Memory
sizeLimit: 64Gi
name: dshm
- hostPath:
path: /data/ymk/ds41/fake-cargo
type: File
name: fake-cargo
- dependencies:
- p
- d
minReadySeconds: 0
name: gateway
podManagementPolicy: Parallel
replicas: 1
standalonePattern:
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: "29000"
prometheus.io/scheme: http
prometheus.io/scrape: "true"
labels:
app: ds41-l3ssd-gw
kongming.io/history-revision: hrev-b90013fa6cc5a2a0e6515115d188135ea41e48b7
kongming.io/history-run: hrun-4ca5f278e80f5e20e03b98351874ad7cd859d5e7
kongming.io/managed-by: kongming
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kongming.io/resource-id: cw-dblft6sgrhagmpdt8ef0
kongming.io/resource-type: custom-workload
kongming.io/user-id: usr-d7upbr3j0sq2fgdrmcr0
spec:
containers:
- command:
- /bin/bash
- -c
- |
exec /opt/sglang/bin/sgl-model-gateway launch \
--pd-disaggregation \
--prefill http://127.0.0.1:30000 8998 \
--decode http://127.0.0.1:30300 \
--prefill-policy cache_aware --decode-policy cache_aware \
--g9i-enabled --g9i-max-new-tokens 4096 --g9i-est-td-enabled --g9i-shadow-enabled --g9i-td-admit off --g9i-p-queue-reqs-cap 0 --g9i-p-queue-token-cap 0 --g9i-p-qps 0 --g9i-p-refresh-ms 250 --g9i-window-fixed 256 \
\
\
--g9i-regular-p-queue-reject-secs 0 \
--g9i-decode-max-running 1000000 \
--pool-idle-timeout-secs 2 --pool-max-idle-per-host 0 \
--health-check-timeout-secs 30 --health-check-interval-secs 30 \
--cb-failure-threshold 50 --disable-circuit-breaker --disable-health-check \
--model-path /data/models/DeepSeek-V4.1-Flash-1017 \
--tokenizer-path /data/models/DeepSeek-V4.1-Flash-1017 \
--host 0.0.0.0 --port 30400
image: sealos.hub:5000/prj-d7te67bj0sq2fgdrmbig/sglang:recipe-ns-l3ssd-433257c7b-baked-20261115
imagePullPolicy: IfNotPresent
name: gateway
ports:
- containerPort: 30400
name: http
protocol: TCP
- containerPort: 29000
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 60
httpGet:
path: /health
port: http
initialDelaySeconds: 20
periodSeconds: 10
timeoutSeconds: 5
resources:
limits:
cpu: "16"
memory: 32Gi
requests:
cpu: "2"
memory: 8Gi
volumeMounts:
- mountPath: /data
name: data
- mountPath: /opt/sglang/bin/sgl-model-gateway
name: gw-bin
readOnly: true
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
nodeSelector:
kongming.io/node-group-id: ng-damg6d4grhalvvbgbbrg
kongming.io/project-id: prj-d7te67bj0sq2fgdrmbig
kubernetes.io/hostname: b300-01
volumes:
- hostPath:
path: /data
type: Directory
name: data
- hostPath:
path: /data/ymk/gw-ns-est-td/sgl-model-gateway
type: File
name: gw-bin
status:
conditions:
- lastTransitionTime: "2026-11-17T01:58:12Z"
message: All roles are ready
observedGeneration: 6
reason: AllRolesReady
status: "True"
type: Ready
observedGeneration: 6
roleStatuses:
- name: mooncake-master
readyReplicas: 1
replicas: 1
updatedReplicas: 1
- name: p
readyReplicas: 1
replicas: 1
updatedReplicas: 1
- name: d
readyReplicas: 1
replicas: 1
updatedReplicas: 1
- name: gateway
readyReplicas: 1
replicas: 1
updatedReplicas: 1
@@ -98,6 +98,7 @@ class Parallel(msgspec.Struct):
"follow_bootstrap_room",
"total_requests",
"total_tokens",
"uncached_tokens",
],
),
] = "auto"
@@ -51,6 +51,7 @@ from sglang.srt.disaggregation.utils import (
)
from sglang.srt.environ import envs
from sglang.srt.mem_cache.page_interleave import get_kv_shard_group
from sglang.srt.observability import batch_telemetry
from sglang.srt.observability.trace import TraceNullContext, TraceReqContext
from sglang.srt.runtime_context import (
get_device,
@@ -2091,6 +2092,10 @@ class CommonKVSender(BaseKVSender):
self._transfer_num_state_indices = 0
self._transfer_start_time: Optional[float] = None
self._early_send_wait_event: Optional[object] = None
# Detail tracing: chunk classification only. Enqueue timestamps,
# chunk_seq and the per-request first-enqueue are assigned by the
# manager at the real queue.put() point, after all early returns.
self._send_kind = ""
self.trace_ctx = TraceNullContext()
if self.kv_mgr.sender_trace_module and get_observability().enable_trace:
self.trace_ctx = TraceReqContext(
@@ -2289,6 +2294,19 @@ class CommonKVSender(BaseKVSender):
):
self._transfer_start_time = time.perf_counter()
obs = None
if batch_telemetry.mooncake_detail_enabled():
# Metadata only; the manager stamps chunk_seq and enqueue times
# when it is certain the chunk is actually put on the queue.
obs = {
"loop_seq": int(batch_telemetry.current_loop_seq()),
"send_kind": str(self._send_kind or ""),
"num_pages": int(len(kv_indices)),
"approx_bytes": int(
len(kv_indices) * getattr(self.kv_mgr, "kv_item_lens_sum", 0)
),
}
# The early-send event belongs to exactly one chunk; the Mooncake and
# Mori workers synchronize on it before reading source KV.
wait_event = self._early_send_wait_event
@@ -2304,9 +2322,27 @@ class CommonKVSender(BaseKVSender):
wait_event=wait_event,
indices_event=indices_event,
trace_ctx=self.trace_ctx.copy_for_thread(),
obs=obs,
)
self._record_transfer_indices(kv_indices, state_indices)
def pop_chunk_join_stats(self) -> Optional[dict]:
"""Take (and free) this room's chunk join summary."""
state = getattr(self.kv_mgr, "_room_detail", None)
if state is None:
return None
return state.pop_join(self.bootstrap_room)
def reset_chunk_detail_state(self) -> None:
"""Drop per-room detail state and mark it cancelled (abort / retract).
The mark is what prevents a worker attempt that already passed its
status check from recreating stats after this cleanup.
"""
state = getattr(self.kv_mgr, "_room_detail", None)
if state is not None:
state.reset(self.bootstrap_room)
def poll(self) -> KVPoll:
return self._poll_with_status(self.kv_mgr.check_status)
@@ -2425,6 +2461,9 @@ class CommonKVSender(BaseKVSender):
def abort(self):
if self.kv_mgr.deferred_bootstrap is not None and not self._owns_bootstrap_room:
return
# Drop per-room detail bookkeeping: an aborted request never reaches
# the poll path that normally consumes it (review finding).
self.reset_chunk_detail_state()
self.kv_mgr.record_failure(
self.bootstrap_room,
"Aborted by AbortReq.",
@@ -36,6 +36,24 @@ class TransferKVChunk:
# Must complete before reading asynchronously copied host indices.
indices_event: Optional[object] = None
# --- Optional observation fields (SGLANG_TRACE_MOONCAKE_DETAIL=1) ------
# Filled at the real enqueue point (add_transfer_request), never at the
# send_kv_chunk entry, so skipped paths cannot create holes in chunk_seq.
chunk_seq: int = 0 # per-sender monotonic enqueue index
loop_seq: int = 0 # scheduler loop that produced this chunk
send_kind: str = "" # cached_prefix / middle / last
enqueue_mono_ns: int = 0 # first enqueue of this chunk, monotonic clock
enqueue_wall_ns: int = 0 # first enqueue of this chunk, wall clock
last_enqueue_mono_ns: int = 0 # updated on every (re-)enqueue
last_enqueue_wall_ns: int = 0 # wall clock of the last (re-)enqueue
dequeue_mono_ns: int = 0 # current attempt dequeue; 0 outside the worker
queue_wait_accum_ns: int = 0 # completed queue waits from earlier attempts
queue_id: int = -1
requeue_count: int = 0
num_pages: int = 0
approx_bytes: int = 0
observe: bool = False # True when detail tracing populated this chunk
def pack_list_of_buffers(buffers: List[bytes]) -> bytes:
if not buffers:
@@ -60,6 +60,7 @@ from sglang.srt.disaggregation.utils import (
)
from sglang.srt.distributed.parallel_state import get_mooncake_transfer_engine
from sglang.srt.environ import envs
from sglang.srt.observability import batch_telemetry
from sglang.srt.observability.mooncake_trace import (
MooncakeRequestStage,
mooncake_trace_func,
@@ -356,6 +357,28 @@ def _uses_flat_entry_layout(
return mla_backend or force_flat or has_layer_ids or is_single_region_swa
def materialize_state_indices(state_indices, indices_event):
"""Convert pinned state tensors to numpy after the event has completed.
``indices_event`` must already be synchronized by the caller. A tensor
without an event is a configuration/implementation error: reading it would
race the scheduler's non-blocking D2H copy.
"""
if not state_indices:
return state_indices
import torch
if not any(isinstance(t, torch.Tensor) for t in state_indices):
return state_indices
if indices_event is None:
raise RuntimeError(
"async state indices require an indices_event before materialization"
)
return [
t.numpy() if isinstance(t, torch.Tensor) else t for t in state_indices
]
class MooncakeKVManager(StagingManagerMixin, CommonKVManager):
sender_trace_module = "mooncake"
@@ -386,6 +409,10 @@ class MooncakeKVManager(StagingManagerMixin, CommonKVManager):
# transfer blocks over zmq when the intra-node NVLink transport is
# active (it cannot reach host memory). Populated on bootstrap.
self._session_endpoint_map = {}
# Detail tracing (SGLANG_TRACE_MOONCAKE_DETAIL=1; empty otherwise).
# Lock-protected per-room state: abort/retract and the worker's join
# update must not interleave into resurrected stats.
self._room_detail = batch_telemetry.RoomDetailState()
self.enable_staging = envs.SGLANG_DISAGG_STAGING_BUFFER.get()
self.max_transfer_batch_indices = (
envs.SGLANG_MOONCAKE_MAX_TRANSFER_BATCH_INDICES.get()
@@ -762,6 +789,10 @@ class MooncakeKVManager(StagingManagerMixin, CommonKVManager):
# block other rooms on this single worker thread.
with self._staging_ctx.watermark_cv:
self._staging_ctx.watermark_cv.wait(STAGING_WATERMARK_WAIT_S)
if getattr(kv_chunk, "observe", False):
batch_telemetry.note_chunk_requeue(
kv_chunk, time.monotonic_ns(), time.time_ns()
)
queue.put(kv_chunk)
return (-1, True)
@@ -2716,6 +2747,65 @@ class MooncakeKVManager(StagingManagerMixin, CommonKVManager):
if self.enable_deferred_decode_kv_release:
self.poison_deferred_ack_room(room)
def _emit_chunk_attempt(
self,
kv_chunk: TransferKVChunk,
*,
status: str,
queue_wait_ns: int,
queue_wait_total_ns: int,
source_wait_ns: int,
indices_wait_ns: int,
kv_transfer_ns: int,
state_transfer_ns: int,
aux_transfer_ns: int,
work_start_ns: int,
dequeue_wall_ns: int,
n_destinations: int,
source_wait_wall=None,
indices_wait_wall=None,
) -> None:
"""Close one worker attempt: chunk span + per-room join summary."""
now_mono_ns = time.monotonic_ns()
now_wall_ns = time.time_ns()
attempt_wall_ns = max(0, now_mono_ns - int(work_start_ns))
worker_busy_ns = max(
0, attempt_wall_ns - int(source_wait_ns) - int(indices_wait_ns)
)
batch_telemetry.emit_mooncake_chunk(
chunk=kv_chunk,
status=status,
queue_wait_ns=queue_wait_ns,
queue_wait_total_ns=queue_wait_total_ns,
source_wait_ns=source_wait_ns,
indices_wait_ns=indices_wait_ns,
kv_transfer_ns=kv_transfer_ns,
state_transfer_ns=state_transfer_ns,
aux_transfer_ns=aux_transfer_ns,
attempt_wall_ns=attempt_wall_ns,
dequeue_wall_ns=dequeue_wall_ns,
end_wall_ns=now_wall_ns,
n_destinations=n_destinations,
source_wait_wall=source_wait_wall,
indices_wait_wall=indices_wait_wall,
dp_rank=self.attn_dp_rank,
tp_rank=self.attn_tp_rank,
)
# The status was checked before this call; the registry re-checks the
# cancellation mark under its lock, which is what makes an abort that
# lands in between unable to resurrect stats (review finding).
self._room_detail.record_join(
kv_chunk.room,
status=status,
room_failed=self.check_status(kv_chunk.room) == KVPoll.Failed,
now_wall_ns=now_wall_ns,
queue_wait_total_ns=queue_wait_total_ns,
worker_busy_ns=worker_busy_ns,
enqueue_wall_ns=kv_chunk.enqueue_wall_ns,
is_last_chunk=kv_chunk.is_last_chunk,
chunk_seq=kv_chunk.chunk_seq,
)
def transfer_worker(
self,
queue: FastQueue,
@@ -2740,6 +2830,34 @@ class MooncakeKVManager(StagingManagerMixin, CommonKVManager):
# _stopped alone can never wake a parked worker during a role switch.
if kv_chunk is None:
break
_obs = bool(getattr(kv_chunk, "observe", False))
_dequeue_mono_ns = time.monotonic_ns() if _obs else 0
_dequeue_wall_ns = time.time_ns() if _obs else 0
if _obs:
# Needed by note_chunk_requeue(): queue wait ends here and
# must not include worker/watermark time.
kv_chunk.dequeue_mono_ns = _dequeue_mono_ns
_queue_wait_ns = 0
_source_wait_ns = 0
_source_wait_wall = None
_indices_wait_ns = 0
_indices_wait_wall = None
_kv_transfer_ns = 0
_state_transfer_ns = 0
_aux_transfer_ns = 0
_n_dest = 0
if _obs:
# queue wait is not part of worker busy time (design §5.2).
# Keep the attempt window (child span) separate from the
# cumulative value across deferred attempts (review finding).
(
_attempt_queue_wait_ns,
_queue_wait_ns,
) = batch_telemetry.split_queue_wait(
kv_chunk.queue_wait_accum_ns,
kv_chunk.last_enqueue_mono_ns,
_dequeue_mono_ns,
)
if self.enable_trace:
kv_chunk.trace_ctx.rebuild_thread_context()
kv_chunk.trace_ctx.trace_slice_start(
@@ -2765,13 +2883,32 @@ class MooncakeKVManager(StagingManagerMixin, CommonKVManager):
kv_chunk.room in self.request_status
and self.check_status(kv_chunk.room) != KVPoll.Failed
):
_wait_t0 = time.monotonic_ns() if _obs else 0
_wait_wall0 = time.time_ns() if _obs else 0
kv_chunk.wait_event.synchronize()
if _wait_t0:
_source_wait_ns += time.monotonic_ns() - _wait_t0
_wait_wall1 = time.time_ns()
_source_wait_wall = (_wait_wall0, _wait_wall1)
kv_chunk.wait_event = None
if kv_chunk.indices_event is not None:
kv_chunk.indices_event.synchronize()
_ind_t0 = time.monotonic_ns() if _obs else 0
_ind_wall0 = time.time_ns() if _obs else 0
indices_event = kv_chunk.indices_event
if indices_event is not None:
indices_event.synchronize()
kv_chunk.indices_event = None
if isinstance(kv_chunk.prefill_kv_indices, torch.Tensor):
kv_chunk.prefill_kv_indices = kv_chunk.prefill_kv_indices.numpy()
# First async-state version: the pinned SWA state tensor is
# materialized only after the event above. Raises if a tensor
# arrives without one (would race the scheduler's D2H).
kv_chunk.state_indices = materialize_state_indices(
kv_chunk.state_indices, indices_event
)
if _ind_t0:
# synchronize + host indices availability, per design §5.2.
_indices_wait_ns += time.monotonic_ns() - _ind_t0
_indices_wait_wall = (_ind_wall0, time.time_ns())
if (
kv_chunk.room not in self.request_status
@@ -2787,6 +2924,23 @@ class MooncakeKVManager(StagingManagerMixin, CommonKVManager):
thread_finish_flag=True,
)
self._staging_outstanding.pop(kv_chunk.room, None)
if _obs:
self._emit_chunk_attempt(
kv_chunk,
status="skipped",
queue_wait_ns=_attempt_queue_wait_ns,
queue_wait_total_ns=_queue_wait_ns,
source_wait_ns=_source_wait_ns,
indices_wait_ns=_indices_wait_ns,
kv_transfer_ns=_kv_transfer_ns,
state_transfer_ns=_state_transfer_ns,
aux_transfer_ns=_aux_transfer_ns,
work_start_ns=_dequeue_mono_ns,
dequeue_wall_ns=_dequeue_wall_ns,
n_destinations=_n_dest,
source_wait_wall=_source_wait_wall,
indices_wait_wall=_indices_wait_wall,
)
if self.enable_deferred_decode_kv_release:
# Skipped => nothing written for this aborted room; ack.
self._maybe_ack_drained_abort(kv_chunk.room)
@@ -2882,6 +3036,7 @@ class MooncakeKVManager(StagingManagerMixin, CommonKVManager):
skip_kv, skip_state = self._get_dsa_cache_transfer_skip_flags(
target_rank_registration_info
)
_kv_t0 = time.monotonic_ns() if _obs else 0
if (
len(kv_chunk.prefill_kv_indices) == 0
or not self.kv_args.kv_data_ptrs
@@ -2954,6 +3109,9 @@ class MooncakeKVManager(StagingManagerMixin, CommonKVManager):
)
if deferred:
staging_deferred = True
if _kv_t0:
_kv_transfer_ns += time.monotonic_ns() - _kv_t0
_n_dest += 1
# Chunk re-enqueued; stop processing remaining reqs for this chunk
break
else:
@@ -2968,6 +3126,9 @@ class MooncakeKVManager(StagingManagerMixin, CommonKVManager):
executor,
target_rank_registration_info.dst_kv_layer_ids,
)
if _kv_t0:
_kv_transfer_ns += time.monotonic_ns() - _kv_t0
_n_dest += 1
if ret != 0:
with self.session_lock:
self.session_failures[req.mooncake_session_id] += 1
@@ -2989,12 +3150,15 @@ class MooncakeKVManager(StagingManagerMixin, CommonKVManager):
if kv_chunk.is_last_chunk:
if kv_chunk.state_indices and not skip_state:
_state_t0 = time.monotonic_ns() if _obs else 0
state_rc = self.maybe_send_extra(
req,
kv_chunk.state_indices,
executor,
target_rank_registration_info,
)
if _state_t0:
_state_transfer_ns += time.monotonic_ns() - _state_t0
# Before the Failed check: an ABORT during the
# send also marks the room Failed.
if state_rc != 0:
@@ -3025,11 +3189,14 @@ class MooncakeKVManager(StagingManagerMixin, CommonKVManager):
break
# Only the last chunk we need to send the aux data
_aux_t0 = time.monotonic_ns() if _obs else 0
ret = self.send_aux(
req,
kv_chunk.prefill_aux_index,
target_rank_registration_info.dst_aux_ptrs,
)
if _aux_t0:
_aux_transfer_ns += time.monotonic_ns() - _aux_t0
if ret != 0:
# The aux write targets decode's metadata
# slot, which the deferred release also frees.
@@ -3071,8 +3238,45 @@ class MooncakeKVManager(StagingManagerMixin, CommonKVManager):
)
if staging_deferred:
if _obs:
self._emit_chunk_attempt(
kv_chunk,
status="deferred",
queue_wait_ns=_attempt_queue_wait_ns,
queue_wait_total_ns=_queue_wait_ns,
source_wait_ns=_source_wait_ns,
indices_wait_ns=_indices_wait_ns,
kv_transfer_ns=_kv_transfer_ns,
state_transfer_ns=_state_transfer_ns,
aux_transfer_ns=_aux_transfer_ns,
work_start_ns=_dequeue_mono_ns,
dequeue_wall_ns=_dequeue_wall_ns,
n_destinations=_n_dest,
source_wait_wall=_source_wait_wall,
indices_wait_wall=_indices_wait_wall,
)
continue
if _obs:
self._emit_chunk_attempt(
kv_chunk,
status="ok"
if self.check_status(kv_chunk.room) != KVPoll.Failed
else "failed",
queue_wait_ns=_attempt_queue_wait_ns,
queue_wait_total_ns=_queue_wait_ns,
source_wait_ns=_source_wait_ns,
indices_wait_ns=_indices_wait_ns,
kv_transfer_ns=_kv_transfer_ns,
state_transfer_ns=_state_transfer_ns,
aux_transfer_ns=_aux_transfer_ns,
work_start_ns=_dequeue_mono_ns,
dequeue_wall_ns=_dequeue_wall_ns,
n_destinations=_n_dest,
source_wait_wall=_source_wait_wall,
indices_wait_wall=_indices_wait_wall,
)
self._staging_outstanding[kv_chunk.room] -= 1
if self.enable_deferred_decode_kv_release:
# In-flight write finished; if aborted and nothing outstanding,
@@ -3096,6 +3300,15 @@ class MooncakeKVManager(StagingManagerMixin, CommonKVManager):
self._session_endpoint_map.pop(sid, None)
self.transfer_infos.pop(kv_chunk.room)
self.req_to_decode_prefix_len.pop(kv_chunk.room, None)
# Success: the scheduler poll still needs the join summary.
# Failed/absent: nothing will consume it, drop it here.
self._room_detail.teardown(
kv_chunk.room,
keep_join=(
self.request_status.get(kv_chunk.room)
== KVPoll.Success
),
)
if self.enable_staging:
# Purge prefetch bookkeeping for the finished room.
# Snapshot first: the scheduler thread adds concurrently.
@@ -3332,6 +3545,7 @@ class MooncakeKVManager(StagingManagerMixin, CommonKVManager):
wait_event: Optional[object] = None,
indices_event: Optional[object] = None,
trace_ctx: Optional[Union[TraceReqContext, TraceNullContext]] = None,
obs: Optional[dict] = None,
):
assert self.disaggregation_mode == DisaggregationMode.PREFILL
assert not is_last_chunk or (is_last_chunk and aux_index is not None)
@@ -3361,6 +3575,23 @@ class MooncakeKVManager(StagingManagerMixin, CommonKVManager):
if trace_ctx is None:
trace_ctx = TraceNullContext()
if obs is not None:
# Stamp only now: every earlier return (failed room, dummy rank,
# zero-page skip) must not consume a chunk_seq or a first-enqueue
# timestamp (review finding).
now_mono_ns = time.monotonic_ns()
now_wall_ns = time.time_ns()
chunk_seq, _first_wall_ns = self._room_detail.begin_chunk(
bootstrap_room, now_mono_ns, now_wall_ns
)
obs = dict(obs)
obs.update(
chunk_seq=chunk_seq,
enqueue_mono_ns=now_mono_ns,
enqueue_wall_ns=now_wall_ns,
last_enqueue_wall_ns=now_wall_ns,
)
self.transfer_queues[shard_idx].put(
TransferKVChunk(
room=bootstrap_room,
@@ -3373,6 +3604,10 @@ class MooncakeKVManager(StagingManagerMixin, CommonKVManager):
trace_ctx=trace_ctx,
wait_event=wait_event,
indices_event=indices_event,
**(obs or {}),
queue_id=int(shard_idx),
observe=obs is not None,
last_enqueue_mono_ns=int((obs or {}).get("enqueue_mono_ns", 0)),
)
)
@@ -3551,6 +3786,10 @@ class MooncakeKVManager(StagingManagerMixin, CommonKVManager):
class MooncakeKVSender(CommonKVSender):
supports_host_tensor_indices = True
# First async state-payload version (SWA-only): the scheduler copies the
# SWA state indices to a pinned buffer and the worker materializes them
# after kv_chunk.indices_event.
supports_async_state_indices = True
@mooncake_trace_func(MooncakeRequestStage.MOONCAKE_SEND)
def send(
@@ -804,6 +804,7 @@ class MoriKVManager(CommonKVManager):
wait_event: Optional[object] = None,
indices_event: Optional[object] = None,
trace_ctx=None,
obs: Optional[dict] = None,
) -> None:
assert self.disaggregation_mode == DisaggregationMode.PREFILL
assert not is_last_chunk or (is_last_chunk and aux_index is not None)
@@ -3267,6 +3267,7 @@ class NixlKVManager(StagingManagerMixin, CommonKVManager):
wait_event=None,
indices_event=None,
trace_ctx=None,
obs: Optional[dict] = None,
):
assert self.disaggregation_mode == DisaggregationMode.PREFILL
assert not is_last_chunk or (is_last_chunk and aux_index is not None)
+404 -14
View File
@@ -77,6 +77,7 @@ from sglang.srt.mem_cache.base_prefix_cache import CacheRequestOutcome
from sglang.srt.mem_cache.common import (
checkpoint_kv_cache,
kv_to_page_indices,
kv_to_page_indices_pinned,
kv_to_page_num,
release_kv_cache,
suffix_page_indices,
@@ -84,6 +85,7 @@ from sglang.srt.mem_cache.common import (
from sglang.srt.mem_cache.deepseek_v4_memory_pool import DeepSeekV4TokenToKVPool
from sglang.srt.mem_cache.memory_pool import HybridLinearKVPool
from sglang.srt.mem_cache.unified_radix_cache import UnifiedRadixCache
from sglang.srt.observability import batch_telemetry
from sglang.srt.observability.req_time_stats import set_schedule_time_batch
from sglang.srt.observability.scheduler_stage_metrics import (
SCHEDULER_STAGE_GET_NEXT_BATCH,
@@ -563,6 +565,40 @@ class PrefillBootstrapQueue:
self.kv_manager.register_buffer_to_engine()
def should_async_state_payload(
*,
last_chunk: bool,
supports_async_state_indices: bool,
kv_shard_size: int,
enable_all_cp_ranks_for_transfer: bool,
enable_dsa_cache_layer_split: bool,
) -> bool:
"""Gate for the first async state-payload version (SWA-only).
``kv_shard_size > 1`` and all-CP-rank transfer **without DSA cache layer
split** keep the old synchronous path: their indices are filtered before
the transfer event, and deferring that to the worker would widen the
pre-event shard-filter risk tracked by #45492.
"""
if not last_chunk or not supports_async_state_indices:
return False
if kv_shard_size != 1:
return False
if enable_all_cp_ranks_for_transfer and not enable_dsa_cache_layer_split:
return False
return True
def require_async_state_event(
async_state: bool, segment_is_last: bool, indices_event
) -> None:
"""Fail loudly if an async state payload has no event to protect its copy."""
if async_state and segment_is_last and indices_event is None:
raise RuntimeError(
"async state payload requires an indices_event for the final segment"
)
class SchedulerDisaggregationPrefillMixin:
"""
Mixin for Scheduler to handle disaggregation prefill
@@ -616,11 +652,15 @@ class SchedulerDisaggregationPrefillMixin:
self.attn_tp_cpu_group,
)
failed = set()
bootstrap_candidates = len(candidates)
pending_bootstrap = 0
for req, poll in zip(candidates, polls):
if poll == KVPoll.Failed:
self.handle_bootstrap_failure(req)
failed.add(req)
elif (
bootstrap_candidates -= 1
continue
if (
poll == KVPoll.WaitingForInput
and req.pending_bootstrap
and not should_force_retry(req)
@@ -629,6 +669,10 @@ class SchedulerDisaggregationPrefillMixin:
# finalize cannot fail here; if it ever does, the request stays
# pending and the post-forward check resolves it.
self.disagg_prefill_bootstrap_queue.finalize_bootstrap(req)
pending_bootstrap += int(req.pending_bootstrap)
batch_telemetry.set_bootstrap_pending(
pending_bootstrap, bootstrap_candidates
)
if failed:
self.waiting_queue = [
req for req in self.waiting_queue if req not in failed
@@ -647,32 +691,71 @@ class SchedulerDisaggregationPrefillMixin:
last_batch: Optional[ScheduleBatch],
should_retry_storage_prefetch: bool = True,
) -> NextBatchPlan:
bt_pass = batch_telemetry.begin_pass()
if bt_pass is not None:
# Capture before process_prefill_chunk()/admission can consume it:
# the last chunk of a resumed request clears self.chunked_req.
batch_telemetry.note_had_chunked(self.chunked_req is not None)
if bt_pass is not None:
_bt_t = time.monotonic()
self.process_pending_chunked_abort()
if bt_pass is not None:
batch_telemetry.add_pending_abort(time.monotonic() - _bt_t)
_bt_t = time.monotonic()
self._process_hicache_events(
should_retry_storage_prefetch=should_retry_storage_prefetch
)
if bt_pass is not None:
batch_telemetry.add_hicache(time.monotonic() - _bt_t)
# HACK (byronhsu): reset the batch_is_full flag because we never enter update_running_batch which resets it
# Otherwise, it hangs under high concurrency
running_batch.batch_is_full = False
if bt_pass is not None:
_bt_t = time.monotonic()
self.resolve_waiting_queue_bootstrap()
if bt_pass is not None:
batch_telemetry.add_bootstrap(time.monotonic() - _bt_t)
# Skip repeated chunk processing while polling a pending batch.
_chunk_t0 = time.monotonic() if bt_pass is not None else None
_had_chunk = self.chunked_req is not None
if last_batch is None or not last_batch.prefill_chunk_processed:
self.process_prefill_chunk(
last_batch=last_batch, running_batch=running_batch
)
if last_batch is not None:
last_batch.prefill_chunk_processed = True
if _chunk_t0 is not None:
batch_telemetry.add_chunk(
time.monotonic() - _chunk_t0, 1 if _had_chunk else 0
)
_nb_t0 = time.monotonic() if bt_pass is not None else None
prefill_plan = self.get_new_batch_prefill(running_batch)
batch = prefill_plan.batch_to_run
running_batch = prefill_plan.running_batch
if _nb_t0 is not None:
batch_telemetry.add_new_batch(time.monotonic() - _nb_t0)
if bt_pass is not None:
# Drop any gather value left by an earlier pass; only a gather that
# actually runs inside the call below may be attributed to this one.
batch_telemetry.note_dp_gather(None)
_bt_t = time.monotonic()
batch = self.dp_attn_adapter.maybe_prepare_mlp_sync_batch(batch)
if bt_pass is not None:
batch_telemetry.add_mlp_sync(time.monotonic() - _bt_t)
if batch:
_ss_t0 = time.monotonic() if bt_pass is not None else None
set_schedule_time_batch(batch)
if _ss_t0 is not None:
batch_telemetry.add_set_sched(time.monotonic() - _ss_t0)
if bt_pass is not None:
batch_telemetry.end_pass(dp_rank=get_parallel().attn_dp_rank)
return NextBatchPlan(batch_to_run=batch, running_batch=running_batch)
@@ -765,10 +848,94 @@ class SchedulerDisaggregationPrefillMixin:
batch = None
batch_result = None
def _forward_active() -> bool:
reporter = getattr(self, "metrics_reporter", None)
timer = (
getattr(reporter, "forward_pass_device_timer", None)
if reporter is not None
else None
)
return timer.is_active() if timer is not None else False
def _pre_has_work() -> bool:
bootstrap_queue = getattr(
self.disagg_prefill_bootstrap_queue, "queue", None
)
return bool(
self.last_batch is not None
or len(self.result_queue) > 0
or len(self.waiting_queue) > 0
or len(self._deferred_input_requests) > 0
or self._engine_paused
or (
self.disagg_prefill_inflight_queue is not None
and len(self.disagg_prefill_inflight_queue) > 0
)
or (bootstrap_queue is not None and len(bootstrap_queue) > 0)
)
def _loop_state(
waiting_for_result: bool = False, paused: bool = False
) -> dict:
bootstrap_queue = getattr(
self.disagg_prefill_bootstrap_queue, "queue", None
)
inflight = self.disagg_prefill_inflight_queue
return {
"waiting": len(self.waiting_queue),
"result_queue": len(self.result_queue),
"result_limit": 2,
"inflight": len(inflight) if inflight is not None else 0,
"bootstrap": len(bootstrap_queue) if bootstrap_queue is not None else 0,
"deferred": len(self._deferred_input_requests),
"paused": bool(paused or self._engine_paused),
"waiting_for_result": bool(waiting_for_result),
"skip_batch_creation": bool(skip_batch_creation),
"forward_active": _forward_active(),
}
def _end_loop(
lt,
has_batch: bool,
loop_batch,
waiting_for_result: bool = False,
paused: bool = False,
) -> None:
if lt is None:
return
try:
batch_telemetry.end_loop(
lt,
dp_rank=get_parallel().attn_dp_rank,
has_batch=has_batch,
batch=loop_batch,
state=_loop_state(
waiting_for_result=waiting_for_result, paused=paused
),
)
except Exception:
# Telemetry must never break the scheduler loop.
pass
while True:
if self.gracefully_exit:
break
try:
from sglang.srt.observability import fwd_timeline
fwd_timeline.drain()
except Exception:
pass
_lt = None
if batch_telemetry.loop_accounting_enabled():
_lt = batch_telemetry.begin_loop(
"prefill",
pre_has_work=_pre_has_work(),
forward_active=_forward_active,
)
loop_batch = None
# Stop dispatching at pause requests while waiting for a batch result.
received_inputs = self.ingest_requests(
stop_at_pause=waiting_for_batch_result
@@ -776,6 +943,7 @@ class SchedulerDisaggregationPrefillMixin:
if self._engine_paused:
self._record_scheduler_state_for_paused_engine()
self._yield_gil_if_needed()
_end_loop(_lt, False, None, paused=True)
continue
bootstrapped_reqs = self.disagg_prefill_bootstrap_queue.pop_bootstrapped()
self.waiting_queue.extend(bootstrapped_reqs)
@@ -783,6 +951,14 @@ class SchedulerDisaggregationPrefillMixin:
if not waiting_for_batch_result or received_inputs or bootstrapped_reqs:
skip_batch_creation = False
# Preserve the launch decision before processing a result mutates the
# queues; end-of-loop depths cannot explain why this loop did not launch.
if _lt is not None:
if len(self.result_queue) >= 2:
batch_telemetry.note_launch_blocked(_lt, "result_backpressure")
elif self._deferred_input_requests:
batch_telemetry.note_launch_blocked(_lt, "deferred_input")
# Keep at most two batches in flight.
if (
len(self.result_queue) < 2
@@ -802,6 +978,7 @@ class SchedulerDisaggregationPrefillMixin:
batch = self.ngram_embedding_manager.prepare_for_forward(
batch, chunked_req=self.chunked_req
)
loop_batch = batch
self.cur_batch_for_debug = batch
# Skip batch creation next round to avoid repeated preparation failures.
skip_batch_creation = batch is None
@@ -810,8 +987,31 @@ class SchedulerDisaggregationPrefillMixin:
if batch:
if self.enable_staging:
self.maybe_prefetch_staging_for_batch(batch)
_rb_t0 = (
time.time_ns()
if _lt is not None and _lt.span is not None
else 0
)
batch_result = self.run_batch(batch)
self._apply_war_barrier()
if _rb_t0:
try:
batch_telemetry.mark_run_batch(
_rb_t0,
time.time_ns(),
{
"sglang.forward_mode": str(batch.forward_mode),
"sglang.bs": batch.batch_size(),
"sglang.num_tokens": int(
getattr(batch, "extend_num_tokens", 0) or 0
),
},
)
except Exception:
pass
# WAR barrier is outside run_batch's timing boundary, so it is
# recorded as its own loop phase (design §4.3).
with batch_telemetry.detail_phase("war_barrier"):
self._apply_war_barrier()
self.result_queue.append((batch.copy(), batch_result))
else:
batch_result = None
@@ -828,24 +1028,67 @@ class SchedulerDisaggregationPrefillMixin:
):
waiting_for_batch_result = True
self._yield_gil_if_needed()
_end_loop(
_lt,
loop_batch is not None,
loop_batch,
waiting_for_result=True,
)
continue
# Process the oldest batch result.
oldest_batch, oldest_batch_result = self.result_queue.popleft()
_pr_t0 = (
time.time_ns() if _lt is not None and _lt.span is not None else 0
)
# Scope KV phases to this call: sends issued later in the loop
# (inflight-queue pending bootstrap) must not be charged here.
_proc_kv_phases = (
"kv_state_payload",
"kv_page_indices",
"kv_sender_send",
)
_proc_detail = batch_telemetry.loop_detail_enabled()
_proc_kv_before = (
batch_telemetry.detail_phase_total_ns(_proc_kv_phases)
if _proc_detail
else 0
)
self.process_batch_result(oldest_batch, oldest_batch_result)
_pr_attrs = {"sglang.n_reqs": len(oldest_batch.reqs)}
if _proc_detail:
_pr_attrs["sglang.process_kv_ms"] = (
batch_telemetry.detail_phase_total_ns(_proc_kv_phases)
- _proc_kv_before
) / 1e6
if _pr_t0:
try:
batch_telemetry.mark_process(
_pr_t0,
time.time_ns(),
_pr_attrs,
)
except Exception:
pass
elif batch is None:
# When the server is idle, do self-check and re-init some states
self.on_idle()
self.process_disagg_prefill_inflight_queue()
# `loop other` (outside run_batch/process_result/get_next) hosts the
# inflight poll + release_kv_cache(checkpoint/insert) and the sample
# launch. Timed as its own phases so the loop-other residual shrinks.
with batch_telemetry.detail_phase("inflight_poll"):
self.process_disagg_prefill_inflight_queue()
# Run sample of the current batch
# It depends on the result of the last batch (e.g., grammar), so we run it after the last batch is processed.
self.launch_batch_sample_if_needed(batch_result, batch)
with batch_telemetry.detail_phase("launch_sample"):
self.launch_batch_sample_if_needed(batch_result, batch)
# Update last_batch and scheduler status
self._sched_idled = batch is None
self.last_batch = batch
waiting_for_batch_result = False
_end_loop(_lt, loop_batch is not None, loop_batch)
def is_disagg_prefill_batch_result_ready(
self: Scheduler,
@@ -897,7 +1140,11 @@ class SchedulerDisaggregationPrefillMixin:
)
if copy_done is not None:
copy_done.synchronize()
with batch_telemetry.detail_phase("copy_done_wait"):
copy_done.synchronize()
_aux_t0 = (
time.monotonic_ns() if batch_telemetry.loop_detail_enabled() else None
)
auxiliary_output_starts = (
self.batch_result_processor.snapshot_auxiliary_output_starts(batch, result)
)
@@ -943,10 +1190,29 @@ class SchedulerDisaggregationPrefillMixin:
if extend_logprob_start_len < extend_input_len:
logprob_pt += extend_input_len - extend_logprob_start_len
# Auxiliary finalize / logits-to-CPU work is a contiguous host phase.
if _aux_t0 is not None:
batch_telemetry.detail_add(
"auxiliary_and_logprob", _aux_t0, time.monotonic_ns()
)
# Per-request result handling is everything in this loop except the KV
# phases; the delta below keeps `continue` paths accounted for without
# measuring each branch.
_pr_loop_t0 = (
time.monotonic_ns() if batch_telemetry.loop_detail_enabled() else None
)
_kv_phases = ("kv_state_payload", "kv_page_indices", "kv_sender_send")
_kv_before = (
batch_telemetry.detail_phase_total_ns(_kv_phases)
if _pr_loop_t0 is not None
else 0
)
for i, (req, next_token_id) in enumerate(
zip(batch.reqs, next_token_ids, strict=True)
):
if req.inflight_middle_chunks <= 0:
batch_telemetry.detail_count("n_last_reqs")
req.time_stats.set_prefill_finished_time()
# Its KV never landed: abort instead of sending it to decode.
@@ -1007,6 +1273,7 @@ class SchedulerDisaggregationPrefillMixin:
self.checkpoint_disagg_prefill(req)
self.disagg_prefill_inflight_queue.append(req)
batch_telemetry.note_request_completed("prefill")
if self.spec_algorithm.is_eagle() and draft_input is not None:
req.output_topk_p = draft_input.topk_p[i]
req.output_topk_index = draft_input.topk_index[i]
@@ -1050,6 +1317,7 @@ class SchedulerDisaggregationPrefillMixin:
else:
# being chunked reqs' prefill is not finished
batch_telemetry.detail_count("n_middle_reqs")
req.inflight_middle_chunks -= 1
# Still chunking iff its next chunk was launched: either it is
@@ -1100,12 +1368,27 @@ class SchedulerDisaggregationPrefillMixin:
self.send_kv_chunk(req, last_chunk=False, end_idx=req.tmp_end_idx)
req.time_stats.set_last_chunked_prefill_finish_time()
if _pr_loop_t0 is not None:
_kv_ns = batch_telemetry.detail_phase_total_ns(_kv_phases) - _kv_before
batch_telemetry.detail_add(
"per_request_result",
_pr_loop_t0 + max(0, _kv_ns),
time.monotonic_ns(),
)
if auxiliary_output is not None:
_aux2_t0 = (
time.monotonic_ns() if batch_telemetry.loop_detail_enabled() else None
)
self.batch_result_processor.consume_auxiliary_output(
batch,
auxiliary_output,
auxiliary_output_starts,
)
if _aux2_t0 is not None:
batch_telemetry.detail_add(
"auxiliary_and_logprob", _aux2_t0, time.monotonic_ns()
)
if aborted_reqs:
self.output_streamer.stream_output(
@@ -1114,12 +1397,13 @@ class SchedulerDisaggregationPrefillMixin:
)
can_run_cuda_graph = result.can_run_cuda_graph
self.metrics_reporter.report_prefill_stats(
batch=batch,
prefill_stats=batch.prefill_stats,
can_run_cuda_graph=can_run_cuda_graph,
dp_cooperation_info=batch.dp_cooperation_info,
)
with batch_telemetry.detail_phase("report_stats"):
self.metrics_reporter.report_prefill_stats(
batch=batch,
prefill_stats=batch.prefill_stats,
can_run_cuda_graph=can_run_cuda_graph,
dp_cooperation_info=batch.dp_cooperation_info,
)
self.maybe_send_health_check_signal()
@scheduler_stage_method(SCHEDULER_STAGE_PROCESS_QUEUE)
@@ -1200,15 +1484,44 @@ class SchedulerDisaggregationPrefillMixin:
if not isinstance(req.finished_reason, FINISH_ABORT):
req.finished_reason = FINISH_LENGTH(length=0)
# unlock the tree
_release_t0 = (
time.monotonic_ns()
if batch_telemetry.loop_detail_enabled()
else None
)
release_kv_cache(req, self.tree_cache, checkpoint=True)
self.tree_cache.finish(
req.cache_request_handle, CacheRequestOutcome.SUCCESS
)
if _release_t0 is not None:
# Nested in inflight_poll: checkpoint + radix/hicache insert.
batch_telemetry.detail_add(
"inflight_release", _release_t0, time.monotonic_ns()
)
# Consume the join summary before clear() releases the room.
_join_stats = None
if batch_telemetry.mooncake_detail_enabled():
_pop_stats = getattr(
req.disagg_kv_sender, "pop_chunk_join_stats", None
)
_join_stats = _pop_stats() if _pop_stats is not None else None
# FIXME: clean up req's data in transfer engine
req.disagg_kv_sender.clear()
done_reqs.append(req)
req.time_stats.set_prefill_kv_transfer_finish_time()
if batch_telemetry.mooncake_detail_enabled():
batch_telemetry.emit_request_kv_join(
req=req,
stats=_join_stats,
dp_rank=get_parallel().attn_dp_rank,
)
elif poll == KVPoll.Failed:
if batch_telemetry.mooncake_detail_enabled():
_pop_stats = getattr(
req.disagg_kv_sender, "pop_chunk_join_stats", None
)
if _pop_stats is not None:
_pop_stats()
self.handle_inflight_transfer_failure(req)
done_reqs.append(req)
else:
@@ -1467,14 +1780,21 @@ class SchedulerDisaggregationPrefillMixin:
ev = torch.cuda.Event()
ev.record(self.forward_stream)
req.disagg_kv_sender._early_send_wait_event = ev
self.send_kv_chunk(req, last_chunk=False, end_idx=cached_end)
if batch_telemetry.loop_detail_enabled():
batch_telemetry.detail_count("early_send_reqs")
self.send_kv_chunk(
req, last_chunk=False, end_idx=cached_end, send_kind="cached_prefix"
)
def send_kv_chunk(
self,
req: Req,
last_chunk: bool = False,
end_idx: Optional[int] = None,
send_kind: Optional[str] = None,
) -> None:
if send_kind is None:
send_kind = "last" if last_chunk else "middle"
computer: Optional[KvChecksumComputer] = self.kv_checksum_computer
if last_chunk and computer is not None:
if is_health_check_req(req):
@@ -1486,13 +1806,21 @@ class SchedulerDisaggregationPrefillMixin:
state_indices = state_indices_for_request(self, req, end_idx)
value = computer.compute(page_indices_gpu, state_indices)
self.disagg_metadata_buffers.set_kv_checksum(req, value)
self._send_kv_chunk(req, last_chunk=last_chunk, end_idx=end_idx)
_kv_t0 = time.time_ns() if batch_telemetry.kv_send_span_enabled() else 0
if batch_telemetry.loop_detail_enabled():
batch_telemetry.detail_count("n_kv_calls")
self._send_kv_chunk(
req, last_chunk=last_chunk, end_idx=end_idx, send_kind=send_kind
)
if _kv_t0:
batch_telemetry.add_kv_send(_kv_t0, time.time_ns())
def _send_kv_chunk(
self: Scheduler,
req: Req,
last_chunk: bool = False,
end_idx: Optional[int] = None,
send_kind: str = "last",
) -> None:
"""
Send a prefilled chunk to the decode server
@@ -1526,7 +1854,27 @@ class SchedulerDisaggregationPrefillMixin:
)
return
# First async state-payload version: SWA only, with the shard/CP guards
# in should_async_state_payload(). Other builders stay synchronous.
sender = req.disagg_kv_sender
async_state = should_async_state_payload(
last_chunk=last_chunk,
supports_async_state_indices=getattr(
sender, "supports_async_state_indices", False
),
kv_shard_size=getattr(sender.kv_mgr, "kv_shard_size", 1),
enable_all_cp_ranks_for_transfer=getattr(
sender.kv_mgr, "enable_all_cp_ranks_for_transfer", False
),
enable_dsa_cache_layer_split=get_parallel().enable_dsa_cache_layer_split,
)
state_indices: Optional[List] = None
_state_tok = (
batch_telemetry.detail_begin("kv_state_payload." + send_kind)
if last_chunk
else None
)
if last_chunk:
self.disagg_metadata_buffers.set_buf(req)
@@ -1572,6 +1920,10 @@ class SchedulerDisaggregationPrefillMixin:
window_kv_indices_full
)
)
if async_state:
return kv_to_page_indices_pinned(
window_kv_indices_swa, page_size
)
return kv_to_page_indices(window_kv_indices_swa, page_size)
def _full_kv_pages_payload():
@@ -1665,6 +2017,12 @@ class SchedulerDisaggregationPrefillMixin:
state_indices = [
payloads[st]() if st in payloads else None for st in state_types
]
if _state_tok is not None:
batch_telemetry.detail_attr(
"state_types",
",".join(sorted(str(getattr(st, "name", st)) for st in state_types)),
)
batch_telemetry.detail_end("kv_state_payload." + send_kind, _state_tok)
transfer_chunk_tokens = req.disagg_kv_sender.get_max_transfer_tokens()
if self.enable_staging:
@@ -1685,7 +2043,16 @@ class SchedulerDisaggregationPrefillMixin:
else:
segments = [(start_idx, end_idx)]
# Classify the chunk for detail accounting (cached_prefix/middle/last).
# Set on the sender so backend send() signatures stay unchanged.
req.disagg_kv_sender._send_kind = send_kind
for seg_start, seg_end in segments:
_seg_t0 = (
time.monotonic_ns()
if batch_telemetry.loop_detail_enabled()
else None
)
is_final_segment = seg_end == end_idx
raw_kv_indices = self.req_to_token_pool.req_to_token[
req.kv.req_pool_idx, seg_start:seg_end
@@ -1697,7 +2064,6 @@ class SchedulerDisaggregationPrefillMixin:
raw_kv_indices
)
)
sender = req.disagg_kv_sender
indices_event = None
if getattr(sender, "supports_host_tensor_indices", False) and not (
sender.kv_mgr.enable_all_cp_ranks_for_transfer
@@ -1717,10 +2083,23 @@ class SchedulerDisaggregationPrefillMixin:
indices_event.record(torch.cuda.current_stream())
else:
page_indices = kv_to_page_indices(kv_indices, page_size)
if _seg_t0 is not None:
batch_telemetry.detail_add(
"kv_page_indices." + send_kind, _seg_t0, time.monotonic_ns()
)
batch_telemetry.detail_count("n_pages", len(page_indices))
segment_is_last = last_chunk and is_final_segment
if not sender.should_send_kv_chunk(len(page_indices), segment_is_last):
continue
require_async_state_event(
async_state, segment_is_last, indices_event
)
send_state_indices = state_indices if segment_is_last else None
_send_t0 = (
time.monotonic_ns()
if batch_telemetry.loop_detail_enabled()
else None
)
if indices_event is not None:
sender.send(
page_indices,
@@ -1734,6 +2113,11 @@ class SchedulerDisaggregationPrefillMixin:
send_state_indices,
num_kv_tokens=seg_end - seg_start,
)
if _send_t0 is not None:
batch_telemetry.detail_add(
"kv_sender_send." + send_kind, _send_t0, time.monotonic_ns()
)
batch_telemetry.detail_count("n_segments")
req.start_send_idx = end_idx
# A last chunk needs no entry: every `last_chunk=True` call site has
# already put the request on `disagg_prefill_inflight_queue`.
@@ -1744,6 +2128,12 @@ class SchedulerDisaggregationPrefillMixin:
def optimistic_release_and_requeue(self: Scheduler, req: Req) -> None:
"""Release KV cache and requeue an optimistic prefill request."""
if batch_telemetry.mooncake_detail_enabled():
# A retry may reuse the same room; stale chunk stats would
# otherwise be merged into the next attempt's join summary.
_reset = getattr(req.disagg_kv_sender, "reset_chunk_detail_state", None)
if _reset is not None:
_reset()
max_attempts = get_disagg().optimistic_prefill_attempts
uses_write_through_cache = _uses_write_through_cache(self.tree_cache)
if not uses_write_through_cache:
@@ -56,6 +56,7 @@ class AnthropicUsage(BaseModel):
class TextBlock(BaseModel):
type: Literal["text"] = "text"
text: str
cache_control: Optional[dict[str, Any]] = None
class ImageBlock(BaseModel):
@@ -91,6 +92,50 @@ class ToolReferenceBlock(BaseModel):
id: Optional[str] = None
class McpToolReference(BaseModel):
type: Literal["mcp_tool_reference"]
server_name: str
name: str
class McpToolsetReference(BaseModel):
type: Literal["mcp_toolset_reference"]
server_name: str
class ToolDefinition(BaseModel):
"""A tool given by value: `definition` is a `tools` entry."""
type: Literal["tool_definition"]
definition: dict[str, Any]
class ToolAdditionBlock(BaseModel):
"""Anthropic beta block: a ``system`` message offers a tool from its position
on. Rendered through deepseek-recipe, which reads only a ``tool_reference``
target."""
type: Literal["tool_addition"] = "tool_addition"
tool: Annotated[
Union[
ToolReferenceBlock, McpToolReference, McpToolsetReference, ToolDefinition
],
Field(discriminator="type"),
]
cache_control: Optional[dict[str, Any]] = None
class ToolRemovalBlock(BaseModel):
"""Anthropic beta block: a ``system`` message withdraws a tool."""
type: Literal["tool_removal"] = "tool_removal"
tool: Annotated[
Union[ToolReferenceBlock, McpToolReference, McpToolsetReference],
Field(discriminator="type"),
]
cache_control: Optional[dict[str, Any]] = None
class SearchResultBlock(BaseModel):
type: Literal["search_result"] = "search_result"
# ``source`` here is a URL/identifier string (unlike ImageBlock.source).
@@ -117,6 +162,8 @@ AnthropicContentBlock = Annotated[
ToolUseBlock,
ToolResultBlock,
ToolReferenceBlock,
ToolAdditionBlock,
ToolRemovalBlock,
SearchResultBlock,
ThinkingBlock,
RedactedThinkingBlock,
@@ -128,6 +175,8 @@ AnthropicContentBlock = Annotated[
class AnthropicMessage(BaseModel):
role: Literal["user", "assistant", "system"]
content: Union[str, list[AnthropicContentBlock]]
clear_at: Optional[Literal["next_user_message", "never"]] = None
output_config: Optional[dict[str, Any]] = None
# ---------- Tools (discriminated by ``type`` family) ----------
@@ -20,8 +20,10 @@ from sglang.srt.entrypoints.anthropic.protocol import (
AnthropicContentBlock,
AnthropicCountTokensRequest,
AnthropicCountTokensResponse,
AnthropicCustomTool,
AnthropicError,
AnthropicErrorResponse,
AnthropicMessage,
AnthropicMessageEndDelta,
AnthropicMessagesRequest,
AnthropicMessagesResponse,
@@ -32,6 +34,8 @@ from sglang.srt.entrypoints.anthropic.protocol import (
ContentBlockStopEvent,
ErrorEvent,
InputJsonDelta,
McpToolReference,
McpToolsetReference,
MessageDeltaEvent,
MessageStartEvent,
MessageStopEvent,
@@ -40,6 +44,8 @@ from sglang.srt.entrypoints.anthropic.protocol import (
TextDelta,
ThinkingBlock,
ThinkingDelta,
ToolDefinition,
ToolReferenceBlock,
ToolUseBlock,
is_server_tool,
)
@@ -152,6 +158,207 @@ def _extract_system_text(
return "\n".join(texts) if texts else None
def _block_types(request: AnthropicMessagesRequest) -> set[str]:
"""The block types of the request's system and messages, tool result contents
included."""
types = set()
for content in [request.system, *(message.content for message in request.messages)]:
for block in content if isinstance(content, list) else []:
types.add(block.type)
if block.type == "tool_result" and isinstance(block.content, list):
types.update(item.type for item in block.content)
return types
def _without_cleared_messages(
request: AnthropicMessagesRequest,
) -> AnthropicMessagesRequest:
"""`request` without the turn-scoped system messages (`clear_at:
"next_user_message"`) that a later user message has cleared."""
last_user = max(
(
index
for index, message in enumerate(request.messages)
if message.role == "user"
),
default=-1,
)
rendered = []
for index, message in enumerate(request.messages):
if message.clear_at is not None and message.role != "system":
raise ValueError(
f"messages.{index}.clear_at: clear_at is only permitted on role "
"'system' messages"
)
if message.clear_at == "next_user_message":
_check_turn_scoped(index=index, message=message)
_check_turn_scoped_placement(index=index, messages=request.messages)
if index < last_user:
continue
rendered.append(message)
return request.model_copy(update={"messages": rendered})
def _check_turn_scoped(index: int, message: AnthropicMessage) -> None:
scope = "a turn-scoped system message (clear_at: 'next_user_message')"
if message.output_config is not None:
raise ValueError(f"messages.{index}: output_config is not permitted on {scope}")
for position, block in enumerate(
message.content if isinstance(message.content, list) else []
):
if block.type != "text":
raise ValueError(
f"messages.{index}: a turn-scoped system message supports text "
"blocks only (clear_at: 'next_user_message')"
)
if block.cache_control is not None:
raise ValueError(
f"messages.{index}.content.{position}: cache_control is not "
f"permitted on {scope}"
)
def _check_turn_scoped_placement(index: int, messages: list[AnthropicMessage]) -> None:
# Consecutive system messages are placed as one group.
before = next(
(m.role for m in reversed(messages[:index]) if m.role != "system"), None
)
after = next((m.role for m in messages[index + 1 :] if m.role != "system"), None)
if before != "user" or after == "user":
raise ValueError(
f"messages.{index}: a turn-scoped system message (clear_at: "
"'next_user_message') must follow a user message and either end "
"messages or precede an assistant message"
)
def _declare_inline_tools(
request: AnthropicMessagesRequest,
) -> AnthropicMessagesRequest:
"""`request` with each custom tool a `tool_addition` block defines by value
declared in `tools` as deferred, and the block naming it by reference. A
definition of a tool already offered unchanged is dropped."""
tools = list(request.tools or [])
declared = {tool.name: tool for tool in tools}
offered = {tool.name for tool in tools if not tool.defer_loading}
unresolved: dict[str, str] = {}
messages = []
for index, message in enumerate(request.messages):
if isinstance(message.content, str):
messages.append(message)
continue
blocks = []
for position, block in enumerate(message.content):
where = f"messages.{index}.content.{position}"
for name in _referenced_tool_names(block):
if name not in declared:
unresolved.setdefault(name, where)
if block.type == "tool_addition" and block.tool.type != "tool_reference":
if message.role != "system":
raise ValueError(
f"{where}: tool_addition blocks can only be in system messages"
)
tool = _inline_custom_tool(where=where, target=block.tool)
# Declaring the tool makes it resolvable from the first message on.
if tool.name in unresolved:
raise ValueError(
f"{unresolved[tool.name]}: tool reference {tool.name!r} "
f"precedes the tool_definition that defines it at {where}"
)
earlier = declared.setdefault(tool.name, tool)
if earlier is tool:
tools.append(tool)
elif is_server_tool(earlier) or _definition(earlier) != _definition(
tool
):
raise ValueError(
f"{where}: tool_definition "
f"reuses the name {tool.name!r} with a different "
"definition; replacing a tool mid-conversation is not "
"supported"
)
if tool.name in offered:
continue
offered.add(tool.name)
block = block.model_copy(
update={"tool": ToolReferenceBlock(name=tool.name)}
)
blocks.append(block)
# recipe renders a system message left without blocks as an empty turn.
if blocks or not message.content:
messages.append(message.model_copy(update={"content": blocks}))
return request.model_copy(update={"tools": tools, "messages": messages})
def _referenced_tool_names(block: AnthropicContentBlock) -> list[Optional[str]]:
if block.type == "tool_addition" and block.tool.type == "tool_reference":
return [block.tool.name or block.tool.tool_name]
if block.type == "tool_reference":
return [block.tool_name or block.name]
if block.type == "tool_result" and isinstance(block.content, list):
return [name for item in block.content for name in _referenced_tool_names(item)]
return []
def _inline_custom_tool(
where: str, target: Union[McpToolReference, McpToolsetReference, ToolDefinition]
) -> AnthropicCustomTool:
if target.type != "tool_definition":
raise ValueError(
f"{where}: tool_addition with a {target.type} is not supported; this "
"server runs no MCP connector"
)
kind = target.definition.get("type")
if kind not in (None, "custom"):
raise ValueError(
f"{where}: tool_definition supports custom tools only, got a "
f"definition of type {kind!r}"
)
tool = AnthropicCustomTool.model_validate(target.definition)
if tool.defer_loading:
raise ValueError(
f"{where}: a tool_definition with defer_loading is not supported; "
"declare the tool in tools and add it by reference"
)
return tool.model_copy(update={"defer_loading": True})
def _definition(tool: AnthropicCustomTool) -> dict[str, Any]:
return tool.model_dump(exclude={"type", "defer_loading"})
def _native_block(block: dict[str, Any]) -> dict[str, Any]:
"""`block` with SGLang's accepted aliases spelled as deepseek-recipe reads them."""
kind = block["type"]
if kind == "tool_result":
content = block.get("content")
fields = {
"type": kind,
"tool_use_id": block.get("tool_use_id") or block.get("id"),
"content": (
[_native_block(item) for item in content]
if isinstance(content, list)
else content
),
"is_error": block.get("is_error"),
}
return {key: value for key, value in fields.items() if value is not None}
if kind == "tool_reference":
return {"type": kind, "tool_name": block.get("tool_name") or block.get("name")}
if kind == "tool_addition":
tool = block["tool"]
name = tool.get("name") or tool.get("tool_name")
return {"type": kind, "tool": {"type": "tool_reference", "name": name}}
return block
def _native_message(message: AnthropicMessage) -> dict[str, Any]:
data = message.model_dump(exclude_none=True)
if isinstance(data["content"], list):
data["content"] = [_native_block(block) for block in data["content"]]
return data
def _wrap_sse_event(data: str, event_type: str) -> str:
"""Format an Anthropic SSE event with event type and data lines."""
return f"event: {event_type}\ndata: {data}\n\n"
@@ -218,6 +425,24 @@ class AnthropicServing:
self, anthropic_request: AnthropicMessagesRequest
) -> ChatCompletionRequest:
"""Convert an Anthropic Messages request to an OpenAI ChatCompletion request."""
anthropic_request = _declare_inline_tools(
_without_cleared_messages(anthropic_request)
)
block_types = _block_types(anthropic_request)
if "tool_removal" in block_types:
raise ValueError(
"tool_removal blocks are not supported; a tool cannot be withdrawn "
"mid-conversation"
)
native = self.openai_serving_chat.native_tool_loading and (
bool(block_types & {"tool_reference", "tool_addition"})
or any(
not is_server_tool(tool) and tool.defer_loading
for tool in anthropic_request.tools or []
)
)
if not native and "tool_addition" in block_types:
raise ValueError("tool_addition blocks are not supported by this model")
openai_messages = []
def _convert_anthropic_image_source_to_openai_part(
@@ -719,8 +944,51 @@ class AnthropicServing:
elif chat_request.tools:
chat_request.tool_choice = "auto"
if native:
self._prepare_native_prompt(anthropic_request, chat_request)
return chat_request
def _prepare_native_prompt(
self,
anthropic_request: AnthropicMessagesRequest,
chat_request: ChatCompletionRequest,
) -> None:
"""Render the request's own wire form into `chat_request`, whose tools
become the ones the history has loaded."""
body = {
"messages": [
_native_message(message) for message in anthropic_request.messages
],
# recipe reads a tool with a `type` as a server tool.
"tools": [
tool.model_dump(exclude_none=True, exclude={"type"})
for tool in anthropic_request.tools or []
if not is_server_tool(tool)
],
}
system = anthropic_request.system
if system:
body["system"] = (
system
if isinstance(system, str)
else [block.model_dump(exclude_none=True) for block in system]
)
choice = anthropic_request.tool_choice
if choice is not None and choice.type == "none":
body["tool_choice"] = {"type": "none"}
tools = self.openai_serving_chat.prepare_native_prompt(
chat_request, protocol="messages", body=body
).tools
if (
choice is not None
and choice.type == "tool"
and not any(tool.function.name == choice.name for tool in tools)
):
raise ValueError(
f"tool_choice names {choice.name!r}, a deferred tool no "
"tool_reference or tool_addition block has loaded"
)
async def _handle_non_streaming(
self,
chat_request: ChatCompletionRequest,
@@ -3,7 +3,7 @@ concrete adapters load only when selected."""
from __future__ import annotations
from typing import TYPE_CHECKING, Any, Callable, Literal, Optional, Protocol
from typing import TYPE_CHECKING, Any, Callable, Literal, Optional, Protocol, Union
import msgspec
@@ -15,8 +15,27 @@ if TYPE_CHECKING:
ToolCall,
)
from sglang.srt.function_call.core_types import ToolCallItem
from sglang.srt.multimodal.media_io import ImageData
EncodingBackend = Literal["builtin", "recipe"]
HistoryToolCallArguments = Literal["strict", "lenient", "as_sent"]
NativeProtocol = Literal["responses", "messages"]
class NativePrompt(msgspec.Struct, frozen=True, kw_only=True):
"""A Responses or Messages request rendered from its own wire form."""
prompt_ids: list[int]
tools: list[Tool]
"""The tools the model can call after the request's history: those loaded from
the start and those the history has loaded since."""
image_data: list[ImageData] = []
"""The images the prompt's placeholders stand for, in order."""
namespace_names: dict[str, tuple[str, str]] = {}
"""The call name of each namespace member, mapped to its namespace and the
name the request gives it there."""
continued_turn: Optional[dict[str, Any]] = None
"""The final assistant message the prompt continues, as a chat message."""
class ChatOutputHooks(msgspec.Struct, frozen=True, kw_only=True):
@@ -78,6 +97,25 @@ class ChatEncodingOutputAdapter(Protocol):
) -> ResponsesOutputStream: ...
class NativeRequestEncoder(Protocol):
def encode_native(
self,
*,
protocol: NativeProtocol,
body: dict[str, Any],
thinking: bool,
effort: Union[str, int, float, None],
response_schema: Optional[dict[str, Any]],
continue_final_message: bool,
) -> NativePrompt:
"""``body`` carries the request's conversation, tools and, for
``tool_choice="none"``, its choice; ``thinking`` and ``effort`` are the
handler's decisions, as for ``encode_messages``. ``response_schema`` is
the JSON schema the answer follows; a trailing assistant turn is continued
under ``continue_final_message`` and otherwise re-sent as a user turn."""
...
class ChatEncodingAdapter(Protocol):
collects_openai_media: bool
"""Whether media still has to be collected from the OpenAI messages after
@@ -92,6 +130,12 @@ class ChatEncodingAdapter(Protocol):
"""None when the configured parsers leave output to the text parsers."""
...
@property
def native_encoder(self) -> Optional[NativeRequestEncoder]:
"""None when Responses and Messages requests that load tools mid-conversation
have no rendering of their own wire form."""
...
def encode_messages(
self,
*,
@@ -0,0 +1,105 @@
# SPDX-License-Identifier: Apache-2.0
"""Compatibility shim for Dynamo's SGLang chat processor.
Upstream SGLang builds expose ``sglang.srt.entrypoints.openai.encoding_dsv4``
with ``encode_messages(messages, *, thinking_mode, reasoning_effort) -> str``.
Dynamo's frontend (``dynamo.frontend.sglang_prepost``) imports that module to
render DeepSeek-V4/V4.1 prompts when the model ships no HF chat template.
Our DeepSeek-V4 fork replaced the encoder with the ``deepseek-recipe``
renderer (``sglang.srt.parser.deepseek_recipe_renderer``). This module exposes
the upstream API on top of the recipe, so ``--dyn-chat-processor sglang``
works without installing a second SGLang build.
The returned string is tokenized by the caller. For the V4.1 checkpoint the
resulting ids match ``DeepSeekRecipe.encode()`` (the native serving path)
byte-for-byte; the renderer is the same code path, only the tokenizer call
moves to the caller.
"""
from __future__ import annotations
import copy
import json
from functools import lru_cache
from typing import Any, Optional
from sglang.srt.parser.deepseek_recipe_renderer import (
IMAGE_TOKEN,
MODEL_NAMES,
load_deepseek_recipe,
pre_normalize,
)
# The Dynamo frontend uses this encoder for the V4 family. Our deployment is
# DeepSeek-V4.1-Flash; V4-Pro checkpoints need the ``dsv4`` spec instead.
_CHAT_ENCODING_SPEC = "dsv41"
@lru_cache(maxsize=1)
def _renderer():
recipe = load_deepseek_recipe()
special_tokens = recipe.SpecialTokens(image=IMAGE_TOKEN)
encoding = recipe.DeepseekV41Encoding().with_special_tokens(special_tokens)
options = recipe.ConversionOptions().with_special_tokens(special_tokens)
return recipe, special_tokens, encoding, options
def encode_messages(
messages: list[dict[str, Any]],
*,
thinking_mode: str = "chat",
reasoning_effort: Optional[str] = None,
) -> str:
"""Render OpenAI-style ``messages`` to a DeepSeek-V4.1 prompt string.
Mirrors upstream ``encoding_dsv4.encode_messages``:
* ``thinking_mode`` is ``"thinking"`` or ``"chat"``;
* ``reasoning_effort`` is a named tier (``"high"`` / ``"max"``) or None;
* tool schemas are attached to the first system message by the caller
(Dynamo's convention), not passed as a separate argument.
Raises ``ValueError`` for prompt shapes the recipe rejects.
"""
recipe, _special_tokens, encoding, options = _renderer()
encoding_messages = copy.deepcopy(messages)
tools = None
if encoding_messages:
tools = encoding_messages[0].pop("tools", None)
if not tools:
tools = None
# Dynamo already flattens message content and re-serializes tool-call
# arguments; keep this defensive normalization for direct callers.
for message in encoding_messages:
if message.get("content") is None:
message["content"] = ""
for call in message.get("tool_calls") or []:
function = call.get("function") if isinstance(call, dict) else None
if isinstance(function, dict) and isinstance(
function.get("arguments"), (dict, list)
):
function["arguments"] = json.dumps(
function["arguments"], ensure_ascii=False
)
body: dict[str, Any] = {
"messages": encoding_messages,
"thinking": thinking_mode == "thinking",
}
if tools:
body["tools"] = tools
if reasoning_effort is not None:
body["reasoning_effort"] = reasoning_effort
normalized = pre_normalize(body)
normalized["model"] = MODEL_NAMES[_CHAT_ENCODING_SPEC]
try:
converted = recipe.ChatCompletionRequest(normalized).convert(options)
except recipe.ConversionError as error: # type: ignore[attr-defined]
detail = getattr(error, "body", None) or str(error)
raise ValueError(f"DeepSeek-V4.1 prompt conversion failed: {detail}") from error
return encoding.render_conversation(converted.conversation).prompt
@@ -17,6 +17,7 @@ from __future__ import annotations
import logging
import time
import msgspec
import unicodedata
import uuid
from dataclasses import dataclass
@@ -743,6 +744,10 @@ class ChatCompletionMessageGenericParam(BaseModel):
name: Optional[str] = None
phase: Optional[Literal["commentary", "final_answer"]] = None
reasoning_content: Optional[str] = None
action: Optional[str] = Field(
default=None,
description="Action tag for assistant messages (used by deepseek-recipe).",
)
tool_calls: Optional[List[ToolCall]] = Field(default=None, examples=[None])
tools: Optional[List[Tool]] = Field(default=None, examples=[None])
# DeepSeek API prefix completion: continue this final assistant message.
@@ -796,14 +801,22 @@ class Function(BaseModel):
description: Optional[str] = Field(default=None, examples=[None])
name: str
parameters: Optional[object] = None
strict: bool = False
strict: Optional[bool] = None
defer_loading: Optional[bool] = None
namespace: Optional[str] = Field(
default=None,
description="Tool namespace for disambiguation (used by deepseek-recipe).",
)
@model_serializer(mode="wrap")
def _serialize(self, handler):
data = handler(self)
if self.defer_loading is None:
data.pop("defer_loading", None)
if self.strict is None:
data.pop("strict", None)
if self.namespace is None:
data.pop("namespace", None)
return data
@@ -821,6 +834,26 @@ class Tool(BaseModel):
return self
class ChatCompletionNamespaceTool(BaseModel):
"""A Responses-style namespace grouping sent to the Chat Completions API.
Chat Completions has no namespace field in its tool-call shape, so the
members are flattened to ``<namespace>__<function>`` before rendering; the
model calls that flat name and the response reports it back unchanged.
"""
type: Literal["namespace"]
name: str
description: Optional[str] = None
tools: List[Tool]
@model_validator(mode="after")
def validate_members(self) -> "ChatCompletionNamespaceTool":
if not self.tools:
raise ValueError("namespace tools must list at least one function tool")
return self
# Tool is defined after the message params that reference it, so the forward
# reference has to be resolved explicitly.
ChatCompletionMessageGenericParam.model_rebuild()
@@ -908,11 +941,22 @@ class ChatCompletionRequest(PDRoutingFields):
temperature: Optional[float] = None
top_p: Optional[float] = None
user: Optional[str] = None
tools: Optional[List[Tool]] = Field(default=None, examples=[None])
tools: Optional[List[Union[Tool, ChatCompletionNamespaceTool]]] = Field(
default=None, examples=[None]
)
# Set by /v1/responses when the request has namespace tools: the chat template
# renders these, while ``tools`` (flattened to ``<namespace>.<function>``) still
# drives tool-call parsing.
_template_tools: Optional[List[Dict[str, Any]]] = PrivateAttr(default=None)
# Set with ``input_ids`` by a caller that rendered them itself: the images the
# prompt's placeholders stand for, in order.
_input_image_data: Optional[List[Any]] = PrivateAttr(default=None)
# Whether ``tools`` are the only tools the rendered prompt shows, so a call
# to any other tool is dropped rather than returned.
_drops_unlisted_tool_calls: bool = PrivateAttr(default=False)
# Set with ``input_ids`` under ``continue_final_message``: the final assistant
# message the prompt continues, where output parsing resumes.
_continued_turn: Optional[Dict[str, Any]] = PrivateAttr(default=None)
tool_choice: Union[ToolChoice, Literal["auto", "required", "none"]] = Field(
default="auto", examples=["none"]
) # noqa
@@ -1140,6 +1184,39 @@ class ChatCompletionRequest(PDRoutingFields):
return values
@model_validator(mode="after")
def flatten_namespace_tools(self) -> "ChatCompletionRequest":
"""Render namespace groups as flat ``ns__function`` chat tools."""
if not self.tools:
return self
flattened: List[Tool] = []
for tool in self.tools:
if isinstance(tool, ChatCompletionNamespaceTool):
for member in tool.tools:
name = f"{tool.name}__{member.function.name}"
description = member.function.description
if tool.description:
description = (
tool.description + chr(10) + description
if description
else tool.description
)
flattened.append(
Tool(
type="function",
function=Function(
name=name,
description=description,
parameters=member.function.parameters,
strict=member.function.strict,
),
)
)
else:
flattened.append(tool)
self.tools = flattened
return self
def to_sampling_params(
self,
stop: List[str],
@@ -1252,6 +1329,7 @@ class ChatMessage(BaseModel):
role: Optional[str] = None
content: Optional[str] = None
reasoning_content: Optional[str] = None
action: Optional[str] = None
tool_calls: Optional[List[ToolCall]] = Field(default=None, examples=[None])
@model_serializer(mode="wrap")
@@ -1259,6 +1337,8 @@ class ChatMessage(BaseModel):
data = handler(self)
if self.tool_calls is None:
data.pop("tool_calls", None)
if self.action is None:
data.pop("action", None)
return data
@@ -1811,6 +1891,8 @@ RESPONSE_TOOL_TYPES = Literal[
class ResponseTool(BaseModel):
"""Tool definition for responses."""
model_config = ConfigDict(extra="allow")
type: RESPONSE_TOOL_TYPES = Field(description="Type of tool to enable")
name: Optional[str] = None
description: Optional[str] = None
@@ -1830,6 +1912,40 @@ class ResponseTool(BaseModel):
return self
class ResponseToolState(msgspec.Struct, frozen=True):
"""Public tool metadata and name mappings for one translated request."""
public_tools: list[ResponseTool] | None = None
public_choice: str | dict[str, Any] | None = None
names: dict[str, dict[str, str]] = {}
# The tools the model can call after the history, from a request rendered in
# its own wire form; None when `tools` alone declares them.
chat_tools: list[Tool] | None = None
# ``custom`` tools a ``tool_search_output`` loaded, beside the declared ones.
discovered_custom_names: frozenset[str] = frozenset()
class ResponseToolSearchCall(BaseModel):
"""A client-executed ``tool_search`` call; the pinned openai SDK has no type."""
id: str
type: Literal["tool_search_call"] = "tool_search_call"
call_id: str
execution: Literal["client"] = "client"
arguments: Dict[str, Any]
status: Literal["in_progress", "completed", "incomplete"]
class ResponseToolSearchCallEvent(BaseModel):
"""``response.output_item.added`` / ``.done`` for a tool search call, which the
SDK's event types reject."""
type: Literal["response.output_item.added", "response.output_item.done"]
sequence_number: int
output_index: int
item: ResponseToolSearchCall
class ResponseInputMessageParam(EasyInputMessageParam, total=False):
phase: Optional[Literal["commentary", "final_answer"]]
@@ -1856,6 +1972,10 @@ ResponseInputOutputItem: TypeAlias = Union[
class ResponsesRequest(PDRoutingFields):
"""Request body for v1/responses endpoint."""
_tool_state: ResponseToolState = PrivateAttr(default_factory=ResponseToolState)
# By call_id, the name the model called a tool by when the output item
# reports another; the stored history replays the call under it.
_called_tool_names: Dict[str, str] = PrivateAttr(default_factory=dict)
_response_parser_prefix: str = PrivateAttr(default="")
_response_parser_grammar_start: Optional[str] = PrivateAttr(default=None)
_response_parser_json_response_format: bool = PrivateAttr(default=False)
@@ -2052,7 +2172,10 @@ class ResponsesRequest(PDRoutingFields):
"name"
)
if tool_choice.get("type") in ("function", "custom") and name:
return {"type": "function", "name": name}
choice = {"type": tool_choice["type"], "name": name}
if "namespace" in tool_choice:
choice["namespace"] = tool_choice["namespace"]
return choice
return "auto"
def to_sampling_params(
@@ -2176,6 +2299,7 @@ class ResponsesResponse(BaseModel):
ResponseOutputItem,
ResponseReasoningItem,
ResponseFunctionToolCall,
ResponseToolSearchCall,
]
] = Field(default_factory=list)
status: Literal[
@@ -2241,6 +2365,7 @@ class ResponsesResponse(BaseModel):
ResponseOutputItem,
ResponseReasoningItem,
ResponseFunctionToolCall,
ResponseToolSearchCall,
]
],
status: str,
@@ -957,12 +957,13 @@ class OpenAIServingChat(OpenAIServingBase):
The reasoning parser is told the same thing, so channel-framed formats
keep their framing intact exactly when a tool-call parser consumes it.
"""
return bool(
request.tool_choice != "none"
and self._effective_tools(request)
and self.tool_call_parser
and not request._response_parser_json_response_format
)
if not self.tool_call_parser or request._response_parser_json_response_format:
return False
if request._drops_unlisted_tool_calls and request.tool_choice == "none":
# The history showed the model tools, so it can still write a call;
# parsing drops it, where skipping the parser would return its markup.
return True
return bool(request.tool_choice != "none" and self._effective_tools(request))
def _validate_request(self, request: ChatCompletionRequest) -> str | None:
"""Validate that the input is valid."""
@@ -1367,18 +1368,7 @@ class OpenAIServingChat(OpenAIServingBase):
self, request: ChatCompletionRequest, is_multimodal: bool
) -> MessageProcessingResult:
"""Process chat messages and apply chat template"""
if self.default_chat_template_kwargs:
ctk = dict(request.chat_template_kwargs or {})
for k, v in self.default_chat_template_kwargs.items():
ctk.setdefault(k, v)
request.chat_template_kwargs = ctk
effort = ctk.get("reasoning_effort")
if effort is not None and request.reasoning_effort is None:
request.reasoning_effort = effort
normalize_hunyuan_reasoning_effort(
request, self.reasoning_parser, self.template_manager.reasoning_config
)
self._apply_request_defaults(request)
# GptOss model needs to keep special tokens for harmony parsing
if self.is_gpt_oss or self.is_gemma4:
@@ -1484,7 +1474,7 @@ class OpenAIServingChat(OpenAIServingBase):
result = MessageProcessingResult(
prompt="",
prompt_ids=request.input_ids,
image_data=None,
image_data=request._input_image_data,
audio_data=None,
video_data=None,
modalities=[],
@@ -1540,6 +1530,64 @@ class OpenAIServingChat(OpenAIServingBase):
result.reasoning_end_token_ids = list(token_ids)
return result
def _apply_request_defaults(self, request: ChatCompletionRequest) -> None:
"""Settle the thinking and effort that rendering reads; repeating it
changes nothing."""
if self.default_chat_template_kwargs:
ctk = dict(request.chat_template_kwargs or {})
for k, v in self.default_chat_template_kwargs.items():
ctk.setdefault(k, v)
request.chat_template_kwargs = ctk
effort = ctk.get("reasoning_effort")
if effort is not None and request.reasoning_effort is None:
request.reasoning_effort = effort
normalize_hunyuan_reasoning_effort(
request, self.reasoning_parser, self.template_manager.reasoning_config
)
@property
def native_tool_loading(self) -> bool:
"""Whether Responses and Messages requests that load tools mid-conversation
render from their own wire form (see `prepare_native_prompt`)."""
return (
self._encoding_adapter is not None
and self._encoding_adapter.native_encoder is not None
)
def prepare_native_prompt(
self,
request: ChatCompletionRequest,
*,
protocol: chat_encoding_adapters.NativeProtocol,
body: dict[str, Any],
) -> chat_encoding_adapters.NativePrompt:
"""Render `body`, the Responses or Messages form of `request`, into its
`input_ids` and images, and set its `tools` to the tools the model can call
after the history; rendering then keeps only grammar, stops and parsing."""
self._apply_request_defaults(request)
response_format = request.response_format
prompt = self._encoding_adapter.native_encoder.encode_native(
protocol=protocol,
body=body,
thinking=self._get_reasoning_from_request(request),
effort=request.reasoning_effort,
response_schema=(
response_format.json_schema.schema_
if response_format is not None and response_format.json_schema
else None
),
continue_final_message=request.continue_final_message,
)
if prompt.image_data and not self.tokenizer_manager.model_config.is_multimodal:
raise ValueError("image input is not supported for this model")
request.input_ids = prompt.prompt_ids
request.tools = prompt.tools or None
request._input_image_data = prompt.image_data or None
request._drops_unlisted_tool_calls = True
request._continued_turn = prompt.continued_turn
return prompt
def _apply_jinja_template(
self,
request: ChatCompletionRequest,
@@ -5,15 +5,20 @@ request semantics."""
from __future__ import annotations
from collections.abc import Sequence
import re
from typing import TYPE_CHECKING, Any, Optional, Union
from sglang.srt.entrypoints.openai.chat_encoding_adapters import (
ADAPTER_OUTPUT_PARSERS,
NativePrompt,
NativeProtocol,
select_new_output_ids,
token_output_sse_chunks,
)
from sglang.srt.entrypoints.openai.protocol import (
Function,
FunctionResponse,
Tool,
ToolCall,
)
from sglang.srt.environ import envs
@@ -26,7 +31,9 @@ from sglang.srt.parser.deepseek_recipe_output import (
whole_tool_calls,
)
from sglang.srt.parser.deepseek_recipe_renderer import (
BUDGET_TIER,
DeepSeekRecipe,
loaded_tools,
parse_effort_budget,
)
from sglang.srt.runtime_context import get_serving
@@ -37,7 +44,6 @@ if TYPE_CHECKING:
from sglang.srt.entrypoints.openai.protocol import (
ChatCompletionRequest,
ResponsesRequest,
Tool,
)
DEFAULT_EFFORT_ENVS = {
@@ -45,6 +51,12 @@ DEFAULT_EFFORT_ENVS = {
"dsv41": envs.SGLANG_DSV41_REASONING_EFFORT,
}
JSON_RESPONSE_FORMATS = ("json_object", "json_schema")
# V4 renders deferred tools up front and never loads one mid-conversation.
NATIVE_TOOL_LOADING_SPEC = "dsv41"
NATIVE_EFFORT_FIELDS: dict[str, str] = {
"responses": "reasoning",
"messages": "output_config",
}
class DeepSeekRecipeChatEncodingAdapter:
@@ -96,6 +108,13 @@ class DeepSeekRecipeChatEncodingAdapter:
def output(self) -> Optional[DeepSeekRecipeChatEncodingAdapter]:
return self if self._token_output else None
def validate_request(self, request: ChatCompletionRequest) -> Optional[str]:
return None
@property
def native_encoder(self) -> Optional[DeepSeekRecipeChatEncodingAdapter]:
return self if self._spec == NATIVE_TOOL_LOADING_SPEC else None
def encode_messages(
self,
*,
@@ -149,6 +168,66 @@ class DeepSeekRecipeChatEncodingAdapter:
prompt_ids += self.deepseek_recipe.encode_text(task_prefix)
return prompt_ids
def encode_native(
self,
*,
protocol: NativeProtocol,
body: dict[str, Any],
thinking: bool,
effort: Union[str, int, float, None],
response_schema: Optional[dict[str, Any]],
continue_final_message: bool,
) -> NativePrompt:
tier, budget = self._effort(effort)
# recipe reads an effort as a request to think.
if thinking and tier is not None:
body = {**body, NATIVE_EFFORT_FIELDS[protocol]: {"effort": tier}}
conversation = self.deepseek_recipe.with_host_settings(
self.deepseek_recipe.convert_native(protocol, body, thinking=thinking),
response_schema=response_schema,
continue_final_message=continue_final_message,
)
last = conversation.messages[-1] if conversation.messages else None
# recipe's Messages conversion drops `strict`, which SGLang's grammar enforces.
strict = {tool.get("name") for tool in body["tools"] if tool.get("strict")}
tools = [
Tool(
type="function",
function=Function(
name=tool.name,
description=tool.description,
parameters=tool.parameters,
strict=True if tool.name in strict else tool.strict,
),
)
for tool in loaded_tools(conversation)
]
return NativePrompt(
prompt_ids=self.deepseek_recipe.encode_conversation(
conversation, effort_budget=budget
),
tools=tools,
image_data=[
ImageData(url=url)
for url in self.deepseek_recipe.image_urls(conversation)
],
namespace_names=(
self.deepseek_recipe.namespace_tool_names(_responses_namespaces(body))
if protocol == "responses"
else {}
),
continued_turn=(
{"role": "assistant", "content": last.content}
| (
{"reasoning_content": last.reasoning_content}
if last.reasoning_content is not None
else {}
)
if last is not None and last.prefix
else None
),
)
def _encode(
self,
body: dict[str, Any],
@@ -157,21 +236,29 @@ class DeepSeekRecipeChatEncodingAdapter:
task: Optional[str] = None,
image_urls: Optional[list[str]] = None,
) -> list[int]:
if effort is None or effort == "none":
effort = self._default_effort
budget = None
if effort is not None and not isinstance(effort, str):
if self._spec != "dsv41":
raise ValueError(
f"DeepSeek-V4 reasoning_effort must be a named tier, got {effort!r}"
)
budget = parse_effort_budget(effort)
elif effort is not None:
body["reasoning_effort"] = effort
tier, budget = self._effort(effort)
if budget is None and tier is not None:
body["reasoning_effort"] = tier
return self.deepseek_recipe.encode(
body, task=task, effort_budget=budget, image_urls=image_urls
)
def _effort(
self, effort: Union[str, int, float, None]
) -> tuple[Optional[str], Optional[int]]:
"""The named tier recipe renders and, for a V4.1 integer budget, the budget
that replaces the tier's line."""
effort = _numeric_effort(effort)
if effort is None or effort == "none":
effort = self._default_effort
if effort is None or isinstance(effort, str):
return effort, None
if self._spec != "dsv41":
raise ValueError(
f"DeepSeek-V4 reasoning_effort must be a named tier, got {effort!r}"
)
return BUDGET_TIER, parse_effort_budget(effort)
def parse_chat_response(
self,
*,
@@ -320,7 +407,7 @@ class DeepSeekRecipeChatEncodingAdapter:
self, request: ChatCompletionRequest
) -> DeepSeekRecipeOutputParser:
response_format = request.response_format
continued = (
continued = request._continued_turn or (
_trailing_assistant_turn(
[
message.model_dump(exclude_unset=True, by_alias=True, mode="json")
@@ -344,6 +431,11 @@ class DeepSeekRecipeChatEncodingAdapter:
separate_reasoning=self._separate_reasoning and request.separate_reasoning,
stream_reasoning=request.stream_reasoning,
stop=_stop_strings(request.stop),
callable_names=(
_native_callable_names(request.tools or [], request.tool_choice)
if request._drops_unlisted_tool_calls
else None
),
)
def _new_responses_parser(
@@ -353,12 +445,25 @@ class DeepSeekRecipeChatEncodingAdapter:
separate_reasoning: bool,
parse_tool_calls: bool,
) -> DeepSeekRecipeOutputParser:
native = request._tool_state.chat_tools is not None
return DeepSeekRecipeOutputParser(
self.deepseek_recipe,
thinking_stage=self.deepseek_recipe.parse_start(thinking=require_reasoning),
parse_tool_calls=parse_tool_calls,
# Under `none` the tool parser runs only to drop calls; a JSON answer
# must be read as JSON, or a call marker spelled in a string ends it.
parse_json_output=native
and request.tool_choice == "none"
and request.has_json_schema_constraint(),
separate_reasoning=separate_reasoning,
stop=_stop_strings(request.stop),
callable_names=(
_native_callable_names(
request._tool_state.chat_tools, request.tool_choice
)
if native
else None
),
)
def _tool_call_id(self, index: int, name: str, history_tool_calls_cnt: int) -> str:
@@ -390,6 +495,37 @@ class DeepSeekRecipeResponsesStream:
return delta.reasoning, delta.content, _tool_call_items(delta.tool_calls)
def _responses_namespaces(body: dict[str, Any]) -> list[dict[str, Any]]:
"""The namespace tools a Responses body declares and its history discovers, in
the order recipe's conversion names their members."""
discovered = [
tool
for item in body["input"]
if isinstance(item, dict) and item.get("type") == "tool_search_output"
for tool in item["tools"]
]
return [
tool for tool in [*body["tools"], *discovered] if tool["type"] == "namespace"
]
def _null_content_as_empty(messages: list[dict[str, Any]]) -> list[dict[str, Any]]:
"""As `_process_messages` renders them: a null content is ""."""
return [
message if message.get("content") is not None else {**message, "content": ""}
for message in messages
]
def _numeric_effort(
effort: Union[str, int, float, None],
) -> Union[str, int, float, None]:
"""A numeric string effort ("30", "0.3") as the number it spells."""
if isinstance(effort, str) and re.fullmatch(r"\d+(\.\d+)?", effort.strip()):
return float(effort) if "." in effort else int(effort)
return effort
def _trailing_assistant_turn(
messages: list[dict[str, Any]],
) -> Optional[dict[str, Any]]:
@@ -405,6 +541,18 @@ def _stop_strings(stop: Union[str, list[str], None]) -> list[str]:
return [stop] if isinstance(stop, str) else stop or []
def _native_callable_names(
tools: list[Tool], tool_choice: Any
) -> Optional[frozenset[str]]:
"""The names of `tools`, all that a natively rendered prompt shows the model;
None when calls to other tools are returned. `none` allows no call at all."""
if tool_choice == "none":
return frozenset()
if envs.SGLANG_FORWARD_UNKNOWN_TOOLS.get():
return None
return frozenset(tool.function.name for tool in tools)
def _tool_call_items(
calls: Sequence[Union[RecipeToolCall, RecipeToolCallDelta]],
) -> list[ToolCallItem]:
@@ -36,7 +36,10 @@ if TYPE_CHECKING:
class InklingChatEncodingAdapter:
collects_openai_media = True
strict_tool_call_arguments = True
history_tool_call_arguments = "strict"
renders_response_format = False
uncounted_prompt_tokens = 0
native_encoder = None
def __init__(
self,
@@ -13,6 +13,7 @@ from http import HTTPStatus
from typing import TYPE_CHECKING, Any, AsyncGenerator, AsyncIterator, Optional, Union
import jinja2
import msgspec
import openai.types.responses as openai_responses_types
import orjson
from fastapi import Request
@@ -43,6 +44,7 @@ from openai.types.responses.response_reasoning_summary_part_done_event import (
Part as ResponseReasoningSummaryDonePart,
)
from openai_harmony import Message as OpenAIMessage
from pydantic import BaseModel, TypeAdapter
from sglang.srt.entrypoints.context import (
ConversationContext,
@@ -70,7 +72,12 @@ from sglang.srt.entrypoints.openai.protocol import (
ResponseOutputMessage,
ResponsesRequest,
ResponsesResponse,
ResponseTool,
ResponseToolSearchCall,
ResponseToolSearchCallEvent,
Tool,
ToolChoice,
ToolChoiceFuncName,
UsageInfo,
)
from sglang.srt.entrypoints.openai.responses_adapters import (
@@ -114,6 +121,100 @@ class _MediaInputValidationError(ValueError):
pass
class ResponseToolChoiceError(ValueError):
"""A response tool_choice that names a tool the model cannot call."""
TOOL_SEARCH = "tool_search"
TOOL_SEARCH_ITEMS = ("tool_search_call", "tool_search_output")
APPLY_PATCH = "apply_patch"
NAMESPACE_MEMBERS_ERROR = (
"namespace tools must list function tools, each with a type and a name"
)
def _holds_function_tools(members: Any) -> bool:
return isinstance(members, list) and all(
isinstance(member, dict)
and member.get("type") == "function"
and isinstance(member.get("name"), str)
for member in members
)
def _defers_loading(tool: ResponseTool) -> bool:
if tool.type == "namespace":
return bool(tool.defer_loading) or any(
isinstance(member, dict) and member.get("defer_loading")
for member in tool.tools or []
)
return tool.type in ("function", "custom") and bool(tool.defer_loading)
def _defer_tool_loading(
tools: list[dict[str, Any]], items: list
) -> tuple[list[dict[str, Any]], list]:
"""`tools` without those that defer loading, and `items` whose
`tool_search_output` entries naming a deferred tool take the fields they
leave out from its declaration, so a search result loads it where it returns.
recipe's Responses conversion renders a deferred tool up front and rejects a
search result that repeats a declared name."""
deferred: dict[tuple[Optional[str], str], dict[str, Any]] = {}
shown = []
for tool in tools:
if tool["type"] == "namespace":
members = []
for member in tool["tools"]:
if tool.get("defer_loading") or member.get("defer_loading"):
deferred[(tool["name"], member["name"])] = member
else:
members.append(member)
if members:
shown.append({**tool, "tools": members})
elif tool["type"] in ("function", "custom") and tool.get("defer_loading"):
deferred[(None, tool["name"])] = tool
else:
shown.append(tool)
def loaded(tool: dict[str, Any], namespace: Optional[str] = None) -> dict:
if tool["type"] == "namespace":
members = [loaded(member, tool["name"]) for member in tool["tools"]]
return {**tool, "tools": members}
declared = deferred.get((namespace, tool.get("name")))
if declared is None or declared["type"] != tool["type"]:
return tool
return declared | {k: v for k, v in tool.items() if v is not None}
if deferred:
items = [
(
{**item, "tools": [loaded(tool) for tool in item["tools"]]}
if isinstance(item, dict) and item.get("type") == "tool_search_output"
else item
)
for item in items
]
return shown, items
def _tool_search_call(
item_id: str, call_id: str, arguments: str
) -> ResponseToolSearchCall:
"""A search call reports object arguments; one whose arguments are not a JSON
object is incomplete."""
try:
parsed = json.loads(arguments)
except ValueError:
parsed = None
complete = isinstance(parsed, dict)
return ResponseToolSearchCall(
id=item_id,
call_id=call_id,
arguments=parsed if complete else {},
status="completed" if complete else "incomplete",
)
def _build_output_text_logprobs(meta_info: dict) -> list[Logprob]:
"""Reshape decoded ``meta_info`` logprobs into the Responses logprob type,
covering every generated token."""
@@ -319,9 +420,21 @@ class OpenAIServingResponses(OpenAIServingChat):
# FIXME: If the engine is dead, raise an error
# This is required for the streaming case
# Such a request's callable tools are known once its history renders,
# so `_make_request` checks its tool_choice instead.
native = self._renders_natively(request)
if native:
tools_error = self._native_tools_error(request)
if tools_error is not None:
return self.create_error_response(tools_error, param="tools")
# ``tool_choice="required"`` needs a tool the parser can actually force.
if request.tool_choice == "required" and not any(
tool.type in ("function", "custom") for tool in (request.tools or [])
if (
not native
and request.tool_choice == "required"
and not any(
tool.type in ("function", "custom") for tool in (request.tools or [])
)
):
return self.create_error_response(
'tool_choice="required" requires at least one tool with '
@@ -330,9 +443,13 @@ class OpenAIServingResponses(OpenAIServingChat):
)
tool_choice = request.effective_tool_choice()
if isinstance(tool_choice, dict) and not any(
tool.type in ("function", "custom") and tool.name == tool_choice["name"]
for tool in request.tools or []
if (
not native
and isinstance(tool_choice, dict)
and not any(
tool.type in ("function", "custom") and tool.name == tool_choice["name"]
for tool in request.tools or []
)
):
return self.create_error_response(
f"Tool {tool_choice['name']!r} is not declared in tools",
@@ -415,6 +532,8 @@ class OpenAIServingResponses(OpenAIServingChat):
except _MediaInputValidationError as e:
return self.create_error_response(str(e))
except ResponseToolChoiceError as e:
return self.create_error_response(str(e), param="tool_choice")
except (ValueError, TypeError, RuntimeError, jinja2.TemplateError) as e:
logger.exception("Error in preprocessing prompt inputs")
return self.create_error_response(f"{e} {e.__cause__}")
@@ -684,7 +803,10 @@ class OpenAIServingResponses(OpenAIServingChat):
prev_response: Optional[ResponsesResponse],
tokenizer: Any,
):
messages = self._construct_input_messages(request, prev_response)
native = self._renders_natively(request)
messages = (
[] if native else self._construct_input_messages(request, prev_response)
)
chat_tools = self._response_tools_to_chat_tools(request)
chat_request = ChatCompletionRequest(
@@ -707,6 +829,8 @@ class OpenAIServingResponses(OpenAIServingChat):
chat_template_kwargs=request.chat_template_kwargs,
)
chat_request._template_tools = self._response_tools_to_template_tools(request)
if native:
self._prepare_native_responses_prompt(request, chat_request)
media_error = self._validate_media_content(chat_request)
if media_error:
@@ -906,10 +1030,7 @@ class OpenAIServingResponses(OpenAIServingChat):
else:
self.msg_store[response.id] = self._response_input_history(
request
) + [
item.model_dump(exclude_none=True)
for item in response.output
]
) + self._replayed_output_items(request, response.output)
return response
@@ -952,12 +1073,18 @@ class OpenAIServingResponses(OpenAIServingChat):
name: str,
arguments: str,
custom_names: set[str],
namespace_names: Optional[dict[str, tuple[str, str]]] = None,
) -> Union[ResponseFunctionToolCall, ResponseCustomToolCall]:
tool_names: dict[str, dict[str, str]],
*,
client_tool_search: bool = False,
) -> Union[
ResponseFunctionToolCall, ResponseCustomToolCall, ResponseToolSearchCall
]:
"""A call against a ``custom`` tool reports its freeform payload rather
than the JSON arguments of the shim function tool. A call against a
namespace member reports ``namespace`` and the bare function ``name``."""
call_id = f"call_{random_uuid()[:24]}"
if client_tool_search and name == TOOL_SEARCH:
return _tool_search_call(f"tsc_{random_uuid()[:8]}", call_id, arguments)
if name in custom_names:
return ResponseCustomToolCall(
type="custom_tool_call",
@@ -966,7 +1093,13 @@ class OpenAIServingResponses(OpenAIServingChat):
name=name,
input=decode_custom_tool_input(arguments),
)
namespace, name = (namespace_names or {}).get(name, (None, name))
entry = (tool_names or {}).get(name)
if isinstance(entry, tuple):
namespace, name = entry
elif isinstance(entry, dict):
namespace, name = entry.get("namespace"), entry.get("name", name)
else:
namespace = None
return ResponseFunctionToolCall(
arguments=arguments,
call_id=call_id,
@@ -1063,12 +1196,17 @@ class OpenAIServingResponses(OpenAIServingChat):
finish_reason: Optional[dict[str, Any]] = None,
):
chat_tools = self._response_tools_to_chat_tools(request)
custom_names = custom_tool_names(request.tools)
custom_names = self._custom_tool_names(request)
namespace_names = (
namespace_function_names(request.tools)
if self.tool_call_parser == "muse"
else {}
else {
call_name: (entry.get("namespace"), entry.get("name", call_name))
for call_name, entry in request._tool_state.names.items()
if entry.get("namespace")
}
)
client_tool_search = self._client_tool_search(request)
encoding_output = self._encoding_output()
if encoding_output is not None:
required_json = self._uses_required_tool_json(request, chat_tools)
@@ -1088,12 +1226,17 @@ class OpenAIServingResponses(OpenAIServingChat):
content, custom_names, namespace_names
)
else:
tool_call_items = [
self._make_tool_call_item(
call.name, call.parameters, custom_names, namespace_names
tool_call_items = []
for call in calls:
item = self._make_tool_call_item(
call.name,
call.parameters,
custom_names,
namespace_names,
client_tool_search=client_tool_search,
)
for call in calls
]
self._record_called_name(request, item.call_id, call.name)
tool_call_items.append(item)
else:
reasoning_content, content, tool_call_items = self._parse_text_output(
request,
@@ -1194,14 +1337,15 @@ class OpenAIServingResponses(OpenAIServingChat):
try:
content, call_info_list = parser.parse_non_stream(content)
for call_info in call_info_list:
tool_call_items.append(
self._make_tool_call_item(
call_info.name,
call_info.parameters or "",
custom_names,
namespace_names,
)
item = self._make_tool_call_item(
call_info.name,
call_info.parameters or "",
custom_names,
namespace_names,
client_tool_search=self._client_tool_search(request),
)
self._record_called_name(request, item.call_id, call_info.name)
tool_call_items.append(item)
parsed_via_native = bool(call_info_list)
except Exception as e:
logger.error("Tool call parsing error: %s", e)
@@ -1268,6 +1412,10 @@ class OpenAIServingResponses(OpenAIServingChat):
def _tool_call_parsing_enabled(
self, request: ResponsesRequest, chat_tools: list[Tool]
) -> bool:
if request._tool_state.chat_tools is not None and request.tool_choice == "none":
# The history showed the model tools, so it can still write a call;
# parsing drops it, where skipping the parser would return its markup.
return bool(self.tool_call_parser)
return bool(
chat_tools and self.tool_call_parser and request.tool_choice != "none"
)
@@ -1326,7 +1474,213 @@ class OpenAIServingResponses(OpenAIServingChat):
return tool_choice
return {"type": "function", "function": {"name": tool_choice["name"]}}
def _renders_natively(self, request: ResponsesRequest) -> bool:
"""Whether the served encoding renders the request in its own wire form,
which tool search, namespace tools and deferred tools take."""
return self.native_tool_loading and (
any(
tool.type == "namespace" or _defers_loading(tool)
for tool in request.tools
)
or self._uses_tool_search(request)
)
def _uses_tool_search(self, request: ResponsesRequest) -> bool:
"""Whether the request searches for tools, or its history has."""
if any(tool.type == TOOL_SEARCH for tool in request.tools):
return True
return any(
isinstance(item, dict) and item.get("type") in TOOL_SEARCH_ITEMS
for item in self._history_items(request)
)
def _history_items(self, request: ResponsesRequest) -> list:
"""The input history, before `previous_response_id` is known to exist."""
stored = self.msg_store.get(request.previous_response_id) or []
return list(stored) + (request.input if isinstance(request.input, list) else [])
def _native_tools_error(self, request: ResponsesRequest) -> Optional[str]:
deferred_types = {
tool.name: tool.type
for tool in request.tools
if tool.type in ("function", "custom") and tool.defer_loading
}
for tool in request.tools:
if (
tool.type == TOOL_SEARCH
and (tool.model_extra or {}).get("execution") != "client"
):
return (
'tool_search requires execution "client": this server runs no '
"searches"
)
if tool.type == "namespace" and not _holds_function_tools(tool.tools):
return NAMESPACE_MEMBERS_ERROR
for item in self._history_items(request):
if not isinstance(item, dict) or item.get("type") != "tool_search_output":
continue
tools = item.get("tools")
if not isinstance(tools, list) or not all(
isinstance(tool, dict)
and tool.get("type") in ("function", "namespace", "custom")
for tool in tools
):
return (
"tool_search_output tools must be a list of function, namespace "
"or custom tools"
)
for tool in tools:
if tool["type"] == "namespace" and not _holds_function_tools(
tool.get("tools")
):
return NAMESPACE_MEMBERS_ERROR
# The served encoding renders no other custom tool.
if tool["type"] == "custom" and tool.get("name") != APPLY_PATCH:
return (
f"tool_search_output can load only the {APPLY_PATCH!r} "
"custom tool"
)
# A deferred declaration is loaded as the tool it declares.
declared = deferred_types.get(tool.get("name"), tool["type"])
if tool["type"] != "namespace" and declared != tool["type"]:
return (
f"tool_search_output returns {tool.get('name')!r} as a "
f"{tool['type']} tool, but tools declares it as a deferred "
f"{declared} tool"
)
return None
def _prepare_native_responses_prompt(
self, request: ResponsesRequest, chat_request: ChatCompletionRequest
) -> None:
"""Render the request's own wire form into `chat_request`, whose tools and
choice become the ones the history has loaded."""
choice = request.effective_tool_choice()
tools, items = _defer_tool_loading(
[tool.model_dump(exclude_unset=True) for tool in request.tools],
[
item
for item in map(
self._native_input_item, self._response_input_history(request)
)
if item is not None
],
)
body = {"input": items, "tools": tools}
if request.instructions:
body["instructions"] = request.instructions
if choice == "none":
body["tool_choice"] = "none"
prompt = self.prepare_native_prompt(
chat_request, protocol="responses", body=body
)
tools = prompt.tools
request._tool_state = msgspec.structs.replace(
request._tool_state,
chat_tools=tools,
names={
call_name: {"namespace": namespace, "name": name}
for call_name, (namespace, name) in prompt.namespace_names.items()
},
discovered_custom_names=frozenset(
tool["name"]
for item in body["input"]
if isinstance(item, dict) and item.get("type") == "tool_search_output"
for tool in item["tools"]
if tool["type"] == "custom"
),
)
if isinstance(choice, dict):
name = choice["name"]
if "namespace" in choice:
# `tools` keeps conversion order, declared before discovered, so
# the last match is the member's latest definition.
member = (choice["namespace"], name)
name = next(
(
tool.function.name
for tool in reversed(tools)
if prompt.namespace_names.get(tool.function.name) == member
),
None,
)
if not any(tool.function.name == name for tool in tools) or (
choice["type"] == "custom"
and name not in self._custom_tool_names(request)
):
raise ResponseToolChoiceError(
f"Tool {choice['name']!r} is not declared in tools without "
"defer_loading or loaded by a tool_search_output"
)
chat_request.tool_choice = ToolChoice(
function=ToolChoiceFuncName(name=name)
)
elif choice == "required" and not tools:
raise ResponseToolChoiceError(
'tool_choice="required" needs a tool the model can call'
)
else:
chat_request.tool_choice = choice if tools else "none"
@classmethod
def _native_input_item(cls, item: Any) -> Any:
if isinstance(item, BaseModel):
item = item.model_dump(exclude_none=True)
if isinstance(item, dict) and item.get("type") == "reasoning":
# recipe reads only `content`; keep the text SGLang replays from a
# summary or a store=false client's state.
message = cls._normalize_response_message_for_chat(item)
if message is None:
return None
return {
"type": "reasoning",
"content": [
{"type": "reasoning_text", "text": message["reasoning_content"]}
],
}
return item
@staticmethod
def _record_called_name(request: ResponsesRequest, call_id: str, name: str) -> None:
"""Keep the name the model called a namespace member by: the item reports
its namespace and own name, which several call names can share."""
if request._tool_state.chat_tools is not None and (
name in request._tool_state.names
):
request._called_tool_names[call_id] = name
@staticmethod
def _replayed_output_items(request: ResponsesRequest, output: list) -> list[dict]:
"""`output` as the history a later request replays: a namespace member's
call under the name the model called it by."""
items = []
for item in output:
item = item.model_dump(exclude_none=True)
called_name = request._called_tool_names.get(item.get("call_id"))
if called_name is not None:
item = {**item, "name": called_name}
del item["namespace"]
items.append(item)
return items
@staticmethod
def _custom_tool_names(request: ResponsesRequest) -> set[str]:
declared = custom_tool_names(request.tools)
if request._tool_state.chat_tools is not None:
# A deferred declaration counts once a search result has loaded it.
declared -= {tool.name for tool in request.tools if tool.defer_loading}
return declared | request._tool_state.discovered_custom_names
@staticmethod
def _client_tool_search(request: ResponsesRequest) -> bool:
"""Whether a `tool_search` call is the client-executed search tool."""
return request._tool_state.chat_tools is not None and any(
tool.type == TOOL_SEARCH for tool in request.tools
)
def _response_tools_to_chat_tools(self, request: ResponsesRequest) -> list[Tool]:
if request._tool_state.chat_tools is not None:
return list(request._tool_state.chat_tools)
# ``function`` and ``custom`` tools flow to chat; built-ins go through
# harmony. A custom tool is shimmed into a single-string function tool.
chat_tools = []
@@ -2133,6 +2487,7 @@ class OpenAIServingResponses(OpenAIServingChat):
tool_follows = opens_item and item_type in (
"function_call",
"custom_tool_call",
"tool_search_call",
)
terminal = event_type in (
"response.completed",
@@ -2215,14 +2570,19 @@ class OpenAIServingResponses(OpenAIServingChat):
)
chat_tools = self._response_tools_to_chat_tools(request)
custom_names = custom_tool_names(request.tools)
custom_names = self._custom_tool_names(request)
namespace_names = (
namespace_function_names(request.tools)
if self.tool_call_parser == "muse"
else {}
else {
call_name: (entry.get("namespace"), entry.get("name", call_name))
for call_name, entry in request._tool_state.names.items()
if entry.get("namespace")
}
)
tool_choice = request.effective_tool_choice()
required_json = self._uses_required_tool_json(request, chat_tools)
client_tool_search = self._client_tool_search(request)
encoding_output = self._encoding_output()
token_stream = (
encoding_output.new_responses_stream(
@@ -2453,7 +2813,13 @@ class OpenAIServingResponses(OpenAIServingChat):
return []
arguments = state["arguments"]
events: list = []
if state["custom"]:
done_event = openai_responses_types.ResponseOutputItemDoneEvent
if state["search"]:
completed_item = _tool_search_call(
state["item_id"], state["call_id"], arguments
)
done_event = ResponseToolSearchCallEvent
elif state["custom"]:
payload = decode_custom_tool_input(arguments)
# Deltas cannot be retracted, so a payload that no longer
# extends what was already streamed defers to the streamed text.
@@ -2520,7 +2886,7 @@ class OpenAIServingResponses(OpenAIServingChat):
)
events.append(
_send_event(
openai_responses_types.ResponseOutputItemDoneEvent(
done_event(
type="response.output_item.done",
sequence_number=-1,
output_index=state["output_index"],
@@ -2717,27 +3083,43 @@ class OpenAIServingResponses(OpenAIServingChat):
current_output_index += 1
name = call.name or ""
is_custom = name in custom_names
is_search = client_tool_search and name == TOOL_SEARCH
namespace, name = namespace_names.get(name, (None, name))
prefix = (
"ctc" if is_custom else "tsc" if is_search else "fc"
)
state = {
"item_id": (
f"ctc_{random_uuid()[:8]}"
if is_custom
else f"fc_{random_uuid()[:8]}"
),
"item_id": f"{prefix}_{random_uuid()[:8]}",
"call_id": f"call_{random_uuid()[:24]}",
"output_index": current_output_index,
"name": name,
"namespace": namespace,
"arguments": "",
"custom": is_custom,
"search": is_search,
"payload": "",
"added": False,
"done": False,
}
tool_call_states[tool_index] = state
self._record_called_name(
request, state["call_id"], call.name or ""
)
if not state["added"]:
state["added"] = True
if state["custom"]:
added_event = (
openai_responses_types.ResponseOutputItemAddedEvent
)
if state["search"]:
# Search arguments are reported whole when it completes.
added_item = ResponseToolSearchCall(
id=state["item_id"],
call_id=state["call_id"],
arguments={},
status="in_progress",
)
added_event = ResponseToolSearchCallEvent
elif state["custom"]:
added_item = ResponseCustomToolCall(
type="custom_tool_call",
id=state["item_id"],
@@ -2756,7 +3138,7 @@ class OpenAIServingResponses(OpenAIServingChat):
**_namespace_field(state),
)
yield _send_event(
openai_responses_types.ResponseOutputItemAddedEvent(
added_event(
type="response.output_item.added",
sequence_number=-1,
output_index=state["output_index"],
@@ -2786,7 +3168,7 @@ class OpenAIServingResponses(OpenAIServingChat):
delta=delta,
)
)
else:
elif not state["search"]:
yield _send_event(
openai_responses_types.ResponseFunctionCallArgumentsDeltaEvent(
type="response.function_call_arguments.delta",
@@ -2936,10 +3318,7 @@ class OpenAIServingResponses(OpenAIServingChat):
self.response_store[final_response.id] = final_response
self.msg_store[final_response.id] = self._response_input_history(
request
) + [
item.model_dump(exclude_none=True)
for item in final_response.output
]
) + self._replayed_output_items(request, final_response.output)
response_dict = _sanitize_response_dict(final_response.model_dump())
+690 -59
View File
@@ -18,7 +18,7 @@ import logging
import threading
import time
from dataclasses import dataclass
from queue import Empty, Queue
from queue import Empty, Full, Queue
from typing import TYPE_CHECKING, Any, Callable, List, NamedTuple, Optional
import torch
@@ -42,7 +42,9 @@ from sglang.srt.layers.dp_attention import (
)
from sglang.srt.mem_cache.l2_transfer import L2Transfer, L2TransferEngine
from sglang.srt.mem_cache.memory_pool import MLATokenToKVPool
from sglang.srt.mem_cache.pool_host.base import uses_shared_host_layout
from sglang.srt.mem_cache.utils import get_storage_hash_str
from sglang.srt.observability import batch_telemetry
from sglang.srt.runtime_context import get_parallel
from sglang.srt.utils import get_device_module
@@ -62,6 +64,31 @@ def storage_model_name(
return f"{model_name}-{tag}" if model_name else tag
# [HC_PARALLEL_PATCH] knobs for the pipelined HiCache prefetch/backup lanes
import os as _hc_os
_HC_PREFETCH_WORKERS = int(_hc_os.getenv("SGLANG_HICACHE_PREFETCH_IO_THREADS", "1") or 1)
_HC_BACKUP_WORKERS = int(_hc_os.getenv("SGLANG_HICACHE_BACKUP_IO_THREADS", "1") or 1)
_HC_TIMING = _hc_os.getenv("SGLANG_HICACHE_TIMING", "0") not in ("", "0", "false", "False")
# [ASYNC_BACKUP] Per-op pinned index copy + worker-side merge/submit.
# Only the direct + page_first_direct shape (the one that does a blocking
# device_indices.cpu() on the scheduler thread) is eligible.
_ASYNC_BACKUP_SUBMIT = _hc_os.getenv(
"SGLANG_ASYNC_BACKUP_SUBMIT", "0"
).strip().lower() not in ("", "0", "false", "no", "off")
_ASYNC_BACKUP_ENQUEUE_TIMEOUT = float(
_hc_os.getenv("SGLANG_ASYNC_BACKUP_ENQUEUE_TIMEOUT", "60") or 60
)
_ASYNC_BACKUP_IDLE_TIMEOUT = float(
_hc_os.getenv("SGLANG_ASYNC_BACKUP_IDLE_TIMEOUT", "120") or 120
)
_ASYNC_BACKUP_PINNED_BYTES_MAX = int(
_hc_os.getenv("SGLANG_ASYNC_BACKUP_PINNED_MB", "2048") or 2048
) * 1024 * 1024
_ASYNC_BACKUP_MAX_OPS = int(
_hc_os.getenv("SGLANG_ASYNC_BACKUP_MAX_OPS", "256") or 256
)
class LayerLoadingEvent:
def __init__(self, num_layers: int):
self._num_layers = num_layers
@@ -125,6 +152,9 @@ class CacheOperation:
self.node_ids = [node_id]
self.data = None
self.pool_transfers = pool_transfers
# Set when SGLANG_ASYNC_BACKUP_SUBMIT copied this op's device indices
# into pinned host buffers (see _start_async_index_copy).
self.async_parts = None
self.id = CacheOperation.counter
CacheOperation.counter += 1
@@ -185,6 +215,39 @@ class CacheOperation:
return self.priority < other.priority
@dataclass
class AsyncIndexCopy:
"""One device->pinned-host index copy started where the backup op was made."""
key: tuple
src: torch.Tensor
dst: torch.Tensor
nbytes: int
@dataclass
class AsyncIndexParts:
"""Per-op pinned index parts plus the event that guarantees they are final."""
copies: List[AsyncIndexCopy]
ready_event: object
nbytes: int
@dataclass
class WriteSubmission:
ops: List[CacheOperation]
tracer: object
enqueue_mono_ns: int
@dataclass
class WriteResult:
ack: Optional[HiCacheAck]
ops: List[CacheOperation]
error: Optional[BaseException]
class HiCacheAck(NamedTuple):
start_event: device_module.Event
finish_event: device_module.Event
@@ -376,6 +439,21 @@ class HiCacheController:
self.write_fence_stream = None
self.ack_write_queue: List[HiCacheAck] = []
# [ASYNC_BACKUP] Pinned index copies are started where the backup op is
# created; a single worker waits the per-op events, merges the CPU
# index parts and submits the KV copy. Queues are bounded by the state
# lock/condition below; write_queue/ack_write_queue stay scheduler-only.
self._indices_stream = None
self._write_submit_queue: Optional[Queue] = None
self._write_result_queue: Queue = Queue()
self._write_submit_stop = threading.Event()
self._write_submit_thread: Optional[threading.Thread] = None
self._write_submit_state_lock = threading.Lock()
self._write_submit_cond = threading.Condition(self._write_submit_state_lock)
self._write_submit_active = 0
self._async_pinned_bytes = 0
self._retire_parts: List["AsyncIndexParts"] = []
self.l2_transfer_engine = L2TransferEngine(io_backend)
# If a storage backend is provided at startup, treat it as an implicit attach,
@@ -701,6 +779,9 @@ class HiCacheController:
Requirement: no in-flight requests. This will stop storage threads and release
the backend instance (best-effort close).
"""
# Async backup submissions own pinned index buffers and ACKs; stop them
# before any queue/pool teardown so no result is lost.
self._stop_write_submission_thread()
# Idempotent cleanup: even if `enable_storage` is already False,
# we may still have leftover resources (threads/backend/process group) from a
# previous partial detach. We attempt cleanup whenever possible.
@@ -815,6 +896,7 @@ class HiCacheController:
def reset(self):
# Reuse detach's queue wakeups and bounded joins, and fail if any
# storage thread remains alive before clearing shared state.
self._stop_write_submission_thread()
self._stop_storage_threads()
self.write_queue.clear()
@@ -854,13 +936,311 @@ class HiCacheController:
def has_inflight_device_transfers(self) -> bool:
"""Whether queued or unacknowledged L2 transfers still use device rows."""
with self._write_submit_state_lock:
async_pending = (
self._write_submit_active > 0
or self._async_pinned_bytes > 0
or not self._write_result_queue.empty()
)
return bool(
self.write_queue
or self.load_queue
or self.ack_write_queue
or self.ack_load_queue
or async_pending
)
# ---- async backup submission -----------------------------------------
def async_backup_enabled(self) -> bool:
return (
_ASYNC_BACKUP_SUBMIT
and self.io_backend == "direct"
and getattr(self.mem_pool_host, "layout", None) == "page_first_direct"
)
def _async_submit_applicable(self) -> bool:
return (
self.async_backup_enabled()
and len(self.write_queue) <= _ASYNC_BACKUP_MAX_OPS
and all(op.async_parts is not None for op in self.write_queue)
)
def _ensure_indices_stream(self):
if self._indices_stream is None:
self._indices_stream = device_module.Stream()
return self._indices_stream
def _needs_device_index_copy(self, host_pool, indices) -> bool:
if indices is None or not getattr(indices, "is_cuda", False):
return False
# move_indices() returns shared-layout ids untouched; converting them
# here would hand the transfer a different interface than the sync path.
return not uses_shared_host_layout(host_pool)
def _async_index_targets(self, op: CacheOperation):
targets = []
if self._needs_device_index_copy(
self.mem_pool_host.anchor_entry.host_pool, op.device_indices
):
targets.append(((None, None), op.device_indices))
for transfer in op.pool_transfers or []:
if transfer.device_indices is None:
continue
entry = self.mem_pool_host.entry_map.get(transfer.name)
if entry is None:
continue
if not self._needs_device_index_copy(
entry.host_pool, transfer.device_indices
):
continue
key = (transfer.name, transfer.indices_from_pool)
targets.append((key, transfer.device_indices))
return targets
def _start_async_index_copy(self, op: CacheOperation) -> None:
"""Start the pinned index D2H copy now, off the scheduler stream.
Runs in the scheduler thread at backup-op creation time. The copy runs
on a dedicated stream ordered after this instant, so it does not inherit
later fences (the WAR barrier of a later forward) and the scheduler
never blocks in ``.cpu()``.
"""
if not self.async_backup_enabled():
return
targets = self._async_index_targets(op)
if not targets:
return
nbytes = sum(int(t.numel()) * int(t.element_size()) for _, t in targets)
with self._write_submit_state_lock:
if self._async_pinned_bytes + nbytes > _ASYNC_BACKUP_PINNED_BYTES_MAX:
return
self._async_pinned_bytes += nbytes
try:
stream = self._ensure_indices_stream()
source_stream = device_module.current_stream()
copies: List[AsyncIndexCopy] = []
with device_module.StreamContext(stream):
stream.wait_stream(source_stream)
for key, src in targets:
# Keep the device rows alive until the copy has run, and pin
# the destination so copy_ is a real asynchronous D2H.
src.record_stream(stream)
dst = torch.empty(
src.shape,
dtype=src.dtype,
device="cpu",
pin_memory=True,
)
dst.copy_(src, non_blocking=True)
copies.append(
AsyncIndexCopy(
key=key,
src=src,
dst=dst,
nbytes=int(src.numel()) * int(src.element_size()),
)
)
ready_event = device_module.Event()
ready_event.record(stream)
except BaseException:
with self._write_submit_state_lock:
self._async_pinned_bytes -= nbytes
logger.exception("Async backup index copy failed; keeping sync path")
return
op.async_parts = AsyncIndexParts(
copies=copies, ready_event=ready_event, nbytes=nbytes
)
def _ensure_write_submit_thread(self) -> None:
if self._write_submit_queue is None:
self._write_submit_queue = Queue(maxsize=64)
thread = self._write_submit_thread
if thread is not None and thread.is_alive():
return
self._write_submit_stop.clear()
thread = threading.Thread(
target=self._write_submit_worker,
name="hicache-async-backup",
daemon=True,
)
self._write_submit_thread = thread
thread.start()
def _enqueue_async_write(self, ops, tracer) -> bool:
"""Enqueue one submission; False means "fall back to the sync path".
A full queue or a timeout is not fatal: the caller submits the same ops
synchronously. Before returning False we wait for the worker to become
idle so the fallback cannot race another submission on the shared
device_to_host_stream.
"""
self._ensure_write_submit_thread()
item = WriteSubmission(
ops=ops, tracer=tracer, enqueue_mono_ns=time.monotonic_ns()
)
with self._write_submit_cond:
self._write_submit_active += 1
try:
with batch_telemetry.backup_phase("enqueue"):
self._write_submit_queue.put(
item, timeout=_ASYNC_BACKUP_ENQUEUE_TIMEOUT
)
except Full:
with self._write_submit_cond:
self._write_submit_active -= 1
self._write_submit_cond.notify_all()
logger.warning(
"Async backup submit queue full after %.1fs; falling back to "
"the synchronous path",
_ASYNC_BACKUP_ENQUEUE_TIMEOUT,
)
self.wait_write_submissions_idle()
return False
return True
def _materialize_async_indices(self, ops) -> None:
"""Replace device index tensors with the pinned CPU copies (worker)."""
for op in ops:
parts = op.async_parts
if parts is None:
continue
mapping = {copy.key: copy.dst for copy in parts.copies}
anchor = mapping.get((None, None))
if anchor is not None:
op.device_indices = anchor
for transfer in op.pool_transfers or []:
dst = mapping.get((transfer.name, transfer.indices_from_pool))
if dst is not None:
transfer.device_indices = dst
def _write_submit_worker(self) -> None:
while True:
item = self._write_submit_queue.get()
if item is None:
break
tracer = item.tracer
previous = batch_telemetry.backup_flush_attach(tracer)
try:
with batch_telemetry.backup_phase("worker"):
for op in item.ops:
parts = op.async_parts
if parts is not None:
parts.ready_event.synchronize()
self._materialize_async_indices(item.ops)
merged = CacheOperation.merge_ops(item.ops)
ack = self._submit_write_operation(merged)
result = WriteResult(ack=ack, ops=item.ops, error=None)
except BaseException as exc: # noqa: BLE001 - fail closed
logger.exception("Async backup submission failed")
result = WriteResult(ack=None, ops=item.ops, error=exc)
finally:
batch_telemetry.backup_flush_detach(previous)
self._release_async_parts(item.ops)
try:
batch_telemetry.backup_flush_end(tracer)
except Exception:
pass
# Publish the result before marking the submission done, so an idle
# waiter cannot observe active==0 with the result still missing.
self._write_result_queue.put(result)
with self._write_submit_cond:
self._write_submit_active -= 1
self._write_submit_cond.notify_all()
def _release_async_parts(self, ops) -> None:
released = 0
for op in ops:
parts = op.async_parts
if parts is None:
continue
op.async_parts = None
parts.copies.clear()
released += parts.nbytes
if released:
with self._write_submit_state_lock:
self._async_pinned_bytes -= released
def _retire_async_parts(self, ops) -> None:
"""Release copies that will not be consumed (sync fallback)."""
for op in ops:
parts = op.async_parts
if parts is not None:
self._retire_parts.append(parts)
op.async_parts = None
self._reap_retired_parts()
def _reap_retired_parts(self) -> None:
if not self._retire_parts:
return
pending = []
for parts in self._retire_parts:
try:
done = parts.ready_event.query()
except Exception:
done = True
if done:
parts.copies.clear()
with self._write_submit_state_lock:
self._async_pinned_bytes -= parts.nbytes
else:
pending.append(parts)
self._retire_parts = pending
def drain_write_submission_results(self) -> None:
"""Move finished async ACKs into ack_write_queue (scheduler thread)."""
while True:
try:
result = self._write_result_queue.get_nowait()
except Empty:
break
if result.error is not None:
raise RuntimeError("async backup submission failed") from result.error
if result.ack is not None:
self.ack_write_queue.append(result.ack)
self._reap_retired_parts()
def wait_write_submissions_idle(
self, timeout: float = _ASYNC_BACKUP_IDLE_TIMEOUT
) -> None:
deadline = time.monotonic() + timeout
with self._write_submit_cond:
while self._write_submit_active > 0:
remaining = deadline - time.monotonic()
if remaining <= 0:
raise RuntimeError(
"Timed out waiting for async backup submissions"
)
self._write_submit_cond.wait(remaining)
def _stop_write_submission_thread(self) -> None:
if self._write_submit_thread is None:
self.drain_write_submission_results()
self._reap_retired_parts()
if self._retire_parts:
raise RuntimeError("async backup index copies still in flight")
return
try:
self.wait_write_submissions_idle()
finally:
self._write_submit_stop.set()
queue = self._write_submit_queue
if queue is not None:
try:
queue.put(None, timeout=5.0)
except Exception:
logger.exception("Failed to wake the async backup worker")
thread = self._write_submit_thread
if thread is not None:
thread.join(timeout=30.0)
if thread.is_alive():
raise RuntimeError("async backup submit worker did not stop")
self.drain_write_submission_results()
if self._retire_parts:
raise RuntimeError("async backup index copies still in flight")
self._write_submit_thread = None
self._write_submit_queue = None
def write(
self,
device_indices: torch.Tensor,
@@ -876,47 +1256,144 @@ class HiCacheController:
host_indices = self.mem_pool_host.alloc(len(device_indices))
if host_indices is None:
return None
self.write_queue.append(
CacheOperation(host_indices, device_indices, node_id, priority)
)
op = CacheOperation(host_indices, device_indices, node_id, priority)
self._start_async_index_copy(op)
self.write_queue.append(op)
self.start_writing()
return host_indices
def start_writing(self) -> None:
if len(self.write_queue) == 0:
# Empty flushes are the common case when write-through fires every
# batch: counted so a small `flush_backups` cannot be explained as
# "one big flush per loop" without noticing the empty ones.
batch_telemetry.detail_count("flush_calls")
batch_telemetry.detail_count("flush_empty")
return
op = CacheOperation.merge_ops(self.write_queue)
host_indices, device_indices, pool_transfers = self._move_write_operation(op)
self.write_queue.clear()
ops = list(self.write_queue)
tracer = batch_telemetry.backup_flush_begin()
async_owned = False
try:
batch_telemetry.detail_count("flush_calls")
batch_telemetry.detail_count("flush_nonempty")
self._note_write_queue(ops, len(self.ack_write_queue))
if self._async_submit_applicable():
if self._enqueue_async_write(ops, tracer):
self.write_queue.clear()
batch_telemetry.flush_note(
"sglang.flush_submission_mode", "async"
)
# The worker owns the tracer from here on (it emits once).
async_owned = True
return
# Queue full / enqueue timeout: retire the pinned copies and
# submit synchronously on this thread. `_enqueue_async_write`
# waited for the worker to go idle first, so the two paths
# never submit to device_to_host_stream concurrently.
batch_telemetry.flush_note("sglang.flush_fallback_sync", 1)
self._retire_async_parts(ops)
# Any parts that were started for this queue are not consumed by the
# sync path. Retire them unconditionally (they are released once
# their events complete) so the pinned-byte budget never leaks.
batch_telemetry.flush_note(
"sglang.flush_async_skip_ops",
sum(1 for op in ops if op.async_parts is None),
)
self._retire_async_parts(ops)
batch_telemetry.flush_note("sglang.flush_submission_mode", "sync")
with batch_telemetry.backup_phase("merge_ops"):
op = CacheOperation.merge_ops(self.write_queue)
self.write_queue.clear()
ack = self._submit_write_operation(op)
with batch_telemetry.backup_phase("ack_enqueue"):
self.ack_write_queue.append(ack)
batch_telemetry.flush_note(
"sglang.flush_ack_queue_after", len(self.ack_write_queue)
)
finally:
if not async_owned:
batch_telemetry.backup_flush_end(tracer)
def _note_write_queue(self, ops: List[CacheOperation], ack_before: int) -> None:
"""Queue-level flush dimensions (same for the sync and async paths)."""
batch_telemetry.flush_note("sglang.flush_write_queue_ops", len(ops))
batch_telemetry.flush_note("sglang.flush_io_backend", self.io_backend)
batch_telemetry.flush_note(
"sglang.flush_layout",
str(getattr(self.mem_pool_host, "layout", None) or ""),
)
batch_telemetry.flush_note("sglang.flush_ack_queue_before", ack_before)
def _submit_write_operation(self, op: CacheOperation) -> HiCacheAck:
"""Move indices, submit the D2H copy and build the ACK.
Runs on the scheduler thread (sync path) or on the async submit worker
(async path, with CPU-resident indices). It must not touch
``write_queue`` or ``ack_write_queue``; the caller owns those.
"""
tokens = len(op.device_indices)
indices = op.device_indices
batch_telemetry.flush_note("sglang.flush_tokens", int(tokens))
batch_telemetry.flush_note("sglang.flush_nodes_merged", len(op.node_ids))
batch_telemetry.flush_note(
"sglang.flush_sidecar_pools", len(op.pool_transfers or [])
)
batch_telemetry.flush_note(
"sglang.flush_indices_device", str(indices.device.type)
)
batch_telemetry.flush_note("sglang.flush_indices_dtype", str(indices.dtype))
batch_telemetry.flush_note(
"sglang.flush_indices_numel", int(indices.numel())
)
with batch_telemetry.backup_phase("move_write"):
host_indices, device_indices, pool_transfers = (
self._move_write_operation(op)
)
batch_telemetry.flush_note(
"sglang.flush_host_indices_on_cpu",
bool(getattr(host_indices, "is_cuda", False) is False),
)
batch_telemetry.flush_note(
"sglang.flush_device_indices_on_cpu",
bool(getattr(device_indices, "is_cuda", False) is False),
)
if self.write_fence_stream is not None:
# The copy may read KV a forward is still writing: under overlap
# scheduling a finished request is cached while the next forward
# writes its last token, and an evicted SWA page can belong to an
# unfinished checkpoint. Later forwards still overlap with the copy.
self.l2_transfer_engine.device_to_host_stream.wait_stream(
self.write_fence_stream
with batch_telemetry.backup_phase("write_fence"):
self.l2_transfer_engine.device_to_host_stream.wait_stream(
self.write_fence_stream
)
with batch_telemetry.backup_phase("build_l2"):
transfers = self._l2_transfers(
host_indices, device_indices, pool_transfers
)
batch_telemetry.flush_note("sglang.flush_l2_transfers", len(transfers))
with batch_telemetry.backup_phase("submit_d2h"):
completion = self.l2_transfer_engine.submit_device_to_host(transfers)
with batch_telemetry.backup_phase("set_done_event"):
self.mem_pool_device_allocator.set_hicache_transfer_done_event(
(id(self), "write"), completion.finish_event
)
completion = self.l2_transfer_engine.submit_device_to_host(
self._l2_transfers(host_indices, device_indices, pool_transfers)
)
self.mem_pool_device_allocator.set_hicache_transfer_done_event(
(id(self), "write"), completion.finish_event
)
self.ack_write_queue.append(
HiCacheAck(
start_event=completion.start_event,
finish_event=completion.finish_event,
node_ids=op.node_ids,
num_tokens=len(op.device_indices),
timing_enabled=completion.timing_enabled,
num_tokens_by_pool=self._num_tokens_by_pool(op),
num_bytes=self._transfer_num_bytes(op),
)
return HiCacheAck(
start_event=completion.start_event,
finish_event=completion.finish_event,
node_ids=op.node_ids,
num_tokens=len(op.device_indices),
timing_enabled=completion.timing_enabled,
num_tokens_by_pool=self._num_tokens_by_pool(op),
num_bytes=self._transfer_num_bytes(op),
)
def _transfer_num_bytes(self, op: CacheOperation) -> int:
@@ -948,15 +1425,24 @@ class HiCacheController:
# move indices to GPU if using kernels, to host if using direct indexing
if self.io_backend == "kernel":
if not host_indices.is_cuda:
host_indices = host_indices.to(self.device, non_blocking=True)
with batch_telemetry.backup_phase("indices_h2d"):
host_indices = host_indices.to(self.device, non_blocking=True)
return host_indices, device_indices
elif self.io_backend == "direct":
if self.mem_pool_host.layout == "layer_first":
device_indices = device_indices.cpu()
host_indices, idx = host_indices.sort()
return host_indices, device_indices.index_select(0, idx)
with batch_telemetry.backup_phase("indices_cpu"):
device_indices = device_indices.cpu()
with batch_telemetry.backup_phase("index_sort"):
host_indices, idx = host_indices.sort()
with batch_telemetry.backup_phase("index_select"):
return host_indices, device_indices.index_select(0, idx)
elif self.mem_pool_host.layout == "page_first_direct":
return host_indices, device_indices.cpu()
# Synchronous D2H of the token indices: the host wall time of
# this call is the scheduler blocking on the copy, not a
# device bandwidth measurement.
with batch_telemetry.backup_phase("indices_cpu"):
device_indices = device_indices.cpu()
return host_indices, device_indices
else:
raise ValueError(
f"Unsupported layout {self.mem_pool_host.layout!r} for io backend 'direct'"
@@ -1152,7 +1638,11 @@ class HiCacheController:
count += 1
return count
def _page_transfer(self, operation: PrefetchOperation) -> int:
def _page_transfer(
self,
operation: PrefetchOperation,
ack_sink: Optional[Callable[[PrefetchAck], None]] = None,
) -> int:
# Transfer batch by batch
prefix_keys = operation.prefix_keys
kv_derived_transfers = [
@@ -1202,9 +1692,28 @@ class HiCacheController:
completed_tokens=completed_pages * self.page_size,
operation=operation,
)
self.prefetch_sync_queue.put(ack)
if ack_sink is None:
# Keep the direct-call behavior for existing users. The pipelined
# coordinator always supplies a sink so workers never share the queue.
self.prefetch_sync_queue.put(ack)
else:
ack_sink(ack)
return completed_pages
def _transfer_prefetch_and_collect_acks(
self, operation: PrefetchOperation
) -> List[PrefetchAck]:
"""Run one worker without publishing ACKs to the shared sync queue."""
acks = []
try:
self._page_transfer(operation, ack_sink=acks.append)
except Exception:
logger.exception(
"HiCache prefetch transfer failed for request %s",
operation.request_id,
)
return acks
def _page_transfer_kv_batch(
self,
operation: PrefetchOperation,
@@ -1247,24 +1756,86 @@ class HiCacheController:
def prefetch_io_aux_func(self):
"""
Auxiliary function conducting IO operations for prefetching.
# [HC_PARALLEL_PATCH] pipelined: up to SGLANG_HICACHE_PREFETCH_IO_THREADS
# transfers in flight. Every ACK of one operation is published before any
# ACK of the next operation, keeping per-rank collective pairing unchanged.
"""
while not self.storage_stop_event.is_set():
try:
operation = self.prefetch_buffer.get(block=True, timeout=1)
except Empty:
continue
if operation is None:
continue
try:
self._page_transfer(operation)
finally:
self.prefetch_sync_queue.put(
PrefetchAck(
rid=operation.request_id,
completed_req=True,
operation=operation,
import time as _time
from collections import deque as _deque
from concurrent.futures import ThreadPoolExecutor as _TPE
K = max(1, _HC_PREFETCH_WORKERS)
timing = _HC_TIMING
ex = _TPE(max_workers=K, thread_name_prefix="hc-prefetch-io")
inflight = _deque()
lat = []
pages = 0
try:
while not self.storage_stop_event.is_set():
while len(inflight) < K:
try:
operation = self.prefetch_buffer.get_nowait()
except Empty:
break
if operation is None:
continue
inflight.append(
(
operation,
_time.perf_counter(),
ex.submit(
self._transfer_prefetch_and_collect_acks, operation
),
)
)
)
if not inflight:
try:
operation = self.prefetch_buffer.get(block=True, timeout=1)
except Empty:
continue
if operation is None:
continue
inflight.append(
(
operation,
_time.perf_counter(),
ex.submit(
self._transfer_prefetch_and_collect_acks, operation
),
)
)
operation, t0, fut = inflight[0]
try:
acks = fut.result()
except Exception:
logger.exception("HiCache prefetch transfer failed")
acks = []
finally:
inflight.popleft()
if timing:
lat.append((_time.perf_counter() - t0) * 1000.0)
pages += len(getattr(operation, "hash_value", []) or [])
for ack in acks:
self.prefetch_sync_queue.put(ack)
self.prefetch_sync_queue.put(
PrefetchAck(
rid=operation.request_id,
completed_req=True,
operation=operation,
)
)
if timing and len(lat) >= 100:
lat.sort()
print("[HC_TIMING] prefetch queue=%d inflight=%d n=%d transfer_ms p50=%.1f p95=%.1f pages=%d"
% (self.prefetch_buffer.qsize(), len(inflight), len(lat),
lat[len(lat) // 2], lat[int(len(lat) * 0.95)], pages), flush=True)
lat = []
pages = 0
finally:
# Detach closes the backend after joining this outer thread. Do not
# let pool workers outlive that join and touch released state.
ex.shutdown(wait=True)
def prefetch_rate_limited(self) -> bool:
"""
@@ -1443,22 +2014,82 @@ class HiCacheController:
"HiCache backup failed after %d tokens", operation.completed_tokens
)
def _should_skip_backup(self) -> bool:
"""Whether this rank can ACK without doing any storage IO."""
return self.backup_skip
def backup_thread_func(self):
"""
Manage backup operations from host memory to storage backend.
# [HC_PARALLEL_PATCH] pipelined: up to SGLANG_HICACHE_BACKUP_IO_THREADS
# writes in flight, acked in FIFO order.
"""
while not self.storage_stop_event.is_set():
try:
operation = self.backup_queue.get(block=True, timeout=1)
if operation is None:
continue
import time as _time
from collections import deque as _deque
from concurrent.futures import ThreadPoolExecutor as _TPE
if not self.backup_skip:
self._page_backup_or_log(operation)
K = max(1, _HC_BACKUP_WORKERS)
timing = _HC_TIMING
ex = _TPE(max_workers=K, thread_name_prefix="hc-backup-io")
inflight = _deque()
lat = []
pages = 0
def submit(operation) -> bool:
if self._should_skip_backup():
self.ack_backup_queue.put(operation)
return False
inflight.append(
(
operation,
_time.perf_counter(),
ex.submit(self._page_backup_or_log, operation),
)
)
return True
except Empty:
continue
try:
while not self.storage_stop_event.is_set():
while len(inflight) < K:
try:
operation = self.backup_queue.get_nowait()
except Empty:
break
if operation is None:
continue
submit(operation)
if not inflight:
try:
operation = self.backup_queue.get(block=True, timeout=1)
except Empty:
continue
if operation is None:
continue
if not submit(operation):
continue
operation, t0, fut = inflight[0]
try:
fut.result()
except Exception:
logger.exception("HiCache backup failed")
finally:
inflight.popleft()
if timing:
lat.append((_time.perf_counter() - t0) * 1000.0)
pages += len(getattr(operation, "hash_value", []) or [])
self.ack_backup_queue.put(operation)
if timing and len(lat) >= 100:
lat.sort()
print("[HC_TIMING] backup queue=%d inflight=%d n=%d write_ms p50=%.1f p95=%.1f pages=%d"
% (self.backup_queue.qsize(), len(inflight), len(lat),
lat[len(lat) // 2], lat[int(len(lat) * 0.95)], pages), flush=True)
lat = []
pages = 0
finally:
# The outer thread is joined before backend/process-group teardown.
# Wait for every submitted write before reporting that thread stopped.
ex.shutdown(wait=True)
def prefetch_sync_thread_func(self):
"""Synchronize prefetch results across all PP and TP ranks."""
File diff suppressed because it is too large Load Diff
@@ -201,6 +201,8 @@ class LoadSnapshot(msgspec.Struct, omit_defaults=True):
num_running_reqs: int = 0
num_waiting_reqs: int = 0
num_waiting_uncached_tokens: int = 0
queued_total_input_tokens: int = 0
drain_rate_tokens_per_s: float = 0.0
num_used_tokens: int = 0
num_total_tokens: int = 0
# num_total_tokens minus tokens still awaiting a KV transfer (equal to it
+18 -7
View File
@@ -120,6 +120,7 @@ from sglang.srt.mem_cache.common import (
)
from sglang.srt.mem_cache.memory_pool import HybridReqToTokenPool, ReqToTokenPool
from sglang.srt.mem_cache.unified_cache.component_type import ComponentType
from sglang.srt.observability import batch_telemetry
from sglang.srt.model_executor.encoder_swa_replay import resets_swa_window
from sglang.srt.model_executor.forward_batch_info import (
CaptureHiddenMode,
@@ -1148,6 +1149,10 @@ class Req(ReqDllmMixin):
# Full-KV-derived boundary whose SWA window should be inserted after
# the current prefill pass.
self.swa_branching_seqlen: Optional[int] = None
# Observability: Full-KV tokens rejected by the SWA prefix-match gate.
self.swa_gate_reject_tokens: int = 0
# Longest Full-KV match before component gates (set by match_kv_cache).
self.full_kv_hit_length: int = 0
# for cross-encoder model
self.token_type_ids = token_type_ids
@@ -2083,6 +2088,8 @@ class Req(ReqDllmMixin):
self.num_matched_prefix_tokens = 0
self.lock = None
self.swa_branching_seqlen = None
self.swa_gate_reject_tokens = 0
self.full_kv_hit_length = 0
self.extend_end = None
self.dllm_initialized = False
self.is_retracted = True
@@ -2909,9 +2916,12 @@ class ScheduleBatch(ScheduleBatchDisaggregationDecodeMixin):
)
self.encoder_swa_reset = [resets_swa_window(r) for r in reqs]
# Allocate memory
out_cache_loc, req_pool_indices_tensor, req_pool_indices_cpu = alloc_for_extend(
self
)
with batch_telemetry.detail_phase("alloc_for_extend"):
(
out_cache_loc,
req_pool_indices_tensor,
req_pool_indices_cpu,
) = alloc_for_extend(self)
# Set fields
input_embeds = []
@@ -3127,10 +3137,11 @@ class ScheduleBatch(ScheduleBatchDisaggregationDecodeMixin):
self.prepare_encoder_info_extend(input_ids, seq_lens)
# Build sampling info
self.sampling_info = SamplingBatchInfo.from_schedule_batch(
self,
self.model_config.vocab_size,
)
with batch_telemetry.detail_phase("sampling_batch_info"):
self.sampling_info = SamplingBatchInfo.from_schedule_batch(
self,
self.model_config.vocab_size,
)
def _mamba_radix_cache_v2_req_prepare_for_extend(
self,
@@ -28,6 +28,25 @@ logger = logging.getLogger(__name__)
# See the License for the specific language governing permissions and
# limitations under the License.
# ==============================================================================
def observe_swa_gate_reject_on_admission(req, admitted_prefix_len: int) -> int:
"""Emit the SWA prefix-match gate's cost once per prefill admission.
``match_kv_cache`` also runs on waiting-queue probes, PD decode and HiCache
rematches; only the prefill admission knows the final accepted prefix (after
any host load-back), so the counter is incremented here. A retracted and
re-admitted request is counted again because it is re-prefilled.
"""
gap = max(0, getattr(req, "full_kv_hit_length", 0) - admitted_prefix_len)
req.swa_gate_reject_tokens = gap
if gap > 0:
time_stats = getattr(req, "time_stats", None)
collector = getattr(time_stats, "metrics_collector", None)
observe = getattr(collector, "observe_swa_gate_reject_admission", None)
if observe is not None and getattr(time_stats, "enable_metrics", False):
observe(gap)
return gap
"""Request scheduler policy"""
import os
@@ -1622,6 +1641,7 @@ class PrefillAdder:
) -> None:
assert req.prefix_len == admission.prefix_len
req.extend_end = admission.prefix_len + admission.extend_len
observe_swa_gate_reject_on_admission(req, admission.prefix_len)
req.lock = self.tree_cache.lock(req.last_node)
self.can_run_list.append(req)
if admission.is_chunked:
+215 -49
View File
@@ -304,6 +304,7 @@ from sglang.srt.model_executor.forward_batch_info import PPProxyTensors
from sglang.srt.model_executor.runner_utils.pool import prewarm_graph_pool_borrow
from sglang.srt.model_loader.utils import get_resolved_model_impl
from sglang.srt.multiplex.multiplexing_mixin import SchedulerMultiplexMixin
from sglang.srt.observability import batch_telemetry
from sglang.srt.observability.metrics_collector import SchedulerMetricsCollector
from sglang.srt.observability.req_time_stats import (
flush_trace_batch,
@@ -644,6 +645,14 @@ class Scheduler(
)
else:
self.decode_offload_manager = None
# Prefill read-done lets the schedule stream run ahead of the forward that
# writes a stashed chunk. The cache controller's fence streams above order
# its copies behind that forward; the decode offload controller has none.
self.tp_worker.model_runner.kv_offload_fenced_on_forward_stream = (
self.enable_hierarchical_cache
and self.tree_cache.cache_controller is not None
and self.decode_offload_manager is None
)
# Init running status
self.init_running_status()
@@ -1416,6 +1425,8 @@ class Scheduler(
metrics_collector_context=self.metrics_collector_context,
metrics_collector=self.metrics_collector,
)
# Read the already-running DeviceTimer; adds no CUDA events.
batch_telemetry.install_forward_reporter(self)
def init_schedule_policy(self):
# Init schedule policy and new token estimation
@@ -2178,6 +2189,10 @@ class Scheduler(
# Count successful receive/broadcast time as active; empty polls stay idle.
self.metrics_reporter.record_scheduler_active(recv_start_ns)
self.process_input_requests(recv_reqs)
if recv_reqs:
# The last snapshot may predate these requests (it is published after
# each forward, before the requests sent during it are read).
self.publish_load_snapshot(force=True)
return recv_reqs
@scheduler_stage_method(SCHEDULER_STAGE_PROCESS_REQUESTS)
@@ -2563,6 +2578,7 @@ class Scheduler(
),
get_stats=lambda: self.metrics_reporter.stats,
get_chunked_req=lambda: self.chunked_req,
get_grammar_queue=lambda: self.grammar_manager.grammar_queue,
get_disagg_prefill_bootstrap_queue=lambda: (
self.disagg_prefill_bootstrap_queue
),
@@ -3856,8 +3872,17 @@ class Scheduler(
def get_next_batch_to_run(
self, running_batch: ScheduleBatch, last_batch: Optional[ScheduleBatch]
) -> NextBatchPlan:
bt_pass = batch_telemetry.begin_pass()
if bt_pass is not None:
# Capture before the chunked request can be consumed by this pass.
batch_telemetry.note_had_chunked(self.chunked_req is not None)
self.process_pending_chunked_abort()
if bt_pass is not None:
_bt_t = time.monotonic()
self._process_hicache_events()
if bt_pass is not None:
batch_telemetry.add_hicache(time.monotonic() - _bt_t)
if self.dllm_config is not None:
self.dllm_manager.filter_finished_reqs()
@@ -3948,7 +3973,11 @@ class Scheduler(
# Before merging the new batch into running batch:
# 1. All new batches are none -> need_mlp_sync remains true (sync is needed for decode batch).
# 2. All new batches are some (prefill / idle) -> we do not need prepare mlp sync one more time.
if bt_pass is not None:
_bt_t = time.monotonic()
new_batch = self.dp_attn_adapter.maybe_prepare_mlp_sync_batch(new_batch)
if bt_pass is not None:
batch_telemetry.add_mlp_sync(time.monotonic() - _bt_t)
need_mlp_sync = new_batch is None
if new_batch is not None:
@@ -3967,9 +3996,13 @@ class Scheduler(
ret = None
# Handle DP attention and log stats
if bt_pass is not None:
_bt_t = time.monotonic()
ret = self.dp_attn_adapter.maybe_prepare_mlp_sync_batch(
ret, need_sync=need_mlp_sync
)
if bt_pass is not None:
batch_telemetry.add_mlp_sync(time.monotonic() - _bt_t)
# Decode->extend conversion keeps a heterogeneous dp step replayable.
# A converted running batch stays the running batch; next step it comes
# back as last_batch and is filtered in place, not merged with itself.
@@ -3984,6 +4017,9 @@ class Scheduler(
if ret:
set_schedule_time_batch(ret)
if bt_pass is not None:
batch_telemetry.end_pass(dp_rank=get_parallel().attn_dp_rank)
return NextBatchPlan(batch_to_run=ret, running_batch=running_batch)
def get_num_allocatable_reqs(
@@ -4088,11 +4124,18 @@ class Scheduler(
return None, running_batch
# Get priority queue
_pf_t0 = time.monotonic_ns()
self.policy.calc_priority(
self.waiting_queue,
running_batch,
processed_tokens=self.processed_tokens_counter,
)
try:
from sglang.srt.observability import fwd_timeline
fwd_timeline.detail("prefill_policy", _pf_t0, time.monotonic_ns())
except Exception:
fwd_timeline = None
if TEST_RETRACT and running_bs > TEST_RETRACT_NO_PREFILL_BS:
# If we are testing retraction and the running batch size exceeds
@@ -4116,6 +4159,7 @@ class Scheduler(
else:
prefill_tile_block_m = 64 # Fallback for non-Triton backends
_pf_t1 = time.monotonic_ns()
adder = self.get_prefill_adder_class()(
self.page_size,
self.tree_cache,
@@ -4158,9 +4202,12 @@ class Scheduler(
running_batch.reqs,
)
if fwd_timeline is not None:
fwd_timeline.detail("prefill_adder_init", _pf_t1, time.monotonic_ns())
mamba_allocator = getattr(self.req_to_token_pool, "mamba_allocator", None)
if mamba_allocator is not None:
mamba_allocator.alloc_group_begin(len(self.waiting_queue))
_pf_t2 = time.monotonic_ns()
buffer_pipeline = self.tree_cache.buffer_pipeline
# Get requests from the waiting queue to a new prefill batch
for req in self.waiting_queue:
@@ -4168,6 +4215,7 @@ class Scheduler(
# candidate: the resumed chunk or mixed decode rows can spend it.
if adder.budget_state() == AddReqResult.OTHER:
break
batch_telemetry.mark_candidate()
if self.enable_lora and not self.can_schedule_lora_req(req, running_loras):
continue
@@ -4206,6 +4254,7 @@ class Scheduler(
req.time_stats.observe_storage_prefetch_ready(prefetch_done)
if not prefetch_done:
# skip staging requests that are ongoing prefetch
batch_telemetry.mark_skipped_prefetch()
continue
# Pop the L3-loaded span. Unified cache exposes its absolute
# start so cache-mode L2/L3 attribution survives L3-tail eviction.
@@ -4219,7 +4268,11 @@ class Scheduler(
# marks the staged span below once it is surfaced.
req.host_hit_is_storage = False
if batch_telemetry.stage_timing_enabled():
_bt_t = time.monotonic()
req.init_next_round_input(self.tree_cache)
if batch_telemetry.stage_timing_enabled():
batch_telemetry.add_match(time.monotonic() - _bt_t)
if self.enable_hicache_storage and (
self._prefetch_after_device_hit_loss(req)
):
@@ -4229,11 +4282,15 @@ class Scheduler(
req.time_stats.observe_storage_staging_ready(staging_ready)
if not staging_ready:
continue
if batch_telemetry.stage_timing_enabled():
_bt_t = time.monotonic()
res = adder.add_one_req(
req,
has_chunked_req=(self.chunked_req is not None),
truncation_align_size=self.truncation_align_size,
)
if batch_telemetry.stage_timing_enabled():
batch_telemetry.add_adder(time.monotonic() - _bt_t)
if self.enable_lora:
running_loras.add(req.lora_id)
@@ -4263,6 +4320,15 @@ class Scheduler(
if mamba_allocator is not None:
mamba_allocator.alloc_group_end()
if fwd_timeline is not None:
fwd_timeline.detail("prefill_scan", _pf_t2, time.monotonic_ns())
_pf_t3 = time.monotonic_ns()
_pf_cpu_t3 = (
batch_telemetry.thread_cpu_ns()
if batch_telemetry.loop_detail_enabled()
else None
)
_had_chunked = self.chunked_req is not None
# Update waiting queue
can_run_list: List[Req] = adder.can_run_list
@@ -4271,6 +4337,14 @@ class Scheduler(
# though the request they made room for was not admitted.
for req in adder.preempt_list:
self._add_request_to_queue(req)
batch_telemetry.set_admitted(0)
if fwd_timeline is not None:
fwd_timeline.detail(
"prefill_batch_init",
_pf_t3,
time.monotonic_ns(),
thread_cpu_ns=self._detail_cpu_delta(_pf_cpu_t3),
)
return None, running_batch
can_run_set = set(can_run_list)
@@ -4294,16 +4368,17 @@ class Scheduler(
set_time_batch(can_run_list, "set_forward_entry_time")
# Create a new batch
new_batch = ScheduleBatch.init_new(
can_run_list,
self.req_to_token_pool,
self.token_to_kv_pool_allocator,
self.tree_cache,
self.model_config,
self.enable_overlap,
self.spec_algorithm,
chunked_req=self.chunked_req,
)
with batch_telemetry.detail_phase("schedule_batch_init"):
new_batch = ScheduleBatch.init_new(
can_run_list,
self.req_to_token_pool,
self.token_to_kv_pool_allocator,
self.tree_cache,
self.model_config,
self.enable_overlap,
self.spec_algorithm,
chunked_req=self.chunked_req,
)
new_batch.contains_last_prefill_chunk = (
self.chunked_req is None or len(can_run_list) != 1
@@ -4312,29 +4387,35 @@ class Scheduler(
if self.enable_hierarchical_cache or self.enable_unified_cache_external_linker:
# todo (zhiqiang): disable cuda graph execution if hicache loading triggered
new_batch.hicache_consumer_index = (
self.tree_cache.ready_to_load_host_cache()
)
with batch_telemetry.detail_phase("ready_to_load_host_cache"):
new_batch.hicache_consumer_index = (
self.tree_cache.ready_to_load_host_cache()
)
new_batch.prepare_for_extend()
with batch_telemetry.detail_phase("prepare_for_extend"):
new_batch.prepare_for_extend()
# Snapshot the prefill-only shape before mixed-chunk folding:
# mix_with_running() appends decode rows to extend_num_tokens/prefix_lens,
# so reading them after the branch would mix two different scopes.
_prefill_n_new = int(getattr(new_batch, "extend_num_tokens", 0) or 0)
_prefill_n_cached = int(sum(getattr(new_batch, "prefix_lens", ()) or ()))
if self.tp_worker.model_runner.prefill_aware_swa:
for req in can_run_list:
req.kv.swa_evict_floor = req.extend_end
# Record prefill stats for logging after forward.
new_batch.prefill_stats = PrefillStats.from_adder(
adder,
running_batch.reqs,
self.enable_priority_scheduling,
num_pending_tokens=self.load_inquirer._get_num_pending_tokens(
chunk_deduct=(
self.chunked_req.extend_len if self.chunked_req is not None else 0
),
),
)
with batch_telemetry.detail_phase("prefill_stats"):
new_batch.prefill_stats = PrefillStats.from_adder(
adder,
running_batch.reqs,
self.enable_priority_scheduling,
num_pending_tokens=self._prefill_pending_tokens(),
)
# Mixed-style chunked prefill
_mixed_t0 = None
_mixed_cpu0 = None
if (
self.is_mixed_chunk
and not running_batch.is_empty()
@@ -4344,6 +4425,9 @@ class Scheduler(
# Beam member rows are not supported inside a mixed extend batch.
and all(r.beam_group is None for r in running_batch.reqs)
):
if batch_telemetry.loop_detail_enabled():
_mixed_t0 = time.monotonic_ns()
_mixed_cpu0 = batch_telemetry.thread_cpu_ns()
# TODO (lianmin): support return_logprob + mixed chunked prefill
finishing_reqs = self._reqs_finishing_inflight(running_batch)
running_batch.filter_batch(reqs_to_exclude=finishing_reqs)
@@ -4373,8 +4457,71 @@ class Scheduler(
else:
new_batch.decoding_reqs = None
batch_telemetry.set_admitted(len(can_run_list))
batch_telemetry.note_new_chunk(adder.new_chunked_req is not None)
if _mixed_t0 is not None and fwd_timeline is not None:
fwd_timeline.detail(
"mixed_chunk_tail",
_mixed_t0,
time.monotonic_ns(),
thread_cpu_ns=self._detail_cpu_delta(_mixed_cpu0),
)
if batch_telemetry.loop_detail_enabled():
# Prefill-only scope for the three base counters, even when the
# batch was folded with running decode rows afterwards.
batch_telemetry.detail_attr("prefill_batch_n_reqs", len(can_run_list))
batch_telemetry.detail_attr("prefill_batch_n_new", _prefill_n_new)
batch_telemetry.detail_attr("prefill_batch_n_cached", _prefill_n_cached)
batch_telemetry.detail_attr(
"prefill_batch_chunked", bool(self.chunked_req is not None)
)
batch_telemetry.detail_attr("prefill_batch_resumed_chunked", bool(_had_chunked))
batch_telemetry.detail_attr(
"prefill_batch_new_chunked", bool(adder.new_chunked_req is not None)
)
batch_telemetry.detail_attr(
"prefill_batch_mixed_chunk", bool(_mixed_t0 is not None)
)
if _mixed_t0 is not None:
# Mixed batches carry a second scope; keep it explicit instead
# of folding decode rows into the prefill counters above.
_decode_rows = len(getattr(new_batch, "decoding_reqs", ()) or ())
batch_telemetry.detail_attr(
"prefill_batch_decode_reqs", _decode_rows
)
batch_telemetry.detail_attr(
"prefill_batch_merged_n_reqs", len(can_run_list) + _decode_rows
)
batch_telemetry.detail_attr(
"prefill_batch_merged_n_new",
int(getattr(new_batch, "extend_num_tokens", 0) or 0),
)
if fwd_timeline is not None:
fwd_timeline.detail(
"prefill_batch_init",
_pf_t3,
time.monotonic_ns(),
thread_cpu_ns=self._detail_cpu_delta(_pf_cpu_t3),
)
return new_batch, running_batch
@staticmethod
def _detail_cpu_delta(cpu_start_ns):
"""Thread CPU delta for detail spans; None when the clock is missing."""
if cpu_start_ns is None:
return None
now = batch_telemetry.thread_cpu_ns()
return None if now is None else now - cpu_start_ns
def _prefill_pending_tokens(self) -> int:
"""Pending-token evaluation for PrefillStats, timed as its own phase."""
with batch_telemetry.detail_phase("pending_tokens"):
return self.load_inquirer._get_num_pending_tokens(
chunk_deduct=(
self.chunked_req.extend_len if self.chunked_req is not None else 0
),
)
def can_schedule_lora_req(
self, req: Req, running_loras: set[Optional[str]]
) -> bool:
@@ -4609,8 +4756,9 @@ class Scheduler(
# PD prefill: early-send cached prefix KV, overlapping the suffix forward.
if self.disaggregation_mode == DisaggregationMode.PREFILL:
for req in batch.reqs:
self.maybe_send_cached_prefix_chunk(req)
with batch_telemetry.detail_phase("early_send_host"):
for req in batch.reqs:
self.maybe_send_cached_prefix_chunk(req)
# Run forward
if self.is_generation:
@@ -4637,7 +4785,8 @@ class Scheduler(
# mix_running_indices). Run OUTSIDE isolation so the
# snapshot captures the post-consume state — restoring
# post-forward must not un-consume staging.
resolve_forward_inputs(batch, self.future_map)
with batch_telemetry.detail_phase("resolve_inputs"):
resolve_forward_inputs(batch, self.future_map)
with self._forward_isolation(batch, overlap=True):
future_indices = batch.req_pool_indices
@@ -4660,9 +4809,10 @@ class Scheduler(
)
# FIXME: pp is not compatible with overlap
batch_result = self.model_worker.forward_batch_generation(
batch, **fwd_kwargs
)
with batch_telemetry.detail_phase("model_forward_host"):
batch_result = self.model_worker.forward_batch_generation(
batch, **fwd_kwargs
)
if batch.spec_algorithm.is_none():
self.future_map.publish(future_indices, batch.seq_lens + 1)
# Park any refs the worker wants kept alive 2 iters
@@ -4700,7 +4850,10 @@ class Scheduler(
return_hidden_states=batch.return_hidden_states,
)
else:
self._copy_result_to_cpu(batch_result, batch)
with batch_telemetry.detail_phase(
"copy_result_submit"
):
self._copy_result_to_cpu(batch_result, batch)
else:
batch_result.future_indices = future_indices
@@ -5037,15 +5190,19 @@ class Scheduler(
):
# Flush async trace ops here: in overlap mode this CPU work runs while
# the next batch's GPU forward is in flight, giving free overlap.
flush_trace_batch(batch.reqs)
snapshot = self.publish_load_snapshot(force=batch.forward_mode.is_extend())
# Router-facing gauge on the dedicated PUB socket, reusing the
# snapshot above rather than walking the queues again.
self.load_publisher.publish_load_stat(
self.load_inquirer.get_loads,
force=batch.forward_mode.is_extend(),
snapshot=snapshot,
)
with batch_telemetry.detail_phase("flush_trace"):
flush_trace_batch(batch.reqs)
with batch_telemetry.detail_phase("load_snapshot"):
snapshot = self.publish_load_snapshot(
force=batch.forward_mode.is_extend()
)
# Router-facing gauge on the dedicated PUB socket, reusing the
# snapshot above rather than walking the queues again.
self.load_publisher.publish_load_stat(
self.load_inquirer.get_loads,
force=batch.forward_mode.is_extend(),
snapshot=snapshot,
)
if batch.forward_mode.is_decode():
self.batch_result_processor.process_batch_result_decode(batch, result)
@@ -5062,19 +5219,28 @@ class Scheduler(
self.batch_result_processor.process_batch_result_idle(batch, result)
# Submit this batch's queued host backups before the next scheduler step.
self.tree_cache.flush_pending_backups()
# Tag the origin so the per-flush micro timer can separate these from
# the ones issued by check_hicache_events / write().
_flush_origin = batch_telemetry.set_flush_origin("process_result")
try:
with batch_telemetry.detail_phase("flush_backups"):
self.tree_cache.flush_pending_backups()
finally:
batch_telemetry.set_flush_origin(_flush_origin)
self._record_step_counters(batch, result)
with batch_telemetry.detail_phase("step_counters"):
self._record_step_counters(batch, result)
self.metrics_reporter.log_batch_result_stats(batch, result)
with batch_telemetry.detail_phase("metrics_report"):
self.metrics_reporter.log_batch_result_stats(batch, result)
# Emit forward pass metrics (every iteration when enabled)
if self.enable_fpm:
self.metrics_reporter._emit_forward_pass_metrics(batch, result)
# Emit forward pass metrics (every iteration when enabled)
if self.enable_fpm:
self.metrics_reporter._emit_forward_pass_metrics(batch, result)
self._maybe_clear_mm_inputs(batch)
self.maybe_send_health_check_signal()
self.metrics_reporter.update_device_timer()
self._maybe_clear_mm_inputs(batch)
self.maybe_send_health_check_signal()
self.metrics_reporter.update_device_timer()
def _record_step_counters(
self, batch: ScheduleBatch, result: GenerationBatchResult
@@ -29,6 +29,7 @@ from sglang.srt.managers.schedule_batch import (
ScheduleBatch,
mamba_lazy_spec_in_window,
)
from sglang.srt.observability import batch_telemetry
from sglang.srt.mem_cache.common import (
checkpoint_kv_cache,
release_kv_cache,
@@ -995,6 +996,10 @@ class SchedulerBatchResultProcessor:
if batch.spec_algorithm.is_none() and logits_output is not None:
newly_finished_beam_groups = self.beam_coordinator.commit_decode(batch)
# Keep the hot loop free when telemetry is off: read the gate once and
# skip the per-request finished() check entirely.
track_completions = batch_telemetry.enabled()
finished_reqs = 0
for i, req in enumerate(batch.reqs):
req: Req
@@ -1009,6 +1014,8 @@ class SchedulerBatchResultProcessor:
self._handle_finish_state_updated_req(
req, batch, result, i, logits_output
)
if track_completions and req.finished():
finished_reqs += 1
continue
if (self.enable_overlap or self.enable_overlap_mlx) and (
@@ -1041,10 +1048,16 @@ class SchedulerBatchResultProcessor:
req.update_finish_state(new_accept_len)
if sampling_mask_finish_reason is not None:
# This path finishes the request too (to_finish ->
# finished_reason); count it before the continue.
self._handle_sampling_mask_abort(req)
if track_completions and req.finished():
finished_reqs += 1
continue
self._handle_finish_state_updated_req(req, batch, result, i, logits_output)
if track_completions and req.finished():
finished_reqs += 1
if req.return_logprob:
self._apply_decode_logprobs(
@@ -1081,6 +1094,9 @@ class SchedulerBatchResultProcessor:
self._accept_grammar_tokens(req, next_token_id)
req.grammar.finished = req.finished()
if finished_reqs:
batch_telemetry.note_request_completed("decode", finished_reqs)
if auxiliary_output is not None:
self.consume_auxiliary_output(
batch,
@@ -21,6 +21,7 @@ from sglang.srt.managers.scheduler_components.metrics_reporter import (
from sglang.srt.managers.scheduler_components.recv_skipper import (
SchedulerRecvSkipper,
)
from sglang.srt.observability import batch_telemetry
from sglang.srt.managers.utils import is_health_check_generate_req
from sglang.srt.mem_cache.allocator import BaseTokenToKVPoolAllocator
from sglang.srt.mem_cache.base_prefix_cache import BasePrefixCache
@@ -295,6 +296,14 @@ class MLPSyncBatchInfo:
self.any_health_check = bool(tp0_info_cpu[:, 9].max())
self.global_attention_pairs = tp0_info_cpu[:, 10].tolist()
self.sync_wait_seconds = time.perf_counter() - sync_start
if batch_telemetry.enabled():
batch_telemetry.observe_dp_step(
self.global_num_tokens,
get_parallel().attn_dp_rank,
modes=tp0_info_cpu[:, 5].tolist(),
health=tp0_info_cpu[:, 9].tolist(),
phase="extend" if self.is_extend_in_batch else "decode",
)
if _ENABLE_METRICS_DP_ATTENTION:
self.dp_cooperation_info = DPCooperationInfo.create(
tp0_info_cpu[:, 5].tolist()
@@ -666,6 +675,15 @@ def prepare_mlp_sync_batch_raw(
else:
need_idle_batch = max(mlp_sync_info.global_num_tokens) > 0
# Capture the pre-sync local kind before the idle replacement below: the
# loop span's local_batch_kind describes what this rank brought in, not the
# idle batch it may be asked to run for DP lockstep.
_loop_local_kind = (
"none"
if local_batch is None
else ("idle" if local_batch.forward_mode.is_idle() else "extend")
)
batch_to_gather = local_batch
if need_idle_batch:
if local_batch is None:
@@ -674,6 +692,13 @@ def prepare_mlp_sync_batch_raw(
# NOTE: for prebuilt batch, we add an inner idle batch to run MLP sync
batch_to_gather = local_batch.inner_idle_batch = get_idle_batch()
_loop_global_outcome = (
"idle"
if (_loop_local_kind == "none" and need_idle_batch)
else ("none" if _loop_local_kind == "none" else _loop_local_kind)
)
batch_telemetry.note_mlp_sync_outcome(_loop_local_kind, _loop_global_outcome)
if batch_to_gather is not None:
_update_gather_batch(
batch_to_gather,
@@ -714,6 +739,12 @@ def prepare_mlp_sync_batch_raw(
# No batch yet: this iteration may gather again, or end in on_idle.
sync_wait_carry.seconds = sync_wait_seconds
# Stage timing: publish this pass's DP all-gather wait (None = no gather ran).
if metadata_ready and not skip_all_gather and num_dp_ranks > 1:
batch_telemetry.note_dp_gather(mlp_sync_info.sync_wait_seconds)
else:
batch_telemetry.note_dp_gather(None)
return local_batch
@@ -46,6 +46,7 @@ class SchedulerLoadInquirer:
get_recent_cache_hit_rate: Callable
get_stats: Callable
get_chunked_req: Callable
get_grammar_queue: Callable
get_disagg_prefill_bootstrap_queue: Callable
get_disagg_prefill_inflight_queue: Callable
get_disagg_decode_prealloc_queue: Callable
@@ -138,6 +139,12 @@ class SchedulerLoadInquirer:
for req in queue
)
# Requests waiting for their grammar to compile reach the queues above
# only once it is ready, but they are already this rank's work.
grammar_queue = self.get_grammar_queue()
waiting_queues.append(grammar_queue)
pending_token_queues.append(grammar_queue)
num_waiting_reqs = sum(len(queue) for queue in waiting_queues)
num_used_tokens, kv_token_usage = (
self.pool_stats_observer.get_pool_stats().get_kv_token_stats()
@@ -145,6 +152,10 @@ class SchedulerLoadInquirer:
num_total_tokens = num_used_tokens + sum(
req.seqlen for queue in pending_token_queues for req in queue
)
chunked_req = self.get_chunked_req()
if chunked_req is not None and chunked_req.extend_end is not None:
# KV covers only the chunks run so far; the rest is still pending.
num_total_tokens += max(0, chunked_req.seqlen - chunked_req.extend_end)
num_active_tokens = max(0, num_total_tokens - awaiting_kv_tokens)
memory = None
@@ -226,6 +237,13 @@ class SchedulerLoadInquirer:
totals = self.get_decode_moment_totals()
decode_moments = list(totals) if totals[0] > 0 else None
_waiting_uncached = self.get_num_waiting_uncached_tokens()
_busy_us = self.get_total_prefill_busy_us()
_drain_rate = (
round(self.get_total_prefill_uncached_tokens() * 1e6 / _busy_us, 2)
if _busy_us > 0
else 0.0
)
return LoadSnapshot(
dp_rank=int(get_parallel().dp_rank)
if get_parallel().dp_rank is not None
@@ -233,7 +251,9 @@ class SchedulerLoadInquirer:
timestamp=time.time(),
num_running_reqs=num_running_reqs,
num_waiting_reqs=num_waiting_reqs,
num_waiting_uncached_tokens=self.get_num_waiting_uncached_tokens(),
num_waiting_uncached_tokens=_waiting_uncached,
queued_total_input_tokens=_waiting_uncached,
drain_rate_tokens_per_s=_drain_rate,
num_used_tokens=num_used_tokens,
num_total_tokens=num_total_tokens,
num_active_tokens=num_active_tokens,
@@ -21,6 +21,7 @@ from sglang.srt.observability.forward_pass_metrics import (
WelfordAccumulator,
wrap_forward_with_fpm,
)
from sglang.srt.observability import batch_telemetry
from sglang.srt.observability.metrics_collector import (
DPCooperationInfo,
QueueCount,
@@ -798,6 +799,21 @@ class SchedulerMetricsReporter:
self.forward_pass_device_timer._report()
msg += ", " + self.forward_occupancy_log_string()
if batch_telemetry.enabled():
_batch_trace_id = batch_telemetry.observe_prefill(
dp_rank=get_parallel().attn_dp_rank,
iter_no=batch_iter,
n_reqs=prefill_stats.num_new_seqs,
n_new=prefill_stats.log_input_tokens,
n_cached=prefill_stats.log_hit_tokens,
queue_depth=len(self.scheduler.waiting_queue),
pending_tokens=prefill_stats.num_pending_tokens,
duration_s=gap_latency,
reqs=batch.reqs if batch is not None else None,
)
if _batch_trace_id:
msg += f", batch-trace: {_batch_trace_id}"
if self.is_stats_logging_rank:
logger.info(msg)
if self.current_scheduler_metrics_enabled:
@@ -940,6 +956,20 @@ class SchedulerMetricsReporter:
self.last_decode_stats_tic = time.perf_counter()
self.last_gen_throughput = self.num_generated_tokens / gap_latency
_decode_batch_trace = None
if batch_telemetry.enabled():
_decode_batch_trace = batch_telemetry.observe_decode(
dp_rank=get_parallel().attn_dp_rank,
iter_no=self.forward_ct_decode,
n_reqs=len(batch.reqs) if batch is not None else 0,
n_new=num_generated_tokens,
n_cached=0,
queue_depth=len(self.scheduler.waiting_queue),
pending_tokens=0,
duration_s=gap_latency,
reqs=batch.reqs if batch is not None else None,
)
self.num_generated_tokens = 0
num_running_reqs = len(batch.reqs)
@@ -1064,6 +1094,9 @@ class SchedulerMetricsReporter:
self.forward_pass_device_timer._report()
msg += ", " + self.forward_occupancy_log_string()
if _decode_batch_trace:
msg += f", batch-trace: {_decode_batch_trace}"
if self.is_stats_logging_rank:
logger.info(msg)
if self.current_scheduler_metrics_enabled:
@@ -1,6 +1,7 @@
from __future__ import annotations
import logging
import time
from dataclasses import dataclass
from http import HTTPStatus
from typing import (
@@ -33,6 +34,7 @@ from sglang.srt.managers.mm_utils import (
has_shm_features,
unwrap_shm_features,
)
from sglang.srt.observability import batch_telemetry
from sglang.srt.observability.scheduler_stage_metrics import (
SCHEDULER_STAGE_RECV_REQUESTS,
SchedulerStageMetricsRecorder,
@@ -93,20 +95,54 @@ class SchedulerRequestReceiver:
if not self.recv_skipper.handle(self.get_last_batch()):
return []
_bt = batch_telemetry.recv_timing_enabled()
_t0 = time.monotonic() if _bt else 0.0
recv_reqs = self._pull_raw_reqs()
_t_pull = time.monotonic() if _bt else 0.0
if self.input_blocker is not None:
recv_reqs = self.input_blocker.handle(recv_reqs)
recv_reqs = self._broadcast_reqs_across_ranks(recv_reqs, local_reqs)
_t_bcast = time.monotonic() if _bt else 0.0
if get_parallel().pp_rank == 0:
self.unwrap_pickle_wrapper(recv_reqs)
_t_unwrap = time.monotonic() if _bt else 0.0
recv_reqs = self._apply_mm_receiver(recv_reqs)
_t_mm = time.monotonic() if _bt else 0.0
self._finalize_shm_features(recv_reqs)
if _bt:
n_msgs = 0
n_tokens = 0
for req in recv_reqs or ():
n_msgs += 1
ids = getattr(req, "input_ids", None)
if ids is None:
try:
for sub in req:
n_tokens += len(getattr(sub, "input_ids", ()) or ())
except TypeError:
pass
else:
try:
n_tokens += len(ids)
except TypeError:
pass
batch_telemetry.observe_recv(
dp_rank=get_parallel().attn_dp_rank,
total=time.monotonic() - _t0,
pull=_t_pull - _t0,
broadcast=_t_bcast - _t_pull,
unwrap=_t_unwrap - _t_bcast,
mm=_t_mm - _t_unwrap,
n_msgs=n_msgs,
n_tokens=n_tokens,
)
return recv_reqs
def _pull_raw_reqs(self) -> Optional[List]:
+10
View File
@@ -16,6 +16,7 @@
from __future__ import annotations
import logging
import time
from abc import ABC, abstractmethod
from typing import TYPE_CHECKING, List, Optional, Tuple
@@ -643,12 +644,21 @@ class TpModelWorker(BaseTpWorker):
# Replay reads restored main/indexer KV before the normal extend.
run_encoder_swa_replay(self, batch)
try:
from sglang.srt.observability import fwd_timeline
_ft_t = time.monotonic_ns()
except Exception: # pragma: no cover
fwd_timeline = None
_ft_t = 0
forward_batch = ForwardBatch.init_new(
batch if folded is None else folded.batch,
self.model_runner,
capture_hidden_mode=capture_hidden_mode,
return_hidden_states_before_norm=False,
)
if fwd_timeline is not None:
fwd_timeline.detail("fwd_init_new", _ft_t, time.monotonic_ns())
if folded is not None:
apply_folded_extend(folded, forward_batch)
else:
+7 -3
View File
@@ -24,6 +24,7 @@ from sglang.srt.mem_cache.common import (
evict_from_tree_cache,
)
from sglang.srt.mem_cache.memory_pool import HybridReqToTokenPool, ReqToTokenPool
from sglang.srt.observability import batch_telemetry
from sglang.srt.runtime_context import attention_backends, get_parallel
from sglang.srt.utils import (
is_cpu,
@@ -318,8 +319,10 @@ def ensure_mamba_capacity(
supports_mamba = tree_cache is not None and tree_cache.supports_mamba()
available = allocator.schedulable_available_size()
if available < needed and supports_mamba:
tree_cache.evict_for_alloc(
EvictParams(num_tokens=0, mamba_num=needed - available)
from sglang.srt.mem_cache.common import evict_for_alloc_observed
evict_for_alloc_observed(
tree_cache, EvictParams(num_tokens=0, mamba_num=needed - available)
)
return allocator.schedulable_available_size() >= needed
@@ -384,7 +387,8 @@ def alloc_for_extend(
# free out-of-window swa tokens
batch.maybe_evict_swa()
prefix_tensors = [_prefix_kv_indices(batch, r) for r in batch.reqs]
with batch_telemetry.detail_phase("prefix_device_indices"):
prefix_tensors = [_prefix_kv_indices(batch, r) for r in batch.reqs]
reuse_kv = None
if batch.is_dllm():
@@ -165,7 +165,11 @@ class BaseTokenToKVPoolAllocator(abc.ABC):
return
shortfall = num_tokens - self.available_size()
if shortfall > 0:
tree_cache.evict_for_alloc(EvictParams(num_tokens=shortfall))
from sglang.srt.mem_cache.common import evict_for_alloc_observed
evict_for_alloc_observed(
tree_cache, EvictParams(num_tokens=shortfall)
)
_evict_until_allocatable(tree_cache, self, num_tokens)
def check_decode_capacity(
+9 -4
View File
@@ -189,10 +189,12 @@ class SWATokenToKVPoolAllocator(BaseTokenToKVPoolAllocator):
"""
from sglang.srt.mem_cache.base_prefix_cache import EvictParams
from sglang.srt.mem_cache.common import evict_for_alloc_observed
available = self.swa_available_size()
if available < swa_tokens:
tree_cache.evict_for_alloc(
EvictParams(swa_num_tokens=swa_tokens - available)
evict_for_alloc_observed(
tree_cache, EvictParams(swa_num_tokens=swa_tokens - available)
)
available = self.swa_available_size()
if available < swa_tokens:
@@ -215,8 +217,11 @@ class SWATokenToKVPoolAllocator(BaseTokenToKVPoolAllocator):
full_shortfall = max(0, num_tokens - self.full_available_size())
swa_shortfall = max(0, num_tokens - self.swa_available_size())
if full_shortfall or swa_shortfall:
tree_cache.evict_for_alloc(
EvictParams(num_tokens=full_shortfall, swa_num_tokens=swa_shortfall)
from sglang.srt.mem_cache.common import evict_for_alloc_observed
evict_for_alloc_observed(
tree_cache,
EvictParams(num_tokens=full_shortfall, swa_num_tokens=swa_shortfall),
)
def full_available_size(self):
@@ -1254,7 +1254,10 @@ class UnifiedSWATokenToKVPoolAllocator(UnifiedSWAAllocatorBase):
return
full_reclaim, swa_reclaim = reclaim_plan
if full_reclaim or swa_reclaim:
tree_cache.evict_for_alloc(
from sglang.srt.mem_cache.common import evict_for_alloc_observed
evict_for_alloc_observed(
tree_cache,
EvictParams(num_tokens=full_reclaim, swa_num_tokens=swa_reclaim),
allocation_reclaim_satisfied=lambda: (
self.reclaim_plan(num_tokens, required_swa) == (0, 0)
@@ -1909,7 +1912,10 @@ class UnifiedMambaSWATokenToKVPoolAllocator(UnifiedSWAAllocatorBase):
# precedes state reclaim so Mamba-first tree ordering cannot skip it.
state_freed = 0
if full_reclaim or swa_reclaim:
result = tree_cache.evict_for_alloc(
from sglang.srt.mem_cache.common import evict_for_alloc_observed
result = evict_for_alloc_observed(
tree_cache,
EvictParams(num_tokens=num_tokens, swa_num_tokens=num_tokens),
allocation_reclaim_satisfied=lambda: self._token_reclaim_satisfied(
num_tokens
@@ -1930,7 +1936,10 @@ class UnifiedMambaSWATokenToKVPoolAllocator(UnifiedSWAAllocatorBase):
)
state_quota = max(0, -(-pair_bytes // ma.entry_bytes) - state_freed)
if state_quota and tree_cache.mamba_evictable_size():
result = tree_cache.evict_for_alloc(
from sglang.srt.mem_cache.common import evict_for_alloc_observed
result = evict_for_alloc_observed(
tree_cache,
EvictParams(mamba_num=state_quota),
allocation_reclaim_satisfied=lambda: self._token_reclaim_satisfied(
num_tokens
@@ -25,6 +25,7 @@ from __future__ import annotations
import inspect
import logging
import os
import threading
from typing import (
Callable,
Dict,
@@ -440,6 +441,9 @@ class MultiEndedAllocator(BaseTokenToKVPoolAllocator):
# site; `_flush` materializes the write-set lazily, avoiding a sync here.
self._inflight_forward: Optional[Tuple[torch.cuda.Event, torch.Tensor]] = None
self._hicache_transfer_done_events: Dict[Hashable, torch.cuda.Event] = {}
# The async backup submit worker records transfer events while the
# scheduler may concurrently iterate/clear them for compaction gating.
self._hicache_transfer_done_events_lock = threading.Lock()
# HiCache reserves SWA physical pages during load-back, then binds them
# into the tree before the H2D copy is submitted. Compaction must not
# move those pages in that interval because the queued PoolTransfer still
@@ -557,7 +561,8 @@ class MultiEndedAllocator(BaseTokenToKVPoolAllocator):
self.live_page_count = 0
self._inflight_forward = None
self._latest_forward_done_event = None
self._hicache_transfer_done_events.clear()
with self._hicache_transfer_done_events_lock:
self._hicache_transfer_done_events.clear()
self._pending_hicache_load_pages = 0
def clear_inverse_history(self) -> None:
@@ -2083,7 +2088,8 @@ class MultiEndedAllocator(BaseTokenToKVPoolAllocator):
_pending_hicache_load_pages: _CapacityField[int] = _CapacityField()
def set_hicache_transfer_done_event(self, transfer_key: Hashable, event) -> None:
self._hicache_transfer_done_events[transfer_key] = event
with self._hicache_transfer_done_events_lock:
self._hicache_transfer_done_events[transfer_key] = event
if transfer_key == "load" or (
isinstance(transfer_key, tuple) and transfer_key[-1] == "load"
):
@@ -2095,11 +2101,12 @@ class MultiEndedAllocator(BaseTokenToKVPoolAllocator):
"""Order the current stream after outstanding HiCache copies. Only moves
and frees need this: a copy's source and target are not free pages until
its finish event is observed, so allocation never reuses them."""
if not self._hicache_transfer_done_events:
return
with self._hicache_transfer_done_events_lock:
if not self._hicache_transfer_done_events:
return
events = tuple(self._hicache_transfer_done_events.values())
self._hicache_transfer_done_events.clear()
current_stream = torch.cuda.current_stream()
events = tuple(self._hicache_transfer_done_events.values())
self._hicache_transfer_done_events.clear()
for event in events:
current_stream.wait_event(event)
@@ -703,6 +703,7 @@ class BasePrefixCache(ABC, PrefixCacheTrait):
supports_prefix_sharing=self.supports_prefix_sharing(),
retain_floor=self.swa_retain_floor(req),
eviction_interval=eviction_interval,
reprefill_tail_tokens=self.swa_reprefill_tail_tokens(req),
)
def swa_retain_floor(self, req) -> int | None:
+113 -3
View File
@@ -24,6 +24,7 @@ from sglang.srt.mem_cache.hicache_storage import PoolTransfer
from sglang.srt.mem_cache.memory_pool import ReqToTokenPool
from sglang.srt.mem_cache.radix_cache import RadixKey
from sglang.srt.mem_cache.unified_cache.component_type import ComponentType
from sglang.srt.observability import batch_telemetry
from sglang.srt.runtime_context import get_serving, get_spec
from sglang.srt.utils.common import ceil_align
@@ -47,6 +48,22 @@ def kv_to_page_indices(kv_indices: torch.Tensor, page_size: int) -> np.ndarray:
return (kv_indices[::page_size] // page_size).cpu().numpy()
def kv_to_page_indices_pinned(
kv_indices: torch.Tensor, page_size: int
) -> torch.Tensor:
"""Same values/dtype as :func:`kv_to_page_indices`, in a pinned CPU tensor.
The device-to-host copy is non-blocking on the current stream. The caller
must record a CUDA event on that stream after all such copies and hand it
to the transfer worker, which materializes the tensor only after the event
(see ``materialize_state_indices``).
"""
src = kv_indices[::page_size] // page_size
out = torch.empty(src.shape, dtype=src.dtype, device="cpu", pin_memory=True)
out.copy_(src, non_blocking=True)
return out
def suffix_page_indices(
req_to_token: torch.Tensor,
req_pool_idx: int,
@@ -84,6 +101,7 @@ def free_swa_out_of_window_slots(
component_type: ComponentType = ComponentType.SWA,
free_segment: Callable[..., None] | None = None,
eviction_interval: int = 1,
reprefill_tail_tokens: int = 0,
) -> None:
if not req.kv.holds_kv:
return
@@ -115,6 +133,14 @@ def free_swa_out_of_window_slots(
evict_threshold = pre_len - ceil_align(
max(sliding_window_size, page_size), page_size
)
if reprefill_tail_tokens:
# Matching excludes the replay tail before page alignment. Keep the
# SWA window behind that earlier boundary too, or a first replay
# falls back to an older checkpoint despite a Full-KV cache hit.
replay_boundary = (pre_len - reprefill_tail_tokens) // page_size * page_size
evict_threshold = min(
evict_threshold, replay_boundary - sliding_window_size
)
if retain_floor is not None and supports_prefix_sharing:
# The caller owns where the floor is (see BasePrefixCache.swa_retain_floor);
# this only promises not to free past it. Without prefix sharing a retained
@@ -259,6 +285,17 @@ def match_kv_cache(
max_len = req._compute_max_prefix_len(len(token_ids))
req.num_matched_prefix_tokens = min(req.prefix_len + req.host_hit_length, max_len)
req.swa_branching_seqlen = match_result.swa_branching_seqlen
# Observability: record the Full-KV length the components could have used.
# This function also runs on waiting-queue probes, PD decode and HiCache
# rematches, so it only stores the gap; the metric is emitted exactly once
# per prefill admission in PrefillAdder._commit_prefill_admission, where the
# accepted prefix length is final (after any host load-back).
req.full_kv_hit_length = match_result.full_kv_hit_length
req.swa_gate_reject_tokens = max(
0,
match_result.full_kv_hit_length
- (match_result.device_prefix_len + match_result.host_hit_length),
)
# A probe match keeps what it did not report; a new round resets both.
if match_result.mamba_branching_seqlen is not None:
req.mamba_branching_seqlen = match_result.mamba_branching_seqlen
@@ -279,13 +316,86 @@ def checkpoint_kv_cache(req: Req, tree_cache: BasePrefixCache) -> None:
req.prefix_len = req.extend_end
def _rank_consensus_enabled() -> bool:
"""Whether the rank-consensus checker is active (cheap, best effort)."""
try:
from sglang.srt.utils import rank_consensus_checker
return bool(rank_consensus_checker.enabled())
except Exception:
return False
def evict_for_alloc_observed(tree_cache, params, **kwargs):
"""Time ``tree_cache.evict_for_alloc`` and record what it realized.
Kept at the call site (not inside the callee) so decorator/consensus waits
are part of the measurement. ``kwargs`` forwards the unified allocators'
``allocation_reclaim_satisfied`` predicate.
"""
with batch_telemetry.detail_phase("evict_for_alloc"):
result = tree_cache.evict_for_alloc(params, **kwargs)
if batch_telemetry.loop_detail_enabled() and result is not None:
batch_telemetry.detail_count("evict_for_alloc_calls")
batch_telemetry.detail_attr_sum(
"evict_plan_requested_tokens", int(getattr(params, "num_tokens", 0) or 0)
)
batch_telemetry.detail_attr_sum(
"evict_plan_requested_swa_tokens",
int(getattr(params, "swa_num_tokens", 0) or 0),
)
batch_telemetry.detail_attr_sum(
"evict_plan_requested_mamba",
int(getattr(params, "mamba_num", 0) or 0),
)
batch_telemetry.detail_attr_sum(
"evict_evicted_tokens",
int(getattr(result, "num_tokens_evicted", 0) or 0),
)
batch_telemetry.detail_attr_sum(
"evict_evicted_swa_tokens",
int(getattr(result, "swa_num_tokens_evicted", 0) or 0),
)
# Record the realized Mamba count unconditionally (including zero) so
# "no Mamba demand" and "demand but nothing reclaimed" stay distinct.
batch_telemetry.detail_attr_sum(
"evict_evicted_mamba",
int(getattr(result, "mamba_num_evicted", 0) or 0),
)
return result
def evict_from_tree_cache(
tree_cache: BasePrefixCache | None, num_tokens: int
) -> bool | None:
if tree_cache is not None and tree_cache.supports_prefix_sharing():
return tree_cache.token_to_kv_pool_allocator.evict_to_free_tokens(
tree_cache, num_tokens
)
allocator = tree_cache.token_to_kv_pool_allocator
with batch_telemetry.detail_phase("evict_to_free_tokens"):
if batch_telemetry.loop_detail_enabled():
available_before = allocator.available_size()
batch_telemetry.detail_count("evict_calls")
# Demand is what the batch wants allocatable; shortfall is the
# part that eviction must actually recover. Keep them apart:
# summing demand as "requested eviction" overstates eviction.
batch_telemetry.detail_attr_sum(
"evict_demand_tokens", int(num_tokens)
)
batch_telemetry.detail_attr_sum(
"evict_shortfall_tokens",
max(0, int(num_tokens) - int(available_before)),
)
batch_telemetry.detail_attr_setdefault(
"evict_free_tokens_before", int(available_before)
)
batch_telemetry.detail_attr(
"evict_rank_consensus_checker", _rank_consensus_enabled()
)
result = allocator.evict_to_free_tokens(tree_cache, num_tokens)
if batch_telemetry.loop_detail_enabled():
batch_telemetry.detail_attr(
"evict_free_tokens_after", int(allocator.available_size())
)
return result
def _evict_until_allocatable(
@@ -41,6 +41,7 @@ from sglang.srt.mem_cache.pool_host.base import uses_shared_host_layout
from sglang.srt.mem_cache.pool_host.mha import MHATokenToKVPoolHost
from sglang.srt.mem_cache.pool_host.unified import UnifiedPageEnvelopeHostPool
from sglang.srt.mem_cache.radix_cache import RadixKey
from sglang.srt.observability import batch_telemetry
if TYPE_CHECKING:
from sglang.srt.mem_cache.allocator import BaseTokenToKVPoolAllocator
@@ -555,15 +556,17 @@ class HybridCacheController(BaseHiCacheController):
return None
host_indices, pool_transfers = allocation
self.write_queue.append(
CacheOperation(
host_indices,
device_indices,
node_id,
priority,
pool_transfers=pool_transfers or None,
)
op = CacheOperation(
host_indices,
device_indices,
node_id,
priority,
pool_transfers=pool_transfers or None,
)
# Start the pinned index D2H before the (possibly deferred) flush, so
# the copy can run while the scheduler keeps going.
self._start_async_index_copy(op)
self.write_queue.append(op)
if flush:
self.start_writing()
return host_indices
@@ -1509,26 +1512,30 @@ class HybridCacheController(BaseHiCacheController):
self, host_pool, host_indices, device_indices, *, write_back_jit: bool
) -> tuple[torch.Tensor, torch.Tensor]:
if uses_shared_host_layout(host_pool):
batch_telemetry.flush_count("shared_layout_pools")
return host_indices, device_indices
if write_back_jit and host_pool.can_use_write_back_jit:
if host_indices.is_cuda:
host_indices = host_indices.cpu()
with batch_telemetry.backup_phase("host_indices_cpu"):
host_indices = host_indices.cpu()
return host_indices, device_indices
return self.move_indices(host_indices, device_indices)
def move_hybrid_indices(
self, operation: CacheOperation, *, write_back_jit: bool = False
) -> tuple[torch.Tensor, torch.Tensor, Optional[list[PoolTransfer]]]:
host_indices, device_indices = self._move_pool_indices(
self.mem_pool_host.anchor_entry.host_pool,
operation.host_indices,
operation.device_indices,
write_back_jit=write_back_jit,
)
with batch_telemetry.backup_phase("anchor_indices"):
host_indices, device_indices = self._move_pool_indices(
self.mem_pool_host.anchor_entry.host_pool,
operation.host_indices,
operation.device_indices,
write_back_jit=write_back_jit,
)
resolved_pool_transfers = None
if operation.pool_transfers:
resolved_pool_transfers = []
for transfer in operation.pool_transfers:
_pool_t0 = batch_telemetry.flush_mark()
transfer_host_indices, transfer_device_indices = (
self._move_pool_indices(
self.mem_pool_host.entry_map[transfer.name].host_pool,
@@ -1537,6 +1544,9 @@ class HybridCacheController(BaseHiCacheController):
write_back_jit=write_back_jit,
)
)
batch_telemetry.flush_phase_since(
"sidecar_indices." + str(transfer.name.value), _pool_t0
)
# Tree-owned transfers keep their original radix-tree indices.
resolved_pool_transfers.append(
replace(
@@ -1547,19 +1557,31 @@ class HybridCacheController(BaseHiCacheController):
)
return host_indices, device_indices, resolved_pool_transfers
def _page_transfer(self, operation: PrefetchOperation) -> bool:
def _page_transfer(
self,
operation: PrefetchOperation,
ack_sink: Optional[Callable[[PrefetchAck], None]] = None,
) -> int:
with self.mem_pool_host.layout_lease():
return self._page_transfer_with_stable_layout(operation)
return self._page_transfer_with_stable_layout(operation, ack_sink)
def _page_transfer_with_stable_layout(self, operation: PrefetchOperation) -> None:
def _page_transfer_with_stable_layout(
self,
operation: PrefetchOperation,
ack_sink: Optional[Callable[[PrefetchAck], None]] = None,
) -> int:
# KV pools and KV-derived pools first — determines actual completed page count
kv_completed_pages = super()._page_transfer(operation)
kv_completed_pages = super()._page_transfer(operation, ack_sink)
# Read non-KV derived sidecar pool, e.g. SWA, Mamba.
self._page_transfer_sidecar(operation, kv_completed_pages)
self._page_transfer_sidecar(operation, kv_completed_pages, ack_sink)
return kv_completed_pages
def _page_transfer_sidecar(
self, operation: PrefetchOperation, kv_completed_pages: int
self,
operation: PrefetchOperation,
kv_completed_pages: int,
ack_sink: Optional[Callable[[PrefetchAck], None]] = None,
) -> None:
if operation.pool_transfers is None:
return
@@ -1604,17 +1626,19 @@ class HybridCacheController(BaseHiCacheController):
"HiCache sidecar prefetch failed for request %s",
operation.request_id,
)
# Emit PrefetchAck to prefetch_sync_queue, even the operation has been canceled by the
# scheduler thread. The prefetch sync thread expects the same number of PrefetchAck objects
# to perform all_reduce.
self.prefetch_sync_queue.put(
PrefetchAck(
rid=operation.request_id,
operation=operation,
pool_hits=pool_hits,
)
# Emit PrefetchAck even when the operation has been canceled by the
# scheduler thread. The sync thread expects the same number of ACKs on
# every rank. Pipelined workers buffer this ACK per operation instead of
# publishing it directly to the shared queue.
ack = PrefetchAck(
rid=operation.request_id,
operation=operation,
pool_hits=pool_hits,
)
return
if ack_sink is None:
self.prefetch_sync_queue.put(ack)
else:
ack_sink(ack)
def trim_prefetch_full_head(self, operation, trim_tokens: int) -> None:
"""Drop a device-resident FULL head after the availability query;
@@ -1704,22 +1728,15 @@ class HybridCacheController(BaseHiCacheController):
return False
def backup_thread_func(self):
"""Back up rank-sharded sidecars on every TP rank.
def _should_skip_backup(self) -> bool:
# Replicated primary MLA KV may be TP0-only, but rank-sharded sidecars
# still require IO on every rank. _page_backup applies should_backup()
# to each individual transfer, so the whole operation cannot be skipped.
return False
The base implementation skips the entire operation on non-zero MLA TP
ranks. That optimization is valid for replicated MLA KV, but not for
hybrid rank-sharded pools such as Kimi-K3 Mamba state.
"""
while not self.storage_stop_event.is_set():
try:
operation = self.backup_queue.get(block=True, timeout=1)
if operation is None:
continue
self._page_backup_or_log(operation)
self.ack_backup_queue.put(operation)
except Empty:
continue
def backup_thread_func(self):
"""Use the base pipeline while retaining per-pool writer selection."""
super().backup_thread_func()
def _resolve_sidecar_kv_derived_pool_transfers(self, operation):
for transfer in operation.pool_transfers:
+28 -7
View File
@@ -9,6 +9,7 @@ import torch
from sglang.srt.environ import envs
from sglang.srt.mem_cache.pool_host.base import shared_host_layout_domains
from sglang.srt.observability import batch_telemetry
from sglang.srt.utils import get_device_module
logger = logging.getLogger(__name__)
@@ -71,10 +72,13 @@ class L2TransferEngine:
original_device = transfer.device_indices.device
# The direct backend supplies CPU indices even for a CUDA pool.
# Translate alongside the v2p table, then restore the backend's device.
_t0 = batch_telemetry.flush_mark()
indices = transfer.device_indices.to(
getattr(transfer.device_pool, "device", original_device)
).contiguous()
batch_telemetry.flush_phase_since("resolve_indices.to_device", _t0)
dcp_size = getattr(transfer.host_pool, "dcp_size", 1)
_t0 = batch_telemetry.flush_mark()
if dcp_size > 1:
# The MLA host pool selects this rank and collapses logical IDs.
# Translate in local virtual space, then preserve that widened
@@ -82,16 +86,22 @@ class L2TransferEngine:
resolved = translate(indices // dcp_size) * dcp_size + indices % dcp_size
else:
resolved = translate(indices)
return resolved.to(original_device)
batch_telemetry.flush_phase_since("resolve_indices.translate", _t0)
_t0 = batch_telemetry.flush_mark()
result = resolved.to(original_device)
batch_telemetry.flush_phase_since("resolve_indices.back_to_host", _t0)
return result
def _prepare_transfers(self, transfers: list[L2Transfer]) -> list[L2Transfer]:
prepared = []
for transfer in transfers:
_t0 = batch_telemetry.flush_mark()
host_indices, device_indices = transfer.host_pool.prepare_transfer_indices(
transfer.host_indices,
self._resolve_device_indices(transfer),
self.io_backend,
)
batch_telemetry.flush_phase_since("prepare_transfer_indices", _t0)
prepared.append(
transfer._replace(
host_indices=host_indices,
@@ -102,44 +112,55 @@ class L2TransferEngine:
@contextmanager
def _submission(self, transfers, stream, transfer_key, start_event=None):
_t0 = batch_telemetry.flush_mark()
start_event = self._start_event(start_event)
ack_start, ack_finish, timing_enabled = make_timing_event_pair()
completion = TransferCompletion(ack_start, ack_finish, timing_enabled)
domains = shared_host_layout_domains(t.host_pool for t in transfers)
for domain in domains:
domain.acquire_layout()
batch_telemetry.flush_phase_since("submit_setup", _t0)
finish_recorded = False
try:
with device_module.stream(stream):
# Index preparation can read producer-owned device indices.
start_event.wait(stream)
prepared = self._prepare_transfers(transfers)
ack_start.record()
with batch_telemetry.backup_phase("start_event_wait"):
start_event.wait(stream)
with batch_telemetry.backup_phase("prepare_transfers"):
prepared = self._prepare_transfers(transfers)
with batch_telemetry.backup_phase("ack_events"):
ack_start.record()
yield prepared, completion
ack_finish.record()
finish_recorded = True
self._record_stream(transfers + prepared, stream)
with batch_telemetry.backup_phase("ack_events"):
ack_finish.record()
finish_recorded = True
self._record_stream(transfers + prepared, stream)
except Exception:
stream.synchronize()
raise
finally:
_t0 = batch_telemetry.flush_mark()
for domain in reversed(domains):
domain.release_layout(
ack_finish if finish_recorded else None,
(id(self), transfer_key),
)
batch_telemetry.flush_phase_since("layout_release", _t0)
def submit_device_to_host(self, transfers: list[L2Transfer]) -> TransferCompletion:
with self._submission(
transfers, self.device_to_host_stream, "device_to_host"
) as (transfers, completion):
for transfer in transfers:
_t0 = batch_telemetry.flush_mark()
transfer.host_pool.backup_from_device_all_layer_physical(
transfer.device_pool,
transfer.host_indices,
transfer.device_indices,
self.io_backend,
)
batch_telemetry.flush_phase_since("launch_pool", _t0)
batch_telemetry.flush_count("launch_pools")
return completion
def submit_host_to_device(
@@ -526,6 +526,22 @@ class MooncakeStore(HiCacheStorage, MooncakeBaseStore):
transfer_engine = None
setup_kwargs = {}
# Opt-in client metrics HTTP server (Prometheus). Each rank
# embeds its own Mooncake client, so give every rank its own
# port: base + dp_rank. Off unless the env var is set.
_mc_metrics_port_base = os.environ.get(
"SGLANG_MOONCAKE_CLIENT_METRICS_PORT_BASE"
)
if _mc_metrics_port_base:
_mc_dp_rank = (
int(getattr(storage_config, "dp_rank", 0) or 0)
if storage_config is not None
else 0
)
setup_kwargs["enable_client_http_server"] = True
setup_kwargs["client_http_port"] = (
int(_mc_metrics_port_base) + _mc_dp_rank
)
if self.config.enable_ssd_offload:
setup_kwargs["enable_ssd_offload"] = True
if self.config.ssd_offload_path is not None:
@@ -1046,6 +1046,7 @@ class SWAComponent(TreeComponent):
req_to_token_pool=self.cache.req_to_token_pool,
token_to_kv_pool_allocator=self.cache.token_to_kv_pool_allocator,
retain_floor=self.cache.swa_retain_floor(req),
reprefill_tail_tokens=self.cache.swa_reprefill_tail_tokens(req),
)
def free_out_of_window_slots(
@@ -111,11 +111,13 @@ from sglang.srt.observability.metrics_collector import (
StorageMetrics,
StorageMetricsCollector,
)
from sglang.srt.observability import batch_telemetry
from sglang.srt.runtime_context import (
get_disagg,
get_memory,
get_model,
get_observability,
get_parallel,
)
from sglang.srt.session.streaming_session import StreamingSession, is_virtual_node
from sglang.srt.utils.common import ceil_align
@@ -437,6 +439,8 @@ class UnifiedRadixCache(BasePrefixCache):
CacheRequestHandle, int
] = {}
self.storage_prefetch_retries = StoragePrefetchRetries()
# request -> monotonic stamp of the first in-flight check (ready latency)
self._prefetch_issue_ts: dict[CacheRequestHandle, float] = {}
self.ongoing_backup: dict[int, tuple[NodeId, DecLockRefParams]] = {}
if self.buffer_pipeline is not None:
self.buffer_pipeline.reset()
@@ -1690,6 +1694,16 @@ class UnifiedRadixCache(BasePrefixCache):
# backed up. Skip only when no transfer remains.
if device_value.numel() == 0 and not comp_xfers:
continue
# A split fragment keeps the original node's ack, and component
# transfers can cover other nodes. Check the whole publish set,
# not just ack keys, before assigning any node a new ack owner.
publish_set = set(self._backup_publish_node_ids(node_id, comp_xfers))
# Write-back eviction has its own round-wide completion barrier.
if not write_back and any(
not publish_set.isdisjoint(pending.publish_node_ids)
for pending in self.ongoing_write_through.values()
):
self.writing_check(write_back=True)
sidecar_xfers = self._build_backup_sidecar(device_value, comp_xfers)
host_indices = self._execute_kv_backup(
node_id, device_value, comp_xfers, sidecar_xfers
@@ -2228,6 +2242,8 @@ class UnifiedRadixCache(BasePrefixCache):
anchor_lock_params,
comp_xfers,
)
if batch_telemetry.stage_timing_enabled():
self._prefetch_issue_ts.setdefault(request, time.monotonic())
if buffer_mode:
# The query reads no tree state; pinning across it would hold the
# anchor cap for a storage round trip, so the pin waits for IO commit.
@@ -2405,6 +2421,22 @@ class UnifiedRadixCache(BasePrefixCache):
self._handle_prefetch_result(operation)
return True
def _observe_prefetch_ready(
self, request: CacheRequestHandle, hit: bool
) -> None:
started = self._prefetch_issue_ts.pop(request, None)
if started is None:
return
try:
dp_rank = get_parallel().attn_dp_rank
except Exception:
dp_rank = 0
batch_telemetry.observe_prefetch_ready(
dp_rank=dp_rank,
result="hit" if hit else "miss",
seconds=time.monotonic() - started,
)
def _handle_prefetch_result(self, operation: PrefetchOperation) -> None:
# This function **owns**:
# - host_indices[0 : completed_tokens]
@@ -2499,6 +2531,9 @@ class UnifiedRadixCache(BasePrefixCache):
self.dec_host_lock_ref(last_host_node_id, anchor_lock_params)
del self.ongoing_prefetch[request]
# Success path retires here instead of _retire_ongoing_prefetch(); the
# ready-latency metric must cover it or every L3 hit goes unmeasured.
self._observe_prefetch_ready(request, loaded_from_storage > 0)
self.cache_controller.prefetch_tokens_occupied -= len(prefetch_key)
self.prefetch_loaded_tokens_by_reqid[request] = loaded_from_storage
@@ -2753,6 +2788,7 @@ class UnifiedRadixCache(BasePrefixCache):
@rank_consensus(same_params=True)
def release_aborted_request(self, request: CacheRequestHandle) -> None:
self._prefetch_issue_ts.pop(request, None)
rid = request.rid
self._storage_prefetch_wait.pop(request, None)
if self.linker is not None:
@@ -2800,6 +2836,9 @@ class UnifiedRadixCache(BasePrefixCache):
) -> None:
"""Drop an in-flight prefetch, releasing ``host_indices`` and ``extra_pools``."""
info = self.ongoing_prefetch.pop(request)
self._observe_prefetch_ready(
request, bool(info.operation.storage_hit_count)
)
if info.anchor_lock_params is not None:
self.dec_host_lock_ref(info.anchor_node_id, info.anchor_lock_params)
if self.buffer_pipeline is not None:
@@ -3466,6 +3505,13 @@ class UnifiedRadixCache(BasePrefixCache):
if write_back:
# Blocking: submit what is still queued, then wait for every ack.
cc.start_writing()
# Async submissions cover ACKs asynchronously; wait for them before
# asserting that every write-through node has been acknowledged.
if hasattr(cc, "wait_write_submissions_idle"):
cc.wait_write_submissions_idle()
_drain = getattr(cc, "drain_write_submission_results", None)
if _drain is not None:
_drain()
while self.ongoing_write_through:
for ack in cc.ack_write_queue:
ack.finish_event.synchronize()
@@ -3478,6 +3524,9 @@ class UnifiedRadixCache(BasePrefixCache):
return
if finish_count is None:
_drain = getattr(cc, "drain_write_submission_results", None)
if _drain is not None:
_drain()
# Every rank must enter the all_reduce below; ongoing_write_through can
# diverge across ranks (e.g. write_backup returning 0 on a subset).
finish_count = 0
@@ -3655,9 +3704,16 @@ class UnifiedRadixCache(BasePrefixCache):
# Reap the previous round's PP-sync sends before issuing new ones.
self._drain_async_work()
# Collect ACKs finished by the async submit worker before counting them.
# getattr keeps lightweight controller fakes (unit tests) working.
_drain = getattr(self.cache_controller, "drain_write_submission_results", None)
if _drain is not None:
_drain()
# Backups queued outside process_batch_result: the chunked-prefill stash
# in get_next_batch_to_run, abort_request, and the PD prefill release.
self.flush_pending_backups()
if _drain is not None:
_drain()
(
write_finish_count,
@@ -468,6 +468,8 @@ class ModelRunner:
# Read-done mailbox: the scheduler's WAR barrier reads it from the runner
# its worker names, and treats None as the coarse whole-forward fence.
self.shared_read_done_event: Optional[torch.cuda.Event] = None
# Set by the scheduler when every KV offload copy waits for forward_stream.
self.kv_offload_fenced_on_forward_stream = False
# CPU offload
set_offloader(create_offloader())
@@ -1846,6 +1848,7 @@ class ModelRunner:
else contextlib.nullcontext()
)
_fwd_tail_t = time.monotonic_ns()
with (
canary_ctx,
step_span_ctx,
@@ -1907,6 +1910,12 @@ class ModelRunner:
):
self.maybe_join_ep_ranks()
try:
from sglang.srt.observability import fwd_timeline as _ft
_ft.detail("fwd_runner_tail", _fwd_tail_t, time.monotonic_ns())
except Exception:
pass
return output
def _maybe_execute_deferred_mamba_cow_and_clear(
@@ -2007,6 +2016,7 @@ class ModelRunner:
)
return ModelRunnerOutput(logits_output=ret, can_run_graph=can_run_graph)
_fwd_prep_t = time.monotonic_ns()
# DP / MLP-sync padding + attn-tp normalization. Only the decode
# cuda-graph path above pre-pads its static buffers and returns
# early; split prefill, the prefill cuda graph, and the eager
@@ -2023,6 +2033,21 @@ class ModelRunner:
if dwdp_mgr is not None:
dwdp_mgr.prefetch_first_layers()
try:
from sglang.srt.observability import fwd_timeline
except Exception: # pragma: no cover
fwd_timeline = None
if fwd_timeline is not None:
fwd_timeline.detail(
"fwd_prepare", _fwd_prep_t, time.monotonic_ns()
)
_fwd_exec_t = time.monotonic_ns()
_fwd_ticket = (
fwd_timeline.before_execute()
if fwd_timeline is not None
else None
)
if forward_batch.forward_mode.is_split_prefill():
# Layer-split mode; stays on ModelRunner, not the eager runner.
ret = self.forward_split_prefill(
@@ -2061,11 +2086,20 @@ class ModelRunner:
forward_batch, pp_proxy_tensors=pp_proxy_tensors
)
if fwd_timeline is not None:
fwd_timeline.after_execute(_fwd_ticket)
fwd_timeline.detail(
"fwd_execute", _fwd_exec_t, time.monotonic_ns()
)
_fwd_post_t = time.monotonic_ns()
if (
forward_batch.global_num_tokens_cpu is not None
and self.pp_group.is_last_rank
):
forward_batch.post_forward_mlp_sync_batch(ret)
if fwd_timeline is not None:
fwd_timeline.detail("fwd_post", _fwd_post_t, time.monotonic_ns())
return ModelRunnerOutput(logits_output=ret, can_run_graph=can_run_graph)
@@ -38,6 +38,7 @@ Backend selection comes from cuda_graph_config.prefill:
from __future__ import annotations
import copy
import time
import dataclasses
import inspect
import logging
@@ -2338,9 +2339,18 @@ class PrefillCudaGraphRunner(BaseCudaGraphRunner):
) -> Optional[Union[LogitsProcessorOutput, PPProxyTensors, EmbeddingPoolerOutput]]:
self._validate_capture_hidden_mode(forward_batch)
with self.model_context(), self.backend.replay_session():
try:
from sglang.srt.observability import fwd_timeline
_gl_t = time.monotonic_ns()
except Exception: # pragma: no cover
fwd_timeline = None
static_forward_batch, shared_read_ends = self.load_batch(
forward_batch, **kwargs
)
if fwd_timeline is not None:
fwd_timeline.detail("fwd_graph_load", _gl_t, time.monotonic_ns())
_gr_t = time.monotonic_ns()
static_num_tokens = len(static_forward_batch.input_ids)
raw_num_tokens = self.raw_num_tokens
shape_key = self._shape_key(static_num_tokens, forward_batch)
@@ -2382,4 +2392,8 @@ class PrefillCudaGraphRunner(BaseCudaGraphRunner):
raw_num_tokens,
**kwargs,
)
if fwd_timeline is not None:
fwd_timeline.detail(
"fwd_graph_replay", _gr_t, time.monotonic_ns()
)
return self._finalize_execute_output(output)
@@ -31,12 +31,12 @@ def maybe_publish_prefill_shared_read_done(
if not model_runner.spec_algorithm.supports_prefill_shared_read_done():
return
# Host and storage offload copy a stashed chunk's KV ordered only behind the
# schedule stream, so they need the whole-forward barrier.
# schedule stream, so they need the whole-forward barrier unless every such
# copy waits for the forward stream itself.
memory = get_memory()
if (
memory.enable_hierarchical_cache
or memory.hicache_storage_backend is not None
or memory.enable_lmcache
if memory.enable_lmcache or (
(memory.enable_hierarchical_cache or memory.hicache_storage_backend is not None)
and not model_runner.kv_offload_fenced_on_forward_stream
):
return
logger.info_once(
+102 -13
View File
@@ -3,6 +3,7 @@ from __future__ import annotations
import concurrent.futures
import functools
import logging
import os
import time
from array import array
from contextlib import contextmanager, nullcontext
@@ -273,6 +274,17 @@ def get_mhc_ops() -> MhcOps:
logger = logging.getLogger(__name__)
# ---------------------------------------------------------------------------
# Optional vision-tower timing (env-gated; off by default).
# SGLANG_MM_TRACE=1 emit one OTel span + one log line per get_image_feature call
# SGLANG_MM_TRACE_SYNC=1 synchronize CUDA around ViT/aligner so the times are GPU time
# (only for short vision-focused runs; leave 0 under load)
# ---------------------------------------------------------------------------
_MM_TRACE = os.getenv("SGLANG_MM_TRACE", "0").strip().lower() not in ("", "0", "false", "no", "off")
_MM_TRACE_SYNC = os.getenv("SGLANG_MM_TRACE_SYNC", "0").strip().lower() not in ("", "0", "false", "no", "off")
_MM_TRACE_RIDS = None
_MM_TRACE_ACC = None
_FP8_WO_A_GEMM = envs.SGLANG_OPT_FP8_WO_A_GEMM.get()
_FP8_WO_A_UE8M0 = _FP8_WO_A_GEMM and DEEPGEMM_SCALE_UE8M0
@@ -4754,6 +4766,62 @@ class DeepseekV4ForCausalLM(nn.Module):
return pad_mm_tokens(input_ids, mm_inputs)
def get_image_feature(self, items):
if not _MM_TRACE:
return self._get_image_feature_impl(items)
global _MM_TRACE_ACC
from opentelemetry import trace as _otel_trace
parallel = get_parallel()
rids = list(_MM_TRACE_RIDS or [])
acc = {
"vit_s": 0.0,
"aligner_s": 0.0,
"items": 0,
"patches": 0,
"cuda": torch.cuda.is_available(),
}
prev_acc, _MM_TRACE_ACC = _MM_TRACE_ACC, acc
tracer = _otel_trace.get_tracer("sglang.mm")
try:
if _MM_TRACE_SYNC and acc["cuda"]:
torch.cuda.synchronize()
t0 = time.perf_counter()
with tracer.start_as_current_span("mm.vision_tower") as span:
try:
out = self._get_image_feature_impl(items)
finally:
if _MM_TRACE_SYNC and acc["cuda"]:
torch.cuda.synchronize()
total_s = time.perf_counter() - t0
dp_rank = int(getattr(parallel, "attn_dp_rank", -1))
span.set_attribute("sglang.dp_rank", dp_rank)
span.set_attribute("sglang.n_items", int(acc["items"]))
span.set_attribute("sglang.n_patches", int(acc["patches"]))
span.set_attribute("sglang.vit_ms", round(acc["vit_s"] * 1e3, 3))
span.set_attribute("sglang.aligner_ms", round(acc["aligner_s"] * 1e3, 3))
span.set_attribute("sglang.total_ms", round(total_s * 1e3, 3))
span.set_attribute("sglang.timing", "gpu_sync" if _MM_TRACE_SYNC else "cpu_wall_launch")
span.set_attribute("sglang.n_rids", len(rids))
if rids:
span.set_attribute("sglang.rids", ",".join(map(str, rids[:8])))
logger.info(
"[MMTrace] dp=%d items=%d patches=%d vit_ms=%.3f aligner_ms=%.3f "
"total_ms=%.3f timing=%s rids=%s",
dp_rank,
acc["items"],
acc["patches"],
acc["vit_s"] * 1e3,
acc["aligner_s"] * 1e3,
total_s * 1e3,
"gpu_sync" if _MM_TRACE_SYNC else "cpu_wall_launch",
",".join(map(str, rids[:4])),
)
return out
finally:
_MM_TRACE_ACC = prev_acc
def _get_image_feature_impl(self, items):
"""Return complete spans for the shared MM cache and chunk scheduler."""
spans = []
@@ -4768,7 +4836,21 @@ class DeepseekV4ForCausalLM(nn.Module):
if plan is not None
else pixels.to(dtype)
)
features = self.aligner(self.vision(patches, h, w), h, w)
if _MM_TRACE_ACC is not None:
t_vit = time.perf_counter()
vision_features = self.vision(patches, h, w)
if _MM_TRACE_SYNC and _MM_TRACE_ACC["cuda"]:
torch.cuda.synchronize()
_MM_TRACE_ACC["vit_s"] += time.perf_counter() - t_vit
t_aligner = time.perf_counter()
features = self.aligner(vision_features, h, w)
if _MM_TRACE_SYNC and _MM_TRACE_ACC["cuda"]:
torch.cuda.synchronize()
_MM_TRACE_ACC["aligner_s"] += time.perf_counter() - t_aligner
_MM_TRACE_ACC["items"] += 1
_MM_TRACE_ACC["patches"] += int(h) * int(w)
else:
features = self.aligner(self.vision(patches, h, w), h, w)
r = self.config.vision_downsample_ratio
types = image_token_types((h + r - 1) // r, (w + r - 1) // r).to(device)
span = torch.empty(
@@ -4782,18 +4864,25 @@ class DeepseekV4ForCausalLM(nn.Module):
return spans
def _prepare_mm_embeddings(self, input_ids, forward_batch):
# Keep scheduler hash IDs intact: the shared embedder clamps its input in place.
input_embeds, _ = embed_mm_inputs(
mm_inputs_list=[
item if item is not None else MultimodalInputs(mm_items=[])
for item in forward_batch.mm_inputs
],
extend_prefix_lens=forward_batch.extend_prefix_lens_cpu,
extend_seq_lens=forward_batch.extend_seq_lens_cpu,
input_ids=input_ids.clone(),
input_embedding=self.get_input_embeddings(),
multimodal_model=self,
)
global _MM_TRACE_RIDS
prev_rids = _MM_TRACE_RIDS
if _MM_TRACE:
_MM_TRACE_RIDS = list(getattr(forward_batch, "rids", None) or [])
try:
# Keep scheduler hash IDs intact: the shared embedder clamps its input in place.
input_embeds, _ = embed_mm_inputs(
mm_inputs_list=[
item if item is not None else MultimodalInputs(mm_items=[])
for item in forward_batch.mm_inputs
],
extend_prefix_lens=forward_batch.extend_prefix_lens_cpu,
extend_seq_lens=forward_batch.extend_seq_lens_cpu,
input_ids=input_ids.clone(),
input_embedding=self.get_input_embeddings(),
multimodal_model=self,
)
finally:
_MM_TRACE_RIDS = prev_rids
forward_batch.mm_input_embeds = input_embeds
return input_embeds
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,164 @@
"""Opt-in GPU busy-window probe for prefill forwards (SGLANG_FWD_TIMELINE=1).
The scheduler/loop spans tell us *host* time; they cannot tell whether the GPU
was busy while the host was working. This probe records one CUDA event pair
around the model execute call and writes the resolved wall-clock window of the
GPU work as one JSONL line per forward::
{"seq": 123, "rank": 0, "a_ns": ..., "b_ns": ..., "dur_ms": 42.1}
``a_ns``/``b_ns`` are ``time.time_ns()``-based wall clock nanoseconds, anchored
to a reference event recorded once (after a ``cudaSynchronize``) so they can be
joined with the loop/pass spans from OpenSearch. Everything before ``a_ns``
and after ``b_ns`` in the same loop is GPU-idle (i.e. host-serial) time.
Design constraints:
* never raise into the scheduler -- any error disables the probe permanently;
* the resolve step only reads events that are already complete
(``Event.query()``), so it never blocks the scheduler thread;
* file is per DP rank under ``/tmp`` (the container is hostPath-mounted for
other reasons; ``kubectl exec ... cat`` is enough to collect it).
"""
from __future__ import annotations
import json
import os
import time
_ENABLED = os.environ.get("SGLANG_FWD_TIMELINE", "0") == "1"
try: # pragma: no cover - import guard only
import torch
except Exception: # pragma: no cover
torch = None
_REF = None # reference CUDA event
_REF_WALL_NS = 0
_RANK = -1
_PENDING = [] # [(seq, ev_a, ev_b)]
_FILE = None
_FAILED = False
def enabled() -> bool:
return _ENABLED and not _FAILED
def _init() -> bool:
"""Record the reference event; returns False when the probe must stay off."""
global _REF, _REF_WALL_NS, _RANK, _FILE, _FAILED, _ENABLED
if _REF is not None:
return True
if not _ENABLED or torch is None:
return False
try:
from sglang.srt.distributed import get_parallel
_RANK = int(get_parallel().attn_dp_rank or 0)
except Exception:
_RANK = 0
try:
torch.cuda.synchronize()
_REF_WALL_NS = time.time_ns()
_REF = torch.cuda.Event(enable_timing=True)
_REF.record()
path = f"/tmp/fwd_timeline_rank{_RANK}.jsonl"
_FILE = open(path, "a", buffering=1)
_FILE.write(
json.dumps({"header": True, "rank": _RANK, "ref_ns": _REF_WALL_NS}) + "\n"
)
return True
except Exception:
_ENABLED = False
_FAILED = True
return False
def before_execute():
"""Record the GPU-start marker; returns an opaque ticket (or None)."""
if not enabled():
return None
try:
if not _init():
return None
ev = torch.cuda.Event(enable_timing=True)
ev.record()
return ev
except Exception:
_disable()
return None
def after_execute(ticket) -> None:
if ticket is None or not enabled():
return
try:
ev = torch.cuda.Event(enable_timing=True)
ev.record()
seq = 0
try:
from sglang.srt.observability import batch_telemetry
seq = int(batch_telemetry.current_loop_seq() or 0)
except Exception:
pass
_PENDING.append((seq, ticket, ev))
except Exception:
_disable()
def detail(
name: str, start_ns: int, end_ns: int, thread_cpu_ns=None
) -> None:
"""Attach a host phase window to the current loop span (absolute times)."""
try:
from sglang.srt.observability import batch_telemetry
batch_telemetry.detail_add(
name, start_ns, end_ns, thread_cpu_ns=thread_cpu_ns
)
except Exception:
pass
def drain() -> None:
"""Resolve completed windows (never blocks) and append them to the file."""
if not _PENDING or _FILE is None:
return
try:
keep = []
lines = []
for seq, ev_a, ev_b in _PENDING:
try:
ready = ev_b.query()
except Exception:
ready = False
if not ready:
keep.append((seq, ev_a, ev_b))
continue
a_ns = _REF_WALL_NS + int(_REF.elapsed_time(ev_a) * 1e6)
b_ns = _REF_WALL_NS + int(_REF.elapsed_time(ev_b) * 1e6)
lines.append(
json.dumps(
{
"seq": seq,
"rank": _RANK,
"a_ns": a_ns,
"b_ns": b_ns,
"dur_ms": round((b_ns - a_ns) / 1e6, 3),
}
)
)
_PENDING[:] = keep
if lines:
_FILE.write("\n".join(lines) + "\n")
except Exception:
_disable()
def _disable() -> None:
global _ENABLED, _FAILED
_ENABLED = False
_FAILED = True
@@ -562,6 +562,34 @@ class SchedulerMetricsCollector(_StatLoggerDIMixin):
documentation="Total number of retracted output tokens.",
labelnames=labels.keys(),
)
# Prefix-cache component gate (SWA window): Full-KV tokens that a
# component validator refused to match, so they are re-prefilled.
# Counted once per prefill admission (not on waiting-queue probes, PD
# decode matches or HiCache rematches). With the SWA prefix-match gate
# enabled this is the hit length lost to a missing sliding window; with
# the gate disabled it only captures the host-only / load-back-gap
# fallback, so it should be near 0.
self.swa_gate_reject_tokens_total = Counter(
name="sglang:swa_gate_reject_tokens_total",
documentation=(
"Full-KV matched tokens rejected by the SWA prefix-match gate "
"at prefill admission and therefore re-prefilled."
),
labelnames=labels.keys(),
)
self.swa_gate_reject_admissions_total = Counter(
name="sglang:swa_gate_reject_admissions_total",
documentation=(
"Prefill admissions whose Full-KV match was truncated by the "
"SWA prefix-match gate. A retracted and re-admitted request is "
"counted again because it is re-prefilled."
),
labelnames=labels.keys(),
)
# Create the zero-valued children so a gate-off deployment exposes the
# series before the first rejection.
self.swa_gate_reject_tokens_total.labels(**self.labels).inc(0)
self.swa_gate_reject_admissions_total.labels(**self.labels).inc(0)
self.num_paused_reqs = Gauge(
name="sglang:num_paused_reqs",
documentation="The number of paused requests by async weight sync.",
@@ -1422,6 +1450,12 @@ class SchedulerMetricsCollector(_StatLoggerDIMixin):
if count > 0:
self.num_prefill_retries_total.labels(**self.labels).inc(count)
def observe_swa_gate_reject_admission(self, tokens: int) -> None:
"""Record one prefill admission whose match the SWA gate truncated."""
if tokens > 0:
self.swa_gate_reject_tokens_total.labels(**self.labels).inc(tokens)
self.swa_gate_reject_admissions_total.labels(**self.labels).inc(1)
def observe_kv_transfer_metrics(
self,
latency_ms: float,
@@ -5,6 +5,7 @@ parses them, and tool-call arguments arrive as the API streams them."""
from __future__ import annotations
import json
import logging
from collections.abc import Sequence
from typing import Any, Optional
@@ -17,6 +18,11 @@ from sglang.srt.parser.deepseek_recipe_renderer import (
load_deepseek_recipe,
)
logger = logging.getLogger(__name__)
# The most tokens one UTF-8 character can span, a byte each.
_MAX_UTF8_BYTES = 4
class RecipeToolCallDelta(msgspec.Struct, frozen=True):
"""A piece of one call's JSON arguments; the first piece also names the call."""
@@ -74,6 +80,7 @@ class DeepSeekRecipeOutputParser:
separate_reasoning: bool = True,
stream_reasoning: bool = True,
stop: Sequence[str] = (),
callable_names: Optional[frozenset[str]] = None,
):
recipe = load_deepseek_recipe()
options = recipe.ParsingOptions(
@@ -101,6 +108,10 @@ class DeepSeekRecipeOutputParser:
# out, so a matched stop is cut before recipe has emitted any of it.
self._stop_hold_chars = max(map(len, stop), default=1) - 1
self._pending_ids: list[int] = []
self._callable_names = callable_names
# recipe's index of each call so far, mapped to the index it is reported
# at; None for a dropped call.
self._call_indices: dict[int, Optional[int]] = {}
self.num_consumed_tokens = 0
def feed(self, token_ids: Sequence[int]) -> RecipeOutputDelta:
@@ -192,6 +203,27 @@ class DeepSeekRecipeOutputParser:
+ self._hf_tokenizer.encode(text[:cut], add_special_tokens=False).ids
)
def _reported_call(
self, call: RecipeToolCallDelta
) -> Optional[RecipeToolCallDelta]:
"""`call` at the index it is reported at; None when it belongs to a call
outside `callable_names`."""
if self._callable_names is None:
return call
if call.name is not None:
index = None
if call.name in self._callable_names:
index = sum(kept is not None for kept in self._call_indices.values())
else:
logger.warning(
"Model attempted to call undefined function: %s", call.name
)
self._call_indices[call.index] = index
index = self._call_indices[call.index]
if index is None:
return None
return msgspec.structs.replace(call, index=index)
def _deltas(self, chunks: list[Any]) -> RecipeOutputDelta:
delta = RecipeOutputDelta()
for chunk in chunks:
@@ -202,12 +234,19 @@ class DeepSeekRecipeOutputParser:
reasoning = piece.get("reasoning_content") or ""
content = piece.get("content") or ""
tool_calls = tuple(
RecipeToolCallDelta(
index=call["index"],
arguments=call["function"]["arguments"],
name=call["function"].get("name"),
call
for call in map(
self._reported_call,
(
RecipeToolCallDelta(
index=call["index"],
arguments=call["function"]["arguments"],
name=call["function"].get("name"),
)
for call in piece.get("tool_calls") or ()
),
)
for call in piece.get("tool_calls") or ()
if call is not None
)
if self._reasoning_as_content:
content, reasoning = reasoning + content, ""
@@ -6,6 +6,7 @@ from __future__ import annotations
import copy
import functools
import json
import logging
from types import ModuleType
from typing import Any, Optional, Union
@@ -15,6 +16,8 @@ DEEPSEEK_RECIPE_INSTALL_HINT = (
"`pip install deepseek-recipe`."
)
logger = logging.getLogger(__name__)
MODEL_NAMES = {"dsv4": "DeepSeek-V4-Pro", "dsv41": "DeepSeek-V4.1-Flash"}
# The checkpoint tokenizer's image token; recipe's bundled tokenizer spells it <|image|>.
IMAGE_TOKEN = "<|deepseek_image|>"
@@ -95,6 +98,23 @@ def pre_normalize(body: dict[str, Any]) -> dict[str, Any]:
return request
def loaded_tools(conversation: Any) -> list[Any]:
"""The tools the model can call after `conversation`'s history: those it does
not defer and the deferred ones a system message has loaded."""
recipe = load_deepseek_recipe()
loaded = {
tool.name
for message in conversation.messages
if isinstance(message, recipe.SystemMessage)
for tool in message.tools or []
}
return [
tool
for tool in conversation.tools
if not tool.defer_loading or tool.name in loaded
]
def _error_message(body: Any) -> str:
try:
return json.loads(body)["error"]["message"]
@@ -110,6 +130,20 @@ class DeepSeekRecipe:
recipe = load_deepseek_recipe()
self.model = MODEL_NAMES[chat_encoding_spec]
self.hf_tokenizer = tokenizer
self._fast_prompt_tokenizer = None
if type(tokenizer).__module__.partition(".")[0] == "fastokens":
from fastokens import Tokenizer
if hasattr(Tokenizer, "encode_segments"):
self._fast_prompt_tokenizer = Tokenizer.from_json_str(
tokenizer.to_str()
)
logger.info("DeepSeek recipe prompt encoder: fastokens segmented")
else:
logger.warning(
"deepseek-recipe prompt acceleration requires fastokens.encode_segments; "
"using the reference prompt tokenizer"
)
self.special_tokens = recipe.SpecialTokens(image=IMAGE_TOKEN)
self._options = recipe.ConversionOptions().with_special_tokens(
self.special_tokens
@@ -146,6 +180,19 @@ class DeepSeekRecipe:
) -> list[int]:
"""Prompt ids for `body` (see `pre_normalize`); a request recipe rejects
raises ValueError. `image_urls` receives the images in placeholder order."""
conversation = self.convert(body, effort_budget=effort_budget)
prompt_ids = self.encode_conversation(conversation, effort_budget=effort_budget)
if image_urls is not None:
image_urls.extend(self.image_urls(conversation))
if task is not None:
prompt_ids = self._attach_task(prompt_ids, task, body["thinking"])
return prompt_ids
def convert(
self, body: dict[str, Any], *, effort_budget: Optional[int] = None
) -> Any:
"""Convert `body` to recipe's `Conversation`; invalid requests raise ValueError.
Pass the same `effort_budget` to `encode_conversation`."""
recipe = load_deepseek_recipe()
normalized = pre_normalize(body)
normalized["model"] = self.model
@@ -155,16 +202,104 @@ class DeepSeekRecipe:
converted = recipe.ChatCompletionRequest(normalized).convert(self._options)
except recipe.ConversionError as error:
raise ValueError(_error_message(error.body)) from error
prompt_ids = self._encoding.encode(converted.conversation)
if image_urls is not None:
rendered = self._encoding.render_conversation(converted.conversation)
image_urls.extend(
source.url or source.data_url for source in rendered.image_sources
)
if effort_budget is not None and body["thinking"]:
return converted.conversation
def convert_native(
self, protocol: str, body: dict[str, Any], *, thinking: bool
) -> Any:
"""Convert a Responses or Messages request to recipe's `Conversation` in the
host's thinking mode; invalid requests raise ValueError."""
recipe = load_deepseek_recipe()
request_type = {
"responses": recipe.ResponsesRequest,
"messages": recipe.MessagesRequest,
}[protocol]
options = self._options.with_default_thinking_mode(thinking)
try:
converted = request_type({**body, "model": self.model}).convert(options)
except recipe.ConversionError as error:
raise ValueError(_error_message(error.body)) from error
return converted.conversation
def with_host_settings(
self,
conversation: Any,
*,
response_schema: Optional[dict[str, Any]],
continue_final_message: bool,
) -> Any:
"""`conversation` with what recipe's Responses and Messages conversions
leave out: a JSON response schema, and SGLang's reading of a trailing
assistant turn, continued or else re-sent as a user turn."""
recipe = load_deepseek_recipe()
messages = conversation.messages
last = messages[-1] if messages else None
if last is not None and last.role == "assistant":
# recipe closes the turn and opens another; SGLang continues it or,
# as `_handle_last_assistant_message`, re-sends it as a user turn.
if continue_final_message:
last = recipe.AssistantMessage(
last.content, last.reasoning_content, prefix=True
)
else:
last = recipe.UserMessage(last.content)
messages = [*messages[:-1], last]
elif response_schema is None:
return conversation
return recipe.Conversation(
messages,
thinking_mode=conversation.thinking_mode,
tools=conversation.tools,
tool_choice=conversation.tool_choice,
reasoning_effort=conversation.reasoning_effort,
response_schema=(
response_schema
if response_schema is not None
else conversation.response_schema
),
)
def namespace_tool_names(
self, namespaces: list[dict[str, Any]]
) -> dict[str, tuple[str, str]]:
"""The name recipe's Responses conversion calls each member of `namespaces`
by, mapped to its namespace and its name there. `namespaces` lists the
request's namespace tools in conversion order: declared, then discovered."""
recipe = load_deepseek_recipe()
body = {"model": self.model, "input": "", "tools": namespaces}
try:
return recipe.ResponsesRequest(body).namespace_tool_map(self._options)
except recipe.ConversionError as error:
raise ValueError(_error_message(error.body)) from error
def image_urls(self, conversation: Any) -> list[str]:
"""The images of `conversation` in the order its prompt places them."""
rendered = self._encoding.render_conversation(conversation)
return [source.url or source.data_url for source in rendered.image_sources]
def encode_conversation(
self, conversation: Any, *, effort_budget: Optional[int] = None
) -> list[int]:
"""Encode a recipe `Conversation`, including a prefix of its messages."""
if self._fast_prompt_tokenizer is None:
prompt_ids = self._encoding.encode(conversation)
else:
rendered = self._encoding.render_conversation(conversation)
prompt = rendered.prompt
segments = []
cursor = 0
# Recipe distinguishes control tokens from identical spellings in
# user text. Its Python ranges use code-point offsets, not bytes.
for start, end in rendered.special_ranges:
if start > cursor:
segments.append((prompt[cursor:start], False))
segments.append((prompt[start:end], True))
cursor = end
if cursor < len(prompt):
segments.append((prompt[cursor:], False))
prompt_ids = self._fast_prompt_tokenizer.encode_segments(segments).ids
if effort_budget is not None and conversation.thinking_mode:
prompt_ids = self._replace_effort_budget(prompt_ids, effort_budget)
if task is not None:
prompt_ids = self._attach_task(prompt_ids, task, body["thinking"])
return prompt_ids
def parse_start(
@@ -97,7 +97,9 @@ class AnthropicMessagesMixin:
)
serving = AnthropicServing(
openai_serving_chat=SimpleNamespace(supports_inline_system=True)
openai_serving_chat=SimpleNamespace(
supports_inline_system=True, native_tool_loading=False
)
)
chat_request = serving._convert_to_chat_completion_request(anthropic_request)
converted = chat_request.model_dump()
@@ -0,0 +1,253 @@
"""CUDA ordering/timeline tests for the per-op async backup index copy.
Plan: handoff/prefill-backup-overlap-next-20261107.md §3/§8.2.
Run inside the GPU container with the overlay source:
PYTHONPATH=/tmp/ovr python -m pytest \
test/registered/cuda/test_async_backup_index_copy_cuda.py -q
What is proven on real hardware:
1. ``_start_async_index_copy`` returns while a long kernel is still queued on
the scheduler stream, and the synchronous ``.cpu()`` on that stream blocks;
2. the copy is only released after that fence, but the *next* forward (a
marker kernel on another stream, ordered behind the same fence) runs
concurrently with it: the copy end timestamp falls strictly inside the
marker's [start, end] interval;
3. the host can enqueue the next batch before ``ready_event`` completes and
the event is not reported ready early;
4. with three ops whose copies complete out of order, the worker still merges
them in op order, identical to ``CacheOperation.merge_ops`` over the CPU
reference tensors;
5. the copy stream is distinct from the scheduler stream.
"""
import threading
import time
import unittest
import torch
from sglang.srt.managers import cache_controller as cc_mod
class _FakeHostPool:
"""Not a HostKVCache: uses_shared_host_layout() is false, i.e. the sync path
would call device_indices.cpu()."""
shared_allocation_domain = None
class _FakeEntry:
def __init__(self):
self.host_pool = _FakeHostPool()
class _FakeMemPoolHost:
layout = "page_first_direct"
def __init__(self):
self.anchor_entry = _FakeEntry()
self.entry_map = {}
def _make_controller():
c = cc_mod.HiCacheController.__new__(cc_mod.HiCacheController)
c.io_backend = "direct"
c.mem_pool_host = _FakeMemPoolHost()
c.write_queue = []
c.load_queue = []
c.ack_write_queue = []
c.ack_load_queue = []
c._indices_stream = None
c._write_submit_queue = None
c._write_result_queue = cc_mod.Queue()
c._write_submit_stop = threading.Event()
c._write_submit_thread = None
c._write_submit_state_lock = threading.Lock()
c._write_submit_cond = threading.Condition(c._write_submit_state_lock)
c._write_submit_active = 0
c._async_pinned_bytes = 0
c._retire_parts = []
return c
def _op(device_indices, node_id):
host = torch.empty(device_indices.numel(), dtype=torch.int64)
return cc_mod.CacheOperation(host, device_indices, node_id)
class AsyncBackupIndexCopyCudaTest(unittest.TestCase):
@classmethod
def setUpClass(cls):
cc_mod._ASYNC_BACKUP_SUBMIT = True
cls.saved_pin = cc_mod._ASYNC_BACKUP_PINNED_BYTES_MAX
cc_mod._ASYNC_BACKUP_PINNED_BYTES_MAX = 1 << 30
@classmethod
def tearDownClass(cls):
cc_mod._ASYNC_BACKUP_PINNED_BYTES_MAX = cls.saved_pin
def test_start_does_not_block_and_copy_is_async(self):
torch.cuda.synchronize()
sched = torch.cuda.current_stream()
fwd = torch.cuda.Stream()
indices = torch.arange(4096, dtype=torch.int64, device="cuda")
# Simulated in-flight forward plus the scheduler's WAR fence.
with torch.cuda.stream(fwd):
torch.cuda._sleep(2_000_000_000)
sched.wait_stream(fwd)
c = _make_controller()
op = _op(indices, 11)
t0 = time.perf_counter()
c._start_async_index_copy(op)
start_elapsed = time.perf_counter() - t0
self.assertIsNotNone(op.async_parts)
# Starting the copy must not block on the simulated forward.
self.assertLess(start_elapsed, 0.05, start_elapsed)
self.assertFalse(op.async_parts.ready_event.query())
# The synchronous baseline on the same stream does block.
t1 = time.perf_counter()
reference = indices.cpu()
sync_elapsed = time.perf_counter() - t1
self.assertGreater(sync_elapsed, 0.1, sync_elapsed)
op.async_parts.ready_event.synchronize()
self.assertTrue(op.async_parts.ready_event.query())
c._materialize_async_indices([op])
self.assertTrue(bool(torch.equal(op.device_indices, reference)))
self.assertFalse(op.device_indices.is_cuda)
c._release_async_parts([op])
self.assertEqual(c._async_pinned_bytes, 0)
def test_next_forward_overlaps_the_index_copy(self):
torch.cuda.synchronize()
sched = torch.cuda.current_stream()
fwd = torch.cuda.Stream()
# Large enough that the copy takes long enough to observe (32 MB).
indices = torch.arange(4 * 1024 * 1024, dtype=torch.int64, device="cuda")
# 1. A long fence on the scheduler stream (the in-flight forward).
with torch.cuda.stream(sched):
fence_done = torch.cuda.Event(enable_timing=True)
fence_done.record()
torch.cuda._sleep(1_000_000_000)
# 2. Per-op pinned copy started now; it inherits the fence.
c = _make_controller()
op = _op(indices, 3)
c._start_async_index_copy(op)
self.assertIsNotNone(op.async_parts)
# 3. The next batch launch waits for the scheduler stream (as the real
# forward_stream does) and then runs a long marker kernel.
copy_end = torch.cuda.Event(enable_timing=True)
with torch.cuda.stream(c._indices_stream):
copy_end.record()
launch_host_ns = time.monotonic_ns()
with torch.cuda.stream(fwd):
fwd.wait_stream(sched)
fwd_start = torch.cuda.Event(enable_timing=True)
fwd_start.record()
torch.cuda._sleep(1_500_000_000)
fwd_end = torch.cuda.Event(enable_timing=True)
fwd_end.record()
# 4. The event must not be ready while the fence is in flight, and the
# host has already enqueued the next batch.
ready_seen_ns = [None]
def _wait_ready():
op.async_parts.ready_event.synchronize()
ready_seen_ns[0] = time.monotonic_ns()
waiter = threading.Thread(target=_wait_ready)
waiter.start()
self.assertIsNone(ready_seen_ns[0])
waiter.join(timeout=30)
self.assertIsNotNone(ready_seen_ns[0])
self.assertLess(launch_host_ns, ready_seen_ns[0])
torch.cuda.synchronize()
# CUDA timestamps: copy end strictly inside the next forward.
self.assertGreater(fwd_start.elapsed_time(copy_end), 0.0)
self.assertGreater(copy_end.elapsed_time(fwd_end), 0.0)
self.assertGreater(fwd_start.elapsed_time(fwd_end), 0.5)
c._materialize_async_indices([op])
self.assertTrue(
bool(torch.equal(op.device_indices, indices.cpu()))
)
c._release_async_parts([op])
def test_copy_stream_is_distinct_from_scheduler(self):
torch.cuda.synchronize()
c = _make_controller()
op = _op(torch.arange(8, dtype=torch.int64, device="cuda"), 0)
c._start_async_index_copy(op)
self.assertIsNotNone(c._indices_stream)
self.assertNotEqual(
c._indices_stream.cuda_stream,
torch.cuda.current_stream().cuda_stream,
)
op.async_parts.ready_event.synchronize()
c._release_async_parts([op])
def test_out_of_order_completion_keeps_op_order(self):
torch.cuda.synchronize()
c = _make_controller()
delays = {0: 600_000_000, 1: 150_000_000, 2: 300_000_000}
ops = []
references = []
for i in range(3):
src = torch.arange(8, dtype=torch.int64, device="cuda") + i * 100
op = _op(src, node_id=i)
buf = torch.empty(src.shape, dtype=src.dtype, device="cpu", pin_memory=True)
stream = torch.cuda.Stream()
with torch.cuda.stream(stream):
torch.cuda._sleep(delays[i])
buf.copy_(src, non_blocking=True)
ev = torch.cuda.Event()
ev.record(stream)
op.async_parts = cc_mod.AsyncIndexParts(
[cc_mod.AsyncIndexCopy((None, None), src, buf, buf.numel() * 8)],
ev,
buf.numel() * 8,
)
ops.append(op)
references.append(src.cpu())
captured = []
c._submit_write_operation = lambda merged: (
captured.append(merged.device_indices.clone())
or cc_mod.HiCacheAck(None, None, list(merged.node_ids), num_tokens=8)
)
c._write_submit_queue = cc_mod.Queue(maxsize=4)
worker = threading.Thread(target=c._write_submit_worker, daemon=True)
c._write_submit_thread = worker
worker.start()
try:
c._write_submit_active = 1
c._write_submit_queue.put(cc_mod.WriteSubmission(ops, None, 0))
deadline = time.time() + 30
while c._write_submit_active and time.time() < deadline:
time.sleep(0.01)
self.assertEqual(c._write_submit_active, 0)
finally:
c._write_submit_queue.put(None)
worker.join(timeout=10)
self.assertEqual(len(captured), 1)
expected = torch.cat(references)
self.assertTrue(bool(torch.equal(captured[0], expected)))
self.assertEqual(ops[0].node_ids, [0])
self.assertEqual(ops[1].node_ids, [1])
self.assertEqual(ops[2].node_ids, [2])
if __name__ == "__main__":
unittest.main()
@@ -0,0 +1,550 @@
"""Unit tests for the async HiCache backup submission state machine.
Design: handoff/prefill-backup-overlap-next-20261107.md §8.1.
These tests exercise the scheduler/worker split without a GPU: the index copy
itself is covered by the CUDA test, and here the device stream/event and the
submit path are faked. What is checked:
* disabled / inapplicable shapes keep the sync path;
* per-op pinned parts are materialized per (pool, indices_from_pool) key;
* the worker waits every ready event before touching host buffers;
* exactly-once ACK, fail-closed errors, bounded queues, idle/stop lifecycle;
* has_inflight_device_transfers reflects queued/running results.
"""
import threading
import time
import types
import unittest
from unittest import mock
from sglang.srt.managers import cache_controller as cc_mod
from sglang.srt.mem_cache.hicache_storage import PoolName, PoolTransfer
class _FakeEvent:
def __init__(self, worker_log=None, name="ev"):
self.worker_log = worker_log
self.name = name
self._set = False
self.waited = 0
def record(self, stream=None):
self._set = True
def synchronize(self):
self.waited += 1
if self.worker_log is not None:
self.worker_log.append(("wait", self.name))
self._set = True
def query(self):
return self._set
class _FakeStream:
def __init__(self, name="stream", log=None):
self.name = name
self.log = log
self.waited = 0
def wait_stream(self, other):
self.waited += 1
if self.log is not None:
self.log.append(("wait_stream", self.name, getattr(other, "name", "?")))
def wait_event(self, event):
pass
class _FakeStreamCtx:
def __init__(self, stream):
self.stream = stream
def __enter__(self):
return self.stream
def __exit__(self, *exc):
return False
class _FakeCudaModule:
def __init__(self, log=None):
self.log = log
self.current = _FakeStream("schedule", log)
def Stream(self, *a, **k):
return _FakeStream("indices", self.log)
def StreamContext(self, stream):
return _FakeStreamCtx(stream)
def Event(self, *a, **k):
return _FakeEvent(self.log)
def current_stream(self):
return self.current
class _FakeTensor:
def __init__(self, numel=8, element_size=8, is_cuda=True, tag="dev"):
self._numel = numel
self._element_size = element_size
self.is_cuda = is_cuda
self.tag = tag
self.recorded = []
self.cpu_calls = 0
self.device = types.SimpleNamespace(type="cuda" if is_cuda else "cpu")
self.dtype = "torch.int64"
def numel(self):
return self._numel
def element_size(self):
return self._element_size
def record_stream(self, stream):
self.recorded.append(stream)
def copy_(self, src, non_blocking=False):
self.copied_from = src
@property
def shape(self):
return (self._numel,)
def cpu(self):
self.cpu_calls += 1
return _FakeTensor(self._numel, self._element_size, False, tag=self.tag + "-cpu")
def __len__(self):
return self._numel
class _FakeHostPool:
def __init__(self, shared=False):
self.shared = shared
class _FakeEntry:
def __init__(self, host_pool):
self.host_pool = host_pool
class _FakeMemPoolHost:
layout = "page_first_direct"
def __init__(self, anchor_shared=False, sidecars=None):
self.anchor_entry = _FakeEntry(_FakeHostPool(anchor_shared))
self.entry_map = {
name: _FakeEntry(pool) for name, pool in (sidecars or {}).items()
}
def _make_controller(*, enabled=True, anchor_shared=False, sidecars=None, policy=""):
c = cc_mod.HiCacheController.__new__(cc_mod.HiCacheController)
c.io_backend = "direct"
c.write_policy = policy
c.mem_pool_host = _FakeMemPoolHost(anchor_shared, sidecars)
c.write_queue = []
c.load_queue = []
c.ack_write_queue = []
c.ack_load_queue = []
c._indices_stream = None
c._write_submit_queue = None
c._write_result_queue = cc_mod.Queue()
c._write_submit_stop = threading.Event()
c._write_submit_thread = None
c._write_submit_state_lock = threading.Lock()
c._write_submit_cond = threading.Condition(c._write_submit_state_lock)
c._write_submit_active = 0
c._async_pinned_bytes = 0
c._retire_parts = []
return c
def _op(controller, *, anchor=None, transfers=(), node_id=7):
host = _FakeTensor(8, 8, is_cuda=False, tag="host")
dev = anchor or _FakeTensor(8, 8, is_cuda=True, tag="anchor")
op = cc_mod.CacheOperation(host, dev, node_id, pool_transfers=list(transfers))
return op
class AsyncBackupUnitTest(unittest.TestCase):
def setUp(self):
patcher = mock.patch.object(cc_mod, "_ASYNC_BACKUP_SUBMIT", True)
patcher.start()
self.addCleanup(patcher.stop)
# The real predicate isinstance-checks HostKVCache; the fakes expose a
# plain `shared` flag instead.
shared_patcher = mock.patch.object(
cc_mod, "uses_shared_host_layout", lambda pool: bool(getattr(pool, "shared", False))
)
shared_patcher.start()
self.addCleanup(shared_patcher.stop)
cat_patcher = mock.patch.object(
cc_mod.torch,
"cat",
side_effect=lambda parts: _FakeTensor(
sum(len(p) for p in parts), 8, False, "cat"
),
)
cat_patcher.start()
self.addCleanup(cat_patcher.stop)
# Pinned destination buffers are faked; only the copy_ call is checked.
self.pin_patch = mock.patch.object(
cc_mod.torch, "empty", side_effect=lambda *a, **k: _FakeTensor(8, 8, False, "pin")
)
self.pin_patch.start()
self.addCleanup(self.pin_patch.stop)
def test_disabled_keeps_sync_path(self):
with mock.patch.object(cc_mod, "_ASYNC_BACKUP_SUBMIT", False):
c = _make_controller()
self.assertFalse(c.async_backup_enabled())
self.assertFalse(c._async_submit_applicable())
def test_wrong_backend_or_layout_is_ineligible(self):
c = _make_controller()
c.io_backend = "kernel"
self.assertFalse(c.async_backup_enabled())
c = _make_controller()
c.mem_pool_host.layout = "page_first"
self.assertFalse(c.async_backup_enabled())
def test_index_targets_skip_shared_layout_and_cpu_tensors(self):
shared = PoolName.SWA
c = _make_controller(sidecars={shared: _FakeHostPool(shared=True)})
transfer = PoolTransfer(
name=shared, device_indices=_FakeTensor(4, 8, True, "swa")
)
cpu_anchor = _FakeTensor(8, 8, False, "anchor")
op = _op(c, anchor=cpu_anchor, transfers=[transfer])
self.assertEqual(c._async_index_targets(op), [])
c = _make_controller(sidecars={PoolName.SWA: _FakeHostPool(shared=False)})
transfer = PoolTransfer(
name=PoolName.SWA,
device_indices=_FakeTensor(4, 8, True, "swa"),
indices_from_pool=PoolName.KV,
)
anchor = _FakeTensor(8, 8, True, "anchor")
op = _op(c, anchor=anchor, transfers=[transfer])
targets = c._async_index_targets(op)
self.assertEqual([k for k, _ in targets], [(None, None), (PoolName.SWA, PoolName.KV)])
def test_start_copy_orders_after_scheduler_and_records_event(self):
log = []
c = _make_controller()
op = _op(c)
with mock.patch.object(cc_mod, "device_module", _FakeCudaModule(log)):
c._start_async_index_copy(op)
self.assertIsNotNone(op.async_parts)
self.assertEqual(op.async_parts.ready_event._set, True)
self.assertEqual(op.device_indices.recorded[0].name, "indices")
# The copy stream is ordered after the scheduler stream once.
self.assertIn(("wait_stream", "indices", "schedule"), log)
self.assertEqual(c._async_pinned_bytes, 64)
def test_pinned_cap_falls_back_to_sync(self):
c = _make_controller()
op = _op(c)
with mock.patch.object(cc_mod, "_ASYNC_BACKUP_PINNED_BYTES_MAX", 1), \
mock.patch.object(cc_mod, "device_module", _FakeCudaModule()):
c._start_async_index_copy(op)
self.assertIsNone(op.async_parts)
self.assertEqual(c._async_pinned_bytes, 0)
def test_materialize_uses_pool_keys(self):
c = _make_controller()
anchor_copy = cc_mod.AsyncIndexCopy((None, None), None, _FakeTensor(8, 8, False, "a"), 64)
deriv = PoolTransfer(
name=PoolName.SWA,
device_indices=_FakeTensor(4, 8, True, "swa"),
indices_from_pool=PoolName.KV,
)
side_copy = cc_mod.AsyncIndexCopy(
(PoolName.SWA, PoolName.KV), None, _FakeTensor(4, 8, False, "s"), 32
)
op = _op(c, transfers=[deriv])
op.async_parts = cc_mod.AsyncIndexParts(
copies=[anchor_copy, side_copy], ready_event=_FakeEvent(), nbytes=96
)
c._materialize_async_indices([op])
self.assertIs(op.device_indices, anchor_copy.dst)
self.assertIs(deriv.device_indices, side_copy.dst)
def test_worker_waits_events_then_submits_and_acks_once(self):
log = []
c = _make_controller()
events = []
def _make_op(tag):
op = _op(c, anchor=_FakeTensor(8, 8, True, tag))
ev = _FakeEvent(log, tag)
events.append(ev)
op.async_parts = cc_mod.AsyncIndexParts(
copies=[
cc_mod.AsyncIndexCopy(
(None, None), op.device_indices, _FakeTensor(8, 8, False, tag), 64
)
],
ready_event=ev,
nbytes=64,
)
return op
ops = [_make_op("a"), _make_op("b")]
c._async_pinned_bytes = 128
submitted = []
def _submit(op):
# Every ready event must have been waited before the merge is used.
submitted.append(list(op.node_ids))
return cc_mod.HiCacheAck(None, None, list(op.node_ids), num_tokens=8)
c._submit_write_operation = _submit
c._write_submit_queue = cc_mod.Queue(maxsize=4)
c._ensure_write_submit_thread = lambda: None
worker = threading.Thread(target=c._write_submit_worker, daemon=True)
c._write_submit_thread = worker
worker.start()
try:
item = cc_mod.WriteSubmission(ops=ops, tracer=None, enqueue_mono_ns=0)
c._write_submit_active = 1
c._write_submit_queue.put(item)
deadline = time.time() + 5
while c._write_submit_active and time.time() < deadline:
time.sleep(0.005)
self.assertEqual(c._write_submit_active, 0)
finally:
c._write_submit_queue.put(None)
worker.join(timeout=5)
self.assertEqual([e.waited for e in events], [1, 1])
self.assertEqual(submitted, [[7, 7]])
self.assertEqual(c._async_pinned_bytes, 0)
c.drain_write_submission_results()
self.assertEqual(len(c.ack_write_queue), 1)
# Second drain is a no-op: exactly-once ACK.
c.drain_write_submission_results()
self.assertEqual(len(c.ack_write_queue), 1)
def test_worker_error_is_fail_closed(self):
c = _make_controller()
op = _op(c)
op.async_parts = cc_mod.AsyncIndexParts([], _FakeEvent(), 0)
c._submit_write_operation = mock.Mock(side_effect=RuntimeError("boom"))
c._write_submit_queue = cc_mod.Queue(maxsize=4)
worker = threading.Thread(target=c._write_submit_worker, daemon=True)
c._write_submit_thread = worker
worker.start()
c._write_submit_active = 1
c._write_submit_queue.put(cc_mod.WriteSubmission([op], None, 0))
deadline = time.time() + 5
while c._write_submit_active and time.time() < deadline:
time.sleep(0.005)
c._write_submit_queue.put(None)
worker.join(timeout=5)
with self.assertRaises(RuntimeError):
c.drain_write_submission_results()
self.assertEqual(c.ack_write_queue, [])
def test_enqueue_failure_keeps_active_consistent(self):
c = _make_controller()
c._write_submit_queue = cc_mod.Queue(maxsize=1)
c._write_submit_queue.put("occupied")
c._ensure_write_submit_thread = lambda: None
with mock.patch.object(cc_mod, "_ASYNC_BACKUP_ENQUEUE_TIMEOUT", 0.05):
ok = c._enqueue_async_write([_op(c)], None)
self.assertFalse(ok)
self.assertEqual(c._write_submit_active, 0)
def test_queue_full_falls_back_to_sync_submit(self):
c = _make_controller()
# A permanently-full submit queue forces the sync path.
c._write_submit_queue = cc_mod.Queue(maxsize=1)
c._write_submit_queue.put("occupied")
c._ensure_write_submit_thread = lambda: None
op = _op(c)
retired_event = _FakeEvent()
retired_event.record()
op.async_parts = cc_mod.AsyncIndexParts(
[cc_mod.AsyncIndexCopy((None, None), op.device_indices, _FakeTensor(8, 8, False), 64)],
retired_event,
64,
)
c._async_pinned_bytes = 64
c.write_queue.append(op)
submitted = []
def _submit(merged):
# The fallback must hand over the original device indices, i.e. the
# pinned parts are retired rather than materialized.
submitted.append(merged.device_indices)
return cc_mod.HiCacheAck(None, None, list(merged.node_ids), num_tokens=8)
c._submit_write_operation = _submit
with mock.patch.object(cc_mod, "_ASYNC_BACKUP_ENQUEUE_TIMEOUT", 0.05):
c.start_writing()
self.assertEqual(len(submitted), 1)
self.assertIs(submitted[0], op.device_indices)
self.assertEqual(len(c.ack_write_queue), 1)
self.assertEqual(c.write_queue, [])
self.assertIsNone(op.async_parts)
self.assertEqual(c._write_submit_active, 0)
# The retired copy is released once its event completes.
c._reap_retired_parts()
self.assertEqual(c._async_pinned_bytes, 0)
def test_mixed_queue_sync_path_retires_unconsumed_parts(self):
c = _make_controller()
ev = _FakeEvent()
ev.record()
with_parts = _op(c, node_id=1)
with_parts.async_parts = cc_mod.AsyncIndexParts(
[
cc_mod.AsyncIndexCopy(
(None, None), with_parts.device_indices, _FakeTensor(8, 8, False), 64
)
],
ev,
64,
)
without_parts = _op(c, node_id=2)
c.write_queue.extend([with_parts, without_parts])
c._async_pinned_bytes = 64
submitted = []
c._submit_write_operation = lambda merged: (
submitted.append(list(merged.node_ids))
or cc_mod.HiCacheAck(None, None, list(merged.node_ids), num_tokens=8)
)
c.start_writing()
# Mixed residency () forces the sync path, but the pinned copy started
# for the first op must be retired and its budget released.
self.assertEqual(submitted, [[1, 2]])
self.assertIsNone(with_parts.async_parts)
self.assertEqual(c._async_pinned_bytes, 0)
self.assertEqual(len(c.ack_write_queue), 1)
def test_op_cap_falls_back_to_sync(self):
c = _make_controller()
with mock.patch.object(cc_mod, "_ASYNC_BACKUP_MAX_OPS", 1):
for i in range(2):
op = _op(c, node_id=i)
op.async_parts = cc_mod.AsyncIndexParts([], _FakeEvent(), 0)
c.write_queue.append(op)
self.assertFalse(c._async_submit_applicable())
def test_inflight_reflects_async_state(self):
c = _make_controller()
self.assertFalse(c.has_inflight_device_transfers())
c._write_submit_active = 1
self.assertTrue(c.has_inflight_device_transfers())
c._write_submit_active = 0
c._async_pinned_bytes = 8
self.assertTrue(c.has_inflight_device_transfers())
c._async_pinned_bytes = 0
c._write_result_queue.put(cc_mod.WriteResult(None, [], None))
self.assertTrue(c.has_inflight_device_transfers())
c._write_result_queue.get()
self.assertFalse(c.has_inflight_device_transfers())
def test_idle_wait_times_out(self):
c = _make_controller()
c._write_submit_active = 1
with self.assertRaises(RuntimeError):
c.wait_write_submissions_idle(timeout=0.05)
def test_retired_parts_release_only_after_event(self):
c = _make_controller()
ev = _FakeEvent()
ev._set = False
op = _op(c)
op.async_parts = cc_mod.AsyncIndexParts(
[cc_mod.AsyncIndexCopy((None, None), None, _FakeTensor(8, 8, False), 64)],
ev,
64,
)
c._async_pinned_bytes = 64
c._retire_async_parts([op])
self.assertIsNone(op.async_parts)
self.assertEqual(c._async_pinned_bytes, 64)
ev._set = True
c._reap_retired_parts()
self.assertEqual(c._async_pinned_bytes, 0)
self.assertEqual(c._retire_parts, [])
def test_merge_of_cpu_parts_matches_cpu_reference(self):
c = _make_controller()
ops = []
for i in range(3):
op = _op(c, anchor=_FakeTensor(4, 8, True, f"a{i}"), node_id=i)
op.async_parts = cc_mod.AsyncIndexParts(
[
cc_mod.AsyncIndexCopy(
(None, None),
op.device_indices,
_FakeTensor(4, 8, False, f"p{i}"),
32,
)
],
_FakeEvent(),
32,
)
ops.append(op)
c._materialize_async_indices(ops)
merged = cc_mod.CacheOperation.merge_ops(ops)
self.assertEqual(merged.node_ids, [0, 1, 2])
# Single-op merge must not copy: it returns the operation itself.
single = cc_mod.CacheOperation.merge_ops([ops[0]])
self.assertIs(single, ops[0])
def test_stop_waits_for_inflight_submission_and_drains_ack(self):
c = _make_controller()
release = threading.Event()
started = threading.Event()
def _submit(op):
started.set()
release.wait(10)
return cc_mod.HiCacheAck(None, None, list(op.node_ids), num_tokens=8)
c._submit_write_operation = _submit
op = _op(c)
op.async_parts = cc_mod.AsyncIndexParts([], _FakeEvent(), 0)
c._ensure_write_submit_thread()
c._write_submit_active = 1
c._write_submit_queue.put(cc_mod.WriteSubmission([op], None, 0))
self.assertTrue(started.wait(5))
stopped = threading.Event()
def _stop():
c._stop_write_submission_thread()
stopped.set()
stopper = threading.Thread(target=_stop, daemon=True)
stopper.start()
# The stop must not return while a submission is still in flight.
self.assertFalse(stopped.wait(0.05))
release.set()
self.assertTrue(stopped.wait(10))
self.assertFalse(c._write_submit_thread is not None and c._write_submit_thread.is_alive())
# The ACK produced by the in-flight submission was not dropped.
self.assertEqual(len(c.ack_write_queue), 1)
def test_stop_joins_worker_and_allows_restart(self):
c = _make_controller()
c._ensure_write_submit_thread()
self.assertTrue(c._write_submit_thread.is_alive())
c._stop_write_submission_thread()
self.assertIsNone(c._write_submit_thread)
self.assertIsNone(c._write_submit_queue)
c._ensure_write_submit_thread()
self.assertTrue(c._write_submit_thread.is_alive())
c._stop_write_submission_thread()
if __name__ == "__main__":
unittest.main()
@@ -0,0 +1,504 @@
"""Unit tests for the SWA-only async state-payload first version.
Covers the v2 checklist that can run without a live server:
* should_async_state_payload guards (last chunk / capability / shard / CP);
* require_async_state_event (empty last chunk / multi segment invariant);
* worker materialize_state_indices (event ordering, no-event failure,
non-tensor and None entries preserved);
* kv_to_page_indices_pinned body: pinned CPU buffer, non-blocking copy,
same dtype/shape, no synchronous .cpu();
* a CUDA ordering test (skipped unless torch + CUDA + sglang are importable)
proving the helper returns before a preceding long kernel finishes and
the event-synchronized values are correct.
The functions are extracted from the real source with ``ast`` so the test does
not need to import the heavy SGLang modules. Full DSV4 component / abort /
multi-segment integration cases run in the server harness on b300.
"""
import ast
import pathlib
import sys
import time
import types
import unittest
from unittest import mock
import numpy as np
try:
from sglang.test.test_utils import CustomTestCase
except Exception: # local helper-only runs without sglang installed
CustomTestCase = unittest.TestCase
def _repo_root():
return pathlib.Path(__file__).resolve().parents[4]
def _load_function(rel_path, name, extra_ns=None):
path = _repo_root() / rel_path
tree = ast.parse(path.read_text(encoding="utf-8"))
fn = next(
node
for node in tree.body
if isinstance(node, ast.FunctionDef) and node.name == name
)
namespace = dict(extra_ns or {})
module = ast.Module(body=[fn], type_ignores=[])
exec(compile(module, str(path), "exec"), namespace)
return namespace[name]
def _load_pinned_helper():
"""Execute the helper body from the real source with a fake torch."""
path = _repo_root() / "python/sglang/srt/mem_cache/common.py"
tree = ast.parse(path.read_text(encoding="utf-8"))
fn = next(
node
for node in tree.body
if isinstance(node, ast.FunctionDef) and node.name == "kv_to_page_indices_pinned"
)
calls = {"empty": [], "copy": []}
class _FakeTensor:
def __init__(self, values, dtype="torch.int32", shape=None):
self.values = list(values)
self.dtype = dtype
self.shape = (len(self.values),) if shape is None else shape
def __getitem__(self, sl):
return _FakeTensor(self.values[sl], self.dtype)
def __floordiv__(self, value):
return _FakeTensor([v // value for v in self.values], self.dtype)
def cpu(self):
raise AssertionError("kv_to_page_indices_pinned must not call .cpu()")
def numpy(self):
raise AssertionError(
"kv_to_page_indices_pinned must not call .numpy()"
)
def copy_(self, src, non_blocking=False):
calls["copy"].append((src, non_blocking))
self.values = list(src.values)
class _FakeTorch:
Tensor = _FakeTensor
@staticmethod
def empty(shape, dtype, device, pin_memory):
calls["empty"].append(
{"shape": shape, "dtype": dtype, "device": device, "pin": pin_memory}
)
return _FakeTensor([], dtype=dtype, shape=shape)
namespace = {"torch": _FakeTorch}
module = ast.Module(body=[fn], type_ignores=[])
exec(compile(module, str(path), "exec"), namespace)
return namespace["kv_to_page_indices_pinned"], calls, _FakeTensor
class AsyncStateGateTest(CustomTestCase):
@classmethod
def setUpClass(cls):
cls.mod = types.SimpleNamespace()
base = "python/sglang/srt/disaggregation/prefill.py"
cls.mod.should_async_state_payload = _load_function(
base, "should_async_state_payload"
)
cls.mod.require_async_state_event = _load_function(
base, "require_async_state_event"
)
def test_guards(self):
mod = self.mod
base = dict(
last_chunk=True,
supports_async_state_indices=True,
kv_shard_size=1,
enable_all_cp_ranks_for_transfer=False,
enable_dsa_cache_layer_split=False,
)
self.assertTrue(mod.should_async_state_payload(**base))
self.assertFalse(
mod.should_async_state_payload(**{**base, "last_chunk": False})
)
self.assertFalse(
mod.should_async_state_payload(
**{**base, "supports_async_state_indices": False}
)
)
self.assertFalse(
mod.should_async_state_payload(**{**base, "kv_shard_size": 2})
)
self.assertFalse(
mod.should_async_state_payload(
**{**base, "enable_all_cp_ranks_for_transfer": True}
)
)
self.assertTrue(
mod.should_async_state_payload(
**{
**base,
"enable_all_cp_ranks_for_transfer": True,
"enable_dsa_cache_layer_split": True,
}
)
)
def test_final_event_required(self):
mod = self.mod
mod.require_async_state_event(True, True, object()) # no raise
mod.require_async_state_event(False, True, None) # sync path
mod.require_async_state_event(True, False, None) # non-final segment
with self.assertRaises(RuntimeError):
mod.require_async_state_event(True, True, None)
class MaterializeStateIndicesTest(CustomTestCase):
class _FakeTensor:
def __init__(self, values):
self.values = values
def numpy(self):
return np.array(self.values, dtype=np.int32)
@classmethod
def setUpClass(cls):
cls.fn = staticmethod(
_load_function(
"python/sglang/srt/disaggregation/mooncake/conn.py",
"materialize_state_indices",
)
)
def _fake_torch(self):
module = types.ModuleType("torch")
module.Tensor = self._FakeTensor
return module
def test_event_required_for_tensor(self):
with mock.patch.dict(sys.modules, {"torch": self._fake_torch()}):
with self.assertRaises(RuntimeError):
self.fn([self._FakeTensor([1, 2])], None)
def test_materializes_after_event_and_preserves_others(self):
np_indices = np.array([7, 8], dtype=np.int32)
with mock.patch.dict(sys.modules, {"torch": self._fake_torch()}):
out = self.fn([self._FakeTensor([1, 2]), None, np_indices], object())
self.assertIsInstance(out[0], np.ndarray)
np.testing.assert_array_equal(out[0], np.array([1, 2], dtype=np.int32))
self.assertIsNone(out[1])
self.assertIs(out[2], np_indices)
def test_second_materialization_without_event_is_safe(self):
"""Staging retry: first pass materializes; second pass has no event."""
with mock.patch.dict(sys.modules, {"torch": self._fake_torch()}):
first = self.fn([self._FakeTensor([1, 2])], object())
second = self.fn(first, None)
self.assertIs(second, first)
np.testing.assert_array_equal(first[0], np.array([1, 2], dtype=np.int32))
def test_no_tensor_needs_no_event(self):
np_indices = np.array([1], dtype=np.int32)
with mock.patch.dict(sys.modules, {"torch": self._fake_torch()}):
out = self.fn([None, np_indices], None)
self.assertIsNone(out[0])
self.assertIs(out[1], np_indices)
self.assertIsNone(self.fn(None, None))
class PinnedHelperBodyTest(CustomTestCase):
def test_pinned_nonblocking_no_cpu_sync(self):
helper, calls, fake_tensor = _load_pinned_helper()
src = fake_tensor(range(16), dtype="torch.int32")
out = helper(src, page_size=4)
self.assertEqual(out.values, [0, 1, 2, 3])
self.assertEqual(out.dtype, "torch.int32")
self.assertEqual(out.shape, (4,))
self.assertEqual(len(calls["empty"]), 1)
self.assertEqual(calls["empty"][0]["device"], "cpu")
self.assertTrue(calls["empty"][0]["pin"])
self.assertEqual(len(calls["copy"]), 1)
self.assertTrue(calls["copy"][0][1])
class CudaSendChainTest(CustomTestCase):
"""Real _send_kv_chunk -> sender.send -> transfer_worker call chain."""
@classmethod
def setUpClass(cls):
try:
import torch
if not torch.cuda.is_available():
raise unittest.SkipTest("CUDA required")
from sglang.srt.disaggregation.prefill import ( # noqa: F401
SchedulerDisaggregationPrefillMixin,
)
except unittest.SkipTest:
raise
except Exception as exc: # noqa: BLE001
raise unittest.SkipTest(f"torch/CUDA/sglang unavailable: {exc}")
def test_send_chunk_queues_async_state_and_worker_materializes(self):
import types
import torch
from sglang.srt.disaggregation.base.conn import KVPoll, StateType
from sglang.srt.disaggregation.common.utils import TransferKVChunk
from sglang.srt.disaggregation.mooncake.conn import (
MooncakeKVManager,
MooncakeKVSender,
)
from sglang.srt.disaggregation.prefill import (
SchedulerDisaggregationPrefillMixin,
)
from sglang.srt.observability import batch_telemetry # noqa: F401
seq_len = 32
page_size = 8
req_to_token = torch.arange(0, seq_len, device="cuda", dtype=torch.int32).view(
1, seq_len
)
allocator = types.SimpleNamespace(
page_size=page_size,
get_kvcache=lambda: types.SimpleNamespace(
request_window=None,
sliding_window=seq_len,
request_state_transfer_indices=lambda req_idx, length: np.array(
[req_idx], dtype=np.int32
),
),
translate_swa_indices_for_transfer=lambda x: x,
translate_kv_indices_for_transfer=lambda x: x,
)
req_pool = types.SimpleNamespace(req_to_token=req_to_token)
captured = []
def add_transfer_request(
room,
kv_indices,
index_slice,
is_last_chunk,
aux_index=None,
state_indices=None,
num_kv_tokens=None,
wait_event=None,
indices_event=None,
trace_ctx=None,
obs=None,
):
chunk = TransferKVChunk(
room,
kv_indices,
index_slice,
is_last_chunk,
aux_index,
state_indices,
num_kv_tokens,
)
chunk.indices_event = indices_event
captured.append(chunk)
fake_mgr = types.SimpleNamespace(
kv_shard_size=1,
is_dummy_cp_rank=False,
enable_all_cp_ranks_for_transfer=False,
_dcp_pack_max_tokens=None,
prepare_send_state_indices=lambda state: state,
add_transfer_request=add_transfer_request,
)
sender = object.__new__(MooncakeKVSender)
sender.kv_mgr = fake_mgr
sender.bootstrap_room = 1
sender.aux_index = 0
sender.conclude_state = None
sender.trace_ctx = mock.MagicMock()
sender._early_send_wait_event = None
sender._transfer_start_time = None
sender._transfer_num_kv_indices = 0
sender._transfer_num_state_indices = 0
sender.num_kv_indices = seq_len // page_size
sender.curr_idx = 0
sender._send_kind = "last"
fake_sched = types.SimpleNamespace(
token_to_kv_pool_allocator=allocator,
req_to_token_pool=req_pool,
disagg_metadata_buffers=types.SimpleNamespace(
set_buf=lambda req: None
),
disagg_prefill_bootstrap_queue=types.SimpleNamespace(
kv_manager=types.SimpleNamespace(
kv_args=types.SimpleNamespace(
state_types=[
StateType.DSV4_REQUEST_STATE,
StateType.REQUEST_ROW,
StateType.SWA,
StateType.SWA,
]
)
)
),
disagg_prefill_pending_chunk_rids=set(),
sliding_window_size=seq_len,
enable_staging=False,
)
req = types.SimpleNamespace(
rid="r1",
kv=types.SimpleNamespace(req_pool_idx=0),
start_send_idx=0,
extend_end=seq_len,
origin_input_ids=list(range(seq_len)),
disagg_decode_prefix_len=0,
pending_bootstrap=False,
disagg_kv_sender=sender,
metadata_buffer_index=0,
)
parallel = types.SimpleNamespace(
enable_dsa_cache_layer_split=False, num_dp_ranks=4
)
with mock.patch(
"sglang.srt.disaggregation.prefill.get_parallel", lambda: parallel
), mock.patch(
"sglang.srt.disaggregation.common.conn.get_parallel", lambda: parallel
):
SchedulerDisaggregationPrefillMixin._send_kv_chunk(
fake_sched, req, last_chunk=True
)
self.assertEqual(len(captured), 1)
chunk = captured[0]
self.assertTrue(chunk.is_last_chunk)
self.assertIsNotNone(chunk.indices_event)
self.assertEqual(len(chunk.state_indices), 4)
# DSV4_REQUEST_STATE / REQUEST_ROW stay host arrays; both SWA entries
# are pinned CPU tensors before the worker materializes them.
self.assertIsInstance(chunk.state_indices[0], np.ndarray)
self.assertIsInstance(chunk.state_indices[1], np.ndarray)
self.assertIsInstance(chunk.state_indices[2], torch.Tensor)
self.assertIsInstance(chunk.state_indices[3], torch.Tensor)
self.assertFalse(chunk.state_indices[2].is_cuda)
self.assertFalse(chunk.state_indices[3].is_cuda)
# Now run the real Mooncake worker on the chunk produced above.
peer = types.SimpleNamespace(
is_dummy=False,
mooncake_session_id="peer",
dst_device_kv_indices=None,
dst_kv_indices=[0, 1, 2, 3],
required_dst_info_num=1,
room=1,
endpoint="127.0.0.1",
dst_port=8998,
dst_aux_ptrs=[],
)
manager = types.SimpleNamespace(
enable_trace=False,
enable_staging=False,
enable_deferred_decode_kv_release=False,
_staging_outstanding=__import__("collections").defaultdict(int),
request_status={1: KVPoll.WaitingForInput},
transfer_infos={1: {"peer": peer}},
_prefill_unique_rank=lambda: 0,
session_lock=__import__("threading").Lock(),
failed_sessions=set(),
state_layout_rejections={},
poison_deferred_ack_room=lambda room: None,
decode_kv_args_table={
"peer": types.SimpleNamespace(
requires_dcp_relayout=False,
dst_kv_ptrs=[1000],
dst_kv_layer_ids=[0],
dst_kv_item_len=16,
dst_attn_tp_size=1,
dst_aux_ptrs=[],
)
},
_get_dsa_cache_transfer_skip_flags=lambda reg: (False, False),
kv_args=types.SimpleNamespace(
kv_data_ptrs=[1], state_types=[StateType.SWA]
),
is_mla_backend=True,
send_kvcache=lambda *a, **kw: 0,
send_aux=lambda *a, **kw: 0,
conclude_transfer=lambda *a, **kw: None,
_maybe_ack_drained_abort=lambda room: None,
bootstrap_port=8998,
)
manager.check_status = lambda room: manager.request_status[room]
seen = {}
def maybe_send_extra(req, indices, executor, registration):
seen["types"] = [type(x) for x in indices]
seen["swa_values"] = [indices[2].tolist(), indices[3].tolist()]
return 0
manager.maybe_send_extra = maybe_send_extra
queue = __import__("queue").SimpleQueue()
queue.put(chunk)
queue.put(None)
MooncakeKVManager.transfer_worker(manager, queue, mock.MagicMock())
self.assertEqual(seen["types"], [np.ndarray] * 4)
self.assertEqual(seen["swa_values"], [[0, 1, 2, 3], [0, 1, 2, 3]])
torch.cuda.synchronize()
class CudaOrderingTest(CustomTestCase):
@classmethod
def setUpClass(cls):
try:
import torch
if not torch.cuda.is_available():
raise unittest.SkipTest("CUDA required")
import sglang.srt.mem_cache.common # noqa: F401
except unittest.SkipTest:
raise
except Exception as exc: # noqa: BLE001
raise unittest.SkipTest(f"torch/CUDA/sglang unavailable: {exc}")
def test_pinned_copy_returns_before_kernel_and_event_protects_values(self):
import torch # noqa: F401
from sglang.srt.mem_cache.common import (
kv_to_page_indices,
kv_to_page_indices_pinned,
)
page_size = 8
src = torch.arange(4096, device="cuda", dtype=torch.int32)
# Warm up pinned allocation/CUDA context.
_ = kv_to_page_indices_pinned(src, page_size)
torch.cuda.synchronize()
stream = torch.cuda.Stream()
with torch.cuda.stream(stream):
torch.cuda._sleep(2_000_000_000)
t0 = time.monotonic()
out = kv_to_page_indices_pinned(src, page_size)
enqueue_s = time.monotonic() - t0
event = torch.cuda.Event()
event.record(stream)
t1 = time.monotonic()
event.synchronize()
wait_s = time.monotonic() - t1
# The helper returns from the enqueue without waiting for the kernel.
self.assertLess(enqueue_s, max(0.05, wait_s * 0.5))
# After the event, the pinned values match the synchronous helper.
np.testing.assert_array_equal(
out.numpy(), kv_to_page_indices(src, page_size)
)
if __name__ == "__main__":
unittest.main()
@@ -580,6 +580,109 @@ class TestMooncakeEarlySend(CustomTestCase):
manager.send_kvcache.assert_called_once()
class TestMooncakeAsyncStateMaterialization(CustomTestCase):
"""Real transfer_worker path for pinned state tensors (v2 async state)."""
def _manager(self):
peer = SimpleNamespace(
is_dummy=False,
mooncake_session_id="peer",
dst_device_kv_indices=None,
dst_kv_indices=[10],
required_dst_info_num=1,
room=1,
endpoint="127.0.0.1",
dst_port=8998,
dst_aux_ptrs=[],
)
manager = SimpleNamespace(
enable_trace=False,
enable_staging=False,
enable_deferred_decode_kv_release=False,
_staging_outstanding=defaultdict(int),
request_status={1: KVPoll.WaitingForInput},
transfer_infos={1: {"peer": peer}},
_prefill_unique_rank=lambda: 0,
session_lock=Lock(),
failed_sessions=set(),
state_layout_rejections={},
poison_deferred_ack_room=MagicMock(),
decode_kv_args_table={
"peer": SimpleNamespace(
requires_dcp_relayout=False,
dst_kv_ptrs=[1000],
dst_kv_layer_ids=[0],
dst_kv_item_len=16,
dst_attn_tp_size=1,
dst_aux_ptrs=[],
)
},
_get_dsa_cache_transfer_skip_flags=lambda reg: (False, False),
kv_args=SimpleNamespace(kv_data_ptrs=[1], state_types=[StateType.SWA]),
is_mla_backend=True,
send_kvcache=MagicMock(return_value=0),
send_aux=MagicMock(return_value=0),
maybe_send_extra=MagicMock(return_value=0),
conclude_transfer=MagicMock(),
_maybe_ack_drained_abort=MagicMock(),
bootstrap_port=8998,
)
manager.check_status = lambda room: manager.request_status[room]
return manager
def _run(self, manager, chunk):
queue = SimpleQueue()
queue.put(chunk)
queue.put(None)
MooncakeKVManager.transfer_worker(manager, queue, MagicMock())
def test_worker_materializes_state_after_event(self):
manager = self._manager()
state = torch.tensor([0], dtype=torch.int32)
event = MagicMock()
order = []
def sync():
order.append("sync")
state.fill_(7)
event.synchronize.side_effect = sync
chunk = TransferKVChunk(
1, np.array([0], dtype=np.int32), slice(0, 1), True, 0, [state]
)
chunk.indices_event = event
def maybe_send_extra(req, indices, executor, registration):
order.append("state")
self.assertIsInstance(indices[0], np.ndarray)
self.assertEqual(indices[0].tolist(), [7])
return 0
manager.maybe_send_extra.side_effect = maybe_send_extra
self._run(manager, chunk)
self.assertEqual(order, ["sync", "state"])
self.assertIsInstance(chunk.state_indices[0], np.ndarray)
manager.send_aux.assert_called_once()
def test_worker_rejects_state_tensor_without_event(self):
manager = self._manager()
chunk = TransferKVChunk(
1,
np.array([0], dtype=np.int32),
slice(0, 1),
True,
0,
[torch.tensor([0], dtype=torch.int32)],
)
with self.assertRaises(RuntimeError) as ctx:
self._run(manager, chunk)
self.assertIn(
"async state indices require an indices_event", str(ctx.exception)
)
manager.maybe_send_extra.assert_not_called()
class TestMooncakeStagingDeviceType(unittest.TestCase):
"""Staging init must receive the runtime's device, not a hardcoded "cuda".
@@ -31,6 +31,7 @@ register_cpu_ci(est_time=11, suite="base-a-test-cpu")
class _FakeOpenAIServingChat:
native_reasoning_history = False
native_tool_loading = False
def __init__(self, stream_lines=None, chat_template=None):
self.stream_lines = stream_lines or []
@@ -61,6 +62,7 @@ class _FakeNonStreamingErrorOpenAI:
"""Returns an error response or raises an exception from the OpenAI handler."""
supports_inline_system = False
native_tool_loading = False
def __init__(self, status_code=400, body=None, content=None, exception=None):
self._status_code = status_code
@@ -496,6 +498,26 @@ class TestAnthropicServing(unittest.TestCase):
],
)
def test_tool_addition_fails_where_tools_cannot_load_mid_conversation(self):
"""A tool_addition block must not vanish on a model that cannot render it."""
request = self._anthropic_request(
messages=[
{"role": "user", "content": "hello"},
{
"role": "system",
"content": [
{
"type": "tool_addition",
"tool": {"type": "tool_reference", "name": "Bash"},
}
],
},
]
)
with self.assertRaisesRegex(ValueError, "tool_addition"):
self._serving()._convert_to_chat_completion_request(request)
def test_builtin_web_search_tool_without_schema_is_skipped(self):
request = AnthropicMessagesRequest.model_validate(
{
File diff suppressed because it is too large Load Diff
@@ -265,6 +265,27 @@ class TestDPBudgetUpdateBudget(CustomTestCase):
)
self.assertEqual(budget.total_tokens, [150, 80])
def test_maps_waiting_uncached_tokens_and_hit_rate(self):
budget = DPBudget(num_dp_ranks=2)
budget.update_budget(
[
_load(
dp_rank=0,
timestamp=1.0,
num_waiting_uncached_tokens=900,
cache_hit_rate=0.95,
),
_load(
dp_rank=1,
timestamp=1.0,
num_waiting_uncached_tokens=100,
cache_hit_rate=0.5,
),
]
)
self.assertEqual(budget.total_uncached_tokens, [900, 100])
self.assertEqual(budget.cache_hit_rate, [0.95, 0.5])
def test_partial_update_only_affects_reported_rank(self):
budget = DPBudget(num_dp_ranks=3)
budget.update_budget(
@@ -473,6 +494,44 @@ class TestTotalRequestsScheduler(CustomTestCase):
)
class TestUncachedTokensScheduler(CustomTestCase):
def test_dispatches_to_rank_with_least_uncached_work(self):
ctl = _make_controller(num_dp_ranks=4)
ctl.dp_budget.total_uncached_tokens = [100, 50, 900, 10]
ctl.total_uncached_tokens_scheduler(_req(input_ids=[0] * 1000))
ctl.workers[3].send_pyobj.assert_called_once()
for i in (0, 1, 2):
ctl.workers[i].send_pyobj.assert_not_called()
# No hit rate reported -> the whole prompt is charged as uncached.
self.assertEqual(ctl.dp_budget.total_uncached_tokens, [100, 50, 900, 1010])
self.assertEqual(ctl.dp_budget.total_requests, [0, 0, 0, 1])
def test_charge_uses_fleet_average_miss_rate(self):
ctl = _make_controller(num_dp_ranks=2)
ctl.dp_budget.total_uncached_tokens = [0, 0]
# Fleet-average hit rate 0.5 -> every candidate charges 500 tokens, so
# the pick is decided by queued work alone (no per-rank feedback).
ctl.dp_budget.cache_hit_rate = [0.9, 0.1]
ctl.total_uncached_tokens_scheduler(_req(input_ids=[0] * 1000))
ctl.workers[0].send_pyobj.assert_called_once()
self.assertEqual(ctl.dp_budget.total_uncached_tokens, [500, 0])
def test_pending_backlog_beats_a_cheaper_candidate(self):
ctl = _make_controller(num_dp_ranks=2)
ctl.dp_budget.total_uncached_tokens = [500, 0]
ctl.dp_budget.cache_hit_rate = [0.0, 0.0]
ctl.total_uncached_tokens_scheduler(_req(input_ids=[0] * 100))
ctl.workers[1].send_pyobj.assert_called_once()
self.assertEqual(ctl.dp_budget.total_uncached_tokens, [500, 100])
def test_routed_dp_rank_bypasses_budget(self):
ctl = _make_controller(num_dp_ranks=4)
ctl.dp_budget.total_uncached_tokens = [5, 3, 1, 4]
ctl.total_uncached_tokens_scheduler(_req(routed_dp_rank=0))
ctl.workers[0].send_pyobj.assert_called_once()
self.assertEqual(ctl.dp_budget.total_uncached_tokens, [5, 3, 1, 4])
class TestStatusAwarenessInconsistency(CustomTestCase):
"""Document a divergence: ``round_robin_scheduler`` skips workers whose
``status`` is False, but ``total_requests_scheduler`` /
@@ -510,6 +510,7 @@ class TestSchedulerIdleStepCounters(CustomTestCase):
recv_requests=Mock(side_effect=[[]] * len(schedule) + [StopIteration])
)
scheduler.process_input_requests = Mock()
scheduler.load_snapshot_writer = None
scheduler.disagg_prefill_bootstrap_queue = SimpleNamespace(
pop_bootstrapped=Mock(return_value=[])
)
@@ -1,202 +1,203 @@
import unittest
from dataclasses import replace
from types import SimpleNamespace
from unittest.mock import Mock
from sglang.srt.disaggregation.utils import DisaggregationMode
from sglang.srt.managers.schedule_policy import (
CacheAwarePolicy,
SchedulePolicy,
)
from sglang.srt.managers.scheduler_components.load_inquirer import (
SchedulerLoadInquirer,
)
from sglang.srt.mem_cache.base_prefix_cache import CacheRequestHandle
from sglang.test.ci.ci_register import register_cpu_ci
register_cpu_ci(est_time=10, suite="base-a-test-cpu")
class TestSchedulePolicyWaitingQueueMatching(unittest.TestCase):
def make_policy(self, policy, supports_fast_match_prefix):
schedule_policy = object.__new__(SchedulePolicy)
schedule_policy.policy = policy
schedule_policy.tree_cache = SimpleNamespace(
supports_fast_match_prefix=lambda: supports_fast_match_prefix
)
return schedule_policy
def test_lpm_queue_limit_respects_fast_matching_capability(self):
policy = self.make_policy(CacheAwarePolicy.LPM, False)
self.assertTrue(policy.waiting_queue_prefix_matched([None] * 128))
self.assertFalse(policy.waiting_queue_prefix_matched([None] * 129))
policy.tree_cache = SimpleNamespace(supports_fast_match_prefix=lambda: True)
self.assertTrue(policy.waiting_queue_prefix_matched([None] * 129))
class TestSchedulerLoadInquirer(unittest.TestCase):
def make_inquirer(
self,
waiting_queue_prefix_matched=True,
storage_span=None,
disaggregation_mode=DisaggregationMode.NULL,
):
waiting_req = SimpleNamespace(
seqlen=100,
num_matched_prefix_tokens=20,
cache_request_handle=CacheRequestHandle("waiting", 0),
)
chunked_req = SimpleNamespace(seqlen=50, prefix_len=10)
return SchedulerLoadInquirer(
disaggregation_mode=disaggregation_mode,
server_args=None,
max_total_num_tokens=1000,
max_running_requests=10,
pool_stats_observer=None,
tp_worker=None,
token_to_kv_pool_allocator=None,
spec_algorithm=None,
get_running_batch=None,
get_waiting_queue=lambda: [waiting_req],
waiting_queue_prefix_matched=lambda: waiting_queue_prefix_matched,
get_chunked_req=lambda: chunked_req,
get_recent_cache_hit_rate=lambda: 0.75,
get_stats=None,
get_disagg_prefill_bootstrap_queue=None,
get_disagg_prefill_inflight_queue=None,
get_disagg_decode_prealloc_queue=None,
get_disagg_decode_transfer_queue=None,
get_spec_total_num_accept_tokens=None,
get_spec_total_num_forward_ct=None,
get_total_prefill_uncached_tokens=None,
get_total_prefill_busy_us=None,
get_decode_moment_totals=None,
get_waiting_storage_span=Mock(return_value=storage_span),
)
def test_waiting_tokens_are_estimated_when_prefix_matching_is_skipped(self):
inquirer = self.make_inquirer(waiting_queue_prefix_matched=False)
self.assertEqual(
SchedulerLoadInquirer.get_num_waiting_uncached_tokens(inquirer),
65,
)
def test_waiting_tokens_use_exact_match_when_prefix_matching_is_done(self):
inquirer = self.make_inquirer(waiting_queue_prefix_matched=True)
self.assertEqual(
SchedulerLoadInquirer.get_num_waiting_uncached_tokens(inquirer),
120,
)
def test_storage_exclusion_is_clipped_to_uncached_prompt(self):
for span, expected in (
(None, 120),
((0, 10), 120),
((0, 40), 100),
((20, 80), 60),
((60, 120), 80),
((20, 120), 40),
((20, 20), 120),
((100, 120), 120),
):
with self.subTest(span=span):
inquirer = self.make_inquirer(storage_span=span)
self.assertEqual(inquirer.get_num_waiting_uncached_tokens(), expected)
inquirer.get_waiting_storage_span.assert_called_once_with(
CacheRequestHandle("waiting", 0)
)
def test_storage_exclusion_requires_per_request_matching(self):
for prefix_matched, expected in ((True, 60), (False, 65)):
with self.subTest(prefix_matched=prefix_matched):
inquirer = self.make_inquirer(
waiting_queue_prefix_matched=prefix_matched,
storage_span=(20, 80),
)
self.assertEqual(inquirer.get_num_waiting_uncached_tokens(), expected)
if not prefix_matched:
inquirer.get_waiting_storage_span.assert_not_called()
def test_storage_exclusion_does_not_change_decode(self):
for prefix_matched in (False, True):
with self.subTest(prefix_matched=prefix_matched):
inquirer = self.make_inquirer(
waiting_queue_prefix_matched=prefix_matched,
storage_span=(20, 100),
disaggregation_mode=DisaggregationMode.DECODE,
)
self.assertEqual(inquirer.get_num_waiting_uncached_tokens(), 0)
inquirer.get_waiting_storage_span.assert_not_called()
def test_storage_exclusion_preserves_the_historical_hit_rate_estimate(self):
for span in (None, (20, 50), (20, 80), (80, 100), (0, 120)):
with self.subTest(span=span):
inquirer = self.make_inquirer(
waiting_queue_prefix_matched=False,
storage_span=span,
)
self.assertEqual(inquirer.get_num_waiting_uncached_tokens(), 65)
inquirer.get_waiting_storage_span.assert_not_called()
def test_historical_estimate_already_includes_storage_hits(self):
requests = [
SimpleNamespace(seqlen=100, cache_request_handle=CacheRequestHandle(rid, 0))
for rid in ("storage_hit", "cold")
]
get_storage_span = Mock(
side_effect=lambda handle: (0, 80) if handle.rid == "storage_hit" else None
)
inquirer = replace(
self.make_inquirer(waiting_queue_prefix_matched=False),
get_waiting_queue=lambda: requests,
get_chunked_req=lambda: None,
get_recent_cache_hit_rate=lambda: 0.4,
get_waiting_storage_span=get_storage_span,
)
# One 80-token storage hit and one cold prompt need 20 + 100 tokens.
self.assertEqual(inquirer.get_num_waiting_uncached_tokens(), 120)
get_storage_span.assert_not_called()
def test_storage_exclusion_without_a_provider_keeps_the_original_estimate(self):
inquirer = replace(
self.make_inquirer(),
get_waiting_storage_span=None,
)
self.assertEqual(inquirer.get_num_waiting_uncached_tokens(), 120)
def test_scheduler_wires_storage_provider(self):
from sglang.srt.managers.scheduler import Scheduler
req = self.make_inquirer().get_waiting_queue()[0]
get_storage_span = Mock(return_value=(20, 80))
scheduler = SimpleNamespace(
disaggregation_mode=DisaggregationMode.PREFILL,
server_args=None,
max_total_num_tokens=1000,
kv_shard_widening=1,
max_running_requests=10,
pool_stats_observer=None,
tp_worker=None,
token_to_kv_pool_allocator=None,
spec_algorithm=None,
tree_cache=SimpleNamespace(waiting_storage_span=get_storage_span),
policy=SimpleNamespace(waiting_queue_prefix_matched=lambda queue: True),
metrics_reporter=SimpleNamespace(recent_cache_hit_rate=0.75),
waiting_queue=[req],
chunked_req=None,
)
Scheduler.init_load_inquirer(scheduler)
inquirer = scheduler.load_inquirer
self.assertIs(inquirer.get_waiting_storage_span, get_storage_span)
self.assertEqual(inquirer.get_num_waiting_uncached_tokens(), 20)
get_storage_span.assert_called_once_with(req.cache_request_handle)
if __name__ == "__main__":
unittest.main()
import unittest
from dataclasses import replace
from types import SimpleNamespace
from unittest.mock import Mock
from sglang.srt.disaggregation.utils import DisaggregationMode
from sglang.srt.managers.schedule_policy import (
CacheAwarePolicy,
SchedulePolicy,
)
from sglang.srt.managers.scheduler_components.load_inquirer import (
SchedulerLoadInquirer,
)
from sglang.srt.mem_cache.base_prefix_cache import CacheRequestHandle
from sglang.test.ci.ci_register import register_cpu_ci
register_cpu_ci(est_time=10, suite="base-a-test-cpu")
class TestSchedulePolicyWaitingQueueMatching(unittest.TestCase):
def make_policy(self, policy, supports_fast_match_prefix):
schedule_policy = object.__new__(SchedulePolicy)
schedule_policy.policy = policy
schedule_policy.tree_cache = SimpleNamespace(
supports_fast_match_prefix=lambda: supports_fast_match_prefix
)
return schedule_policy
def test_lpm_queue_limit_respects_fast_matching_capability(self):
policy = self.make_policy(CacheAwarePolicy.LPM, False)
self.assertTrue(policy.waiting_queue_prefix_matched([None] * 128))
self.assertFalse(policy.waiting_queue_prefix_matched([None] * 129))
policy.tree_cache = SimpleNamespace(supports_fast_match_prefix=lambda: True)
self.assertTrue(policy.waiting_queue_prefix_matched([None] * 129))
class TestSchedulerLoadInquirer(unittest.TestCase):
def make_inquirer(
self,
waiting_queue_prefix_matched=True,
storage_span=None,
disaggregation_mode=DisaggregationMode.NULL,
):
waiting_req = SimpleNamespace(
seqlen=100,
num_matched_prefix_tokens=20,
cache_request_handle=CacheRequestHandle("waiting", 0),
)
chunked_req = SimpleNamespace(seqlen=50, prefix_len=10)
return SchedulerLoadInquirer(
disaggregation_mode=disaggregation_mode,
server_args=None,
max_total_num_tokens=1000,
max_running_requests=10,
pool_stats_observer=None,
tp_worker=None,
token_to_kv_pool_allocator=None,
spec_algorithm=None,
get_running_batch=None,
get_waiting_queue=lambda: [waiting_req],
waiting_queue_prefix_matched=lambda: waiting_queue_prefix_matched,
get_chunked_req=lambda: chunked_req,
get_grammar_queue=lambda: [],
get_recent_cache_hit_rate=lambda: 0.75,
get_stats=None,
get_disagg_prefill_bootstrap_queue=None,
get_disagg_prefill_inflight_queue=None,
get_disagg_decode_prealloc_queue=None,
get_disagg_decode_transfer_queue=None,
get_spec_total_num_accept_tokens=None,
get_spec_total_num_forward_ct=None,
get_total_prefill_uncached_tokens=None,
get_total_prefill_busy_us=None,
get_decode_moment_totals=None,
get_waiting_storage_span=Mock(return_value=storage_span),
)
def test_waiting_tokens_are_estimated_when_prefix_matching_is_skipped(self):
inquirer = self.make_inquirer(waiting_queue_prefix_matched=False)
self.assertEqual(
SchedulerLoadInquirer.get_num_waiting_uncached_tokens(inquirer),
65,
)
def test_waiting_tokens_use_exact_match_when_prefix_matching_is_done(self):
inquirer = self.make_inquirer(waiting_queue_prefix_matched=True)
self.assertEqual(
SchedulerLoadInquirer.get_num_waiting_uncached_tokens(inquirer),
120,
)
def test_storage_exclusion_is_clipped_to_uncached_prompt(self):
for span, expected in (
(None, 120),
((0, 10), 120),
((0, 40), 100),
((20, 80), 60),
((60, 120), 80),
((20, 120), 40),
((20, 20), 120),
((100, 120), 120),
):
with self.subTest(span=span):
inquirer = self.make_inquirer(storage_span=span)
self.assertEqual(inquirer.get_num_waiting_uncached_tokens(), expected)
inquirer.get_waiting_storage_span.assert_called_once_with(
CacheRequestHandle("waiting", 0)
)
def test_storage_exclusion_requires_per_request_matching(self):
for prefix_matched, expected in ((True, 60), (False, 65)):
with self.subTest(prefix_matched=prefix_matched):
inquirer = self.make_inquirer(
waiting_queue_prefix_matched=prefix_matched,
storage_span=(20, 80),
)
self.assertEqual(inquirer.get_num_waiting_uncached_tokens(), expected)
if not prefix_matched:
inquirer.get_waiting_storage_span.assert_not_called()
def test_storage_exclusion_does_not_change_decode(self):
for prefix_matched in (False, True):
with self.subTest(prefix_matched=prefix_matched):
inquirer = self.make_inquirer(
waiting_queue_prefix_matched=prefix_matched,
storage_span=(20, 100),
disaggregation_mode=DisaggregationMode.DECODE,
)
self.assertEqual(inquirer.get_num_waiting_uncached_tokens(), 0)
inquirer.get_waiting_storage_span.assert_not_called()
def test_storage_exclusion_preserves_the_historical_hit_rate_estimate(self):
for span in (None, (20, 50), (20, 80), (80, 100), (0, 120)):
with self.subTest(span=span):
inquirer = self.make_inquirer(
waiting_queue_prefix_matched=False,
storage_span=span,
)
self.assertEqual(inquirer.get_num_waiting_uncached_tokens(), 65)
inquirer.get_waiting_storage_span.assert_not_called()
def test_historical_estimate_already_includes_storage_hits(self):
requests = [
SimpleNamespace(seqlen=100, cache_request_handle=CacheRequestHandle(rid, 0))
for rid in ("storage_hit", "cold")
]
get_storage_span = Mock(
side_effect=lambda handle: (0, 80) if handle.rid == "storage_hit" else None
)
inquirer = replace(
self.make_inquirer(waiting_queue_prefix_matched=False),
get_waiting_queue=lambda: requests,
get_chunked_req=lambda: None,
get_recent_cache_hit_rate=lambda: 0.4,
get_waiting_storage_span=get_storage_span,
)
# One 80-token storage hit and one cold prompt need 20 + 100 tokens.
self.assertEqual(inquirer.get_num_waiting_uncached_tokens(), 120)
get_storage_span.assert_not_called()
def test_storage_exclusion_without_a_provider_keeps_the_original_estimate(self):
inquirer = replace(
self.make_inquirer(),
get_waiting_storage_span=None,
)
self.assertEqual(inquirer.get_num_waiting_uncached_tokens(), 120)
def test_scheduler_wires_storage_provider(self):
from sglang.srt.managers.scheduler import Scheduler
req = self.make_inquirer().get_waiting_queue()[0]
get_storage_span = Mock(return_value=(20, 80))
scheduler = SimpleNamespace(
disaggregation_mode=DisaggregationMode.PREFILL,
server_args=None,
max_total_num_tokens=1000,
kv_shard_widening=1,
max_running_requests=10,
pool_stats_observer=None,
tp_worker=None,
token_to_kv_pool_allocator=None,
spec_algorithm=None,
tree_cache=SimpleNamespace(waiting_storage_span=get_storage_span),
policy=SimpleNamespace(waiting_queue_prefix_matched=lambda queue: True),
metrics_reporter=SimpleNamespace(recent_cache_hit_rate=0.75),
waiting_queue=[req],
chunked_req=None,
)
Scheduler.init_load_inquirer(scheduler)
inquirer = scheduler.load_inquirer
self.assertIs(inquirer.get_waiting_storage_span, get_storage_span)
self.assertEqual(inquirer.get_num_waiting_uncached_tokens(), 20)
get_storage_span.assert_called_once_with(req.cache_request_handle)
if __name__ == "__main__":
unittest.main()
@@ -1,16 +1,21 @@
"""on_idle's stalled-path load publish is wall-clock bounded.
"""When the scheduler publishes its load snapshot.
A non-storage no-batch stall spins on_idle without sleeping, so the gate must
cap the O(queue) get_loads for both the DP-balancing writer and the load
socket. CPU-only: builds a bare Scheduler with mocked collaborators, like
test_scheduler_flush_cache.
on_idle's stalled-path publish is wall-clock bounded: a non-storage no-batch
stall spins on_idle without sleeping, so the gate must cap the O(queue)
get_loads for both the DP-balancing writer and the load socket. Ingesting new
requests publishes once they are queued. CPU-only: builds a bare Scheduler with
mocked collaborators, like test_scheduler_flush_cache.
"""
import unittest
from unittest.mock import MagicMock, patch
from unittest.mock import MagicMock, call, patch
from sglang.test.ci.ci_register import register_cpu_ci
from sglang.test.test_utils import CustomTestCase, maybe_stub_sgl_kernel
from sglang.test.test_utils import (
CustomTestCase,
maybe_stub_sgl_kernel,
published_topology,
)
maybe_stub_sgl_kernel()
@@ -52,5 +57,26 @@ class TestOnIdleStallPublish(CustomTestCase):
self.assertEqual(s.publish_load_snapshot.call_count, 2)
class TestIngestPublish(CustomTestCase):
def test_a_snapshot_follows_newly_queued_requests(self):
"""A rank publishes its load after each forward, before it reads the
requests sent meanwhile; without a publish once they are queued, DP
balancing resets its estimate to that stale load and sends it more."""
s = Scheduler.__new__(Scheduler)
s._deferred_input_requests = []
s._poll_timeout_aborts = MagicMock(return_value=[])
s.request_receiver = MagicMock()
s.request_receiver.recv_requests.return_value = ["req"]
s.metrics_reporter = MagicMock()
steps = MagicMock()
s.process_input_requests = steps.queue
s.publish_load_snapshot = steps.publish
with published_topology():
s.ingest_requests()
self.assertEqual(
steps.mock_calls, [call.queue(["req"]), call.publish(force=True)]
)
if __name__ == "__main__":
unittest.main()
@@ -0,0 +1,229 @@
"""CPU regressions for pipelined HiCache storage IO ordering and shutdown."""
import threading
import time
import unittest
from contextlib import nullcontext
from queue import Queue
from types import SimpleNamespace
from unittest.mock import Mock, patch
import torch
import sglang.srt.managers.cache_controller as cache_controller_module
from sglang.srt.managers.cache_controller import (
HiCacheController,
StorageOperation,
)
from sglang.srt.mem_cache.base_prefix_cache import CacheRequestHandle
from sglang.srt.mem_cache.hybrid_cache.hybrid_cache_controller import (
HybridCacheController,
PrefetchOperation,
)
from sglang.test.ci.ci_register import register_cpu_ci
register_cpu_ci(est_time=5, suite="base-a-test-cpu")
def _drain(queue):
items = []
while not queue.empty():
items.append(queue.get_nowait())
return items
def _wait_for_queue_size(queue, size, timeout=5):
deadline = time.monotonic() + timeout
while queue.qsize() < size and time.monotonic() < deadline:
time.sleep(0.005)
return queue.qsize() >= size
class TestPrefetchPipelineOrdering(unittest.TestCase):
def _controller(self, wait_for, peer_done):
controller = HybridCacheController.__new__(HybridCacheController)
controller.page_size = 1
controller.prefetch_buffer = Queue()
controller.prefetch_sync_queue = Queue()
controller.storage_stop_event = threading.Event()
controller.mem_pool_host = SimpleNamespace(
layout_lease=lambda: nullcontext()
)
controller.storage_backend = Mock()
controller.storage_backend.batch_get_v2.return_value = {}
for rid in ("a", "b"):
operation = PrefetchOperation(
CacheRequestHandle(rid, 0),
list(range(2)),
pool_transfers=[],
)
operation.hash_value = [f"{rid}-0", f"{rid}-1"]
operation.host_indices = torch.arange(2)
controller.prefetch_buffer.put(operation)
def page_get(operation, batch_hashes, host_indices, extra_info):
if operation.request_id == wait_for:
if not peer_done.wait(timeout=5):
raise TimeoutError("peer prefetch did not finish")
return len(batch_hashes)
controller.page_get_func = page_get
collect = controller._transfer_prefetch_and_collect_acks
def collect_and_signal(operation):
acks = collect(operation)
if operation.request_id != wait_for:
# This rank's other operation has now completed. The delayed
# peer may resume, producing the opposite worker finish order.
peer_done.set()
return acks
controller._transfer_prefetch_and_collect_acks = collect_and_signal
return controller
def test_two_tp_ranks_keep_operation_ack_sequences_aligned(self):
# Rank 0 finishes A before B; rank 1 finishes B before A. Without
# per-operation buffering their progress/sidecar ACKs enter collectives
# in opposite orders even though completed_req itself remains FIFO.
rank0_done = threading.Event()
rank1_done = threading.Event()
rank0 = self._controller(wait_for="b", peer_done=rank0_done)
rank1 = self._controller(wait_for="a", peer_done=rank1_done)
threads = [
threading.Thread(target=rank.prefetch_io_aux_func, daemon=True)
for rank in (rank0, rank1)
]
with (
patch.object(cache_controller_module, "_HC_PREFETCH_WORKERS", 2),
patch.object(cache_controller_module, "STORAGE_BATCH_SIZE", 1),
):
for thread in threads:
thread.start()
try:
self.assertTrue(_wait_for_queue_size(rank0.prefetch_sync_queue, 8))
self.assertTrue(_wait_for_queue_size(rank1.prefetch_sync_queue, 8))
finally:
rank0.storage_stop_event.set()
rank1.storage_stop_event.set()
rank0_done.set()
rank1_done.set()
for thread in threads:
thread.join(timeout=5)
for thread in threads:
self.assertFalse(thread.is_alive())
rank0_acks = _drain(rank0.prefetch_sync_queue)
rank1_acks = _drain(rank1.prefetch_sync_queue)
def ack_kind(ack):
if ack.completed_req:
return "completed_req"
if ack.completed_tokens is not None:
return "completed_tokens"
if ack.pool_hits is not None:
return "pool_hits"
raise AssertionError(f"unknown ACK: {ack}")
self.assertEqual([ack.rid for ack in rank0_acks], ["a"] * 4 + ["b"] * 4)
self.assertEqual(
[(ack.rid, ack_kind(ack)) for ack in rank0_acks],
[(ack.rid, ack_kind(ack)) for ack in rank1_acks],
)
for acks in (rank0_acks, rank1_acks):
self.assertEqual(
[ack_kind(ack) for ack in acks[:4]],
["completed_tokens", "completed_tokens", "pool_hits", "completed_req"],
)
self.assertEqual(
[ack.completed_tokens for ack in acks[:2]],
[1, 2],
)
class TestBackupPipeline(unittest.TestCase):
@staticmethod
def _controller(controller_type, is_storage_writer, backup):
controller = controller_type.__new__(controller_type)
controller.backup_queue = Queue()
controller.ack_backup_queue = Queue()
controller.storage_stop_event = threading.Event()
controller.storage_config = SimpleNamespace(
is_storage_writer=is_storage_writer
)
controller._page_backup_or_log = backup
return controller
def _run_one(self, controller, operation):
thread = threading.Thread(target=controller.backup_thread_func, daemon=True)
with patch.object(cache_controller_module, "_HC_BACKUP_WORKERS", 1):
controller.backup_queue.put(operation)
thread.start()
try:
ack = controller.ack_backup_queue.get(timeout=5)
finally:
controller.storage_stop_event.set()
thread.join(timeout=5)
self.assertFalse(thread.is_alive())
self.assertIs(ack, operation)
def test_non_writer_never_submits_backup_io(self):
backup = Mock()
controller = self._controller(HiCacheController, False, backup)
operation = StorageOperation(None, [])
self._run_one(controller, operation)
backup.assert_not_called()
def test_hybrid_non_writer_still_runs_rank_sharded_backup(self):
backup = Mock()
controller = self._controller(HybridCacheController, False, backup)
operation = StorageOperation(None, [])
self._run_one(controller, operation)
backup.assert_called_once_with(operation)
def test_stop_waits_for_every_backup_worker(self):
started = [threading.Event(), threading.Event()]
release = [threading.Event(), threading.Event()]
finished = [threading.Event(), threading.Event()]
def backup(operation):
index = operation.token_ids[0]
started[index].set()
if not release[index].wait(timeout=5):
raise TimeoutError(f"backup {index} was not released")
finished[index].set()
controller = self._controller(HiCacheController, True, backup)
for index in range(2):
controller.backup_queue.put(StorageOperation(None, [index]))
thread = threading.Thread(target=controller.backup_thread_func, daemon=True)
with patch.object(cache_controller_module, "_HC_BACKUP_WORKERS", 2):
thread.start()
try:
self.assertTrue(started[0].wait(timeout=5))
self.assertTrue(started[1].wait(timeout=5))
controller.storage_stop_event.set()
release[0].set()
# The coordinator may finish operation 0, but shutdown cannot
# return while operation 1 still uses the storage backend.
thread.join(timeout=0.1)
self.assertTrue(thread.is_alive())
self.assertFalse(finished[1].is_set())
release[1].set()
thread.join(timeout=5)
self.assertFalse(thread.is_alive())
self.assertTrue(finished[0].is_set())
self.assertTrue(finished[1].is_set())
finally:
controller.storage_stop_event.set()
for event in release:
event.set()
thread.join(timeout=5)
if __name__ == "__main__":
unittest.main()
@@ -1,4 +1,4 @@
"""Write-back eviction must wait for the checkpoint's KV producer."""
"""HiCache backups must wait for the checkpoint's KV producer."""
import unittest
from array import array
@@ -109,6 +109,72 @@ class TestHiCacheWriteBackOrdering(UnifiedRadixCacheFixtures, unittest.TestCase)
finally:
forward.synchronize()
def test_write_through_stashed_chunk_waits_for_its_producer(self):
# Prefill read-done lets the scheduler stash a chunk while its forward
# still writes the KV; the write-through backup must copy the final KV.
self.cfg = CacheConfig(
components=(ComponentType.FULL, ComponentType.SWA),
page_size=1,
sliding_window_size=4,
num_layers=2,
full_attention_layer_ids=(0,),
kv_size=64,
max_context_len=64,
head_num=1,
head_dim=64,
)
cache, allocator, req_pool = build_fixture(self.cfg)
self._init_hicache(cache, write_policy="write_through")
cache.write_through_threshold = 1
controller = cache.cache_controller
forward = torch.cuda.Stream()
controller.write_fence_stream = forward
req = self._request(cache, allocator, req_pool, 0, length=12)
kv = allocator.get_kvcache()
buffers = [
buf
for pool in (kv.full_kv_pool, kv.swa_kv_pool)
for buf in pool.k_buffer + pool.v_buffer
]
for buf in buffers:
buf.fill_(-1)
read_done = torch.cuda.Event()
torch.cuda.synchronize()
engine = controller.l2_transfer_engine
submit = engine.submit_device_to_host
snapshots = []
def record_copy(transfers):
completion = submit(transfers)
completion.finish_event.synchronize()
for transfer in transfers:
if transfer.host_indices.numel():
snapshots.append(
transfer.host_pool.get_data_page(
int(transfer.host_indices[0])
).clone()
)
return completion
with torch.cuda.stream(forward):
read_done.record()
torch.cuda._sleep(600_000_000)
for buf in buffers:
buf.fill_(42)
torch.cuda.current_stream().wait_event(read_done)
try:
with patch.object(engine, "submit_device_to_host", side_effect=record_copy):
cache.checkpoint(req, up_to=12)
cache.flush_pending_backups()
self.assertTrue(snapshots)
for snapshot in snapshots:
self.assertTrue(
torch.equal(snapshot, torch.full_like(snapshot, 42)),
f"expected 42, copied {snapshot.unique().tolist()}",
)
finally:
forward.synchronize()
if __name__ == "__main__":
unittest.main()
@@ -121,5 +121,87 @@ class TestMatchKvCacheForceMiss(unittest.TestCase):
self.assertEqual(forced_req.host_hit_length, 0)
class TestSwaGateRejectObservability(unittest.TestCase):
"""The gate gap is recorded on match and emitted once per prefill admission."""
@staticmethod
def _req_with_collector():
class _StubCollector:
def __init__(self):
self.rejected = []
def observe_swa_gate_reject_admission(self, tokens):
self.rejected.append(tokens)
req = _StubReq(list(range(200)))
req.time_stats = SimpleNamespace(
metrics_collector=_StubCollector(), enable_metrics=True
)
return req
@staticmethod
def _tree_cache(device_prefix_len, host_hit_length, full_kv_hit_length):
class _StubTreeCache:
def swa_reprefill_tail_tokens(self, req=None):
return 0
def match_prefix(self, params):
return MatchResult(
device_prefix_len=device_prefix_len,
last_device_node=None,
last_host_node=None,
best_match_node=None,
host_hit_length=host_hit_length,
full_kv_hit_length=full_kv_hit_length,
swa_branching_seqlen=128,
)
return _StubTreeCache()
def test_match_records_gap_but_does_not_emit(self):
req = self._req_with_collector()
tree = self._tree_cache(100, 0, 164)
# Waiting-queue probes and the final init_next_round_input both match.
match_kv_cache(req, tree)
match_kv_cache(req, tree)
self.assertEqual(req.full_kv_hit_length, 164)
self.assertEqual(req.swa_gate_reject_tokens, 64)
self.assertEqual(req.time_stats.metrics_collector.rejected, [])
def test_match_time_gap_accounts_for_host_hit(self):
req = self._req_with_collector()
match_kv_cache(req, self._tree_cache(100, 64, 164))
self.assertEqual(req.swa_gate_reject_tokens, 0)
def test_admission_emits_once_and_counts_retries(self):
from sglang.srt.managers.schedule_policy import (
observe_swa_gate_reject_on_admission,
)
req = self._req_with_collector()
req.full_kv_hit_length = 164
self.assertEqual(observe_swa_gate_reject_on_admission(req, 100), 64)
self.assertEqual(req.time_stats.metrics_collector.rejected, [64])
# A retracted request that is admitted again is re-prefilled: count again.
self.assertEqual(observe_swa_gate_reject_on_admission(req, 100), 64)
self.assertEqual(req.time_stats.metrics_collector.rejected, [64, 64])
def test_host_load_back_closes_the_gap(self):
from sglang.srt.managers.schedule_policy import (
observe_swa_gate_reject_on_admission,
)
req = self._req_with_collector()
req.full_kv_hit_length = 164
# 64 host tokens were loaded back before admission: final prefix is 164.
self.assertEqual(observe_swa_gate_reject_on_admission(req, 164), 0)
self.assertEqual(req.time_stats.metrics_collector.rejected, [])
if __name__ == "__main__":
unittest.main()
@@ -0,0 +1,128 @@
"""Exercise SWA eviction frontiers without accelerator execution or model weights."""
import ast
import unittest
from pathlib import Path
from types import SimpleNamespace
from unittest.mock import Mock
from sglang.srt.mem_cache import common as cache_common
from sglang.test.ci.ci_register import register_cpu_ci
from sglang.test.test_utils import CustomTestCase
register_cpu_ci(est_time=5, suite="base-a-test-cpu")
class ReplayRetentionTest(CustomTestCase):
@classmethod
def setUpClass(cls):
cls.source = Path(cache_common.__file__).parent
cls.free_window = staticmethod(cache_common.free_swa_out_of_window_slots)
def frontier(
self,
length,
window=128,
page=256,
replay=128,
sharing=True,
retain_floor=None,
previous=0,
):
kv = SimpleNamespace(
holds_kv=True,
cache_protected_len=0,
req_pool_idx=0,
swa_dead_lo=lambda _: 0,
get_evicted_seqlen=lambda _: previous,
)
kv.set_evicted_seqlen = lambda _, value: setattr(kv, "evicted", value)
table = Mock()
table.__getitem__ = Mock(return_value=object())
allocator = SimpleNamespace(free_swa_segment=Mock())
self.free_window(
SimpleNamespace(kv=kv),
length - 1,
sliding_window_size=window,
page_size=page,
req_to_token_pool=SimpleNamespace(req_to_token=table),
token_to_kv_pool_allocator=allocator,
supports_prefix_sharing=sharing,
retain_floor=retain_floor,
reprefill_tail_tokens=replay,
)
return kv.evicted
def test_first_replay_keeps_window_before_trimmed_match(self):
for length in (32576, 32448, 269913, 1029986):
with self.subTest(length=length):
boundary = (length - 128) // 256 * 256
self.assertLessEqual(self.frontier(length), boundary - 128)
def test_all_page_alignments_preserve_replay_history(self):
for page, window, replay in (
(256, 128, 128),
(64, 128, 128),
(16, 8, 8),
(1, 4, 4),
):
for remainder in range(page):
length = 64 * page + 2 * window + remainder
with self.subTest(page=page, window=window, remainder=remainder):
boundary = (length - replay) // page * page
self.assertLessEqual(
self.frontier(length, window, page, replay), boundary - window
)
def test_no_replay_keeps_existing_eviction_policy(self):
for length in range(32000, 32512):
expected = (length - 1 - 256) // 256 * 256
self.assertEqual(self.frontier(length, replay=0), expected)
def test_256_token_pages_retain_at_most_one_extra_page(self):
for length in range(32000, 32512):
before = self.frontier(length, replay=0)
after = self.frontier(length)
self.assertGreaterEqual(before - after, 0)
self.assertLessEqual(before - after, 256)
def test_bad_alignment_exposes_original_frontier_mismatch(self):
self.assertEqual(self.frontier(32576, replay=0), 32256)
self.assertEqual(self.frontier(32576), 32000)
self.assertEqual(self.frontier(32448), 32000)
def test_disabled_prefix_sharing_does_not_retain_replay_history(self):
for replay in (0, 128):
self.assertEqual(self.frontier(32576, replay=replay, sharing=False), 32256)
def test_checkpoint_floor_and_eviction_cursor_still_apply(self):
self.assertEqual(self.frontier(32576, retain_floor=30000), 29952)
# Previously freed slots cannot be recovered by changing a frontier.
self.assertEqual(self.frontier(32576, previous=32256), 32256)
def test_scheduler_and_checkpoint_paths_pass_the_replay_guard(self):
for relative, expected in (
("base_prefix_cache.py", "self.swa_reprefill_tail_tokens(req)"),
(
"unified_cache/components/swa.py",
"self.cache.swa_reprefill_tail_tokens(req)",
),
):
tree = ast.parse((self.source / relative).read_text())
calls = [
node
for node in ast.walk(tree)
if isinstance(node, ast.Call)
and ast.unparse(node.func) == "free_swa_out_of_window_slots"
]
self.assertEqual(len(calls), 1)
keyword = next(
(kw for kw in calls[0].keywords if kw.arg == "reprefill_tail_tokens"),
None,
)
self.assertIsNotNone(keyword, relative)
self.assertEqual(ast.unparse(keyword.value), expected)
if __name__ == "__main__":
unittest.main()

Some files were not shown because too many files have changed in this diff Show More