Files
sglang/python/sglang
minke.yu 23053730fc observability: count SWA gate rejections per prefill admission
Review follow-up. match_kv_cache also runs on waiting-queue probes
(calc_priority), PD decode _match_prefix_and_lock and HiCache rematches.
Emitting the counter there counted the same gap once per scheduling round
and attributed "re-prefilled" tokens to decode engines.

- match_kv_cache now only records req.full_kv_hit_length (plus the match-time
  gap); it no longer emits.
- the metric is emitted once in PrefillAdder._commit_prefill_admission, where
  the accepted prefix is final after any host load-back.
- rename the request counter to swa_gate_reject_admissions_total and document
  that a retracted and re-admitted request is counted again (it is re-prefilled).
- pre-create zero-valued children so gate-off deployments expose the series
  before the first rejection.
- tests: repeated match does not emit, admission emits once, a retry counts
  again, and host load-back closes the gap.

Not runtime verified (no GPU run on this build).
2026-11-06 13:18:07 +08:00
..
…

Code Structure

Folders

  • benchmark/: Benchmark implementations and dataset utilities.
  • cli/: Command-line interface commands and entrypoints.
  • kernels/: Kernel interfaces, implementations, selection, and debugging utilities shared by the runtimes.
  • lang/: Deprecated language frontend that is no longer actively maintained.
  • multimodal_gen/: Core runtime for image, video, and audio generation models, most of which are diffusion models.
  • srt/: Core runtime for autoregressive language models. (SRT = SGLang Runtime.)
  • test/: Shared test and evaluation utilities.

Files

  • README.md: This package structure overview.
  • __init__.py: Package initialization and public Python APIs.
  • bench_one_batch_server.py: Deprecated wrapper for the server-based one-batch benchmark.
  • bench_serving.py: Deprecated wrapper for the online serving benchmark.
  • check_env.py: Environment and dependency diagnostics.
  • compile_deep_gemm.py: DeepGEMM kernel precompilation entrypoint.
  • launch_server.py: Compatibility entrypoint for launching an inference server.
  • profiler.py: Client entrypoint for collecting server profiling traces.
  • utils.py: Common package utilities.
  • version.py: Public package version resolution.