minke.yu 85f4f9576c scheduler: add opt-in per-pass prefill stage timing (SGLANG_BATCH_STAGE_TIMING)
Motivation: on the PD prefill node the wall-clock budget is dominated by
get_next_batch_to_run (~50% of the hour on 2026-11-02 16:00-17:00), and
forward_overlap=partial is only 21-87 s/h, i.e. it does not overlap the GPU
forward. We know the total but not which segment inside it costs: radix
match, PrefillAdder admission, the HiCache event collective, the bootstrap
all-reduce, or the DP metadata gather.

Measurement design (one sample per scheduling pass, including passes that
produce no batch; see handoff/prefill-batch-stage-timing-20261103.md v2/v3):

  L1  7 per-pass histograms:
      sglang:batch_prefill_{get_next,match,adder,hicache_events,
      bootstrap,mlp_sync,other}_seconds
      other = total - (match + adder + hicache + bootstrap + mlp_sync),
      clamped to >= 0. Nested stages are not double counted.

  L3  5 per-pass counters:
      candidates / skipped_prefetch / admitted / resumed_chunk / new_chunk

  L2  a dedicated per-pass span scheduler.prefill.pass (not the batch span),
      emitted from end_pass() on every pass. Its sampling decision is a pure
      function of mlp_sync_seq, so a sampled gather round is sampled on all
      DP ranks. Idle ranks and ranks without a local batch still emit, which
      is exactly what the cross-rank skew analysis needs. No links.

Hooks are installed in both scheduling loops:
  * get_next_disagg_prefill_batch_to_run() for --disaggregation-mode prefill
    (hicache / bootstrap / mlp_sync segments), and
  * get_next_batch_to_run() for standalone prefill (hicache / mlp_sync),
so the instrumentation can be validated on ds41-prefill-dp4 without a decode
peer. match/adder/counters live in the shared get_new_batch_prefill path.

Semantics:
  * match times only init_next_round_input; check_prefetch_progress counts as
    skipped_prefetch, never as match.
  * bootstrap and hicache_events are cross-rank sync points measured at the
    outer call, not attributed to local CPU work.
  * dp_gather_ms comes from the existing MLPSyncBatchInfo.sync_wait_seconds;
    sglang:dp_attention_sync_wait_seconds is reused, no duplicate metric.
    The value is cleared before each outer call and mlp_sync_seq only
    increments when a gather actually ran, so stale values cannot leak.
  * resumed vs new chunked requests are captured separately: had_chunked is
    recorded at pass start (before process_prefill_chunk/admission can
    consume the last chunk) and new_chunk from adder.new_chunked_req. This
    fixes both the first-chunk and last-chunk mislabels.

Opt-in:
  * SGLANG_BATCH_STAGE_TIMING=1 (default 0);
    SGLANG_BATCH_PASS_SPAN=1 / SGLANG_BATCH_PASS_SPAN_SAMPLE=0.05.
  * When off every hook is one module-level bool check; no logging, no CUDA
    calls, no new collectives.

Tests: test/registered/unit/observability/test_batch_telemetry_stage_timing.py
covers first/middle/last chunk labelling, all-prefetch-skip passes without a
batch, stale-gather clearing, rank-deterministic pass-span sampling, and the
disabled no-op path. py_compile passes on all touched files.
2026-11-04 14:53:29 +08:00
2026-10-07 01:55:01 -07:00
2026-03-15 21:13:45 +08:00
…

SGLang: Fast inference for LLMs and multimodal models

SGLang

PyPI version License: Apache 2.0 PyPI downloads per month

Docs | Cookbook | Website | Blog | Slack

SGLang is an open-source inference framework for large language, vision-language, and diffusion models, optimized for agentic workloads, large-scale serving, and RL rollouts. SGLang Diffusion is its built-in image and video generation engine, included in this repository and the sglang Python package.

👋 Get started below, or meet the community at SGLang Events, including meetups, developer meetings, workshops, and office hours.

SGLang Summit 2026: Meet the builders of open AI
Our flagship annual gathering · November 12–13 · San Francisco

Go deep on inference, training, and AI hardware through technical talks, live demos, and hands-on workshops. Bring your deployment challenges and meet the engineers and researchers building the stack.

Get Started

Pull the Docker image, which includes SGLang and its dependencies:

docker pull lmsysorg/sglang:latest

Alternatively, install SGLang in an activated Python environment with uv:

uv pip install --prerelease=allow sglang

Next, launch your model:

  • Quickstart: Run your first model and send a request.
  • Cookbook: Choose your model and hardware to get a ready-to-run launch command.

Supported Hardware

SGLang supports a wide range of GPUs, TPUs, NPUs, CPUs, and Apple Silicon platforms.

Platform Representative hardware
NVIDIA B200/B300/GB200/GB300; H100/H200/H800/H20; A100; RTX 30/40/50 series, RTX 6000 Ada / PRO 6000; DGX Spark, Jetson Orin
AMD Instinct MI300X, MI325X, MI350X, MI355X
Google TPU v6e, v7; SGLang-TorchTPU / SGL-JAX
Intel (GPU / CPU) Arc / Arc Pro B-Series GPUs, Xeon CPUs
Apple Silicon Macs via Metal / MLX
Huawei Ascend A2, A3, 950PR/DT NPUs
Moore Threads MTT S5000 GPUs

Integrations in progress: AWS Trainium, Alibaba T-Head PPU, Cambricon MLU, Qualcomm QAIC, MetaX, Hygon HCU/DCU, Iluvatar CoreX, and more.

SGLang-TorchTPU brings the SGLang serving engine to Google TPUs through PyTorch, torch_tpu, and Pallas kernels. It supports LLM and vision-language serving on TPU7x (Ironwood), including tensor and expert parallelism across hosts. SGL-JAX provides the JAX-based TPU serving path.

See the Cookbook and platform guides for model compatibility and setup.

SGL Ecosystem

Area Projects Purpose
Diffusion SGLang Diffusion Built into SGLang for image and video generation with diffusion models.
Audio SGLang Omni Audio model serving for text-to-speech (TTS) and automatic speech recognition (ASR).
Speculative Decoding SpecForge Train draft models for speculative decoding and deploy them with SGLang.
RL and Post-Training Miles, slime, AReaL, Tunix, verl Training frameworks that integrate SGLang for rollout generation.
Deployment and Orchestration SMG, RBG, llm-d, Ray Serve, NVIDIA Dynamo Deploy and scale SGLang inference services with routing, load balancing, and cluster orchestration.
Education Mini-SGLang, zero-to-sglang, DeepLearning.AI course Learn inference engine design and efficient text and image generation through code and hands-on courses.

Development and Contributing

Contributions are welcome, from bug fixes and documentation to model support and performance improvements.

Development setup

Start from the lmsysorg/sglang:dev Docker image, which provides development tools and most dependencies. Clone or mount your SGLang checkout inside the container, then install it in editable mode from the repository root so tests use your local Python changes:

pip install -e "python"

In an activated virtual environment, you can use uv pip install --prerelease=allow -e "python" instead. See the development guide for container setup and testing.

Contribute

  1. Fork the repository and create a branch for your changes. For larger changes, discuss your proposal in a GitHub issue or on Slack.
  2. Make your changes, run the relevant tests, and add regression coverage for fixes or new behavior. Run pre-commit run --all-files before submitting.
  3. Open a pull request describing the change and how you tested it. Include benchmarks or accuracy evaluations when relevant.

See the contributor guide for formatting, testing, and pull request instructions. Documentation contributors can start with the docs guide.

Community and Sponsorship

SGLang is hosted by LMSYS, a non-profit open-source organization.

  • Community discussions: Join Slack for technical questions and development discussions.
  • Events: Find meetups, workshops, and office hours on SGLang Events.
  • Updates: Follow X and LinkedIn for project updates, and the LMSYS Blog for release announcements and technical articles.
  • Project resources: Explore the documentation, Cookbook, roadmap, release notes, issue tracker, and contributor guide.
  • Contact Us: For enterprise adoption and deployment, technical consulting, sponsorship, or partnership inquiries, please contact sglang@lmsys.org.
  • Contributor sponsorship: Long-term active SGLang contributors are eligible for coding agent sponsorship, including Cursor, Claude Code, or OpenAI Codex. To apply, email sglang@lmsys.org with links to your key commits or pull requests.

Trusted by Industry and Research

SGLang serves production workloads across AI labs, cloud platforms, enterprises, and universities.

Organizations adopting SGLang

Acknowledgment

We learned the design and reused code from the following projects: Guidance, vLLM, LightLLM, FlashInfer, Outlines, and LMQL.

S
Description
No description provided
Readme Apache-2.0
596 MiB
Languages
Python 83.7%
Rust 9.5%
Cuda 4.1%
C++ 1.6%
Shell 0.4%
Other 0.3%