The prefill shared-read-done fastpath lets the schedule stream run ahead of the forward that writes a stashed chunk's KV instead of waiting for the whole forward. It was disabled whenever hierarchical cache or a storage backend was enabled, because host and storage offload copies used to be ordered only behind the schedule stream and could have read unfinished KV. The cache controller now orders its device-to-host copies behind the forward stream through write_fence_stream, and its load-back H2D copies through load_fence_stream, so that reason no longer holds when every KV offload copy carries that fence. The scheduler records on the model runner whether that is the case: hierarchical cache with a cache controller and no decode offload manager, which has no such fence. The publish helper keeps the coarse whole-forward barrier for LMCache and for unfenced offload, and publishes read-done otherwise. With the coarse barrier, a synchronous page-index copy issued while preparing the next batch inherited the wait on the entire in-flight forward, leaving the GPU idle between batches. With read-done published, that copy only waits for the shared-table reads to finish; KV ordering is unchanged because the backup copies still wait on the forward stream. Tests: the CPU runner test covers the fenced hierarchical-cache and storage cases publishing, and LMCache keeping the barrier even when hierarchical cache is fenced. The GPU ordering test checks that a write-through backup of a chunk stashed before its forward finished still copies the final KV. Co-authored-by: ZYHowell <ZYHowell@users.noreply.github.com> (cherry picked from commit bb1cb2dded6fb9352d98076280342b10684e061b)
SGLang: Fast inference for LLMs and multimodal models
Docs | Cookbook | Website | Blog | Slack
SGLang is an open-source inference framework for large language, vision-language, and diffusion models, optimized for agentic workloads, large-scale serving, and RL rollouts. SGLang Diffusion is its built-in image and video generation engine, included in this repository and the sglang Python package.
👋 Get started below, or meet the community at SGLang Events, including meetups, developer meetings, workshops, and office hours.
SGLang Summit 2026: Meet the builders of open AI
Our flagship annual gathering · November 12–13 · San FranciscoGo deep on inference, training, and AI hardware through technical talks, live demos, and hands-on workshops. Bring your deployment challenges and meet the engineers and researchers building the stack.
Get Started
Pull the Docker image, which includes SGLang and its dependencies:
docker pull lmsysorg/sglang:latest
Alternatively, install SGLang in an activated Python environment with uv:
uv pip install --prerelease=allow sglang
Next, launch your model:
- Quickstart: Run your first model and send a request.
- Cookbook: Choose your model and hardware to get a ready-to-run launch command.
Supported Hardware
SGLang supports a wide range of GPUs, TPUs, NPUs, CPUs, and Apple Silicon platforms.
| Platform | Representative hardware |
|---|---|
| NVIDIA | B200/B300/GB200/GB300; H100/H200/H800/H20; A100; RTX 30/40/50 series, RTX 6000 Ada / PRO 6000; DGX Spark, Jetson Orin |
| AMD | Instinct MI300X, MI325X, MI350X, MI355X |
| Google TPU | v6e, v7; SGLang-TorchTPU / SGL-JAX |
| Intel (GPU / CPU) | Arc / Arc Pro B-Series GPUs, Xeon CPUs |
| Apple Silicon | Macs via Metal / MLX |
| Huawei Ascend | A2, A3, 950PR/DT NPUs |
| Moore Threads | MTT S5000 GPUs |
Integrations in progress: AWS Trainium, Alibaba T-Head PPU, Cambricon MLU, Qualcomm QAIC, MetaX, Hygon HCU/DCU, Iluvatar CoreX, and more.
SGLang-TorchTPU brings the SGLang serving engine to Google TPUs through PyTorch, torch_tpu, and Pallas kernels. It supports LLM and vision-language serving on TPU7x (Ironwood), including tensor and expert parallelism across hosts. SGL-JAX provides the JAX-based TPU serving path.
See the Cookbook and platform guides for model compatibility and setup.
SGL Ecosystem
| Area | Projects | Purpose |
|---|---|---|
| Diffusion | SGLang Diffusion | Built into SGLang for image and video generation with diffusion models. |
| Audio | SGLang Omni | Audio model serving for text-to-speech (TTS) and automatic speech recognition (ASR). |
| Speculative Decoding | SpecForge | Train draft models for speculative decoding and deploy them with SGLang. |
| RL and Post-Training | Miles, slime, AReaL, Tunix, verl | Training frameworks that integrate SGLang for rollout generation. |
| Deployment and Orchestration | SMG, RBG, llm-d, Ray Serve, NVIDIA Dynamo | Deploy and scale SGLang inference services with routing, load balancing, and cluster orchestration. |
| Education | Mini-SGLang, zero-to-sglang, DeepLearning.AI course | Learn inference engine design and efficient text and image generation through code and hands-on courses. |
Development and Contributing
Contributions are welcome, from bug fixes and documentation to model support and performance improvements.
Development setup
Start from the lmsysorg/sglang:dev Docker image, which provides development tools and most dependencies. Clone or mount your SGLang checkout inside the container, then install it in editable mode from the repository root so tests use your local Python changes:
pip install -e "python"
In an activated virtual environment, you can use uv pip install --prerelease=allow -e "python" instead. See the development guide for container setup and testing.
Contribute
- Fork the repository and create a branch for your changes. For larger changes, discuss your proposal in a GitHub issue or on Slack.
- Make your changes, run the relevant tests, and add regression coverage for fixes or new behavior. Run
pre-commit run --all-filesbefore submitting. - Open a pull request describing the change and how you tested it. Include benchmarks or accuracy evaluations when relevant.
See the contributor guide for formatting, testing, and pull request instructions. Documentation contributors can start with the docs guide.
Community and Sponsorship
SGLang is hosted by LMSYS, a non-profit open-source organization.
- Community discussions: Join Slack for technical questions and development discussions.
- Events: Find meetups, workshops, and office hours on SGLang Events.
- Updates: Follow X and LinkedIn for project updates, and the LMSYS Blog for release announcements and technical articles.
- Project resources: Explore the documentation, Cookbook, roadmap, release notes, issue tracker, and contributor guide.
- Contact Us: For enterprise adoption and deployment, technical consulting, sponsorship, or partnership inquiries, please contact sglang@lmsys.org.
- Contributor sponsorship: Long-term active SGLang contributors are eligible for coding agent sponsorship, including Cursor, Claude Code, or OpenAI Codex. To apply, email sglang@lmsys.org with links to your key commits or pull requests.
Trusted by Industry and Research
SGLang serves production workloads across AI labs, cloud platforms, enterprises, and universities.
Acknowledgment
We learned the design and reused code from the following projects: Guidance, vLLM, LightLLM, FlashInfer, Outlines, and LMQL.
