diff --git a/README.md b/README.md
index 79bfc4fdf4..23002c1903 100644
--- a/README.md
+++ b/README.md
@@ -1,99 +1,115 @@
-
-

+# SGLang: Fast inference for LLMs and multimodal models
-[](https://pypi.org/project/sglang)
-
-[](https://github.com/sgl-project/sglang/tree/main/LICENSE)
-[](https://github.com/sgl-project/sglang/issues)
-[](https://github.com/sgl-project/sglang/issues)
-[](https://deepwiki.com/sgl-project/sglang)
-
-
-
---------------------------------------------------------------------------------
-
-
-🌐 Website |
-Blog |
-Documentation |
-Roadmap |
-Join Slack |
-Weekly Dev Meeting |
-Slides
+
+
-## News
-- [2026/07] 🔥 SGLang and Miles add day-0 support for Kimi K3 ([blog](https://lmsys.org/blog/2026-07-27-kimi-k3-day0-support/)).
-- [2026/07] RadixArk and Google bring full SGLang features to TPUs ([blog](https://lmsys.org/blog/2026-07-30-sglang-google-tpu/)).
-- [2026/07] Serving GLM5.2 NVFP4 agentic workloads with SGLang: Reaching 500 TPS in two weeks ([blog](https://lmsys.org/blog/2026-07-13-glm52-optimization/)).
-- [2026/06] 🔥 The next generation of speculative decoding: DFlash and Spec V2 ([blog](https://lmsys.org/blog/2026-06-15-next-generation-speculative-decoding-dflash-v2/)).
-- [2026/06] SGLang provides day-0 support for latest open models ([Nemotron 3 Ultra](https://lmsys.org/blog/2026-06-04-nvidia-run-nemotron-3-ultra/), [Nemotron 3 Super](https://lmsys.org/blog/2026-03-11-run-nvidia-nemotron-3-super/), [Higgs Audio v3 TTS](https://lmsys.org/blog/2026-06-04-higgs-audio-v3-tts/)).
-- [2026/04] 🔥 DeepSeek-V4 on Day 0: From Fast Inference to Verified RL with SGLang and Miles ([blog](https://lmsys.org/blog/2026-04-25-deepseek-v4/)).
-- [2026/02] 🔥 Unlocking 25x Inference Performance with SGLang on NVIDIA GB300 NVL72 ([blog](https://lmsys.org/blog/2026-02-20-gb300-inferencex/)).
-- [2026/01] SGLang Diffusion accelerates video and image generation ([blog](https://lmsys.org/blog/2026-01-16-sglang-diffusion/)).
+
+
+
+
+
+
-
-More
+
+ Docs |
+ Cookbook |
+ Website |
+ Blog |
+ Slack
+
-- [2025/12] SGLang provides day-0 support for latest open models ([MiMo-V2-Flash](https://lmsys.org/blog/2025-12-16-mimo-v2-flash/), [Nemotron 3 Nano](https://lmsys.org/blog/2025-12-15-run-nvidia-nemotron-3-nano/), [Mistral Large 3](https://github.com/sgl-project/sglang/pull/14213), [LLaDA 2.0 Diffusion LLM](https://lmsys.org/blog/2025-12-19-diffusion-llm/), [MiniMax M2](https://lmsys.org/blog/2025-11-04-miminmax-m2/)).
-- [2025/11] SGLang Diffusion accelerates video and image generation ([blog](https://lmsys.org/blog/2025-11-07-sglang-diffusion/)).
-- [2025/10] SGLang now runs natively on TPU with the SGLang-Jax backend ([blog](https://lmsys.org/blog/2025-10-29-sglang-jax/)).
-- [2025/10] PyTorch Conference 2025 SGLang Talk ([slide](https://github.com/sgl-project/sgl-learning-materials/blob/main/slides/sglang_pytorch_2025.pdf)).
-- [2025/10] SGLang x Nvidia SF Meetup on 10/2 ([recap](https://x.com/lmsysorg/status/1975339501934510231)).
-- [2025/09] Deploying DeepSeek on GB200 NVL72 with PD and Large Scale EP (Part II): 3.8x Prefill, 4.8x Decode Throughput ([blog](https://lmsys.org/blog/2025-09-25-gb200-part-2/)).
-- [2025/09] SGLang Day 0 Support for DeepSeek-V3.2 with Sparse Attention ([blog](https://lmsys.org/blog/2025-09-29-deepseek-V32/)).
-- [2025/08] SGLang x AMD SF Meetup on 8/22: Hands-on GPU workshop, tech talks by AMD/xAI/SGLang, and networking ([Roadmap](https://github.com/sgl-project/sgl-learning-materials/blob/main/slides/amd_meetup_sglang_roadmap.pdf), [Large-scale EP](https://github.com/sgl-project/sgl-learning-materials/blob/main/slides/amd_meetup_sglang_ep.pdf), [Highlights](https://github.com/sgl-project/sgl-learning-materials/blob/main/slides/amd_meetup_highlights.pdf), [AITER/MoRI](https://github.com/sgl-project/sgl-learning-materials/blob/main/slides/amd_meetup_aiter_mori.pdf), [Wave](https://github.com/sgl-project/sgl-learning-materials/blob/main/slides/amd_meetup_wave.pdf)).
-- [2025/08] SGLang provides day-0 support for OpenAI gpt-oss model ([instructions](https://github.com/sgl-project/sglang/issues/8833))
-- [2025/06] SGLang, the high-performance serving infrastructure powering trillions of tokens daily, has been awarded the third batch of the Open Source AI Grant by a16z ([a16z blog](https://a16z.com/advancing-open-source-ai-through-benchmarks-and-bold-experimentation/)).
-- [2025/06] Deploying DeepSeek on GB200 NVL72 with PD and Large Scale EP (Part I): 2.7x Higher Decoding Throughput ([blog](https://lmsys.org/blog/2025-06-16-gb200-part-1/)).
-- [2025/05] Deploying DeepSeek with PD Disaggregation and Large-scale Expert Parallelism on 96 H100 GPUs ([blog](https://lmsys.org/blog/2025-05-05-large-scale-ep/)).
-- [2025/03] Supercharge DeepSeek-R1 Inference on AMD Instinct MI300X ([AMD blog](https://rocm.blogs.amd.com/artificial-intelligence/DeepSeekR1-Part2/README.html))
-- [2025/03] SGLang Joins PyTorch Ecosystem: Efficient LLM Serving Engine ([PyTorch blog](https://pytorch.org/blog/sglang-joins-pytorch/))
-- [2025/02] Unlock DeepSeek-R1 Inference Performance on AMD Instinct™ MI300X GPU ([AMD blog](https://rocm.blogs.amd.com/artificial-intelligence/DeepSeekR1_Perf/README.html))
-- [2025/01] SGLang provides day one support for DeepSeek V3/R1 models on NVIDIA and AMD GPUs with DeepSeek-specific optimizations. ([instructions](https://github.com/sgl-project/sglang/tree/main/benchmark/deepseek_v3), [AMD blog](https://www.amd.com/en/developer/resources/technical-articles/amd-instinct-gpus-power-deepseek-v3-revolutionizing-ai-development-with-sglang.html), [10+ other companies](https://x.com/lmsysorg/status/1887262321636221412))
-- [2024/12] v0.4 Release: Zero-Overhead Batch Scheduler, Cache-Aware Load Balancer, Faster Structured Outputs ([blog](https://lmsys.org/blog/2024-12-04-sglang-v0-4/)).
-- [2024/10] The First SGLang Online Meetup ([slides](https://github.com/sgl-project/sgl-learning-materials?tab=readme-ov-file#the-first-sglang-online-meetup)).
-- [2024/09] v0.3 Release: 7x Faster DeepSeek MLA, 1.5x Faster torch.compile, Multi-Image/Video LLaVA-OneVision ([blog](https://lmsys.org/blog/2024-09-04-sglang-v0-3/)).
-- [2024/07] v0.2 Release: Faster Llama3 Serving with SGLang Runtime (vs. TensorRT-LLM, vLLM) ([blog](https://lmsys.org/blog/2024-07-25-sglang-llama3/)).
-- [2024/02] SGLang enables **3x faster JSON decoding** with compressed finite state machine ([blog](https://lmsys.org/blog/2024-02-05-compressed-fsm/)).
-- [2024/01] SGLang provides up to **5x faster inference** with RadixAttention ([blog](https://lmsys.org/blog/2024-01-17-sglang/)).
-- [2024/01] SGLang powers the serving of the official **LLaVA v1.6** release demo ([usage](https://github.com/haotian-liu/LLaVA?tab=readme-ov-file#demo)).
+SGLang is an open-source inference framework for LLMs and multimodal models, optimized for agentic workloads, RL rollouts, and large-scale serving.
-
+👋 Get started below, or meet the community at [SGLang Events](https://www.sglang.io/events), including meetups, developer meetings, workshops, and office hours.
-## About
-SGLang is a high-performance serving framework for large language models and multimodal models.
-It is designed to deliver low-latency and high-throughput inference across a wide range of setups, from a single GPU to large distributed clusters.
-Its core features include:
+## Get Started
-- **Fast Runtime**: Provides efficient serving with RadixAttention for prefix caching, a zero-overhead CPU scheduler, prefill-decode disaggregation, speculative decoding, continuous batching, paged attention, tensor/pipeline/expert/data parallelism, structured outputs, chunked prefill, quantization (FP4/FP8/INT4/AWQ/GPTQ), and multi-LoRA batching.
-- **Broad Model Support**: Supports a wide range of language models (Llama, Qwen, DeepSeek, Kimi, GLM, GPT, Gemma, Mistral, etc.), embedding models (e5-mistral, gte, mcdse), reward models (Skywork), and diffusion models (WAN, Qwen-Image), with easy extensibility for adding new models. Compatible with most Hugging Face models and OpenAI APIs.
-- **Extensive Hardware Support**: Runs on NVIDIA GPUs (GB200/B300/H100/A100/Spark/5090), AMD GPUs (MI355/MI300), Intel Xeon CPUs, Google TPUs, Ascend NPUs, and more.
-- **Active Community**: SGLang is open-source and supported by a vibrant community with widespread industry adoption, powering over 400,000 GPUs worldwide.
-- **RL & Post-Training Backbone**: SGLang is a proven rollout backend used for training many frontier models, with native RL integrations and adoption by well-known post-training frameworks such as [**AReaL**](https://github.com/inclusionAI/AReaL), [**Miles**](https://github.com/radixark/miles), [**slime**](https://github.com/THUDM/slime), [**Tunix**](https://github.com/google/tunix), [**verl**](https://github.com/volcengine/verl) and more.
+Pull the Docker image, which includes SGLang and its dependencies:
-## Getting Started
-- [Install SGLang](https://docs.sglang.io/get_started/install.html)
-- [Quick Start](https://docs.sglang.io/basic_usage/send_request.html)
-- [Cookbook](https://docs.sglang.io/cookbook) — the deployment command we recommend for each supported model
-- [Backend Tutorial](https://docs.sglang.io/basic_usage/openai_api_completions.html)
-- [Frontend Tutorial](https://docs.sglang.io/references/frontend/frontend_tutorial.html)
-- [Contribution Guide](https://docs.sglang.io/developer_guide/contribution_guide.html)
+```bash
+docker pull lmsysorg/sglang:latest
+```
-## Benchmark and Performance
-Learn more in the release blogs: [v0.2 blog](https://lmsys.org/blog/2024-07-25-sglang-llama3/), [v0.3 blog](https://lmsys.org/blog/2024-09-04-sglang-v0-3/), [v0.4 blog](https://lmsys.org/blog/2024-12-04-sglang-v0-4/), [Large-scale expert parallelism](https://lmsys.org/blog/2025-05-05-large-scale-ep/), [GB200 rack-scale parallelism](https://lmsys.org/blog/2025-09-25-gb200-part-2/), [GB300 long context](https://lmsys.org/blog/2026-02-19-gb300-longctx/).
+Alternatively, install SGLang in an activated Python environment with uv:
-## Adoption and Sponsorship
-SGLang has been deployed at large scale, generating trillions of tokens in production each day. It is trusted and adopted by a wide range of leading enterprises and institutions, including xAI, NVIDIA, AMD, Intel, LinkedIn, Cursor, Oracle Cloud, Google Cloud, Microsoft Azure, AWS, Atlas Cloud, Voltage Park, Nebius, DataCrunch, Novita, RunPod, InnoMatrix, Modal, MIT, UCLA, the University of Washington, Stanford, UC Berkeley, Tsinghua University, Baseten, Baidu, AntGroup, Alibaba, Tencent, and other major technology organizations.
-As an open-source LLM inference engine, SGLang has become the de facto industry standard, with deployments running on over 400,000 GPUs worldwide.
-SGLang is currently hosted under the non-profit open-source organization [LMSYS](https://lmsys.org/about/).
+```bash
+uv pip install --prerelease=allow sglang
+```
-
+Next, launch your model:
-## Contact Us
-For enterprises interested in adopting or deploying SGLang at scale, including technical consulting, sponsorship opportunities, or partnership inquiries, please contact us at [sglang@lmsys.org](mailto:sglang@lmsys.org).
+- [Quickstart](https://docs.sglang.io/docs/get-started/quickstart): Run your first model and send a request.
+- [Cookbook](https://cookbook.sglang.io/): Choose your model and hardware to get a ready-to-run launch command.
-Long-term active SGLang contributors are eligible for coding agent sponsorship, such as Cursor, Claude Code, or OpenAI Codex. Email [sglang@lmsys.org](mailto:sglang@lmsys.org) with your most important commits or pull requests.
+## Supported Hardware
+
+SGLang supports a wide range of GPUs, TPUs, NPUs, CPUs, and Apple Silicon platforms.
+
+| Platform | Representative hardware |
+| --- | --- |
+| [NVIDIA](https://docs.sglang.io/docs/hardware-platforms/nvidia-gpus) | A100; H100/H200/H800/H20; B200/B300/GB200/GB300; select RTX 30/40/50 series, RTX 6000 Ada / PRO 6000; [DGX Spark](https://lmsys.org/blog/2025-11-03-gpt-oss-on-nvidia-dgx-spark/), [Jetson Orin](https://docs.sglang.io/docs/hardware-platforms/nvidia_jetson) |
+| [AMD](https://docs.sglang.io/docs/hardware-platforms/amd_gpu) | Instinct MI300X, MI325X, MI350X, MI355X |
+| [Google TPU](https://docs.sglang.io/docs/hardware-platforms/tpu) | v6e, v7; [SGL-JAX](https://github.com/sgl-project/sglang-jax) / [SGL-torchtpu](https://lmsys.org/blog/2026-07-30-sglang-google-tpu/) |
+| Intel | [Arc / Arc Pro B-Series GPUs](https://docs.sglang.io/docs/hardware-platforms/xpu), [Xeon CPUs](https://docs.sglang.io/docs/hardware-platforms/cpu_server) |
+| [Apple Silicon](https://docs.sglang.io/docs/hardware-platforms/apple_metal) | Macs via Metal / MLX |
+| [Huawei Ascend](https://docs.sglang.io/docs/hardware-platforms/ascend-npus/getting-started/installation) | A2, A3, 950PR/DT NPUs |
+| [Moore Threads](https://docs.sglang.io/docs/hardware-platforms/mthreads_gpu) | MTT S5000 GPUs |
+
+Integrations in progress: AWS Trainium, [Alibaba T-Head PPU](https://github.com/sgl-project/sglang/issues/37519), [Cambricon MLU](https://github.com/sgl-project/sglang/pull/26898), Qualcomm QAIC, MetaX, Hygon HCU/DCU, Iluvatar CoreX, and more.
+
+See the [Cookbook](https://cookbook.sglang.io/) and platform guides for model compatibility and setup.
+
+## SGL Ecosystem
+
+| Area | Projects | Purpose |
+| --- | --- | --- |
+| Education | [Mini-SGLang](https://github.com/sgl-project/mini-sglang), [zero-to-sglang](https://github.com/datawhalechina/zero-to-sglang), [DeepLearning.AI course](https://www.deeplearning.ai/short-courses/efficient-inference-with-sglang-text-and-image-generation/) | Learn inference engine design and efficient text and image generation through code and hands-on courses. |
+| Diffusion | [SGLang Diffusion](https://docs.sglang.io/docs/sglang-diffusion/installation) | Image and video generation with diffusion models. |
+| Audio | [SGLang Omni](https://github.com/sgl-project/sglang-omni) | Audio model serving for text-to-speech (TTS) and automatic speech recognition (ASR). |
+| RL and Post-Training | [Miles](https://github.com/radixark/miles), [slime](https://github.com/THUDM/slime), [AReaL](https://github.com/inclusionAI/AReaL), [Tunix](https://github.com/google/tunix), [verl](https://github.com/volcengine/verl) | Training frameworks that integrate SGLang for rollout generation. |
+| Speculative Decoding | [SpecForge](https://github.com/sgl-project/SpecForge) | Train draft models for speculative decoding and deploy them with SGLang. |
+| KV Cache | [HiCache](https://docs.sglang.io/docs/advanced_features/hicache_design), [Mooncake](https://kvcache-ai.github.io/Mooncake/), [LMCache](https://docs.lmcache.ai/developer_guide/integration.html) | Hierarchical KV caching across GPU memory, host memory, and external storage, with cache transfer and reuse for distributed inference. |
+| Deployment and Orchestration | [SMG](https://github.com/smg-project/smg), [RBG](https://github.com/sgl-project/rbg), [llm-d](https://llm-d.ai/docs/dev/operations/disaggregation/sglang), [Ray Serve](https://docs.ray.io/en/latest/serve/llm/user-guides/sglang.html), [NVIDIA Dynamo](https://docs.nvidia.com/dynamo/backends/sg-lang/reference-guide) | Deploy and scale SGLang inference services with routing, load balancing, and cluster orchestration. |
+
+## Development and Contributing
+
+Contributions are welcome, from bug fixes and documentation to model support and performance improvements.
+
+### Development setup
+
+Start from the `lmsysorg/sglang:dev` Docker image, which provides development tools and most dependencies. Clone or mount your SGLang checkout inside the container, then install it in editable mode from the repository root so tests use your local Python changes:
+
+```bash
+pip install -e "python"
+```
+
+In an activated virtual environment, you can use `uv pip install --prerelease=allow -e "python"` instead. See the [development guide](https://docs.sglang.io/docs/developer_guide/development_guide_using_docker) for container setup and testing.
+
+### Contribute
+
+1. Fork the repository and create a branch for your changes. For larger changes, discuss your proposal in a [GitHub issue](https://github.com/sgl-project/sglang/issues) or on [Slack](https://slack.sglang.io/).
+2. Make your changes, run the relevant tests, and add regression coverage for fixes or new behavior. Run `pre-commit run --all-files` before submitting.
+3. Open a pull request describing the change and how you tested it. Include benchmarks or accuracy evaluations when relevant.
+
+See the [contributor guide](https://docs.sglang.io/docs/developer_guide/contribution_guide) for formatting, testing, and pull request instructions. Documentation contributors can start with the [docs guide](docs/README.md).
+
+## Community and Sponsorship
+
+SGLang is hosted by [LMSYS](https://lmsys.org/about/), a non-profit open-source organization.
+
+- **Community discussions:** Join [Slack](https://slack.sglang.io/) for technical questions and development discussions.
+- **Events:** Find meetups, workshops, and office hours on [SGLang Events](https://www.sglang.io/events).
+- **Updates:** Follow [X](https://x.com/lmsysorg) and [LinkedIn](https://www.linkedin.com/company/sgl-project/) for project updates, and the [LMSYS Blog](https://lmsys.org/blog/) for release announcements and technical articles.
+- **Project resources:** Explore the [documentation](https://docs.sglang.io/), [Cookbook](https://cookbook.sglang.io/), [roadmap](https://roadmap.sglang.io/), [release notes](https://github.com/sgl-project/sglang/releases), [issue tracker](https://github.com/sgl-project/sglang/issues), and [contributor guide](https://docs.sglang.io/docs/developer_guide/contribution_guide).
+- **Contact Us:** For enterprise adoption and deployment, technical consulting, sponsorship, or partnership inquiries, please contact [sglang@lmsys.org](mailto:sglang@lmsys.org).
+- **Contributor sponsorship:** Long-term active SGLang contributors are eligible for coding agent sponsorship, including Cursor, Claude Code, or OpenAI Codex. To apply, email [sglang@lmsys.org](mailto:sglang@lmsys.org) with links to your key commits or pull requests.
+
+## Trusted by Industry and Research
+
+SGLang serves production workloads across AI labs, cloud platforms, enterprises, and universities.
+
+
## Acknowledgment
We learned the design and reused code from the following projects: [Guidance](https://github.com/guidance-ai/guidance), [vLLM](https://github.com/vllm-project/vllm), [LightLLM](https://github.com/ModelTC/lightllm), [FlashInfer](https://github.com/flashinfer-ai/flashinfer), [Outlines](https://github.com/outlines-dev/outlines), and [LMQL](https://github.com/eth-sri/lmql).
diff --git a/docs/docs.json b/docs/docs.json
index ae1b38dd15..88bd729a1f 100644
--- a/docs/docs.json
+++ b/docs/docs.json
@@ -891,9 +891,9 @@
"icon": "play",
"pages": [
"index",
- "docs/get-started/install",
"docs/get-started/quickstart",
- "docs/basic_usage/send_request"
+ "docs/get-started/install",
+ "docs/get-started/examples"
]
}
]
@@ -906,6 +906,7 @@
"icon": "book-open",
"pages": [
"docs/basic_usage/overview",
+ "docs/basic_usage/send_request",
{
"group": "OpenAI-Compatible APIs",
"pages": [
diff --git a/docs/docs/basic_usage/aws_sagemaker.mdx b/docs/docs/basic_usage/aws_sagemaker.mdx
index 63deef77f3..b026518c1b 100644
--- a/docs/docs/basic_usage/aws_sagemaker.mdx
+++ b/docs/docs/basic_usage/aws_sagemaker.mdx
@@ -8,7 +8,7 @@ Deploy SGLang on [Amazon SageMaker AI](https://aws.amazon.com/sagemaker/) endpoi
The SageMaker image variant accepts model configuration via environment variables and serves on port 8080.
This guide uses the pre-built DLC image. To build and deploy your own container instead, see
-[Method 7: Run on AWS SageMaker](/docs/get-started/install#more-3) in the installation guide.
+[Method 7: Run on AWS SageMaker](/docs/get-started/install#aws-sagemaker) in the installation guide.
## Container image
diff --git a/docs/docs/developer_guide/development_guide_using_docker.mdx b/docs/docs/developer_guide/development_guide_using_docker.mdx
index 3c3a568b28..32898f8467 100644
--- a/docs/docs/developer_guide/development_guide_using_docker.mdx
+++ b/docs/docs/developer_guide/development_guide_using_docker.mdx
@@ -4,6 +4,9 @@ sidebarTitle: "Using Docker"
metatags:
description: "SGLang Docker development: VSCode dev container, remote tunnels, debugger setup, nsys profiling."
---
+
+The `lmsysorg/sglang:dev` image is a nightly build from `main` with additional development tools and dependencies for modifying and testing SGLang. For running models without a development environment, use the stable or nightly images described in the [Installation guide](/docs/get-started/install).
+
## Setup VSCode on a Remote Host
(Optional - you can skip this step if you plan to run sglang dev container locally)
diff --git a/docs/docs/get-started/examples.mdx b/docs/docs/get-started/examples.mdx
new file mode 100644
index 0000000000..62cb0f97f0
--- /dev/null
+++ b/docs/docs/get-started/examples.mdx
@@ -0,0 +1,279 @@
+---
+title: "Examples"
+description: "Run a model locally, add chat to an application, process a batch of texts, or build a tool-using assistant."
+---
+
+Choose an example for what you want to do:
+
+| I want to… | Example |
+| --- | --- |
+| Run a model on my own computer | [Local inference](#run-a-model-locally) on an NVIDIA GPU or Apple Silicon Mac. |
+| Serve a larger model on a GPU server | [Deploy Qwen3.8-27B](#serve-qwen3-8-27b) on an NVIDIA H200. |
+| Add chat to an application | [Stream a reply](#stream-a-reply) through the OpenAI-compatible API. |
+| Process a collection of documents | [Summarize multiple texts](#process-multiple-prompts) concurrently. |
+| Build an agent that uses tools | [Look up an order](#build-a-customer-support-agent) and answer a customer’s question. |
+
+## Run a model locally
+
+Run `Qwen/Qwen3-0.6B` on your own computer and ask it a question. This small model is useful for trying the workflow; choose a larger model for more demanding tasks.
+
+
+
+
+This example uses one CUDA 13-compatible NVIDIA GPU, Linux, Docker, and NVIDIA Container Toolkit. For setup, see [Installation](/docs/get-started/install).
+
+Run this command on your computer. Docker pulls the image if needed, and SGLang downloads the model on first use:
+
+```bash
+docker run --rm --gpus all --ipc=host \
+ -p 127.0.0.1:30000:30000 \
+ lmsysorg/sglang:latest \
+ sglang serve Qwen/Qwen3-0.6B \
+ --host 0.0.0.0 --port 30000
+```
+
+If you already installed SGLang with uv, run this in that Python environment instead:
+
+```bash
+sglang serve Qwen/Qwen3-0.6B --host 127.0.0.1 --port 30000
+```
+
+
+
+
+On an Apple Silicon Mac with macOS 14 or newer, first follow the [Apple Silicon installation instructions](/docs/hardware-platforms/apple_metal#install-sglang) to install SGLang with its MLX dependencies. Use that Python environment to launch the model with 4-bit quantization:
+
+```bash
+SGLANG_USE_MLX=1 python -m sglang.launch_server \
+ --model-path Qwen/Qwen3-0.6B \
+ --quantization mlx_q4 \
+ --disable-cuda-graph \
+ --host 127.0.0.1 --port 30000
+```
+
+The model runs on your Mac through MLX. It downloads the weights on first use and quantizes them in memory when loading.
+
+
+
+
+Keep the server running. Once you see `The server is fired up and ready to roll!`, open another terminal on the same computer and send a request:
+
+```bash
+curl http://localhost:30000/v1/chat/completions \
+ -H "Content-Type: application/json" \
+ -d '{
+ "model": "Qwen/Qwen3-0.6B",
+ "messages": [{"role": "user", "content": "What is the capital of France? /no_think"}],
+ "max_tokens": 128
+ }'
+```
+
+Look for the answer in `choices[0].message.content`. Both the model and the API server are running locally. Press `Ctrl+C` in the server terminal when you are done, or leave it running for the chat and batch examples below.
+
+
+
+## Serve Qwen3.8-27B
+
+This configuration runs `Qwen/Qwen3.8-27B-FP8` on one NVIDIA H200. It requires Linux, a CUDA 13-compatible driver, Docker, and NVIDIA Container Toolkit configured for GPU access. For environment setup, see [Installation](/docs/get-started/install). For other hardware configurations, see the [Qwen3.8-27B Cookbook](/cookbook/autoregressive/Qwen/Qwen3.8-27B).
+
+Set `HF_CACHE_DIR` to an absolute directory on the host for downloaded model weights, replacing the placeholder below. Run this command on the GPU host:
+
+```bash
+HF_CACHE_DIR="/path/to/huggingface-cache"
+docker run --rm --gpus all \
+ --shm-size 32g \
+ -p 30000:30000 \
+ -v "$HF_CACHE_DIR:/root/.cache/huggingface" \
+ --ipc=host \
+ lmsysorg/sglang:latest \
+ sglang serve \
+ --trust-remote-code \
+ --model-path Qwen/Qwen3.8-27B-FP8 \
+ --kv-cache-dtype fp8_e4m3 \
+ --mem-fraction-static 0.85 \
+ --attention-backend flashinfer \
+ --chunked-prefill-size 32768 \
+ --max-prefill-tokens 32768 \
+ --reasoning-parser qwen3 \
+ --tool-call-parser qwen3_coder \
+ --mamba-full-memory-ratio 4.59 \
+ --host 0.0.0.0 \
+ --port 30000 \
+ --mamba-radix-cache-strategy extra_buffer \
+ --mamba-ssm-dtype float32
+```
+
+Docker downloads the image if needed. SGLang downloads the model on first use and starts the server. Keep this terminal open and wait for `The server is fired up and ready to roll!` before sending a request.
+
+
+
+Run this command in the activated Python environment where you installed SGLang. It starts the same server without Docker:
+
+```bash
+sglang serve \
+ --trust-remote-code \
+ --model-path Qwen/Qwen3.8-27B-FP8 \
+ --kv-cache-dtype fp8_e4m3 \
+ --mem-fraction-static 0.85 \
+ --attention-backend flashinfer \
+ --chunked-prefill-size 32768 \
+ --max-prefill-tokens 32768 \
+ --reasoning-parser qwen3 \
+ --tool-call-parser qwen3_coder \
+ --mamba-full-memory-ratio 4.59 \
+ --host 0.0.0.0 \
+ --port 30000 \
+ --mamba-radix-cache-strategy extra_buffer \
+ --mamba-ssm-dtype float32
+```
+
+For Python environment setup, see [Install with uv](/docs/get-started/install#method-1-with-pip-or-uv).
+
+
+
+Verify the running server with the [Quickstart request](/docs/get-started/quickstart#send-a-request), using `Qwen/Qwen3.8-27B-FP8` as the model name.
+
+## Connect to a model
+
+The following Python examples connect to the local server above. You can also use a remote SGLang server; only the server needs hardware that can run the model.
+
+Install the Python client in your activated environment:
+
+```bash
+uv pip install openai
+```
+
+Run this setup before each example in the same Python session:
+
+```python
+from openai import OpenAI
+
+client = OpenAI(
+ base_url="http://localhost:30000/v1",
+ api_key="EMPTY",
+)
+model = "Qwen/Qwen3-0.6B"
+```
+
+These settings match the local Qwen3-0.6B server above. For a remote server, replace `localhost` with its reachable address, use its model name, and supply an API key if authentication is enabled.
+
+## Stream a reply
+
+Ask a question and print the answer as it arrives:
+
+```python
+stream = client.chat.completions.create(
+ model=model,
+ messages=[{"role": "user", "content": "Explain prefix caching in two sentences."}],
+ stream=True,
+)
+
+for chunk in stream:
+ if chunk.choices:
+ print(chunk.choices[0].delta.content or "", end="", flush=True)
+print()
+```
+
+The terminal displays the model's explanation incrementally. You can use the same stream to update a chat interface.
+
+See the [API guide](/docs/basic_usage/openai_api_completions) for more request options.
+
+## Process multiple prompts
+
+Send several independent requests concurrently, for example to summarize customer feedback:
+
+```python
+from concurrent.futures import ThreadPoolExecutor
+
+feedback = [
+ "Setup was quick, but I could not find the export button.",
+ "The new search is faster and finds the documents I need.",
+ "Please add a way to share saved reports with my team.",
+]
+
+def summarize(text):
+ response = client.chat.completions.create(
+ model=model,
+ messages=[{"role": "user", "content": f"Summarize in one sentence: {text}"}],
+ )
+ return response.choices[0].message.content
+
+with ThreadPoolExecutor(max_workers=3) as pool:
+ for summary in pool.map(summarize, feedback):
+ print(summary)
+```
+
+You get one summary per input, in the original order. For batch processing inside a Python process without an HTTP server, see the [offline engine API](/docs/basic_usage/offline_engine_api).
+
+
+
+## Build a customer support agent
+
+Let an agent answer “Where is my order?” by looking up order data. The model chooses a tool, your application executes it, and the model uses the returned data to answer the customer.
+
+Use the [Qwen3.8-27B server above](#serve-qwen3-8-27b), which enables a tool parser, or configure another model with the [tool-calling guide](/docs/advanced_features/tool_parser). The small local example does not configure tool calling. This script uses the OpenAI client installed above; update the endpoint and model ID to match your server.
+
+Save this as `order_agent.py` and run `python order_agent.py`. The order data is a local fixture, so no external service or API key is needed for the lookup.
+
+```python
+import json
+from openai import OpenAI
+
+client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
+model = "Qwen/Qwen3.8-27B-FP8"
+
+orders = {
+ "SGL-1042": {"status": "shipped", "estimated_delivery": "Friday"},
+}
+
+def lookup_order(order_id):
+ return orders.get(order_id, {"error": "Order not found"})
+
+tools = [{
+ "type": "function",
+ "function": {
+ "name": "lookup_order",
+ "description": "Look up an order's shipping status and estimated delivery.",
+ "parameters": {
+ "type": "object",
+ "properties": {"order_id": {"type": "string"}},
+ "required": ["order_id"],
+ "additionalProperties": False,
+ },
+ },
+}]
+
+messages = [
+ {"role": "system", "content": "Use lookup_order for order status. Do not invent order details."},
+ {"role": "user", "content": "Where is order SGL-1042, and when should it arrive?"},
+]
+
+for _ in range(3):
+ response = client.chat.completions.create(
+ model=model, messages=messages, tools=tools, tool_choice="auto",
+ )
+ message = response.choices[0].message
+ messages.append(message.model_dump(exclude_none=True))
+ if not message.tool_calls:
+ print(message.content)
+ break
+
+ for call in message.tool_calls:
+ try:
+ if call.function.name != "lookup_order":
+ raise ValueError("Unknown tool")
+ arguments = json.loads(call.function.arguments)
+ result = lookup_order(**arguments)
+ except (ValueError, TypeError) as error:
+ result = {"error": str(error)}
+ print(f"{call.function.name}: {result}")
+ messages.append({
+ "role": "tool",
+ "tool_call_id": call.id,
+ "content": json.dumps(result),
+ })
+else:
+ print("Stopped after three model turns without a final answer.")
+```
+
+You should see the lookup result followed by an answer that the order has shipped and is expected on Friday. The wording can vary. SGLang serves the model; the Python application runs the tools and manages the conversation. Replace `lookup_order` with an authorized order-system query to use real data.
diff --git a/docs/docs/get-started/install.mdx b/docs/docs/get-started/install.mdx
index b7d7155669..efd7233e6d 100644
--- a/docs/docs/get-started/install.mdx
+++ b/docs/docs/get-started/install.mdx
@@ -7,109 +7,166 @@ keywords:
- pip
- docker
---
-You can install SGLang using one of the methods below.
-This page primarily applies to common NVIDIA GPU platforms.
-For other or newer platforms, please refer to the dedicated pages for [AMD GPUs](../hardware-platforms/amd_gpu), [Apple Metal](../hardware-platforms/apple_metal), [Intel Xeon CPUs](../hardware-platforms/cpu_server), [Google TPU](../hardware-platforms/tpu), [NVIDIA DGX Spark](https://lmsys.org/blog/2025-11-03-gpt-oss-on-nvidia-dgx-spark/), [NVIDIA Jetson](../hardware-platforms/nvidia_jetson), [Ascend NPUs](../hardware-platforms/ascend-npus/getting-started/installation), and [Intel XPU](../hardware-platforms/xpu).
+Choose one installation method below. These instructions target Linux with NVIDIA GPUs; for other devices, see [Hardware platforms](/docs/hardware-platforms/overview).
-
-Prerequisites: Python 3.10 or higher.
-
+## Install SGLang
-## Method 1: With pip or uv
+
+
-It is recommended to use uv for faster installation:
+Docker includes Python, SGLang, and its dependencies; no separate host Python installation is required.
-```bash Command
+The docker images are available on Docker Hub at [lmsysorg/sglang](https://hub.docker.com/r/lmsysorg/sglang/tags), built from [Dockerfile](https://github.com/sgl-project/sglang/tree/main/docker).
+Use Linux with a supported NVIDIA GPU, a CUDA 13-compatible driver, [Docker Engine](https://docs.docker.com/engine/install/), and [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html). Complete the Toolkit guide's Docker runtime configuration before starting a GPU container.
+
+SGLang images ship a CUDA 13 environment. CUDA 12 images (`-cu12` / `-cu129`) are no longer published; `lmsysorg/sglang:v0.5.19-cu129` is the last CUDA 12 tag.
+
+**Pull the image**
+
+`latest` contains the newest stable release:
+
+
+```bash
+docker pull lmsysorg/sglang:latest
+```
+
+**Image options**
+
+The examples use `latest`. To use another image, replace the tag in the pull and run commands:
+
+- Nightly: Built daily from the latest `main`, including changes not yet in a stable release. Choose a tag from the [nightly image list](https://hub.docker.com/r/lmsysorg/sglang/tags?name=nightly).
+- Runtime: A smaller image of the same stable release, with inference dependencies and fewer development utilities. Use the `latest-runtime` tag.
+
+For reproducible deployments, use a specific release or dated nightly tag from [Docker Hub](https://hub.docker.com/r/lmsysorg/sglang/tags).
+
+**Start a container**
+
+Set `HF_CACHE_DIR` to an absolute directory path on the host for downloaded model weights, replacing the placeholder below.
+
+For gated or private models, export your [Hugging Face token](https://huggingface.co/docs/hub/en/security-tokens) as `HF_TOKEN` on the host and add `--env HF_TOKEN` before the image name in this command.
+
+
+```bash
+HF_CACHE_DIR="/path/to/huggingface-cache"
+docker run -it --gpus all \
+ --shm-size 32g \
+ -p 30000:30000 \
+ -v "$HF_CACHE_DIR:/root/.cache/huggingface" \
+ --ipc=host \
+ lmsysorg/sglang:latest bash
+```
+
+The container shell is ready for `sglang serve`.
+
+
+
+
+Use Python 3.10 or higher and a compatible CUDA 13 environment.
+
+
+
+SGLang requires CUDA 13. CUDA 12 (`cu129`) wheels and images are retired because PyTorch 2.14 publishes no CUDA 12.9 builds. SGLang 0.5.19 is the last release with a CUDA 12 lane.
+
+
+
+**Prepare the environment**
+
+Use uv for faster installation. Install it with pip, or follow the [uv installation instructions](https://docs.astral.sh/uv/getting-started/installation/):
+
+```bash
pip install --upgrade pip
pip install uv
+```
+
+Create and activate a virtual environment:
+
+```bash Command
+uv venv --python 3.12
+source .venv/bin/activate
+```
+
+**Install SGLang**
+
+```bash
uv pip install --prerelease=allow sglang
```
-
+
+
Some of SGLang's dependencies only publish pre-releases on PyPI, so without `--prerelease=allow` uv older than 0.12.0 silently installs SGLang 0.5.9. On [uv 0.12.0](https://github.com/astral-sh/uv/releases/tag/0.12.0) and newer the flag is a harmless no-op.
-
+
-
-SGLang requires CUDA 13. The CUDA 12 (`cu129`) wheels and images are retired,
-because PyTorch 2.14 publishes no CUDA 12.9 builds. SGLang 0.5.19 is the last
-release with a CUDA 12 lane.
-
-
-### Nightly builds
+
To pick up the latest features and fixes before the next stable release, install a nightly build. Nightly wheels are built from the latest `main` and published to the SGLang wheel index. Add that index with `--extra-index-url`, and combine `--prerelease=allow` with `--index-strategy unsafe-best-match` so uv considers the nightly (pre-release) version alongside PyPI:
```bash Command
-pip install --upgrade pip
-pip install uv
uv pip install --prerelease=allow --index-strategy unsafe-best-match --extra-index-url https://docs.sglang.ai/whl/cu130/ sglang
```
-### Quick fixes to common problems
-- If you encounter `OSError: CUDA_HOME environment variable is not set`. Please set it to your CUDA install root with either of the following solutions:
- 1. Use `export CUDA_HOME=/usr/local/cuda-` to set the `CUDA_HOME` environment variable.
- 2. Install FlashInfer first following [FlashInfer installation doc](https://docs.flashinfer.ai/installation.html), then install SGLang as described above.
+
-## Method 2: From source
+For `CUDA_HOME` errors, see [Troubleshooting](#cuda_home).
-```bash Command
-# Use the last release branch
+
+
+
+Use Python 3.10 or higher in an activated Python environment, with CUDA dependencies appropriate for the version you build.
+
+**Get the source**
+
+Clone a release tag. This example uses `v0.5.18`; choose another tag from [GitHub releases](https://github.com/sgl-project/sglang/releases) as needed:
+
+```bash
git clone -b v0.5.18 https://github.com/sgl-project/sglang.git
cd sglang
+```
-# Install the python packages
+**Install SGLang**
+
+Install the Python package in editable mode so local source changes are reflected in your environment:
+
+```bash
pip install --upgrade pip
pip install -e "python"
```
-**Quick fixes to common problems**
+
-- If you want to develop SGLang, you can try the dev docker image. Please refer to [setup docker container](../developer_guide/development_guide_using_docker#setup-docker-container). The docker image is `lmsysorg/sglang:dev`.
+For contributing to SGLang, follow the [development environment guide](/docs/developer_guide/development_guide_using_docker#setup-docker-container), which covers a container with development tools and dependencies installed.
-## Method 3: Using docker
+
-The docker images are available on Docker Hub at [lmsysorg/sglang](https://hub.docker.com/r/lmsysorg/sglang/tags), built from [Dockerfile](https://github.com/sgl-project/sglang/tree/main/docker).
-Replace `` below with your huggingface hub [token](https://huggingface.co/docs/hub/en/security-tokens).
+For `CUDA_HOME` errors, see [Troubleshooting](#cuda_home).
-
-`latest` and `dev` are **mutable** tags: `latest` always points at the newest stable release, while `dev` is rebuilt daily from the latest `main` and includes build/development tools. Because they are overwritten over time, pin an immutable version tag for reproducible deployments — e.g. `lmsysorg/sglang:v0.5.18`. Browse all released versions on [Docker Hub](https://hub.docker.com/r/lmsysorg/sglang/tags).
-
+
+
-```bash Command
-docker run --gpus all \
- --shm-size 32g \
- -p 30000:30000 \
- -v ~/.cache/huggingface:/root/.cache/huggingface \
- --env "HF_TOKEN=" \
- --ipc=host \
- lmsysorg/sglang:latest \
- python3 -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct --host 0.0.0.0 --port 30000
+## Start a model server
+
+Start a model server inside your Docker container or activated Python environment. Replace `MODEL_PATH` with a Hugging Face model ID or a local model directory:
+
+```bash
+sglang serve MODEL_PATH --host 0.0.0.0 --port 30000
```
-For production deployments, use the `runtime` variant which is significantly smaller (~40% reduction) by excluding build tools and development dependencies:
+Additional launch arguments depend on the model and your hardware.
-```bash Command
-docker run --gpus all \
- --shm-size 32g \
- -p 30000:30000 \
- -v ~/.cache/huggingface:/root/.cache/huggingface \
- --env "HF_TOKEN=" \
- --ipc=host \
- lmsysorg/sglang:latest-runtime \
- python3 -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct --host 0.0.0.0 --port 30000
-```
+- [Quickstart](/docs/get-started/quickstart): Launch a model and send a request to verify the server.
+- [Cookbook](/cookbook/intro): Find model-specific launch arguments for your hardware.
-You can also find the nightly docker images [here](https://hub.docker.com/r/lmsysorg/sglang/tags?name=nightly).
+## Deployment options
-Notes:
-- SGLang images ship a CUDA 13 environment. The `-cu12` / `-cu129` images are no longer published; `lmsysorg/sglang:v0.5.19-cu129` is the last CUDA 12 tag.
+Choose how to run SGLang as a service. Expand an option for setup instructions.
-## Method 4: Using Kubernetes
+
+
+
+
+
Please check out [OME](https://github.com/sgl-project/ome), a Kubernetes operator for enterprise-grade management and serving of large language models (LLMs).
-
-
1. Option 1: For single node serving (typically when the model size fits into GPUs on one node)
Execute command `kubectl apply -f docker/k8s-sglang-service.yaml`, to create k8s deployment and service, with llama-31-8b as example.
@@ -120,9 +177,9 @@ Please check out [OME](https://github.com/sgl-project/ome), a Kubernetes operato
-## Method 5: Using docker compose
+
-
+
> This method is recommended if you plan to serve it as a service.
> A better approach is to use the [k8s-sglang-service.yaml](https://github.com/sgl-project/sglang/blob/main/docker/k8s-sglang-service.yaml).
@@ -132,16 +189,16 @@ Please check out [OME](https://github.com/sgl-project/ome), a Kubernetes operato
-## Method 6: Run on Kubernetes or Clouds with SkyPilot
+
-
+
To deploy on Kubernetes or 12+ clouds, you can use [SkyPilot](https://github.com/skypilot-org/skypilot).
1. Install SkyPilot and set up Kubernetes cluster or cloud access: see [SkyPilot's documentation](https://skypilot.readthedocs.io/en/latest/getting-started/installation.html).
2. Deploy on your own infra with a single command and get the HTTP API endpoint:
-SkyPilot YAML: sglang.yaml>}>
+
```yaml Config
# sglang.yaml
@@ -175,9 +232,9 @@ sky status --endpoint 30000 sglang
-## Method 7: Run on AWS SageMaker
+
-
+
To deploy on SGLang on AWS SageMaker, check out [AWS SageMaker Inference](https://aws.amazon.com/sagemaker/ai/deploy)
@@ -190,7 +247,7 @@ To host a model with your own container, follow the following steps:
1. Build a docker container with [sagemaker.Dockerfile](https://github.com/sgl-project/sglang/blob/main/docker/sagemaker.Dockerfile) alongside the [serve](https://github.com/sgl-project/sglang/blob/main/docker/serve) script.
2. Push your container onto AWS ECR.
-Dockerfile Build Script: build-and-push.sh>}>
+
```bash Command
#!/bin/bash
@@ -228,7 +285,23 @@ echo "Build and push completed successfully!"
-## Common Notes
+
+
+
+
+## Troubleshooting
+
+### CUDA_HOME
+
+If installation fails with `OSError: CUDA_HOME environment variable is not set`, set `CUDA_HOME` to your CUDA Toolkit installation directory. Replace the path below with the actual path on your machine:
+
+```bash
+export CUDA_HOME="/path/to/cuda"
+```
+
+For a versioned installation, this is typically `/usr/local/cuda-VERSION`. Alternatively, install FlashInfer first using the [FlashInfer installation instructions](https://docs.flashinfer.ai/installation.html), then retry the SGLang installation.
+
+### FlashInfer
- [FlashInfer](https://github.com/flashinfer-ai/flashinfer) is the default attention kernel backend. It only supports sm75 and above. If you encounter any FlashInfer-related issues on sm75+ devices (e.g., T4, A10, A100, L4, L40S, H100), please switch to other kernels by adding `--attention-backend triton --sampling-backend pytorch` and open an issue on GitHub.
- To reinstall flashinfer locally, use the following command: `pip3 install --upgrade flashinfer-python --force-reinstall --no-deps` and then delete the cache with `rm -rf ~/.cache/flashinfer`.
diff --git a/docs/docs/get-started/quickstart.mdx b/docs/docs/get-started/quickstart.mdx
index 57773eaed4..0837c34547 100644
--- a/docs/docs/get-started/quickstart.mdx
+++ b/docs/docs/get-started/quickstart.mdx
@@ -1,332 +1,67 @@
---
title: "Quickstart"
-description: "Get up and running with SGLang in minutes: install, launch a server, and send your first request."
+description: "Start a model server and verify it with one request."
---
-## Overview
+## Install and start
-This guide walks you through the entire flow of getting started with SGLang:
-
-1. **Install** SGLang
-2. **Launch** an inference server
-3. **Send requests** using cURL, OpenAI Python client, Python `requests`, or the native SGLang API
-
-By the end, you'll have a working SGLang server responding to your prompts.
-
----
-
-## Prerequisites
-
-- **Python**: 3.10 or higher
-- **GPU**: NVIDIA GPU with CUDA support (sm80 and above, e.g., A10, A100, L4, L40S, H100)
-- **OS**: Linux (recommended)
-
-
-For other platforms, see the dedicated guides for [AMD GPUs](../hardware-platforms/amd_gpu), [Intel Xeon CPUs](../hardware-platforms/cpu_server), [Google TPUs](../hardware-platforms/tpu), [NVIDIA Jetson](../hardware-platforms/nvidia_jetson), [Ascend NPUs](../hardware-platforms/ascend-npus/getting-started/installation), and [Intel XPU](../hardware-platforms/xpu).
-
-
----
-
-## Installation
+These commands target Linux with an NVIDIA GPU and a CUDA 13-compatible driver. See [Installation](/docs/get-started/install) for environment requirements and other setup options.
-
- We recommend using **uv** for faster installation:
+
+
+With Docker and NVIDIA Container Toolkit configured, pull the image. Skip this command if you have already pulled it:
```bash
-pip install --upgrade pip
-pip install uv
+docker pull lmsysorg/sglang:latest
+```
+
+Start the server on the GPU host. Replace `MODEL_PATH` with your Hugging Face model ID (for example, `Qwen/Qwen3-0.6B`):
+
+```bash
+docker run --rm --gpus all --ipc=host \
+ -p 30000:30000 \
+ lmsysorg/sglang:latest \
+ sglang serve MODEL_PATH --host 0.0.0.0 --port 30000
+```
+
+
+
+
+With uv installed, create a Python environment and install SGLang. Skip this block if you already have an environment with SGLang installed:
+
+```bash
+uv venv --python 3.12
+source .venv/bin/activate
uv pip install --prerelease=allow sglang
```
-
-
+
+Start the server in that environment. Replace `MODEL_PATH` with your Hugging Face model ID (for example, `Qwen/Qwen3-0.6B`):
+
```bash
-# Clone and install from source
-git clone https://github.com/sgl-project/sglang.git
-cd sglang
-pip install --upgrade pip
-pip install -e "python"
+sglang serve MODEL_PATH --host 0.0.0.0 --port 30000
```
-
-
- The Docker images are available on Docker Hub at [lmsysorg/sglang](https://hub.docker.com/r/lmsysorg/sglang/tags).
- Replace `` with your [Hugging Face token](https://huggingface.co/docs/hub/en/security-tokens):
-
- ```bash
- docker run --gpus all \
- --shm-size 32g \
- -p 30000:30000 \
- -v ~/.cache/huggingface:/root/.cache/huggingface \
- --env "HF_TOKEN=" \
- --ipc=host \
- lmsysorg/sglang:latest \
- sglang serve --model-path meta-llama/Llama-3.1-8B-Instruct --host 0.0.0.0 --port 30000
- ```
-
- For production deployments, use the smaller **runtime** variant (~40% size reduction):
-
- ```bash
- docker run --gpus all \
- --shm-size 32g \
- -p 30000:30000 \
- -v ~/.cache/huggingface:/root/.cache/huggingface \
- --env "HF_TOKEN=" \
- --ipc=host \
- lmsysorg/sglang:latest-runtime \
- sglang serve --model-path meta-llama/Llama-3.1-8B-Instruct --host 0.0.0.0 --port 30000
- ```
-
+
-
-If you encounter `OSError: CUDA_HOME environment variable is not set`, set it with:
-```bash
-export CUDA_HOME=/usr/local/cuda-
-```
-
+For additional model-specific launch arguments, see the [Cookbook](/cookbook/intro). Wait for `The server is fired up and ready to roll!`.
----
+## Send a request
-## Launch a Server
-
-Start the SGLang server with a model. Here we use `qwen/qwen2.5-0.5b-instruct` as a lightweight example:
-
-```bash
-sglang serve --model-path qwen/qwen2.5-0.5b-instruct --host 0.0.0.0 --port 30000
-```
-
-Wait until you see `The server is fired up and ready to roll!` in the terminal output.
-
-
-Once the server is running, API documentation is available at:
-- **Swagger UI**: `http://localhost:30000/docs`
-- **ReDoc**: `http://localhost:30000/redoc`
-- **OpenAPI Spec**: `http://localhost:30000/openapi.json`
-
-
-
-The server automatically applies the chat template from the Hugging Face tokenizer. You can override it with `--chat-template` when launching.
-
-
----
-
-## Send Requests
-
-SGLang is fully **OpenAI API-compatible**, so you can use the same tools and libraries you already know.
-
-### Using cURL
+In another terminal on the same host, send a request to verify the server. Use the same model ID for `MODEL_PATH`:
```bash
curl http://localhost:30000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
- "model": "qwen/qwen2.5-0.5b-instruct",
+ "model": "MODEL_PATH",
"messages": [
{"role": "user", "content": "What is the capital of France?"}
]
}'
```
-### Using OpenAI Python Client
+Check `choices[0].message.content` for the answer. Press `Ctrl+C` in the server terminal to stop.
-Install the OpenAI Python library if you haven't:
-
-```bash
-pip install openai
-```
-
-Then send a request:
-
-```python Example
-import openai
-
-client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="None")
-
-response = client.chat.completions.create(
- model="qwen/qwen2.5-0.5b-instruct",
- messages=[
- {"role": "user", "content": "List 3 countries and their capitals."},
- ],
- temperature=0,
- max_tokens=64,
-)
-
-print(response.choices[0].message.content)
-```
-
-#### Streaming
-
-```python Example
-import openai
-
-client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="None")
-
-response = client.chat.completions.create(
- model="qwen/qwen2.5-0.5b-instruct",
- messages=[
- {"role": "user", "content": "List 3 countries and their capitals."},
- ],
- temperature=0,
- max_tokens=64,
- stream=True,
-)
-
-for chunk in response:
- if chunk.choices[0].delta.content:
- print(chunk.choices[0].delta.content, end="", flush=True)
-```
-
-### Using Python Requests
-
-```python Example
-import requests
-
-url = "http://localhost:30000/v1/chat/completions"
-
-data = {
- "model": "qwen/qwen2.5-0.5b-instruct",
- "messages": [{"role": "user", "content": "What is the capital of France?"}],
-}
-
-response = requests.post(url, json=data)
-print(response.json())
-```
-
-### Using the Native `/generate` API
-
-SGLang also provides a native `/generate` endpoint for more flexibility.
-
-```python Example
-import requests
-
-response = requests.post(
- "http://localhost:30000/generate",
- json={
- "text": "The capital of France is",
- "sampling_params": {
- "temperature": 0,
- "max_new_tokens": 32,
- },
- },
-)
-
-print(response.json())
-```
-
-#### Streaming with `/generate`
-
-```python Example
-import requests
-import json
-
-response = requests.post(
- "http://localhost:30000/generate",
- json={
- "text": "The capital of France is",
- "sampling_params": {
- "temperature": 0,
- "max_new_tokens": 32,
- },
- "stream": True,
- },
- stream=True,
-)
-
-prev = 0
-for chunk in response.iter_lines(decode_unicode=False):
- chunk = chunk.decode("utf-8")
- if chunk and chunk.startswith("data:"):
- if chunk == "data: [DONE]":
- break
- data = json.loads(chunk[5:].strip("\n"))
- output = data["text"]
- print(output[prev:], end="", flush=True)
- prev = len(output)
-```
-
----
-
-## Offline Batch Inference (No Server)
-
-SGLang also supports offline batch inference using the `Engine` class directly -- no HTTP server required.
-
-```python Example
-import sglang as sgl
-
-llm = sgl.Engine(model_path="qwen/qwen2.5-0.5b-instruct")
-
-prompts = [
- "Hello, my name is",
- "The president of the United States is",
- "The capital of France is",
- "The future of AI is",
-]
-
-sampling_params = {"temperature": 0.8, "top_p": 0.95}
-
-outputs = llm.generate(prompts, sampling_params)
-
-for prompt, output in zip(prompts, outputs):
- print(f"Prompt: {prompt}\nGenerated text: {output['text']}\n")
-
-llm.shutdown()
-```
-
----
-
-## Common Troubleshooting
-
-
-
- Set the `CUDA_HOME` environment variable to your CUDA install root:
- ```bash
- export CUDA_HOME=/usr/local/cuda-
- ```
-
-
- Switch to alternative backends by adding these flags when launching the server:
- ```bash
- --attention-backend triton --sampling-backend pytorch
- ```
-
-
- ```bash
- pip3 install --upgrade flashinfer-python --force-reinstall --no-deps
- rm -rf ~/.cache/flashinfer
- ```
-
-
- ```bash
- export TRITON_PTXAS_PATH=/usr/local/cuda/bin/ptxas
- ```
-
-
-
----
-
-{/*
-WIP, TBD linked later
-## What's Next?
-
-
-
- Explore the full Chat Completions and Completions APIs, including multi-turn conversations.
-
-
- Send image inputs alongside text using OpenAI-compatible vision APIs.
-
-
- Fine-tune generation with temperature, top-p, frequency penalty, and more.
-
-
- Customize server behavior with advanced launch arguments like tensor parallelism.
-
-
- Constrain model output to JSON, regex, or EBNF grammars.
-
-
- Use the familiar Ollama CLI and Python library with SGLang as the backend.
-
-
-*/}
+See [Examples](/docs/get-started/examples) for a complete 27B deployment, streaming, and tool calls.
diff --git a/docs/index.mdx b/docs/index.mdx
index e871f613c9..741f7ffc40 100644
--- a/docs/index.mdx
+++ b/docs/index.mdx
@@ -68,15 +68,17 @@ SGLang is an inference framework meant for production level serving.
It is designed to deliver low-latency and high-throughput inference across a wide range of setups, from a single GPU to large distributed clusters.
-
- Install SGLang with pip, from source, or via Docker on your preferred hardware platform.
+
+ Start a model server and send a request to verify it is running.
-
- Launch your first model server and send requests in minutes with OpenAI-compatible APIs.
+
+ Stream replies, request tool calls, and process multiple prompts from Python.
+For environment setup and other installation methods, see the [Installation guide](/docs/get-started/install).
+
## News and latest blogs
{/* BEGIN_LMSYS_SGLANG_BLOG_CARDS */}
@@ -425,80 +427,10 @@ It is designed to deliver low-latency and high-throughput inference across a wid
---
-## Learn more and join the community
+## Community and resources
-
-
-
- Stay connected
-
-
-
-
-
-
-
- {" "}
-
Slack
-
for questions, feedback, and community support.
-
-
-
-
-
- {" "}
-
LMSYS blog
-
for release notes, benchmarks, and technical deep dives.
-
-
-
-
-
+- [Slack](https://slack.sglang.io/): Technical questions and development discussions.
+- Events: [Meetups, workshops, and office hours](https://www.sglang.io/events) | [Developer meetings](https://meet.sglang.io/).
+- Updates: [X](https://x.com/lmsysorg) | [LinkedIn](https://www.linkedin.com/company/sgl-project/) | [LMSYS Blog](https://lmsys.org/blog/).
+- Contribute: [Contributor guide](/docs/developer_guide/contribution_guide) | [Roadmap](https://roadmap.sglang.io/).
+- [Release notes](https://github.com/sgl-project/sglang/releases): Changes in each release.