Readme refresh (#41787)
Co-authored-by: zijiexia <37504505+zijiexia@users.noreply.github.com> Co-authored-by: Zijie Xia <zijie.xia@radixark.ai>
This commit is contained in:
co-authored by
zijiexia
Zijie Xia
parent
eb9c9ee99d
commit
24ee67d2fa
@@ -8,7 +8,7 @@ Deploy SGLang on [Amazon SageMaker AI](https://aws.amazon.com/sagemaker/) endpoi
|
||||
The SageMaker image variant accepts model configuration via environment variables and serves on port 8080.
|
||||
|
||||
This guide uses the pre-built DLC image. To build and deploy your own container instead, see
|
||||
[Method 7: Run on AWS SageMaker](/docs/get-started/install#more-3) in the installation guide.
|
||||
[Method 7: Run on AWS SageMaker](/docs/get-started/install#aws-sagemaker) in the installation guide.
|
||||
|
||||
## Container image
|
||||
|
||||
|
||||
@@ -4,6 +4,9 @@ sidebarTitle: "Using Docker"
|
||||
metatags:
|
||||
description: "SGLang Docker development: VSCode dev container, remote tunnels, debugger setup, nsys profiling."
|
||||
---
|
||||
|
||||
The `lmsysorg/sglang:dev` image is a nightly build from `main` with additional development tools and dependencies for modifying and testing SGLang. For running models without a development environment, use the stable or nightly images described in the [Installation guide](/docs/get-started/install).
|
||||
|
||||
## Setup VSCode on a Remote Host
|
||||
(Optional - you can skip this step if you plan to run sglang dev container locally)
|
||||
|
||||
|
||||
@@ -0,0 +1,279 @@
|
||||
---
|
||||
title: "Examples"
|
||||
description: "Run a model locally, add chat to an application, process a batch of texts, or build a tool-using assistant."
|
||||
---
|
||||
|
||||
Choose an example for what you want to do:
|
||||
|
||||
| I want to… | Example |
|
||||
| --- | --- |
|
||||
| Run a model on my own computer | [Local inference](#run-a-model-locally) on an NVIDIA GPU or Apple Silicon Mac. |
|
||||
| Serve a larger model on a GPU server | [Deploy Qwen3.8-27B](#serve-qwen3-8-27b) on an NVIDIA H200. |
|
||||
| Add chat to an application | [Stream a reply](#stream-a-reply) through the OpenAI-compatible API. |
|
||||
| Process a collection of documents | [Summarize multiple texts](#process-multiple-prompts) concurrently. |
|
||||
| Build an agent that uses tools | [Look up an order](#build-a-customer-support-agent) and answer a customer’s question. |
|
||||
|
||||
## Run a model locally
|
||||
|
||||
Run `Qwen/Qwen3-0.6B` on your own computer and ask it a question. This small model is useful for trying the workflow; choose a larger model for more demanding tasks.
|
||||
|
||||
<Tabs>
|
||||
<Tab title="NVIDIA GPU (Linux)">
|
||||
|
||||
This example uses one CUDA 13-compatible NVIDIA GPU, Linux, Docker, and NVIDIA Container Toolkit. For setup, see [Installation](/docs/get-started/install).
|
||||
|
||||
Run this command on your computer. Docker pulls the image if needed, and SGLang downloads the model on first use:
|
||||
|
||||
```bash
|
||||
docker run --rm --gpus all --ipc=host \
|
||||
-p 127.0.0.1:30000:30000 \
|
||||
lmsysorg/sglang:latest \
|
||||
sglang serve Qwen/Qwen3-0.6B \
|
||||
--host 0.0.0.0 --port 30000
|
||||
```
|
||||
|
||||
If you already installed SGLang with uv, run this in that Python environment instead:
|
||||
|
||||
```bash
|
||||
sglang serve Qwen/Qwen3-0.6B --host 127.0.0.1 --port 30000
|
||||
```
|
||||
|
||||
</Tab>
|
||||
<Tab title="Apple Silicon Mac">
|
||||
|
||||
On an Apple Silicon Mac with macOS 14 or newer, first follow the [Apple Silicon installation instructions](/docs/hardware-platforms/apple_metal#install-sglang) to install SGLang with its MLX dependencies. Use that Python environment to launch the model with 4-bit quantization:
|
||||
|
||||
```bash
|
||||
SGLANG_USE_MLX=1 python -m sglang.launch_server \
|
||||
--model-path Qwen/Qwen3-0.6B \
|
||||
--quantization mlx_q4 \
|
||||
--disable-cuda-graph \
|
||||
--host 127.0.0.1 --port 30000
|
||||
```
|
||||
|
||||
The model runs on your Mac through MLX. It downloads the weights on first use and quantizes them in memory when loading.
|
||||
|
||||
</Tab>
|
||||
</Tabs>
|
||||
|
||||
Keep the server running. Once you see `The server is fired up and ready to roll!`, open another terminal on the same computer and send a request:
|
||||
|
||||
```bash
|
||||
curl http://localhost:30000/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "Qwen/Qwen3-0.6B",
|
||||
"messages": [{"role": "user", "content": "What is the capital of France? /no_think"}],
|
||||
"max_tokens": 128
|
||||
}'
|
||||
```
|
||||
|
||||
Look for the answer in `choices[0].message.content`. Both the model and the API server are running locally. Press `Ctrl+C` in the server terminal when you are done, or leave it running for the chat and batch examples below.
|
||||
|
||||
<a id="serve-qwen3-8-27b" />
|
||||
|
||||
## Serve Qwen3.8-27B
|
||||
|
||||
This configuration runs `Qwen/Qwen3.8-27B-FP8` on one NVIDIA H200. It requires Linux, a CUDA 13-compatible driver, Docker, and NVIDIA Container Toolkit configured for GPU access. For environment setup, see [Installation](/docs/get-started/install). For other hardware configurations, see the [Qwen3.8-27B Cookbook](/cookbook/autoregressive/Qwen/Qwen3.8-27B).
|
||||
|
||||
Set `HF_CACHE_DIR` to an absolute directory on the host for downloaded model weights, replacing the placeholder below. Run this command on the GPU host:
|
||||
|
||||
```bash
|
||||
HF_CACHE_DIR="/path/to/huggingface-cache"
|
||||
docker run --rm --gpus all \
|
||||
--shm-size 32g \
|
||||
-p 30000:30000 \
|
||||
-v "$HF_CACHE_DIR:/root/.cache/huggingface" \
|
||||
--ipc=host \
|
||||
lmsysorg/sglang:latest \
|
||||
sglang serve \
|
||||
--trust-remote-code \
|
||||
--model-path Qwen/Qwen3.8-27B-FP8 \
|
||||
--kv-cache-dtype fp8_e4m3 \
|
||||
--mem-fraction-static 0.85 \
|
||||
--attention-backend flashinfer \
|
||||
--chunked-prefill-size 32768 \
|
||||
--max-prefill-tokens 32768 \
|
||||
--reasoning-parser qwen3 \
|
||||
--tool-call-parser qwen3_coder \
|
||||
--mamba-full-memory-ratio 4.59 \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000 \
|
||||
--mamba-radix-cache-strategy extra_buffer \
|
||||
--mamba-ssm-dtype float32
|
||||
```
|
||||
|
||||
Docker downloads the image if needed. SGLang downloads the model on first use and starts the server. Keep this terminal open and wait for `The server is fired up and ready to roll!` before sending a request.
|
||||
|
||||
<Accordion title="Already installed with uv?">
|
||||
|
||||
Run this command in the activated Python environment where you installed SGLang. It starts the same server without Docker:
|
||||
|
||||
```bash
|
||||
sglang serve \
|
||||
--trust-remote-code \
|
||||
--model-path Qwen/Qwen3.8-27B-FP8 \
|
||||
--kv-cache-dtype fp8_e4m3 \
|
||||
--mem-fraction-static 0.85 \
|
||||
--attention-backend flashinfer \
|
||||
--chunked-prefill-size 32768 \
|
||||
--max-prefill-tokens 32768 \
|
||||
--reasoning-parser qwen3 \
|
||||
--tool-call-parser qwen3_coder \
|
||||
--mamba-full-memory-ratio 4.59 \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000 \
|
||||
--mamba-radix-cache-strategy extra_buffer \
|
||||
--mamba-ssm-dtype float32
|
||||
```
|
||||
|
||||
For Python environment setup, see [Install with uv](/docs/get-started/install#method-1-with-pip-or-uv).
|
||||
|
||||
</Accordion>
|
||||
|
||||
Verify the running server with the [Quickstart request](/docs/get-started/quickstart#send-a-request), using `Qwen/Qwen3.8-27B-FP8` as the model name.
|
||||
|
||||
## Connect to a model
|
||||
|
||||
The following Python examples connect to the local server above. You can also use a remote SGLang server; only the server needs hardware that can run the model.
|
||||
|
||||
Install the Python client in your activated environment:
|
||||
|
||||
```bash
|
||||
uv pip install openai
|
||||
```
|
||||
|
||||
Run this setup before each example in the same Python session:
|
||||
|
||||
```python
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
base_url="http://localhost:30000/v1",
|
||||
api_key="EMPTY",
|
||||
)
|
||||
model = "Qwen/Qwen3-0.6B"
|
||||
```
|
||||
|
||||
These settings match the local Qwen3-0.6B server above. For a remote server, replace `localhost` with its reachable address, use its model name, and supply an API key if authentication is enabled.
|
||||
|
||||
## Stream a reply
|
||||
|
||||
Ask a question and print the answer as it arrives:
|
||||
|
||||
```python
|
||||
stream = client.chat.completions.create(
|
||||
model=model,
|
||||
messages=[{"role": "user", "content": "Explain prefix caching in two sentences."}],
|
||||
stream=True,
|
||||
)
|
||||
|
||||
for chunk in stream:
|
||||
if chunk.choices:
|
||||
print(chunk.choices[0].delta.content or "", end="", flush=True)
|
||||
print()
|
||||
```
|
||||
|
||||
The terminal displays the model's explanation incrementally. You can use the same stream to update a chat interface.
|
||||
|
||||
See the [API guide](/docs/basic_usage/openai_api_completions) for more request options.
|
||||
|
||||
## Process multiple prompts
|
||||
|
||||
Send several independent requests concurrently, for example to summarize customer feedback:
|
||||
|
||||
```python
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
|
||||
feedback = [
|
||||
"Setup was quick, but I could not find the export button.",
|
||||
"The new search is faster and finds the documents I need.",
|
||||
"Please add a way to share saved reports with my team.",
|
||||
]
|
||||
|
||||
def summarize(text):
|
||||
response = client.chat.completions.create(
|
||||
model=model,
|
||||
messages=[{"role": "user", "content": f"Summarize in one sentence: {text}"}],
|
||||
)
|
||||
return response.choices[0].message.content
|
||||
|
||||
with ThreadPoolExecutor(max_workers=3) as pool:
|
||||
for summary in pool.map(summarize, feedback):
|
||||
print(summary)
|
||||
```
|
||||
|
||||
You get one summary per input, in the original order. For batch processing inside a Python process without an HTTP server, see the [offline engine API](/docs/basic_usage/offline_engine_api).
|
||||
|
||||
<a id="request-a-tool-call" />
|
||||
|
||||
## Build a customer support agent
|
||||
|
||||
Let an agent answer “Where is my order?” by looking up order data. The model chooses a tool, your application executes it, and the model uses the returned data to answer the customer.
|
||||
|
||||
Use the [Qwen3.8-27B server above](#serve-qwen3-8-27b), which enables a tool parser, or configure another model with the [tool-calling guide](/docs/advanced_features/tool_parser). The small local example does not configure tool calling. This script uses the OpenAI client installed above; update the endpoint and model ID to match your server.
|
||||
|
||||
Save this as `order_agent.py` and run `python order_agent.py`. The order data is a local fixture, so no external service or API key is needed for the lookup.
|
||||
|
||||
```python
|
||||
import json
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(base_url="http://localhost:30000/v1", api_key="EMPTY")
|
||||
model = "Qwen/Qwen3.8-27B-FP8"
|
||||
|
||||
orders = {
|
||||
"SGL-1042": {"status": "shipped", "estimated_delivery": "Friday"},
|
||||
}
|
||||
|
||||
def lookup_order(order_id):
|
||||
return orders.get(order_id, {"error": "Order not found"})
|
||||
|
||||
tools = [{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "lookup_order",
|
||||
"description": "Look up an order's shipping status and estimated delivery.",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {"order_id": {"type": "string"}},
|
||||
"required": ["order_id"],
|
||||
"additionalProperties": False,
|
||||
},
|
||||
},
|
||||
}]
|
||||
|
||||
messages = [
|
||||
{"role": "system", "content": "Use lookup_order for order status. Do not invent order details."},
|
||||
{"role": "user", "content": "Where is order SGL-1042, and when should it arrive?"},
|
||||
]
|
||||
|
||||
for _ in range(3):
|
||||
response = client.chat.completions.create(
|
||||
model=model, messages=messages, tools=tools, tool_choice="auto",
|
||||
)
|
||||
message = response.choices[0].message
|
||||
messages.append(message.model_dump(exclude_none=True))
|
||||
if not message.tool_calls:
|
||||
print(message.content)
|
||||
break
|
||||
|
||||
for call in message.tool_calls:
|
||||
try:
|
||||
if call.function.name != "lookup_order":
|
||||
raise ValueError("Unknown tool")
|
||||
arguments = json.loads(call.function.arguments)
|
||||
result = lookup_order(**arguments)
|
||||
except (ValueError, TypeError) as error:
|
||||
result = {"error": str(error)}
|
||||
print(f"{call.function.name}: {result}")
|
||||
messages.append({
|
||||
"role": "tool",
|
||||
"tool_call_id": call.id,
|
||||
"content": json.dumps(result),
|
||||
})
|
||||
else:
|
||||
print("Stopped after three model turns without a final answer.")
|
||||
```
|
||||
|
||||
You should see the lookup result followed by an answer that the order has shipped and is expected on Friday. The wording can vary. SGLang serves the model; the Python application runs the tools and manages the conversation. Replace `lookup_order` with an authorized order-system query to use real data.
|
||||
@@ -7,109 +7,166 @@ keywords:
|
||||
- pip
|
||||
- docker
|
||||
---
|
||||
You can install SGLang using one of the methods below.
|
||||
This page primarily applies to common NVIDIA GPU platforms.
|
||||
For other or newer platforms, please refer to the dedicated pages for [AMD GPUs](../hardware-platforms/amd_gpu), [Apple Metal](../hardware-platforms/apple_metal), [Intel Xeon CPUs](../hardware-platforms/cpu_server), [Google TPU](../hardware-platforms/tpu), [NVIDIA DGX Spark](https://lmsys.org/blog/2025-11-03-gpt-oss-on-nvidia-dgx-spark/), [NVIDIA Jetson](../hardware-platforms/nvidia_jetson), [Ascend NPUs](../hardware-platforms/ascend-npus/getting-started/installation), and [Intel XPU](../hardware-platforms/xpu).
|
||||
Choose one installation method below. These instructions target Linux with NVIDIA GPUs; for other devices, see [Hardware platforms](/docs/hardware-platforms/overview).
|
||||
|
||||
<Note>
|
||||
Prerequisites: Python 3.10 or higher.
|
||||
</Note>
|
||||
## Install SGLang
|
||||
|
||||
## Method 1: With pip or uv
|
||||
<Tabs>
|
||||
<Tab title="Docker" id="method-3-using-docker">
|
||||
|
||||
It is recommended to use uv for faster installation:
|
||||
Docker includes Python, SGLang, and its dependencies; no separate host Python installation is required.
|
||||
|
||||
```bash Command
|
||||
The docker images are available on Docker Hub at [lmsysorg/sglang](https://hub.docker.com/r/lmsysorg/sglang/tags), built from [Dockerfile](https://github.com/sgl-project/sglang/tree/main/docker).
|
||||
Use Linux with a supported NVIDIA GPU, a CUDA 13-compatible driver, [Docker Engine](https://docs.docker.com/engine/install/), and [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html). Complete the Toolkit guide's Docker runtime configuration before starting a GPU container.
|
||||
|
||||
SGLang images ship a CUDA 13 environment. CUDA 12 images (`-cu12` / `-cu129`) are no longer published; `lmsysorg/sglang:v0.5.19-cu129` is the last CUDA 12 tag.
|
||||
|
||||
**Pull the image**
|
||||
|
||||
`latest` contains the newest stable release:
|
||||
|
||||
|
||||
```bash
|
||||
docker pull lmsysorg/sglang:latest
|
||||
```
|
||||
|
||||
**Image options**
|
||||
|
||||
The examples use `latest`. To use another image, replace the tag in the pull and run commands:
|
||||
|
||||
- Nightly: Built daily from the latest `main`, including changes not yet in a stable release. Choose a tag from the [nightly image list](https://hub.docker.com/r/lmsysorg/sglang/tags?name=nightly).
|
||||
- Runtime: A smaller image of the same stable release, with inference dependencies and fewer development utilities. Use the `latest-runtime` tag.
|
||||
|
||||
For reproducible deployments, use a specific release or dated nightly tag from [Docker Hub](https://hub.docker.com/r/lmsysorg/sglang/tags).
|
||||
|
||||
**Start a container**
|
||||
|
||||
Set `HF_CACHE_DIR` to an absolute directory path on the host for downloaded model weights, replacing the placeholder below.
|
||||
|
||||
For gated or private models, export your [Hugging Face token](https://huggingface.co/docs/hub/en/security-tokens) as `HF_TOKEN` on the host and add `--env HF_TOKEN` before the image name in this command.
|
||||
|
||||
|
||||
```bash
|
||||
HF_CACHE_DIR="/path/to/huggingface-cache"
|
||||
docker run -it --gpus all \
|
||||
--shm-size 32g \
|
||||
-p 30000:30000 \
|
||||
-v "$HF_CACHE_DIR:/root/.cache/huggingface" \
|
||||
--ipc=host \
|
||||
lmsysorg/sglang:latest bash
|
||||
```
|
||||
|
||||
The container shell is ready for `sglang serve`.
|
||||
|
||||
</Tab>
|
||||
<Tab title="pip / uv" id="method-1-with-pip-or-uv">
|
||||
|
||||
Use Python 3.10 or higher and a compatible CUDA 13 environment.
|
||||
|
||||
<Accordion title="CUDA version compatibility">
|
||||
|
||||
SGLang requires CUDA 13. CUDA 12 (`cu129`) wheels and images are retired because PyTorch 2.14 publishes no CUDA 12.9 builds. SGLang 0.5.19 is the last release with a CUDA 12 lane.
|
||||
|
||||
</Accordion>
|
||||
|
||||
**Prepare the environment**
|
||||
|
||||
Use uv for faster installation. Install it with pip, or follow the [uv installation instructions](https://docs.astral.sh/uv/getting-started/installation/):
|
||||
|
||||
```bash
|
||||
pip install --upgrade pip
|
||||
pip install uv
|
||||
```
|
||||
|
||||
Create and activate a virtual environment:
|
||||
|
||||
```bash Command
|
||||
uv venv --python 3.12
|
||||
source .venv/bin/activate
|
||||
```
|
||||
|
||||
**Install SGLang**
|
||||
|
||||
```bash
|
||||
uv pip install --prerelease=allow sglang
|
||||
```
|
||||
|
||||
<Note>
|
||||
<Accordion title="Why --prerelease=allow?">
|
||||
|
||||
Some of SGLang's dependencies only publish pre-releases on PyPI, so without `--prerelease=allow` uv older than 0.12.0 silently installs SGLang 0.5.9. On [uv 0.12.0](https://github.com/astral-sh/uv/releases/tag/0.12.0) and newer the flag is a harmless no-op.
|
||||
</Note>
|
||||
</Accordion>
|
||||
|
||||
<Note>
|
||||
SGLang requires CUDA 13. The CUDA 12 (`cu129`) wheels and images are retired,
|
||||
because PyTorch 2.14 publishes no CUDA 12.9 builds. SGLang 0.5.19 is the last
|
||||
release with a CUDA 12 lane.
|
||||
</Note>
|
||||
|
||||
### Nightly builds
|
||||
<Accordion title="Nightly builds" id="nightly-builds">
|
||||
|
||||
To pick up the latest features and fixes before the next stable release, install a nightly build. Nightly wheels are built from the latest `main` and published to the SGLang wheel index. Add that index with `--extra-index-url`, and combine `--prerelease=allow` with `--index-strategy unsafe-best-match` so uv considers the nightly (pre-release) version alongside PyPI:
|
||||
|
||||
```bash Command
|
||||
pip install --upgrade pip
|
||||
pip install uv
|
||||
uv pip install --prerelease=allow --index-strategy unsafe-best-match --extra-index-url https://docs.sglang.ai/whl/cu130/ sglang
|
||||
```
|
||||
|
||||
### Quick fixes to common problems
|
||||
- If you encounter `OSError: CUDA_HOME environment variable is not set`. Please set it to your CUDA install root with either of the following solutions:
|
||||
1. Use `export CUDA_HOME=/usr/local/cuda-<your-cuda-version>` to set the `CUDA_HOME` environment variable.
|
||||
2. Install FlashInfer first following [FlashInfer installation doc](https://docs.flashinfer.ai/installation.html), then install SGLang as described above.
|
||||
</Accordion>
|
||||
|
||||
## Method 2: From source
|
||||
For `CUDA_HOME` errors, see [Troubleshooting](#cuda_home).
|
||||
|
||||
```bash Command
|
||||
# Use the last release branch
|
||||
</Tab>
|
||||
<Tab title="From source" id="method-2-from-source">
|
||||
|
||||
Use Python 3.10 or higher in an activated Python environment, with CUDA dependencies appropriate for the version you build.
|
||||
|
||||
**Get the source**
|
||||
|
||||
Clone a release tag. This example uses `v0.5.18`; choose another tag from [GitHub releases](https://github.com/sgl-project/sglang/releases) as needed:
|
||||
|
||||
```bash
|
||||
git clone -b v0.5.18 https://github.com/sgl-project/sglang.git
|
||||
cd sglang
|
||||
```
|
||||
|
||||
# Install the python packages
|
||||
**Install SGLang**
|
||||
|
||||
Install the Python package in editable mode so local source changes are reflected in your environment:
|
||||
|
||||
```bash
|
||||
pip install --upgrade pip
|
||||
pip install -e "python"
|
||||
```
|
||||
|
||||
**Quick fixes to common problems**
|
||||
<Accordion title="Development environment">
|
||||
|
||||
- If you want to develop SGLang, you can try the dev docker image. Please refer to [setup docker container](../developer_guide/development_guide_using_docker#setup-docker-container). The docker image is `lmsysorg/sglang:dev`.
|
||||
For contributing to SGLang, follow the [development environment guide](/docs/developer_guide/development_guide_using_docker#setup-docker-container), which covers a container with development tools and dependencies installed.
|
||||
|
||||
## Method 3: Using docker
|
||||
</Accordion>
|
||||
|
||||
The docker images are available on Docker Hub at [lmsysorg/sglang](https://hub.docker.com/r/lmsysorg/sglang/tags), built from [Dockerfile](https://github.com/sgl-project/sglang/tree/main/docker).
|
||||
Replace `<secret>` below with your huggingface hub [token](https://huggingface.co/docs/hub/en/security-tokens).
|
||||
For `CUDA_HOME` errors, see [Troubleshooting](#cuda_home).
|
||||
|
||||
<Note>
|
||||
`latest` and `dev` are **mutable** tags: `latest` always points at the newest stable release, while `dev` is rebuilt daily from the latest `main` and includes build/development tools. Because they are overwritten over time, pin an immutable version tag for reproducible deployments — e.g. `lmsysorg/sglang:v0.5.18`. Browse all released versions on [Docker Hub](https://hub.docker.com/r/lmsysorg/sglang/tags).
|
||||
</Note>
|
||||
</Tab>
|
||||
</Tabs>
|
||||
|
||||
```bash Command
|
||||
docker run --gpus all \
|
||||
--shm-size 32g \
|
||||
-p 30000:30000 \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
--env "HF_TOKEN=<secret>" \
|
||||
--ipc=host \
|
||||
lmsysorg/sglang:latest \
|
||||
python3 -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct --host 0.0.0.0 --port 30000
|
||||
## Start a model server
|
||||
|
||||
Start a model server inside your Docker container or activated Python environment. Replace `MODEL_PATH` with a Hugging Face model ID or a local model directory:
|
||||
|
||||
```bash
|
||||
sglang serve MODEL_PATH --host 0.0.0.0 --port 30000
|
||||
```
|
||||
|
||||
For production deployments, use the `runtime` variant which is significantly smaller (~40% reduction) by excluding build tools and development dependencies:
|
||||
Additional launch arguments depend on the model and your hardware.
|
||||
|
||||
```bash Command
|
||||
docker run --gpus all \
|
||||
--shm-size 32g \
|
||||
-p 30000:30000 \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
--env "HF_TOKEN=<secret>" \
|
||||
--ipc=host \
|
||||
lmsysorg/sglang:latest-runtime \
|
||||
python3 -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct --host 0.0.0.0 --port 30000
|
||||
```
|
||||
- [Quickstart](/docs/get-started/quickstart): Launch a model and send a request to verify the server.
|
||||
- [Cookbook](/cookbook/intro): Find model-specific launch arguments for your hardware.
|
||||
|
||||
You can also find the nightly docker images [here](https://hub.docker.com/r/lmsysorg/sglang/tags?name=nightly).
|
||||
## Deployment options
|
||||
|
||||
Notes:
|
||||
- SGLang images ship a CUDA 13 environment. The `-cu12` / `-cu129` images are no longer published; `lmsysorg/sglang:v0.5.19-cu129` is the last CUDA 12 tag.
|
||||
Choose how to run SGLang as a service. Expand an option for setup instructions.
|
||||
|
||||
## Method 4: Using Kubernetes
|
||||
<AccordionGroup>
|
||||
|
||||
<a id="method-4-using-kubernetes" />
|
||||
|
||||
<Accordion title="Kubernetes" id="kubernetes" description="Serve models on a Kubernetes cluster, on one node or across multiple nodes.">
|
||||
|
||||
Please check out [OME](https://github.com/sgl-project/ome), a Kubernetes operator for enterprise-grade management and serving of large language models (LLMs).
|
||||
|
||||
<Accordion title="More">
|
||||
|
||||
1. Option 1: For single node serving (typically when the model size fits into GPUs on one node)
|
||||
|
||||
Execute command `kubectl apply -f docker/k8s-sglang-service.yaml`, to create k8s deployment and service, with llama-31-8b as example.
|
||||
@@ -120,9 +177,9 @@ Please check out [OME](https://github.com/sgl-project/ome), a Kubernetes operato
|
||||
|
||||
</Accordion>
|
||||
|
||||
## Method 5: Using docker compose
|
||||
<a id="method-5-using-docker-compose" />
|
||||
|
||||
<Accordion title="More">
|
||||
<Accordion title="Docker Compose" id="docker-compose" description="Run a model service on a single Docker host with a Compose configuration.">
|
||||
|
||||
> This method is recommended if you plan to serve it as a service.
|
||||
> A better approach is to use the [k8s-sglang-service.yaml](https://github.com/sgl-project/sglang/blob/main/docker/k8s-sglang-service.yaml).
|
||||
@@ -132,16 +189,16 @@ Please check out [OME](https://github.com/sgl-project/ome), a Kubernetes operato
|
||||
|
||||
</Accordion>
|
||||
|
||||
## Method 6: Run on Kubernetes or Clouds with SkyPilot
|
||||
<a id="method-6-run-on-kubernetes-or-clouds-with-skypilot" />
|
||||
|
||||
<Accordion title="More">
|
||||
<Accordion title="SkyPilot" id="skypilot" description="Deploy on Kubernetes or across cloud providers from one YAML configuration.">
|
||||
|
||||
To deploy on Kubernetes or 12+ clouds, you can use [SkyPilot](https://github.com/skypilot-org/skypilot).
|
||||
|
||||
1. Install SkyPilot and set up Kubernetes cluster or cloud access: see [SkyPilot's documentation](https://skypilot.readthedocs.io/en/latest/getting-started/installation.html).
|
||||
2. Deploy on your own infra with a single command and get the HTTP API endpoint:
|
||||
|
||||
<Accordion title={<>SkyPilot YAML: <code>sglang.yaml</code></>}>
|
||||
<Accordion title="SkyPilot YAML: sglang.yaml">
|
||||
|
||||
```yaml Config
|
||||
# sglang.yaml
|
||||
@@ -175,9 +232,9 @@ sky status --endpoint 30000 sglang
|
||||
|
||||
</Accordion>
|
||||
|
||||
## Method 7: Run on AWS SageMaker
|
||||
<a id="method-7-run-on-aws-sagemaker" />
|
||||
|
||||
<Accordion title="More">
|
||||
<Accordion title="AWS SageMaker" id="aws-sagemaker" description="Deploy on SageMaker with a pre-built SGLang image or your own container.">
|
||||
|
||||
To deploy on SGLang on AWS SageMaker, check out [AWS SageMaker Inference](https://aws.amazon.com/sagemaker/ai/deploy)
|
||||
|
||||
@@ -190,7 +247,7 @@ To host a model with your own container, follow the following steps:
|
||||
1. Build a docker container with [sagemaker.Dockerfile](https://github.com/sgl-project/sglang/blob/main/docker/sagemaker.Dockerfile) alongside the [serve](https://github.com/sgl-project/sglang/blob/main/docker/serve) script.
|
||||
2. Push your container onto AWS ECR.
|
||||
|
||||
<Accordion title={<>Dockerfile Build Script: <code>build-and-push.sh</code></>}>
|
||||
<Accordion title="Build script: build-and-push.sh">
|
||||
|
||||
```bash Command
|
||||
#!/bin/bash
|
||||
@@ -228,7 +285,23 @@ echo "Build and push completed successfully!"
|
||||
|
||||
</Accordion>
|
||||
|
||||
## Common Notes
|
||||
</AccordionGroup>
|
||||
|
||||
<a id="common-notes" />
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### CUDA_HOME
|
||||
|
||||
If installation fails with `OSError: CUDA_HOME environment variable is not set`, set `CUDA_HOME` to your CUDA Toolkit installation directory. Replace the path below with the actual path on your machine:
|
||||
|
||||
```bash
|
||||
export CUDA_HOME="/path/to/cuda"
|
||||
```
|
||||
|
||||
For a versioned installation, this is typically `/usr/local/cuda-VERSION`. Alternatively, install FlashInfer first using the [FlashInfer installation instructions](https://docs.flashinfer.ai/installation.html), then retry the SGLang installation.
|
||||
|
||||
### FlashInfer
|
||||
|
||||
- [FlashInfer](https://github.com/flashinfer-ai/flashinfer) is the default attention kernel backend. It only supports sm75 and above. If you encounter any FlashInfer-related issues on sm75+ devices (e.g., T4, A10, A100, L4, L40S, H100), please switch to other kernels by adding `--attention-backend triton --sampling-backend pytorch` and open an issue on GitHub.
|
||||
- To reinstall flashinfer locally, use the following command: `pip3 install --upgrade flashinfer-python --force-reinstall --no-deps` and then delete the cache with `rm -rf ~/.cache/flashinfer`.
|
||||
|
||||
@@ -1,332 +1,67 @@
|
||||
---
|
||||
title: "Quickstart"
|
||||
description: "Get up and running with SGLang in minutes: install, launch a server, and send your first request."
|
||||
description: "Start a model server and verify it with one request."
|
||||
---
|
||||
|
||||
## Overview
|
||||
## Install and start
|
||||
|
||||
This guide walks you through the entire flow of getting started with SGLang:
|
||||
|
||||
1. **Install** SGLang
|
||||
2. **Launch** an inference server
|
||||
3. **Send requests** using cURL, OpenAI Python client, Python `requests`, or the native SGLang API
|
||||
|
||||
By the end, you'll have a working SGLang server responding to your prompts.
|
||||
|
||||
---
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- **Python**: 3.10 or higher
|
||||
- **GPU**: NVIDIA GPU with CUDA support (sm80 and above, e.g., A10, A100, L4, L40S, H100)
|
||||
- **OS**: Linux (recommended)
|
||||
|
||||
<Note>
|
||||
For other platforms, see the dedicated guides for [AMD GPUs](../hardware-platforms/amd_gpu), [Intel Xeon CPUs](../hardware-platforms/cpu_server), [Google TPUs](../hardware-platforms/tpu), [NVIDIA Jetson](../hardware-platforms/nvidia_jetson), [Ascend NPUs](../hardware-platforms/ascend-npus/getting-started/installation), and [Intel XPU](../hardware-platforms/xpu).
|
||||
</Note>
|
||||
|
||||
---
|
||||
|
||||
## Installation
|
||||
These commands target Linux with an NVIDIA GPU and a CUDA 13-compatible driver. See [Installation](/docs/get-started/install) for environment requirements and other setup options.
|
||||
|
||||
<Tabs>
|
||||
<Tab title="Pip / uv (Recommended)">
|
||||
We recommend using **uv** for faster installation:
|
||||
<Tab title="Docker">
|
||||
|
||||
With Docker and NVIDIA Container Toolkit configured, pull the image. Skip this command if you have already pulled it:
|
||||
|
||||
```bash
|
||||
pip install --upgrade pip
|
||||
pip install uv
|
||||
docker pull lmsysorg/sglang:latest
|
||||
```
|
||||
|
||||
Start the server on the GPU host. Replace `MODEL_PATH` with your Hugging Face model ID (for example, `Qwen/Qwen3-0.6B`):
|
||||
|
||||
```bash
|
||||
docker run --rm --gpus all --ipc=host \
|
||||
-p 30000:30000 \
|
||||
lmsysorg/sglang:latest \
|
||||
sglang serve MODEL_PATH --host 0.0.0.0 --port 30000
|
||||
```
|
||||
|
||||
</Tab>
|
||||
<Tab title="uv">
|
||||
|
||||
With uv installed, create a Python environment and install SGLang. Skip this block if you already have an environment with SGLang installed:
|
||||
|
||||
```bash
|
||||
uv venv --python 3.12
|
||||
source .venv/bin/activate
|
||||
uv pip install --prerelease=allow sglang
|
||||
```
|
||||
</Tab>
|
||||
<Tab title="From Source">
|
||||
|
||||
Start the server in that environment. Replace `MODEL_PATH` with your Hugging Face model ID (for example, `Qwen/Qwen3-0.6B`):
|
||||
|
||||
```bash
|
||||
# Clone and install from source
|
||||
git clone https://github.com/sgl-project/sglang.git
|
||||
cd sglang
|
||||
pip install --upgrade pip
|
||||
pip install -e "python"
|
||||
sglang serve MODEL_PATH --host 0.0.0.0 --port 30000
|
||||
```
|
||||
</Tab>
|
||||
<Tab title="Docker">
|
||||
The Docker images are available on Docker Hub at [lmsysorg/sglang](https://hub.docker.com/r/lmsysorg/sglang/tags).
|
||||
|
||||
Replace `<secret>` with your [Hugging Face token](https://huggingface.co/docs/hub/en/security-tokens):
|
||||
|
||||
```bash
|
||||
docker run --gpus all \
|
||||
--shm-size 32g \
|
||||
-p 30000:30000 \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
--env "HF_TOKEN=<secret>" \
|
||||
--ipc=host \
|
||||
lmsysorg/sglang:latest \
|
||||
sglang serve --model-path meta-llama/Llama-3.1-8B-Instruct --host 0.0.0.0 --port 30000
|
||||
```
|
||||
|
||||
For production deployments, use the smaller **runtime** variant (~40% size reduction):
|
||||
|
||||
```bash
|
||||
docker run --gpus all \
|
||||
--shm-size 32g \
|
||||
-p 30000:30000 \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
--env "HF_TOKEN=<secret>" \
|
||||
--ipc=host \
|
||||
lmsysorg/sglang:latest-runtime \
|
||||
sglang serve --model-path meta-llama/Llama-3.1-8B-Instruct --host 0.0.0.0 --port 30000
|
||||
```
|
||||
</Tab>
|
||||
</Tab>
|
||||
</Tabs>
|
||||
|
||||
<Tip>
|
||||
If you encounter `OSError: CUDA_HOME environment variable is not set`, set it with:
|
||||
```bash
|
||||
export CUDA_HOME=/usr/local/cuda-<your-cuda-version>
|
||||
```
|
||||
</Tip>
|
||||
For additional model-specific launch arguments, see the [Cookbook](/cookbook/intro). Wait for `The server is fired up and ready to roll!`.
|
||||
|
||||
---
|
||||
## Send a request
|
||||
|
||||
## Launch a Server
|
||||
|
||||
Start the SGLang server with a model. Here we use `qwen/qwen2.5-0.5b-instruct` as a lightweight example:
|
||||
|
||||
```bash
|
||||
sglang serve --model-path qwen/qwen2.5-0.5b-instruct --host 0.0.0.0 --port 30000
|
||||
```
|
||||
|
||||
Wait until you see `The server is fired up and ready to roll!` in the terminal output.
|
||||
|
||||
<Note>
|
||||
Once the server is running, API documentation is available at:
|
||||
- **Swagger UI**: `http://localhost:30000/docs`
|
||||
- **ReDoc**: `http://localhost:30000/redoc`
|
||||
- **OpenAPI Spec**: `http://localhost:30000/openapi.json`
|
||||
</Note>
|
||||
|
||||
<Info>
|
||||
The server automatically applies the chat template from the Hugging Face tokenizer. You can override it with `--chat-template` when launching.
|
||||
</Info>
|
||||
|
||||
---
|
||||
|
||||
## Send Requests
|
||||
|
||||
SGLang is fully **OpenAI API-compatible**, so you can use the same tools and libraries you already know.
|
||||
|
||||
### Using cURL
|
||||
In another terminal on the same host, send a request to verify the server. Use the same model ID for `MODEL_PATH`:
|
||||
|
||||
```bash
|
||||
curl http://localhost:30000/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "qwen/qwen2.5-0.5b-instruct",
|
||||
"model": "MODEL_PATH",
|
||||
"messages": [
|
||||
{"role": "user", "content": "What is the capital of France?"}
|
||||
]
|
||||
}'
|
||||
```
|
||||
|
||||
### Using OpenAI Python Client
|
||||
Check `choices[0].message.content` for the answer. Press `Ctrl+C` in the server terminal to stop.
|
||||
|
||||
Install the OpenAI Python library if you haven't:
|
||||
|
||||
```bash
|
||||
pip install openai
|
||||
```
|
||||
|
||||
Then send a request:
|
||||
|
||||
```python Example
|
||||
import openai
|
||||
|
||||
client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="None")
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="qwen/qwen2.5-0.5b-instruct",
|
||||
messages=[
|
||||
{"role": "user", "content": "List 3 countries and their capitals."},
|
||||
],
|
||||
temperature=0,
|
||||
max_tokens=64,
|
||||
)
|
||||
|
||||
print(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
#### Streaming
|
||||
|
||||
```python Example
|
||||
import openai
|
||||
|
||||
client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="None")
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="qwen/qwen2.5-0.5b-instruct",
|
||||
messages=[
|
||||
{"role": "user", "content": "List 3 countries and their capitals."},
|
||||
],
|
||||
temperature=0,
|
||||
max_tokens=64,
|
||||
stream=True,
|
||||
)
|
||||
|
||||
for chunk in response:
|
||||
if chunk.choices[0].delta.content:
|
||||
print(chunk.choices[0].delta.content, end="", flush=True)
|
||||
```
|
||||
|
||||
### Using Python Requests
|
||||
|
||||
```python Example
|
||||
import requests
|
||||
|
||||
url = "http://localhost:30000/v1/chat/completions"
|
||||
|
||||
data = {
|
||||
"model": "qwen/qwen2.5-0.5b-instruct",
|
||||
"messages": [{"role": "user", "content": "What is the capital of France?"}],
|
||||
}
|
||||
|
||||
response = requests.post(url, json=data)
|
||||
print(response.json())
|
||||
```
|
||||
|
||||
### Using the Native `/generate` API
|
||||
|
||||
SGLang also provides a native `/generate` endpoint for more flexibility.
|
||||
|
||||
```python Example
|
||||
import requests
|
||||
|
||||
response = requests.post(
|
||||
"http://localhost:30000/generate",
|
||||
json={
|
||||
"text": "The capital of France is",
|
||||
"sampling_params": {
|
||||
"temperature": 0,
|
||||
"max_new_tokens": 32,
|
||||
},
|
||||
},
|
||||
)
|
||||
|
||||
print(response.json())
|
||||
```
|
||||
|
||||
#### Streaming with `/generate`
|
||||
|
||||
```python Example
|
||||
import requests
|
||||
import json
|
||||
|
||||
response = requests.post(
|
||||
"http://localhost:30000/generate",
|
||||
json={
|
||||
"text": "The capital of France is",
|
||||
"sampling_params": {
|
||||
"temperature": 0,
|
||||
"max_new_tokens": 32,
|
||||
},
|
||||
"stream": True,
|
||||
},
|
||||
stream=True,
|
||||
)
|
||||
|
||||
prev = 0
|
||||
for chunk in response.iter_lines(decode_unicode=False):
|
||||
chunk = chunk.decode("utf-8")
|
||||
if chunk and chunk.startswith("data:"):
|
||||
if chunk == "data: [DONE]":
|
||||
break
|
||||
data = json.loads(chunk[5:].strip("\n"))
|
||||
output = data["text"]
|
||||
print(output[prev:], end="", flush=True)
|
||||
prev = len(output)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Offline Batch Inference (No Server)
|
||||
|
||||
SGLang also supports offline batch inference using the `Engine` class directly -- no HTTP server required.
|
||||
|
||||
```python Example
|
||||
import sglang as sgl
|
||||
|
||||
llm = sgl.Engine(model_path="qwen/qwen2.5-0.5b-instruct")
|
||||
|
||||
prompts = [
|
||||
"Hello, my name is",
|
||||
"The president of the United States is",
|
||||
"The capital of France is",
|
||||
"The future of AI is",
|
||||
]
|
||||
|
||||
sampling_params = {"temperature": 0.8, "top_p": 0.95}
|
||||
|
||||
outputs = llm.generate(prompts, sampling_params)
|
||||
|
||||
for prompt, output in zip(prompts, outputs):
|
||||
print(f"Prompt: {prompt}\nGenerated text: {output['text']}\n")
|
||||
|
||||
llm.shutdown()
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Common Troubleshooting
|
||||
|
||||
<AccordionGroup>
|
||||
<Accordion title="CUDA_HOME not set">
|
||||
Set the `CUDA_HOME` environment variable to your CUDA install root:
|
||||
```bash
|
||||
export CUDA_HOME=/usr/local/cuda-<your-cuda-version>
|
||||
```
|
||||
</Accordion>
|
||||
<Accordion title="FlashInfer issues on sm75+ devices">
|
||||
Switch to alternative backends by adding these flags when launching the server:
|
||||
```bash
|
||||
--attention-backend triton --sampling-backend pytorch
|
||||
```
|
||||
</Accordion>
|
||||
<Accordion title="Reinstalling FlashInfer">
|
||||
```bash
|
||||
pip3 install --upgrade flashinfer-python --force-reinstall --no-deps
|
||||
rm -rf ~/.cache/flashinfer
|
||||
```
|
||||
</Accordion>
|
||||
<Accordion title="ptxas error on B300/GB300 (sm_103a)">
|
||||
```bash
|
||||
export TRITON_PTXAS_PATH=/usr/local/cuda/bin/ptxas
|
||||
```
|
||||
</Accordion>
|
||||
</AccordionGroup>
|
||||
|
||||
---
|
||||
|
||||
{/*
|
||||
WIP, TBD linked later
|
||||
## What's Next?
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="OpenAI-Compatible APIs" href="../basic_usage/openai_api_completions">
|
||||
Explore the full Chat Completions and Completions APIs, including multi-turn conversations.
|
||||
</Card>
|
||||
<Card title="Vision Language Models" href="../basic_usage/openai_api_vision">
|
||||
Send image inputs alongside text using OpenAI-compatible vision APIs.
|
||||
</Card>
|
||||
<Card title="Sampling Parameters" href="../basic_usage/sampling_params">
|
||||
Fine-tune generation with temperature, top-p, frequency penalty, and more.
|
||||
</Card>
|
||||
<Card title="Server Arguments" href="../advanced_features/server_arguments">
|
||||
Customize server behavior with advanced launch arguments like tensor parallelism.
|
||||
</Card>
|
||||
<Card title="Structured Outputs" href="../advanced_features/structured_outputs">
|
||||
Constrain model output to JSON, regex, or EBNF grammars.
|
||||
</Card>
|
||||
<Card title="Ollama-Compatible API" href="../basic_usage/ollama_api">
|
||||
Use the familiar Ollama CLI and Python library with SGLang as the backend.
|
||||
</Card>
|
||||
</CardGroup>
|
||||
*/}
|
||||
See [Examples](/docs/get-started/examples) for a complete 27B deployment, streaming, and tool calls.
|
||||
|
||||
Reference in New Issue
Block a user