309 lines
13 KiB
Plaintext
309 lines
13 KiB
Plaintext
---
|
|
title: Installation
|
|
description: Install SGLang with pip/uv, source, Docker, Kubernetes, and cloud deployment options.
|
|
keywords:
|
|
- installation
|
|
- sglang
|
|
- pip
|
|
- docker
|
|
---
|
|
Choose one installation method below. These instructions target Linux with NVIDIA GPUs; for other devices, see [Hardware platforms](/docs/hardware-platforms/overview).
|
|
|
|
## Install SGLang
|
|
|
|
<Tabs>
|
|
<Tab title="Docker" id="method-3-using-docker">
|
|
|
|
Docker includes Python, SGLang, and its dependencies; no separate host Python installation is required.
|
|
|
|
The docker images are available on Docker Hub at [lmsysorg/sglang](https://hub.docker.com/r/lmsysorg/sglang/tags), built from [Dockerfile](https://github.com/sgl-project/sglang/tree/main/docker).
|
|
Use Linux with a supported NVIDIA GPU, a CUDA 13-compatible driver, [Docker Engine](https://docs.docker.com/engine/install/), and [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html). Complete the Toolkit guide's Docker runtime configuration before starting a GPU container.
|
|
|
|
SGLang images ship a CUDA 13 environment. CUDA 12 images (`-cu12` / `-cu129`) are no longer published; `lmsysorg/sglang:v0.5.19-cu129` is the last CUDA 12 tag.
|
|
|
|
**Pull the image**
|
|
|
|
`latest` contains the newest stable release:
|
|
|
|
|
|
```bash
|
|
docker pull lmsysorg/sglang:latest
|
|
```
|
|
|
|
**Image options**
|
|
|
|
The examples use `latest`. To use another image, replace the tag in the pull and run commands:
|
|
|
|
- Nightly: Built daily from the latest `main`, including changes not yet in a stable release. Choose a tag from the [nightly image list](https://hub.docker.com/r/lmsysorg/sglang/tags?name=nightly).
|
|
- Runtime: A smaller image of the same stable release, with inference dependencies and fewer development utilities. Use the `latest-runtime` tag.
|
|
- AWS EFA: Any of the images above plus the AWS EFA userspace stack, for EFA-enabled EC2 instances. Add `-efa` to the tag: `latest-efa`, `latest-runtime-efa`, or the dated `nightly-dev-efa-<date>-<sha>` (moving tag `dev-efa`). See [AWS EFA](/docs/advanced_features/pd_disaggregation#aws-efa) for what the images include and how to run them.
|
|
|
|
For reproducible deployments, use a specific release or dated nightly tag from [Docker Hub](https://hub.docker.com/r/lmsysorg/sglang/tags).
|
|
|
|
**Start a container**
|
|
|
|
Set `HF_CACHE_DIR` to an absolute directory path on the host for downloaded model weights, replacing the placeholder below.
|
|
|
|
For gated or private models, export your [Hugging Face token](https://huggingface.co/docs/hub/en/security-tokens) as `HF_TOKEN` on the host and add `--env HF_TOKEN` before the image name in this command.
|
|
|
|
|
|
```bash
|
|
HF_CACHE_DIR="/path/to/huggingface-cache"
|
|
docker run -it --gpus all \
|
|
--shm-size 32g \
|
|
-p 30000:30000 \
|
|
-v "$HF_CACHE_DIR:/root/.cache/huggingface" \
|
|
--ipc=host \
|
|
lmsysorg/sglang:latest bash
|
|
```
|
|
|
|
The container shell is ready for `sglang serve`.
|
|
|
|
</Tab>
|
|
<Tab title="pip / uv" id="method-1-with-pip-or-uv">
|
|
|
|
Use Python 3.11 or higher and a compatible CUDA 13 environment.
|
|
|
|
<Accordion title="CUDA version compatibility">
|
|
|
|
SGLang requires CUDA 13. CUDA 12 (`cu129`) wheels and images were retired with the upgrade to PyTorch 2.14, which publishes no CUDA 12.9 builds. SGLang 0.5.19 is the last release with a CUDA 12 lane (PyTorch 2.13).
|
|
|
|
</Accordion>
|
|
|
|
**Prepare the environment**
|
|
|
|
Use uv for faster installation. Install it with pip, or follow the [uv installation instructions](https://docs.astral.sh/uv/getting-started/installation/):
|
|
|
|
```bash
|
|
pip install --upgrade pip
|
|
pip install uv
|
|
```
|
|
|
|
Create and activate a virtual environment:
|
|
|
|
```bash Command
|
|
uv venv --python 3.12
|
|
source .venv/bin/activate
|
|
```
|
|
|
|
**Install SGLang**
|
|
|
|
```bash
|
|
uv pip install --prerelease=allow sglang
|
|
```
|
|
|
|
<Accordion title="Why --prerelease=allow?">
|
|
|
|
Some of SGLang's dependencies only publish pre-releases on PyPI, so without `--prerelease=allow` uv older than 0.12.0 silently installs SGLang 0.5.9. On [uv 0.12.0](https://github.com/astral-sh/uv/releases/tag/0.12.0) and newer the flag is a harmless no-op.
|
|
</Accordion>
|
|
|
|
<Accordion title="Nightly builds" id="nightly-builds">
|
|
|
|
To pick up the latest features and fixes before the next stable release, install a nightly build. Nightly wheels are built from the latest `main` and published to the SGLang wheel index. Add that index with `--extra-index-url`, and combine `--prerelease=allow` with `--index-strategy unsafe-best-match` so uv considers the nightly (pre-release) version alongside PyPI:
|
|
|
|
```bash Command
|
|
uv pip install --prerelease=allow --index-strategy unsafe-best-match --extra-index-url https://docs.sglang.ai/whl/cu130/ sglang
|
|
```
|
|
|
|
</Accordion>
|
|
|
|
For `CUDA_HOME` errors, see [Troubleshooting](#cuda_home).
|
|
|
|
</Tab>
|
|
<Tab title="From source" id="method-2-from-source">
|
|
|
|
Use Python 3.11 or higher in an activated Python environment, with CUDA dependencies appropriate for the version you build.
|
|
|
|
**Get the source**
|
|
|
|
Clone a release tag (the example below pins one); choose another from [GitHub releases](https://github.com/sgl-project/sglang/releases) as needed:
|
|
|
|
```bash
|
|
git clone -b v0.5.21 https://github.com/sgl-project/sglang.git
|
|
cd sglang
|
|
```
|
|
|
|
**Install SGLang**
|
|
|
|
Install the Python package in editable mode so local source changes are reflected in your environment:
|
|
|
|
```bash
|
|
pip install --upgrade pip
|
|
pip install -e "python"
|
|
```
|
|
|
|
<Accordion title="Development environment">
|
|
|
|
For contributing to SGLang, follow the [development environment guide](/docs/developer_guide/development_guide_using_docker#setup-docker-container), which covers a container with development tools and dependencies installed.
|
|
|
|
</Accordion>
|
|
|
|
For `CUDA_HOME` errors, see [Troubleshooting](#cuda_home).
|
|
|
|
</Tab>
|
|
</Tabs>
|
|
|
|
## Start a model server
|
|
|
|
Start a model server inside your Docker container or activated Python environment. Replace `MODEL_PATH` with a Hugging Face model ID or a local model directory:
|
|
|
|
```bash
|
|
sglang serve MODEL_PATH --host 0.0.0.0 --port 30000
|
|
```
|
|
|
|
Additional launch arguments depend on the model and your hardware.
|
|
|
|
- [Quickstart](/docs/get-started/quickstart): Launch a model and send a request to verify the server.
|
|
- [Cookbook](/cookbook/intro): Find model-specific launch arguments for your hardware.
|
|
|
|
## Deployment options
|
|
|
|
Choose how to run SGLang as a service. Expand an option for setup instructions.
|
|
|
|
<AccordionGroup>
|
|
|
|
<a id="method-4-using-kubernetes" />
|
|
|
|
<Accordion title="Kubernetes" id="kubernetes" description="Serve models on a Kubernetes cluster, on one node or across multiple nodes.">
|
|
|
|
Please check out [OME](https://github.com/sgl-project/ome), a Kubernetes operator for enterprise-grade management and serving of large language models (LLMs).
|
|
|
|
1. Option 1: For single node serving (typically when the model size fits into GPUs on one node)
|
|
|
|
Execute command `kubectl apply -f docker/k8s-sglang-service.yaml`, to create k8s deployment and service, with llama-31-8b as example.
|
|
|
|
2. Option 2: For multi-node serving (usually when a large model requires more than one GPU node, such as `DeepSeek-R1`)
|
|
|
|
Modify the LLM model path and arguments as necessary, then execute command `kubectl apply -f docker/k8s-sglang-distributed-sts.yaml`, to create two nodes k8s statefulset and serving service.
|
|
|
|
</Accordion>
|
|
|
|
<a id="method-5-using-docker-compose" />
|
|
|
|
<Accordion title="Docker Compose" id="docker-compose" description="Run a model service on a single Docker host with a Compose configuration.">
|
|
|
|
> This method is recommended if you plan to serve it as a service.
|
|
> A better approach is to use the [k8s-sglang-service.yaml](https://github.com/sgl-project/sglang/blob/main/docker/k8s-sglang-service.yaml).
|
|
|
|
1. Copy the [compose.yml](https://github.com/sgl-project/sglang/blob/main/docker/compose.yaml) to your local machine
|
|
2. Execute the command `docker compose up -d` in your terminal.
|
|
|
|
</Accordion>
|
|
|
|
<a id="method-6-run-on-kubernetes-or-clouds-with-skypilot" />
|
|
|
|
<Accordion title="SkyPilot" id="skypilot" description="Deploy on Kubernetes or across cloud providers from one YAML configuration.">
|
|
|
|
To deploy on Kubernetes or 12+ clouds, you can use [SkyPilot](https://github.com/skypilot-org/skypilot).
|
|
|
|
1. Install SkyPilot and set up Kubernetes cluster or cloud access: see [SkyPilot's documentation](https://skypilot.readthedocs.io/en/latest/getting-started/installation.html).
|
|
2. Deploy on your own infra with a single command and get the HTTP API endpoint:
|
|
|
|
<Accordion title="SkyPilot YAML: sglang.yaml">
|
|
|
|
```yaml Config
|
|
# sglang.yaml
|
|
envs:
|
|
HF_TOKEN: null
|
|
|
|
resources:
|
|
image_id: docker:lmsysorg/sglang:latest
|
|
accelerators: A100
|
|
ports: 30000
|
|
|
|
run: |
|
|
conda deactivate
|
|
python3 -m sglang.launch_server \
|
|
--model-path meta-llama/Llama-3.1-8B-Instruct \
|
|
--host 0.0.0.0 \
|
|
--port 30000
|
|
```
|
|
|
|
</Accordion>
|
|
|
|
```bash Command
|
|
# Deploy on any cloud or Kubernetes cluster. Use --cloud <cloud> to select a specific cloud provider.
|
|
HF_TOKEN=<secret> sky launch -c sglang --env HF_TOKEN sglang.yaml
|
|
|
|
# Get the HTTP API endpoint
|
|
sky status --endpoint 30000 sglang
|
|
```
|
|
|
|
3. To further scale up your deployment with autoscaling and failure recovery, check out the [SkyServe + SGLang guide](https://github.com/skypilot-org/skypilot/tree/master/llm/sglang#serving-llama-2-with-sglang-for-more-traffic-using-skyserve).
|
|
|
|
</Accordion>
|
|
|
|
<a id="method-7-run-on-aws-sagemaker" />
|
|
|
|
<Accordion title="AWS SageMaker" id="aws-sagemaker" description="Deploy on SageMaker with a pre-built SGLang image or your own container.">
|
|
|
|
To deploy on SGLang on AWS SageMaker, check out [AWS SageMaker Inference](https://aws.amazon.com/sagemaker/ai/deploy)
|
|
|
|
Amazon Web Services provide supports for SGLang containers along with routine security patching. For available SGLang containers, check out [AWS SGLang DLCs](https://aws.github.io/deep-learning-containers/reference/available_images/#sglang).
|
|
|
|
To deploy a pre-built SGLang Deep Learning Container without building your own image, see [Amazon SageMaker AI](/docs/basic_usage/aws_sagemaker).
|
|
|
|
To host a model with your own container, follow the following steps:
|
|
|
|
1. Build a docker container with [sagemaker.Dockerfile](https://github.com/sgl-project/sglang/blob/main/docker/sagemaker.Dockerfile) alongside the [serve](https://github.com/sgl-project/sglang/blob/main/docker/serve) script.
|
|
2. Push your container onto AWS ECR.
|
|
|
|
<Accordion title="Build script: build-and-push.sh">
|
|
|
|
```bash Command
|
|
#!/bin/bash
|
|
AWS_ACCOUNT="<YOUR_AWS_ACCOUNT>"
|
|
AWS_REGION="<YOUR_AWS_REGION>"
|
|
REPOSITORY_NAME="<YOUR_REPOSITORY_NAME>"
|
|
IMAGE_TAG="<YOUR_IMAGE_TAG>"
|
|
|
|
ECR_REGISTRY="${AWS_ACCOUNT}.dkr.ecr.${AWS_REGION}.amazonaws.com"
|
|
IMAGE_URI="${ECR_REGISTRY}/${REPOSITORY_NAME}:${IMAGE_TAG}"
|
|
|
|
echo "Starting build and push process..."
|
|
|
|
# Login to ECR
|
|
echo "Logging into ECR..."
|
|
aws ecr get-login-password --region ${AWS_REGION} | docker login --username AWS --password-stdin ${ECR_REGISTRY}
|
|
|
|
# Build the image
|
|
echo "Building Docker image..."
|
|
docker build -t ${IMAGE_URI} -f sagemaker.Dockerfile .
|
|
|
|
echo "Pushing ${IMAGE_URI}"
|
|
docker push ${IMAGE_URI}
|
|
|
|
echo "Build and push completed successfully!"
|
|
```
|
|
|
|
</Accordion>
|
|
|
|
3. Deploy a model for serving on AWS Sagemaker, refer to [deploy_and_serve_endpoint.py](https://github.com/sgl-project/sglang/blob/main/examples/sagemaker/deploy_and_serve_endpoint.py). For more information, check out [sagemaker-python-sdk](https://github.com/aws/sagemaker-python-sdk).
|
|
1. By default, the model server on SageMaker will run with the following command: `python3 -m sglang.launch_server --model-path opt/ml/model --host 0.0.0.0 --port 8080`. This is optimal for hosting your own model with SageMaker.
|
|
2. To modify your model serving parameters, the [serve](https://github.com/sgl-project/sglang/blob/main/docker/serve) script allows for all available options within `python3 -m sglang.launch_server --help` cli by specifying environment variables with prefix `SM_SGLANG_`.
|
|
3. The serve script will automatically convert all environment variables with prefix `SM_SGLANG_` from `SM_SGLANG_INPUT_ARGUMENT` into `--input-argument` to be parsed into `python3 -m sglang.launch_server` cli.
|
|
4. For example, to run [Qwen/Qwen3-0.6B](https://huggingface.co/Qwen/Qwen3-0.6B) with reasoning parser, simply add additional environment variables `SM_SGLANG_MODEL_PATH=Qwen/Qwen3-0.6B` and `SM_SGLANG_REASONING_PARSER=qwen3`.
|
|
|
|
</Accordion>
|
|
|
|
</AccordionGroup>
|
|
|
|
<a id="common-notes" />
|
|
|
|
## Troubleshooting
|
|
|
|
### CUDA_HOME
|
|
|
|
If installation fails with `OSError: CUDA_HOME environment variable is not set`, set `CUDA_HOME` to your CUDA Toolkit installation directory. Replace the path below with the actual path on your machine:
|
|
|
|
```bash
|
|
export CUDA_HOME="/path/to/cuda"
|
|
```
|
|
|
|
For a versioned installation, this is typically `/usr/local/cuda-VERSION`. Alternatively, install FlashInfer first using the [FlashInfer installation instructions](https://docs.flashinfer.ai/installation.html), then retry the SGLang installation.
|
|
|
|
### FlashInfer
|
|
|
|
- [FlashInfer](https://github.com/flashinfer-ai/flashinfer) is the default attention kernel backend. It only supports sm75 and above. If you encounter any FlashInfer-related issues on sm75+ devices (e.g., T4, A10, A100, L4, L40S, H100), please switch to other kernels by adding `--attention-backend triton --sampling-backend pytorch` and open an issue on GitHub.
|
|
- To reinstall flashinfer locally, use the following command: `pip3 install --upgrade flashinfer-python --force-reinstall --no-deps` and then delete the cache with `rm -rf ~/.cache/flashinfer`.
|