minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity

[Docs] Update installation and TPU documentation to fix the render problem (#23344)

Browse Source
This commit is contained in:
zijiexia
2026-04-21 01:12:50 -07:00
committed by GitHub
parent 47c4b38257
commit e22dfe8fc2
2 changed files with 17 additions and 22 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs_new/docs/get-started/install.mdx
+15 -19
View File
@@ -50,7 +50,7 @@ uv pip install "https://github.com/sgl-project/whl/releases/download/vX.Y.Z/sgla
export TRITON_PTXAS_PATH=/usr/local/cuda/bin/ptxas export TRITON_PTXAS_PATH=/usr/local/cuda/bin/ptxas
``` ```
### **Quick fixes to common problems** ### Quick fixes to common problems
- If you encounter `OSError: CUDA_HOME environment variable is not set`. Please set it to your CUDA install root with either of the following solutions: - If you encounter `OSError: CUDA_HOME environment variable is not set`. Please set it to your CUDA install root with either of the following solutions:
1. Use `export CUDA_HOME=/usr/local/cuda-<your-cuda-version>` to set the `CUDA_HOME` environment variable. 1. Use `export CUDA_HOME=/usr/local/cuda-<your-cuda-version>` to set the `CUDA_HOME` environment variable.
2. Install FlashInfer first following [FlashInfer installation doc](https://docs.flashinfer.ai/installation.html), then install SGLang as described above. 2. Install FlashInfer first following [FlashInfer installation doc](https://docs.flashinfer.ai/installation.html), then install SGLang as described above.
@@ -109,8 +109,7 @@ Notes:
Please check out [OME](https://github.com/sgl-project/ome), a Kubernetes operator for enterprise-grade management and serving of large language models (LLMs). Please check out [OME](https://github.com/sgl-project/ome), a Kubernetes operator for enterprise-grade management and serving of large language models (LLMs).
<details> <Accordion title="More">
<summary>More</summary>
1. Option 1: For single node serving (typically when the model size fits into GPUs on one node) 1. Option 1: For single node serving (typically when the model size fits into GPUs on one node)
@@ -120,31 +119,30 @@ Please check out [OME](https://github.com/sgl-project/ome), a Kubernetes operato
Modify the LLM model path and arguments as necessary, then execute command `kubectl apply -f docker/k8s-sglang-distributed-sts.yaml`, to create two nodes k8s statefulset and serving service. Modify the LLM model path and arguments as necessary, then execute command `kubectl apply -f docker/k8s-sglang-distributed-sts.yaml`, to create two nodes k8s statefulset and serving service.
</details> </Accordion>
## Method 5: Using docker compose ## Method 5: Using docker compose
<details> <Accordion title="More">
<summary>More</summary>
> This method is recommended if you plan to serve it as a service. > This method is recommended if you plan to serve it as a service.
> A better approach is to use the [k8s-sglang-service.yaml](https://github.com/sgl-project/sglang/blob/main/docker/k8s-sglang-service.yaml). > A better approach is to use the [k8s-sglang-service.yaml](https://github.com/sgl-project/sglang/blob/main/docker/k8s-sglang-service.yaml).
1. Copy the [compose.yml](https://github.com/sgl-project/sglang/blob/main/docker/compose.yaml) to your local machine 1. Copy the [compose.yml](https://github.com/sgl-project/sglang/blob/main/docker/compose.yaml) to your local machine
2. Execute the command `docker compose up -d` in your terminal. 2. Execute the command `docker compose up -d` in your terminal.
</details>
</Accordion>
## Method 6: Run on Kubernetes or Clouds with SkyPilot ## Method 6: Run on Kubernetes or Clouds with SkyPilot
<details> <Accordion title="More">
<summary>More</summary>
To deploy on Kubernetes or 12+ clouds, you can use [SkyPilot](https://github.com/skypilot-org/skypilot). To deploy on Kubernetes or 12+ clouds, you can use [SkyPilot](https://github.com/skypilot-org/skypilot).
1. Install SkyPilot and set up Kubernetes cluster or cloud access: see [SkyPilot's documentation](https://skypilot.readthedocs.io/en/latest/getting-started/installation.html). 1. Install SkyPilot and set up Kubernetes cluster or cloud access: see [SkyPilot's documentation](https://skypilot.readthedocs.io/en/latest/getting-started/installation.html).
2. Deploy on your own infra with a single command and get the HTTP API endpoint: 2. Deploy on your own infra with a single command and get the HTTP API endpoint:
<details>
<summary>SkyPilot YAML: <code>sglang.yaml</code></summary> <Accordion title={<>SkyPilot YAML: <code>sglang.yaml</code></>}>
```yaml Config ```yaml Config
# sglang.yaml # sglang.yaml
@@ -164,7 +162,7 @@ run: |
--port 30000 --port 30000
``` ```
</details> </Accordion>
```bash Command ```bash Command
# Deploy on any cloud or Kubernetes cluster. Use --cloud <cloud> to select a specific cloud provider. # Deploy on any cloud or Kubernetes cluster. Use --cloud <cloud> to select a specific cloud provider.
@@ -176,12 +174,11 @@ sky status --endpoint 30000 sglang
3. To further scale up your deployment with autoscaling and failure recovery, check out the [SkyServe + SGLang guide](https://github.com/skypilot-org/skypilot/tree/master/llm/sglang#serving-llama-2-with-sglang-for-more-traffic-using-skyserve). 3. To further scale up your deployment with autoscaling and failure recovery, check out the [SkyServe + SGLang guide](https://github.com/skypilot-org/skypilot/tree/master/llm/sglang#serving-llama-2-with-sglang-for-more-traffic-using-skyserve).
</details> </Accordion>
## Method 7: Run on AWS SageMaker ## Method 7: Run on AWS SageMaker
<details> <Accordion title="More">
<summary>More</summary>
To deploy on SGLang on AWS SageMaker, check out [AWS SageMaker Inference](https://aws.amazon.com/sagemaker/ai/deploy) To deploy on SGLang on AWS SageMaker, check out [AWS SageMaker Inference](https://aws.amazon.com/sagemaker/ai/deploy)
@@ -192,8 +189,7 @@ To host a model with your own container, follow the following steps:
1. Build a docker container with [sagemaker.Dockerfile](https://github.com/sgl-project/sglang/blob/main/docker/sagemaker.Dockerfile) alongside the [serve](https://github.com/sgl-project/sglang/blob/main/docker/serve) script. 1. Build a docker container with [sagemaker.Dockerfile](https://github.com/sgl-project/sglang/blob/main/docker/sagemaker.Dockerfile) alongside the [serve](https://github.com/sgl-project/sglang/blob/main/docker/serve) script.
2. Push your container onto AWS ECR. 2. Push your container onto AWS ECR.
<details> <Accordion title={<>Dockerfile Build Script: <code>build-and-push.sh</code></>}>
<summary>Dockerfile Build Script: <code>build-and-push.sh</code></summary>
```bash Command ```bash Command
#!/bin/bash #!/bin/bash
@@ -221,7 +217,7 @@ docker push ${IMAGE_URI}
echo "Build and push completed successfully!" echo "Build and push completed successfully!"
``` ```
</details> </Accordion>
3. Deploy a model for serving on AWS Sagemaker, refer to [deploy_and_serve_endpoint.py](https://github.com/sgl-project/sglang/blob/main/examples/sagemaker/deploy_and_serve_endpoint.py). For more information, check out [sagemaker-python-sdk](https://github.com/aws/sagemaker-python-sdk). 3. Deploy a model for serving on AWS Sagemaker, refer to [deploy_and_serve_endpoint.py](https://github.com/sgl-project/sglang/blob/main/examples/sagemaker/deploy_and_serve_endpoint.py). For more information, check out [sagemaker-python-sdk](https://github.com/aws/sagemaker-python-sdk).
1. By default, the model server on SageMaker will run with the following command: `python3 -m sglang.launch_server --model-path opt/ml/model --host 0.0.0.0 --port 8080`. This is optimal for hosting your own model with SageMaker. 1. By default, the model server on SageMaker will run with the following command: `python3 -m sglang.launch_server --model-path opt/ml/model --host 0.0.0.0 --port 8080`. This is optimal for hosting your own model with SageMaker.
@@ -229,7 +225,7 @@ echo "Build and push completed successfully!"
3. The serve script will automatically convert all environment variables with prefix `SM_SGLANG_` from `SM_SGLANG_INPUT_ARGUMENT` into `--input-argument` to be parsed into `python3 -m sglang.launch_server` cli. 3. The serve script will automatically convert all environment variables with prefix `SM_SGLANG_` from `SM_SGLANG_INPUT_ARGUMENT` into `--input-argument` to be parsed into `python3 -m sglang.launch_server` cli.
4. For example, to run [Qwen/Qwen3-0.6B](https://huggingface.co/Qwen/Qwen3-0.6B) with reasoning parser, simply add additional environment variables `SM_SGLANG_MODEL_PATH=Qwen/Qwen3-0.6B` and `SM_SGLANG_REASONING_PARSER=qwen3`. 4. For example, to run [Qwen/Qwen3-0.6B](https://huggingface.co/Qwen/Qwen3-0.6B) with reasoning parser, simply add additional environment variables `SM_SGLANG_MODEL_PATH=Qwen/Qwen3-0.6B` and `SM_SGLANG_REASONING_PARSER=qwen3`.
</details> </Accordion>
## Common Notes ## Common Notes
docs_new/docs/hardware-platforms/tpu.mdx
+2 -3
View File
@@ -247,8 +247,7 @@ uv pip install -e "python[all]"
2. Create a SkyPilot configuration file: 2. Create a SkyPilot configuration file:
<details> <Accordion title={<>SkyPilot YAML: <code>sglang-jax.sky.yaml</code></>}>
<summary>SkyPilot YAML: <code>sglang-jax.sky.yaml</code></summary>
```yaml Config ```yaml Config
# sglang-jax.sky.yaml # sglang-jax.sky.yaml
@@ -266,7 +265,7 @@ run: |
uv pip install -e "python[all]" uv pip install -e "python[all]"
``` ```
</details> </Accordion>
3. Launch your TPU cluster: 3. Launch your TPU cluster: