From e22dfe8fc20538c78d097404c1a98b56f47a33c8 Mon Sep 17 00:00:00 2001 From: zijiexia <37504505+zijiexia@users.noreply.github.com> Date: Tue, 21 Apr 2026 01:12:50 -0700 Subject: [PATCH] [Docs] Update installation and TPU documentation to fix the render problem (#23344) --- docs_new/docs/get-started/install.mdx | 34 +++++++++++------------- docs_new/docs/hardware-platforms/tpu.mdx | 5 ++-- 2 files changed, 17 insertions(+), 22 deletions(-) diff --git a/docs_new/docs/get-started/install.mdx b/docs_new/docs/get-started/install.mdx index 5f2e6b1b2..5199c865d 100644 --- a/docs_new/docs/get-started/install.mdx +++ b/docs_new/docs/get-started/install.mdx @@ -50,7 +50,7 @@ uv pip install "https://github.com/sgl-project/whl/releases/download/vX.Y.Z/sgla export TRITON_PTXAS_PATH=/usr/local/cuda/bin/ptxas ``` -### **Quick fixes to common problems** +### Quick fixes to common problems - If you encounter `OSError: CUDA_HOME environment variable is not set`. Please set it to your CUDA install root with either of the following solutions: 1. Use `export CUDA_HOME=/usr/local/cuda-` to set the `CUDA_HOME` environment variable. 2. Install FlashInfer first following [FlashInfer installation doc](https://docs.flashinfer.ai/installation.html), then install SGLang as described above. @@ -109,8 +109,7 @@ Notes: Please check out [OME](https://github.com/sgl-project/ome), a Kubernetes operator for enterprise-grade management and serving of large language models (LLMs). -
-More + 1. Option 1: For single node serving (typically when the model size fits into GPUs on one node) @@ -120,31 +119,30 @@ Please check out [OME](https://github.com/sgl-project/ome), a Kubernetes operato Modify the LLM model path and arguments as necessary, then execute command `kubectl apply -f docker/k8s-sglang-distributed-sts.yaml`, to create two nodes k8s statefulset and serving service. -
+ ## Method 5: Using docker compose -
-More + > This method is recommended if you plan to serve it as a service. > A better approach is to use the [k8s-sglang-service.yaml](https://github.com/sgl-project/sglang/blob/main/docker/k8s-sglang-service.yaml). 1. Copy the [compose.yml](https://github.com/sgl-project/sglang/blob/main/docker/compose.yaml) to your local machine 2. Execute the command `docker compose up -d` in your terminal. -
+ + ## Method 6: Run on Kubernetes or Clouds with SkyPilot -
-More + To deploy on Kubernetes or 12+ clouds, you can use [SkyPilot](https://github.com/skypilot-org/skypilot). 1. Install SkyPilot and set up Kubernetes cluster or cloud access: see [SkyPilot's documentation](https://skypilot.readthedocs.io/en/latest/getting-started/installation.html). 2. Deploy on your own infra with a single command and get the HTTP API endpoint: -
-SkyPilot YAML: sglang.yaml + +SkyPilot YAML: sglang.yaml}> ```yaml Config # sglang.yaml @@ -164,7 +162,7 @@ run: | --port 30000 ``` -
+ ```bash Command # Deploy on any cloud or Kubernetes cluster. Use --cloud to select a specific cloud provider. @@ -176,12 +174,11 @@ sky status --endpoint 30000 sglang 3. To further scale up your deployment with autoscaling and failure recovery, check out the [SkyServe + SGLang guide](https://github.com/skypilot-org/skypilot/tree/master/llm/sglang#serving-llama-2-with-sglang-for-more-traffic-using-skyserve). -
+ ## Method 7: Run on AWS SageMaker -
-More + To deploy on SGLang on AWS SageMaker, check out [AWS SageMaker Inference](https://aws.amazon.com/sagemaker/ai/deploy) @@ -192,8 +189,7 @@ To host a model with your own container, follow the following steps: 1. Build a docker container with [sagemaker.Dockerfile](https://github.com/sgl-project/sglang/blob/main/docker/sagemaker.Dockerfile) alongside the [serve](https://github.com/sgl-project/sglang/blob/main/docker/serve) script. 2. Push your container onto AWS ECR. -
-Dockerfile Build Script: build-and-push.sh +Dockerfile Build Script: build-and-push.sh}> ```bash Command #!/bin/bash @@ -221,7 +217,7 @@ docker push ${IMAGE_URI} echo "Build and push completed successfully!" ``` -
+ 3. Deploy a model for serving on AWS Sagemaker, refer to [deploy_and_serve_endpoint.py](https://github.com/sgl-project/sglang/blob/main/examples/sagemaker/deploy_and_serve_endpoint.py). For more information, check out [sagemaker-python-sdk](https://github.com/aws/sagemaker-python-sdk). 1. By default, the model server on SageMaker will run with the following command: `python3 -m sglang.launch_server --model-path opt/ml/model --host 0.0.0.0 --port 8080`. This is optimal for hosting your own model with SageMaker. @@ -229,7 +225,7 @@ echo "Build and push completed successfully!" 3. The serve script will automatically convert all environment variables with prefix `SM_SGLANG_` from `SM_SGLANG_INPUT_ARGUMENT` into `--input-argument` to be parsed into `python3 -m sglang.launch_server` cli. 4. For example, to run [Qwen/Qwen3-0.6B](https://huggingface.co/Qwen/Qwen3-0.6B) with reasoning parser, simply add additional environment variables `SM_SGLANG_MODEL_PATH=Qwen/Qwen3-0.6B` and `SM_SGLANG_REASONING_PARSER=qwen3`. -
+ ## Common Notes diff --git a/docs_new/docs/hardware-platforms/tpu.mdx b/docs_new/docs/hardware-platforms/tpu.mdx index 5e2c75bfa..b3d7f2516 100644 --- a/docs_new/docs/hardware-platforms/tpu.mdx +++ b/docs_new/docs/hardware-platforms/tpu.mdx @@ -247,8 +247,7 @@ uv pip install -e "python[all]" 2. Create a SkyPilot configuration file: -
-SkyPilot YAML: sglang-jax.sky.yaml +SkyPilot YAML: sglang-jax.sky.yaml}> ```yaml Config # sglang-jax.sky.yaml @@ -266,7 +265,7 @@ run: | uv pip install -e "python[all]" ``` -
+ 3. Launch your TPU cluster: