minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 13 Packages Projects Releases Wiki Activity

[Intel CPU/XPU] SGL doc updates (#23547)

Browse Source 
merge this one as doc change only.
This commit is contained in:
Zaili Wang
2026-04-24 09:23:27 +08:00
committed by GitHub
parent 23e4d381f0
commit cbc2bee547
2 changed files with 72 additions and 47 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs_new/docs/hardware-platforms/cpu_server.mdx
+44 -44
View File
@@ -222,12 +222,12 @@ Notes:
Example command to launch SGLang serving: Example command to launch SGLang serving:
```bash Launch Server ```bash Launch Server
python -m sglang.launch_server \ sglang serve \
--model <MODEL_ID_OR_PATH> \ --model-path <MODEL_ID_OR_PATH> \
--trust-remote-code \ --trust-remote-code \
--disable-overlap-schedule \ --disable-overlap-schedule \
--device cpu \ --device cpu \
--host 0.0.0.0 \ --host 0.0.0.0 \
--tp 6 --tp 6
``` ```
@@ -320,64 +320,64 @@ or MoE (Mixture of Experts) models with fewer than 10B activated parameters, can
An example command to launch service of W8A8_INT8 DeepSeek-V3.1-Terminus on a Xeon® 6980P server: An example command to launch service of W8A8_INT8 DeepSeek-V3.1-Terminus on a Xeon® 6980P server:
```bash W8A8_INT8 ```bash W8A8_INT8
python -m sglang.launch_server \ sglang serve \
--model IntervitensInc/DeepSeek-V3.1-Terminus-Channel-int8 \ --model-path IntervitensInc/DeepSeek-V3.1-Terminus-Channel-int8 \
--trust-remote-code \ --trust-remote-code \
--disable-overlap-schedule \ --disable-overlap-schedule \
--device cpu \ --device cpu \
--quantization w8a8_int8 \ --quantization w8a8_int8 \
--host 0.0.0.0 \ --enable-torch-compile \
--enable-torch-compile \ --torch-compile-max-bs 4 \
--torch-compile-max-bs 4 \ --host 0.0.0.0 \
--tp 6 --tp 6
``` ```
Similarly, an example command to launch service of FP8 DeepSeek-V3.1-Terminus would be: Similarly, an example command to launch service of FP8 DeepSeek-V3.1-Terminus would be:
```bash FP8 ```bash FP8
python -m sglang.launch_server \ sglang serve \
--model deepseek-ai/DeepSeek-V3.1-Terminus \ --model-path deepseek-ai/DeepSeek-V3.1-Terminus \
--trust-remote-code \ --trust-remote-code \
--disable-overlap-schedule \ --disable-overlap-schedule \
--device cpu \ --device cpu \
--host 0.0.0.0 \ --enable-torch-compile \
--enable-torch-compile \ --torch-compile-max-bs 4 \
--torch-compile-max-bs 4 \ --host 0.0.0.0 \
--tp 6 --tp 6
``` ```
Note: Please set `--torch-compile-max-bs` to the maximum desired batch size for your deployment, Note: Please set `--torch-compile-max-bs` to the maximum desired batch size for your deployment.
which can be up to 16. The value `4` in the examples is illustrative. The value `4` in the examples is illustrative.
### Example: Running Llama-3.2-3B ### Example: Running Llama-3.2-3B
An example command to launch service of Llama-3.2-3B with BF16 precision: An example command to launch service of Llama-3.2-3B with BF16 precision:
```bash BF16 ```bash BF16
python -m sglang.launch_server \ sglang serve \
--model meta-llama/Llama-3.2-3B-Instruct \ --model-path meta-llama/Llama-3.2-3B-Instruct \
--trust-remote-code \ --trust-remote-code \
--disable-overlap-schedule \ --disable-overlap-schedule \
--device cpu \ --device cpu \
--host 0.0.0.0 \ --enable-torch-compile \
--enable-torch-compile \ --torch-compile-max-bs 16 \
--torch-compile-max-bs 16 \ --host 0.0.0.0 \
--tp 2 --tp 3
``` ```
The example command to launch service of W8A8_INT8 version of Llama-3.2-3B: The example command to launch service of W8A8_INT8 version of Llama-3.2-3B:
```bash W8A8_INT8 ```bash W8A8_INT8
python -m sglang.launch_server \ sglang serve \
--model RedHatAI/Llama-3.2-3B-quantized.w8a8 \ --model-path RedHatAI/Llama-3.2-3B-quantized.w8a8 \
--trust-remote-code \ --trust-remote-code \
--disable-overlap-schedule \ --disable-overlap-schedule \
--device cpu \ --device cpu \
--quantization w8a8_int8 \ --quantization w8a8_int8 \
--host 0.0.0.0 \ --enable-torch-compile \
--enable-torch-compile \ --torch-compile-max-bs 16 \
--torch-compile-max-bs 16 \ --host 0.0.0.0 \
--tp 2 --tp 3
``` ```
Note: The `--torch-compile-max-bs` and `--tp` settings are examples that should be adjusted for your setup. Note: The `--torch-compile-max-bs` and `--tp` settings are examples that should be adjusted for your setup.
docs_new/docs/hardware-platforms/xpu.mdx
+28 -3
View File
@@ -71,15 +71,40 @@ pip install -v . --extra-index-url https://download.pytorch.org/whl/xpu
### Install Using Docker ### Install Using Docker
The docker for XPU is under active development. Please stay tuned. [The SGLang XPU Dockerfile](https://github.com/sgl-project/sglang/blob/main/docker/xpu.Dockerfile) is provided to facilitate the installation.
Replace `<secret>` below with your [HuggingFace access token](https://huggingface.co/docs/hub/en/security-tokens).
```bash Command
# Clone the SGLang repository
git clone https://github.com/sgl-project/sglang.git
cd sglang/docker
# Build the docker image
docker build -t sglang-xpu:latest -f xpu.Dockerfile .
# Initiate a docker container
docker run \
-it \
--privileged \
--ipc=host \
--network=host \
--group-add $(getent group video | cut -d: -f3) \
--device /dev/dri \
-v /dev/dri/by-path:/dev/dri/by-path \
-v /dev/shm:/dev/shm \
-v ~/.cache/huggingface:/root/.cache/huggingface \
-p 30000:30000 \
-e "HF_TOKEN=<secret>" \
sglang-xpu:latest /bin/bash
```
## Launch of the Serving Engine ## Launch of the Serving Engine
Example command to launch SGLang serving: Example command to launch SGLang serving:
```bash ```bash
python -m sglang.launch_server \ sglang serve \
--model <MODEL_ID_OR_PATH> \ --model-path <MODEL_ID_OR_PATH> \
--trust-remote-code \ --trust-remote-code \
--disable-overlap-schedule \ --disable-overlap-schedule \
--device xpu \ --device xpu \