minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity

[Intel CPU/XPU] SGL doc updates (#23547)

Browse Source 
merge this one as doc change only.
This commit is contained in:
Zaili Wang
2026-04-24 09:23:27 +08:00
committed by GitHub
parent 23e4d381f0
commit cbc2bee547
2 changed files with 72 additions and 47 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs_new/docs/hardware-platforms/cpu_server.mdx
+44 -44
View File
@@ -222,12 +222,12 @@ Notes:
Example command to launch SGLang serving:
```bash Launch Server
python -m sglang.launch_server \
--model <MODEL_ID_OR_PATH> \
--trust-remote-code \
--disable-overlap-schedule \
--device cpu \
--host 0.0.0.0 \
sglang serve \
--model-path <MODEL_ID_OR_PATH> \
--trust-remote-code \
--disable-overlap-schedule \
--device cpu \
--host 0.0.0.0 \
--tp 6
```
@@ -320,64 +320,64 @@ or MoE (Mixture of Experts) models with fewer than 10B activated parameters, can
An example command to launch service of W8A8_INT8 DeepSeek-V3.1-Terminus on a Xeon® 6980P server:
```bash W8A8_INT8
python -m sglang.launch_server \
--model IntervitensInc/DeepSeek-V3.1-Terminus-Channel-int8 \
--trust-remote-code \
--disable-overlap-schedule \
--device cpu \
--quantization w8a8_int8 \
--host 0.0.0.0 \
--enable-torch-compile \
--torch-compile-max-bs 4 \
sglang serve \
--model-path IntervitensInc/DeepSeek-V3.1-Terminus-Channel-int8 \
--trust-remote-code \
--disable-overlap-schedule \
--device cpu \
--quantization w8a8_int8 \
--enable-torch-compile \
--torch-compile-max-bs 4 \
--host 0.0.0.0 \
--tp 6
```
Similarly, an example command to launch service of FP8 DeepSeek-V3.1-Terminus would be:
```bash FP8
python -m sglang.launch_server \
--model deepseek-ai/DeepSeek-V3.1-Terminus \
--trust-remote-code \
--disable-overlap-schedule \
--device cpu \
--host 0.0.0.0 \
--enable-torch-compile \
--torch-compile-max-bs 4 \
sglang serve \
--model-path deepseek-ai/DeepSeek-V3.1-Terminus \
--trust-remote-code \
--disable-overlap-schedule \
--device cpu \
--enable-torch-compile \
--torch-compile-max-bs 4 \
--host 0.0.0.0 \
--tp 6
```
Note: Please set `--torch-compile-max-bs` to the maximum desired batch size for your deployment,
which can be up to 16. The value `4` in the examples is illustrative.
Note: Please set `--torch-compile-max-bs` to the maximum desired batch size for your deployment.
The value `4` in the examples is illustrative.
### Example: Running Llama-3.2-3B
An example command to launch service of Llama-3.2-3B with BF16 precision:
```bash BF16
python -m sglang.launch_server \
--model meta-llama/Llama-3.2-3B-Instruct \
--trust-remote-code \
--disable-overlap-schedule \
--device cpu \
--host 0.0.0.0 \
--enable-torch-compile \
--torch-compile-max-bs 16 \
--tp 2
sglang serve \
--model-path meta-llama/Llama-3.2-3B-Instruct \
--trust-remote-code \
--disable-overlap-schedule \
--device cpu \
--enable-torch-compile \
--torch-compile-max-bs 16 \
--host 0.0.0.0 \
--tp 3
```
The example command to launch service of W8A8_INT8 version of Llama-3.2-3B:
```bash W8A8_INT8
python -m sglang.launch_server \
--model RedHatAI/Llama-3.2-3B-quantized.w8a8 \
--trust-remote-code \
--disable-overlap-schedule \
--device cpu \
--quantization w8a8_int8 \
--host 0.0.0.0 \
--enable-torch-compile \
--torch-compile-max-bs 16 \
--tp 2
sglang serve \
--model-path RedHatAI/Llama-3.2-3B-quantized.w8a8 \
--trust-remote-code \
--disable-overlap-schedule \
--device cpu \
--quantization w8a8_int8 \
--enable-torch-compile \
--torch-compile-max-bs 16 \
--host 0.0.0.0 \
--tp 3
```
Note: The `--torch-compile-max-bs` and `--tp` settings are examples that should be adjusted for your setup.