minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 14 Packages Projects Releases Wiki Activity

[Docs] Sync docs_new with legacy docs and update migration redirects (#23337)

Browse Source 
Co-authored-by: Mingyi <wisclmy0611@gmail.com>
This commit is contained in:
zijiexia
2026-04-21 00:15:17 -07:00
committed by GitHub
co-authored by Mingyi
parent f63def8510
commit 900aad5f72
179 changed files with 16014 additions and 8162 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs_new/docs/hardware-platforms/amd-gpus.mdx
-194
View File
@@ -1,194 +0,0 @@
---
title: "AMD GPUs"
---
This document describes how run SGLang on AMD GPUs. If you encounter issues or have questions, please [open an issue](https://github.com/sgl-project/sglang/issues).
## System Configuration
When using AMD GPUs (such as MI300X), certain system-level optimizations help ensure stable performance. Here we take MI300X as an example. AMD provides official documentation for MI300X optimization and system tuning:
* [AMD MI300X Tuning Guides](https://rocm.docs.amd.com/en/latest/how-to/tuning-guides/mi300x/index.html)
* [LLM inference performance validation on AMD Instinct MI300X](https://rocm.docs.amd.com/en/latest/how-to/rocm-for-ai/inference/vllm-benchmark.html)
* [AMD Instinct MI300X System Optimization](https://rocm.docs.amd.com/en/latest/how-to/system-optimization/mi300x.html)
* [AMD Instinct MI300X Workload Optimization](https://rocm.docs.amd.com/en/latest/how-to/rocm-for-ai/inference-optimization/workload.html)
* [Supercharge DeepSeek-R1 Inference on AMD Instinct MI300X](https://rocm.blogs.amd.com/artificial-intelligence/DeepSeekR1-Part2/README.html)
<Note>
We strongly recommend reading these docs and guides entirely to fully utilize your system.
</Note>
Below are a few key settings to confirm or enable for SGLang:
### Update GRUB Settings
In `/etc/default/grub`, append the following to `GRUB_CMDLINE_LINUX`:
<CodeGroup>
```text GRUB Configuration
pci=realloc=off iommu=pt
```
</CodeGroup>
Afterward, run `sudo update-grub` (or your distro's equivalent) and reboot.
### Disable NUMA Auto-Balancing
<CodeGroup>
```bash Disable NUMA
sudo sh -c 'echo 0 > /proc/sys/kernel/numa_balancing'
```
</CodeGroup>
You can automate or verify this change using [this helpful script](https://github.com/ROCm/triton/blob/rocm_env/scripts/amd/env_check.sh).
Again, please go through the entire documentation to confirm your system is using the recommended configuration.
## Install SGLang
<Tabs>
<Tab title="Docker (Recommended)">
The docker images are available on Docker Hub at [lmsysorg/sglang](https://hub.docker.com/r/lmsysorg/sglang/tags), built from [rocm.Dockerfile](https://github.com/sgl-project/sglang/tree/main/docker).
1. **Build the docker image**
If you use pre-built images, you can skip this step and replace `sglang_image` with the pre-built image names in the steps below.
<CodeGroup>
```bash Build Image
docker build -t sglang_image -f rocm.Dockerfile .
```
</CodeGroup>
2. **Create a convenient alias**
<CodeGroup>
```bash Create Alias
alias drun='docker run -it --rm --network=host --privileged --device=/dev/kfd --device=/dev/dri \
--ipc=host --shm-size 16G --group-add video --cap-add=SYS_PTRACE \
--security-opt seccomp=unconfined \
-v $HOME/dockerx:/dockerx \
-v /data:/data'
```
</CodeGroup>
If you are using RDMA, please note that:
* `--network host` and `--privileged` are required by RDMA. If you don't need RDMA, you can remove them.
* You may need to set `NCCL_IB_GID_INDEX` if you are using RoCE, for example: `export NCCL_IB_GID_INDEX=3`.
3. **Launch the server**
<Note>
Replace `<secret>` below with your [huggingface hub token](https://huggingface.co/docs/hub/en/security-tokens).
</Note>
<CodeGroup>
```bash Launch Server
drun -p 30000:30000 \
-v ~/.cache/huggingface:/root/.cache/huggingface \
--env "HF_TOKEN=<secret>" \
sglang_image \
python3 -m sglang.launch_server \
--model-path NousResearch/Meta-Llama-3.1-8B \
--host 0.0.0.0 \
--port 30000
```
</CodeGroup>
4. **Verify the installation**
You can run a benchmark in another terminal or refer to [other docs](../basic_usage/openai_api_completions) to send requests to the engine.
<CodeGroup>
```bash Run Benchmark
drun sglang_image \
python3 -m sglang.bench_serving \
--backend sglang \
--dataset-name random \
--num-prompts 4000 \
--random-input 128 \
--random-output 128
```
</CodeGroup>
With your AMD system properly configured and SGLang installed, you can now fully leverage AMD hardware to power SGLang's machine learning capabilities.
</Tab>
<Tab title="From Source">
1. **Clone the repository**
Clone the SGLang repository.
<CodeGroup>
```bash
git clone https://github.com/sgl-project/sglang.git
cd sglang
```
</CodeGroup>
2. **Compile sgl-kernel**
Upgrade pip and compile the sgl-kernel for ROCm support.
<CodeGroup>
```bash
pip install --upgrade pip
cd sgl-kernel
python setup_rocm.py install
```
</CodeGroup>
3. **Install sglang package**
Install the SGLang Python package with HIP and diffusion support.
<CodeGroup>
```bash
cd ..
rm -rf python/pyproject.toml && mv python/pyproject_other.toml python/pyproject.toml
pip install -e "python[all_hip]"
```
</CodeGroup>
</Tab>
</Tabs>
## Examples
### Running DeepSeek-V3
The only difference when running DeepSeek-V3 is in how you start the server.
<CodeGroup>
```bash DeepSeek-V3
drun -p 30000:30000 \
-v ~/.cache/huggingface:/root/.cache/huggingface \
--ipc=host \
--env "HF_TOKEN=<secret>" \
sglang_image \
python3 -m sglang.launch_server \
--model-path deepseek-ai/DeepSeek-V3 \
--tp 8 \
--trust-remote-code \
--host 0.0.0.0 \
--port 30000
```
</CodeGroup>
[Running DeepSeek-R1 on a single NDv5 MI300X VM](https://techcommunity.microsoft.com/blog/azurehighperformancecomputingblog/running-deepseek-r1-on-a-single-ndv5-mi300x-vm/4372726) could also be a good reference.
### Running Llama3.1
Running Llama3.1 is nearly identical to running DeepSeek-V3. The only difference is in the model specified when starting the server.
<CodeGroup>
```bash Llama3.1
drun -p 30000:30000 \
-v ~/.cache/huggingface:/root/.cache/huggingface \
--ipc=host \
--env "HF_TOKEN=<secret>" \
sglang_image \
python3 -m sglang.launch_server \
--model-path meta-llama/Meta-Llama-3.1-8B-Instruct \
--tp 8 \
--trust-remote-code \
--host 0.0.0.0 \
--port 30000
```
</CodeGroup>
### Warmup Step
When the server displays `The server is fired up and ready to roll!`, it means the startup is successful.
docs_new/docs/hardware-platforms/amd_gpu.mdx
+196
View File
@@ -0,0 +1,196 @@
---
title: "AMD GPUs"
---
This document describes how to run SGLang on AMD GPUs. If you encounter issues or have questions, please [open an issue](https://github.com/sgl-project/sglang/issues).
## System Configuration
When using AMD GPUs (such as MI300X), certain system-level optimizations help ensure stable performance. Here we take MI300X as an example. AMD provides official documentation for MI300X optimization and system tuning:
- [AMD MI300X Tuning Guides](https://rocm.docs.amd.com/en/latest/how-to/tuning-guides/mi300x/index.html)
- [LLM inference performance validation on AMD Instinct MI300X](https://rocm.docs.amd.com/en/latest/how-to/rocm-for-ai/inference/vllm-benchmark.html)
- [AMD Instinct MI300X System Optimization](https://rocm.docs.amd.com/en/latest/how-to/system-optimization/mi300x.html)
- [AMD Instinct MI300X Workload Optimization](https://rocm.docs.amd.com/en/latest/how-to/rocm-for-ai/inference-optimization/workload.html)
- [Supercharge DeepSeek-R1 Inference on AMD Instinct MI300X](https://rocm.blogs.amd.com/artificial-intelligence/DeepSeekR1-Part2/README.html)
**NOTE:** We strongly recommend reading these docs and guides entirely to fully utilize your system.
Below are a few key settings to confirm or enable for SGLang:
### Update GRUB Settings
In `/etc/default/grub`, append the following to `GRUB_CMDLINE_LINUX`:
```text GRUB Configuration
pci=realloc=off iommu=pt
```
Afterward, run `sudo update-grub` (or your distro’s equivalent) and reboot.
### Disable NUMA Auto-Balancing
```bash Disable NUMA
sudo sh -c 'echo 0 > /proc/sys/kernel/numa_balancing'
```
You can automate or verify this change using [this helpful script](https://github.com/ROCm/triton/blob/rocm_env/scripts/amd/env_check.sh).
Again, please go through the entire documentation to confirm your system is using the recommended configuration.
## Install SGLang
You can install SGLang using one of the methods below.
### Install from Source
```bash Command
# Use the last release branch
git clone -b v0.5.9 https://github.com/sgl-project/sglang.git
cd sglang
# Compile sgl-kernel
pip install --upgrade pip
cd sgl-kernel
python setup_rocm.py install
# Install sglang python package along with diffusion support
cd ..
rm -rf python/pyproject.toml && mv python/pyproject_other.toml python/pyproject.toml
pip install -e "python[all_hip]"
```
### Install Using Docker (Recommended)
The docker images are available on Docker Hub at [lmsysorg/sglang](https://hub.docker.com/r/lmsysorg/sglang/tags), built from [rocm.Dockerfile](https://github.com/sgl-project/sglang/tree/main/docker).
The steps below show how to build and use an image.
1. Build the docker image.
If you use pre-built images, you can skip this step and replace `sglang_image` with the pre-built image names in the steps below.
```bash Command
docker build -t sglang_image -f rocm.Dockerfile .
```
2. Create a convenient alias.
```bash Command
alias drun='docker run -it --rm --network=host --privileged --device=/dev/kfd --device=/dev/dri \
--ipc=host --shm-size 16G --group-add video --cap-add=SYS_PTRACE \
--security-opt seccomp=unconfined \
-v $HOME/dockerx:/dockerx \
-v /data:/data'
```
If you are using RDMA, please note that:
- `--network host` and `--privileged` are required by RDMA. If you don't need RDMA, you can remove them.
- You may need to set `NCCL_IB_GID_INDEX` if you are using RoCE, for example: `export NCCL_IB_GID_INDEX=3`.
3. Launch the server.
**NOTE:** Replace `<secret>` below with your [huggingface hub token](https://huggingface.co/docs/hub/en/security-tokens).
```bash Command
drun -p 30000:30000 \
-v ~/.cache/huggingface:/root/.cache/huggingface \
--env "HF_TOKEN=<secret>" \
sglang_image \
python3 -m sglang.launch_server \
--model-path NousResearch/Meta-Llama-3.1-8B \
--host 0.0.0.0 \
--port 30000
```
4. To verify the utility, you can run a benchmark in another terminal or refer to [other docs](../basic_usage/openai_api_completions) to send requests to the engine.
```bash Command
drun sglang_image \
python3 -m sglang.bench_serving \
--backend sglang \
--dataset-name random \
--num-prompts 4000 \
--random-input 128 \
--random-output 128
```
With your AMD system properly configured and SGLang installed, you can now fully leverage AMD hardware to power SGLang’s machine learning capabilities.
## Quantization on AMD GPUs
The [Quantization documentation](../advanced_features/quantization#platform-compatibility) has a full compatibility matrix. The short version: FP8, AWQ, MXFP4, W8A8, GPTQ, compressed-tensors, Quark, and **petit_nvfp4** (NVFP4 on ROCm via [Petit](https://github.com/causalflow-ai/petit-kernel)) all work on AMD. Methods that depend on Marlin or NVIDIA-specific kernels (`awq_marlin`, `gptq_marlin`, `gguf`, `modelopt_fp8`, `modelopt_fp4`) do not.
A few things to keep in mind:
- FP8 works via Aiter or Triton. Pre-quantized FP8 models like DeepSeek-V3/R1 work out of the box.
- AWQ uses Triton dequantization kernels on AMD. The faster Marlin path is not available.
- MXFP4 requires CDNA3/CDNA4 and `SGLANG_USE_AITER=1`.
- `petit_nvfp4` enables NVFP4 models (e.g., [Llama 3.3 70B FP4](https://huggingface.co/nvidia/Llama-3.3-70B-Instruct-FP4)) on MI250/MI300X via [Petit](https://github.com/causalflow-ai/petit-kernel). Install with `pip install petit-kernel`; no `--quantization` flag needed when loading pre-quantized NVFP4 models.
- `quark_int4fp8_moe` is an AMD-only online quantization method for MoE models on CDNA3/CDNA4.
Several of these backends are accelerated by [Aiter](https://github.com/ROCm/aiter). Enable it with:
```bash Command
export SGLANG_USE_AITER=1
```
Example -- serving an AWQ model:
```bash Command
python3 -m sglang.launch_server \
--model-path hugging-quants/Mixtral-8x7B-Instruct-v0.1-AWQ-INT4 \
--trust-remote-code \
--port 30000 --host 0.0.0.0
```
Example -- FP8 online quantization:
```bash Command
python3 -m sglang.launch_server \
--model-path meta-llama/Meta-Llama-3.1-8B-Instruct \
--quantization fp8 \
--port 30000 --host 0.0.0.0
```
## Examples
### Running DeepSeek-V3
The only difference when running DeepSeek-V3 is in how you start the server. Here's an example command:
```bash Command
drun -p 30000:30000 \
-v ~/.cache/huggingface:/root/.cache/huggingface \
--ipc=host \
--env "HF_TOKEN=<secret>" \
sglang_image \
python3 -m sglang.launch_server \
--model-path deepseek-ai/DeepSeek-V3 \ # <- here
--tp 8 \
--trust-remote-code \
--host 0.0.0.0 \
--port 30000
```
[Running DeepSeek-R1 on a single NDv5 MI300X VM](https://techcommunity.microsoft.com/blog/azurehighperformancecomputingblog/running-deepseek-r1-on-a-single-ndv5-mi300x-vm/4372726) could also be a good reference.
### Running Llama3.1
Running Llama3.1 is nearly identical to running DeepSeek-V3. The only difference is in the model specified when starting the server, shown by the following example command:
```bash Command
drun -p 30000:30000 \
-v ~/.cache/huggingface:/root/.cache/huggingface \
--ipc=host \
--env "HF_TOKEN=<secret>" \
sglang_image \
python3 -m sglang.launch_server \
--model-path meta-llama/Meta-Llama-3.1-8B-Instruct \ # <- here
--tp 8 \
--trust-remote-code \
--host 0.0.0.0 \
--port 30000
```
### Warmup Step
When the server displays `The server is fired up and ready to roll!`, it means the startup is successful.
docs_new/docs/hardware-platforms/apple_metal.mdx
+24
View File
@@ -0,0 +1,24 @@
---
title: "Apple Silicon with Metal"
metatags:
description: "Run SGLang on Apple Silicon using the Metal backend."
---
This document describes how run SGLang on Apple Silicon using [Metal](https://developer.apple.com/metal/). If you encounter issues or have questions, please [open an issue](https://github.com/sgl-project/sglang/issues).
## Install SGLang
You can install SGLang using one of the methods below.
### Install from Source
```bash
# Use the default branch
git clone https://github.com/sgl-project/sglang.git
cd sglang
# Install sglang python package
pip install --upgrade pip
rm -f python/pyproject.toml && mv python/pyproject_other.toml python/pyproject.toml
uv pip install -e "python[all_mps]"
```
docs_new/docs/hardware-platforms/ascend-npus/DeepSeek-Examples.mdx
-309
View File
@@ -1,309 +0,0 @@
## Running DeepSeek-V3
### Running DeepSeek in PD mixed mode on 1 x Atlas 800I A3
W4A8 Model weights could be found [here](https://modelers.cn/models/Modelers_Park/DeepSeek-R1-0528-w4a8).
<CodeGroup>
```shell Launch Server
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export STREAMS_PER_DEVICE=32
#Deepep communication settings
export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=32
export HCCL_BUFFSIZE=1600
#spec overlap
export SGLANG_ENABLE_SPEC_V2=1
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
#npu acceleration operator
export SGLANG_NPU_USE_MLAPO=1
export SGLANG_USE_FIA_NZ=1
python3 -m sglang.launch_server \
--model-path ${MODEL_PATH} \
--tp 16 \
--trust-remote-code \
--attention-backend ascend \
--device npu \
--quantization modelslim \
--watchdog-timeout 9000 \
--cuda-graph-bs 8 16 24 28 32 \
--mem-fraction-static 0.68 \
--max-running-requests 128 \
--context-length 8188 \
--disable-radix-cache \
--chunked-prefill-size -1 \
--max-prefill-tokens 16384 \
--moe-a2a-backend deepep \
--deepep-mode auto \
--enable-dp-attention \
--dp-size 4 \
--enable-dp-lm-head \
--speculative-algorithm NEXTN \
--speculative-num-steps 3 \
--speculative-eagle-topk 1 \
--speculative-num-draft-tokens 4 \
--dtype bfloat16
```
</CodeGroup>
### Running DeepSeek with PD disaggregation mode on 2 x Atlas 800I A3
W4A8 Model weights could be found [here](https://modelers.cn/models/Modelers_Park/DeepSeek-R1-0528-w4a8).
<Tabs>
<Tab title="Prefill">
```shell Command
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export STREAMS_PER_DEVICE=32
#memfabric config store
export ASCEND_MF_STORE_URL="tcp://<PREFILL_HOST_IP>:<PORT>"
#Deepep communication settings
export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
export HCCL_BUFFSIZE=1536
#npu acceleration operator
export SGLANG_NPU_USE_MLAPO=1
export SGLANG_USE_FIA_NZ=1
export TASK_QUEUE_ENABLE=2
python -m sglang.launch_server \
--model-path ${MODEL_PATH} \
--host $PREFILL_HOST_IP \
--port 8000 \
--disaggregation-mode prefill \
--disaggregation-bootstrap-port 8996 \
--disaggregation-transfer-backend ascend \
--trust-remote-code \
--nnodes 1 \
--node-rank 0 \
--tp-size 16 \
--mem-fraction-static 0.6 \
--attention-backend ascend \
--device npu \
--quantization modelslim \
--load-balance-method round_robin \
--max-running-requests 8 \
--context-length 8192 \
--disable-radix-cache \
--chunked-prefill-size -1 \
--max-prefill-tokens 28680 \
--moe-a2a-backend deepep \
--deepep-mode normal \
--speculative-algorithm NEXTN \
--speculative-num-steps 3 \
--speculative-eagle-topk 1 \
--speculative-num-draft-tokens 4 \
--dp-size 2 \
--enable-dp-attention \
--disable-shared-experts-fusion \
--dtype bfloat16
```
</Tab>
<Tab title="Decode">
```shell Command
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export STREAMS_PER_DEVICE=32
#memfabric config store
export ASCEND_MF_STORE_URL="tcp://<PREFILL_HOST_IP>:<PORT>"
#Deepep communication settings
export HCCL_BUFFSIZE=720
export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=88
#spec overlap
export SGLANG_ENABLE_SPEC_V2=1
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
#npu acceleration operator
unset TASK_QUEUE_ENABLE
export SGLANG_NPU_USE_MLAPO=1
export SGLANG_USE_FIA_NZ=1
export ENABLE_MOE_NZ=1
# suggest max-running-requests <= max-cuda-graph-bs * dp_size, Because when this value is exceeded, performance will significantly degrade.
python -m sglang.launch_server \
--model-path ${MODEL_PATH} \
--disaggregation-mode decode \
--host $DECODE_HOST_IP \
--port 8001 \
--trust-remote-code \
--nnodes 1 \
--node-rank 0 \
--tp-size 16 \
--dp-size 16 \
--mem-fraction-static 0.8 \
--max-running-requests 352 \
--attention-backend ascend \
--device npu \
--quantization modelslim \
--prefill-round-robin-balance \
--moe-a2a-backend deepep \
--enable-dp-attention \
--deepep-mode low_latency \
--enable-dp-lm-head \
--cuda-graph-bs 8 10 12 14 16 18 20 22 \
--disaggregation-transfer-backend ascend \
--watchdog-timeout 9000 \
--context-length 8192 \
--speculative-algorithm NEXTN \
--speculative-num-steps 3 \
--speculative-eagle-topk 1 \
--speculative-num-draft-tokens 4 \
--disable-shared-experts-fusion \
--dtype bfloat16 \
--tokenizer-worker-num 4
```
</Tab>
<Tab title="Router">
```shell Command
python -m sglang_router.launch_router \
--pd-disaggregation \
--policy cache_aware \
--prefill http://<PREFILL_HOST_IP>:8000 8996 \
--decode http://<DECODE_HOST_IP>:8001 \
--host 127.0.0.1 \
--port 6688
```
</Tab>
</Tabs>
### Running DeepSeek with PD disaggregation on 4 x Atlas 800I A3
W8A8 Model weights could be found [here](https://modelers.cn/models/State_Cloud/Deepseek-R1-bf16-hfd-w8a8).
<Tabs>
<Tab title="Prefill & Decode">
```shell Command
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000
export SGLANG_SET_CPU_AFFINITY=1
unset ASCEND_LAUNCH_BLOCKING
source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh
export PATH=/usr/local/Ascend/8.5.0/compiler/bishengir/bin:$PATH
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export STREAMS_PER_DEVICE=32
export ASCEND_MF_STORE_URL="tcp://your prefill ip1:24669"
P_IP=('your prefill ip1' 'your prefill ip2')
D_IP=('your decode ip1' 'your decode ip2')
MODEL_PATH=xxx
export SGLANG_NPU_USE_MLAPO=1
export SGLANG_USE_FIA_NZ=1
LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'`
LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'`
echo "${LOCAL_HOST1}"
echo "${LOCAL_HOST2}"
# prefill
for i in "${!P_IP[@]}";
do
if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]];
then
echo "${P_IP[$i]}"
export HCCL_BUFFSIZE=1536
export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
export TASK_QUEUE_ENABLE=2
export HCCL_SOCKET_IFNAME=lo
export GLOO_SOCKET_IFNAME=lo
python -m sglang.launch_server --model-path ${MODEL_PATH} --disaggregation-mode prefill --host ${P_IP[$i]} \
--port 8000 --disaggregation-bootstrap-port $((8998+$i)) --trust-remote-code --nnodes 1 --node-rank 0 \
--tp-size 16 --mem-fraction-static 0.81 --attention-backend ascend --device npu --quantization modelslim \
--disaggregation-transfer-backend ascend --max-running-requests 8 --context-length 8192 --disable-radix-cache \
--chunked-prefill-size -1 --max-prefill-tokens 28680 --moe-a2a-backend deepep --deepep-mode normal \
--speculative-algorithm NEXTN --speculative-num-steps 1 --speculative-eagle-topk 1 --speculative-num-draft-tokens 2 \
--dp-size 2 --enable-dp-attention --disable-shared-experts-fusion --dtype bfloat16 --enable-attn-tp-input-scattered
NODE_RANK=$i
break
fi
done
# decode
for i in "${!D_IP[@]}";
do
if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]];
then
echo "${D_IP[$i]}"
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
export SGLANG_ENABLE_SPEC_V2=1
export HCCL_BUFFSIZE=650
export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=78
export TASK_QUEUE_ENABLE=1
export SGLANG_SCHEDULER_SKIP_ALL_GATHER=1
export HCCL_SOCKET_IFNAME=xxx
export GLOO_SOCKET_IFNAME=xxx
python -m sglang.launch_server --model-path ${MODEL_PATH} --disaggregation-mode decode --host ${D_IP[$i]} \
--port 8001 --trust-remote-code --dist-init-addr ${D_IP[0]}:5000 --nnodes 2 --node-rank $i --tp-size 32 --dp-size 32 \
--mem-fraction-static 0.815 --max-running-requests 832 --attention-backend ascend --device npu --quantization modelslim \
--moe-a2a-backend deepep --enable-dp-attention --deepep-mode low_latency --enable-dp-lm-head --moe-dense-tp 1 \
--cuda-graph-bs 12 14 16 18 20 22 24 26 --disaggregation-transfer-backend ascend --watchdog-timeout 9000 --context-length 8192 \
--speculative-algorithm NEXTN --speculative-num-steps 2 --speculative-eagle-topk 1 --speculative-num-draft-tokens 3 \
--tokenizer-worker-num 4 --prefill-round-robin-balance --disable-shared-experts-fusion --dtype bfloat16 \
--load-balance-method decode_round_robin
NODE_RANK=$i
break
fi
done
```
</Tab>
<Tab title="Router">
```shell Command
export SGLANG_DP_ROUND_ROBIN=1
python -m sglang_router.launch_router \
--pd-disaggregation \
--policy cache_aware \
--prefill http://P_IP:8000 8998 \
--prefill http://P_IP:8000 8999 \
--decode http://D_IP:8001 \
--host 127.0.0.1 \
--port 6688 \
--mini-lb
```
</Tab>
</Tabs>
### Test GSM8K
<CodeGroup>
```python Test GSM8K
from types import SimpleNamespace
from sglang.test.few_shot_gsm8k import run_eval
def gsm8k():
args = SimpleNamespace(
num_shots=5,
data_path=None,
num_questions=200,
max_new_tokens=512,
parallel=32,
host=f"http://127.0.0.1",
port=6688,
)
metrics = run_eval(args)
print(f"{metrics=}")
print(f"{metrics['accuracy']=}")
if __name__ == "__main__":
gsm8k()
```
</CodeGroup>
docs_new/docs/hardware-platforms/ascend-npus/Qwen3.5.mdx
-106
View File
@@ -1,106 +0,0 @@
## Environment Preparation
### Installation
The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the quay.io platform. You can directly pull it.
<CodeGroup>
```bash Pull and Start Container
#Atlas 800 A3
docker pull swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:cann8.5.0-a3-qwen3.5
#Atlas 800 A2
docker pull swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:cann8.5.0-910b-qwen3.5
#start container
docker run -itd --shm-size=16g --privileged=true --name ${NAME} \
--privileged=true --net=host \
-v /var/queue_schedule:/var/queue_schedule \
-v /etc/ascend_install.info:/etc/ascend_install.info \
-v /usr/local/sbin:/usr/local/sbin \
-v /usr/local/Ascend/driver:/usr/local/Ascend/driver \
-v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \
--device=/dev/davinci0:/dev/davinci0 \
--device=/dev/davinci1:/dev/davinci1 \
--device=/dev/davinci2:/dev/davinci2 \
--device=/dev/davinci3:/dev/davinci3 \
--device=/dev/davinci4:/dev/davinci4 \
--device=/dev/davinci5:/dev/davinci5 \
--device=/dev/davinci6:/dev/davinci6 \
--device=/dev/davinci7:/dev/davinci7 \
--device=/dev/davinci8:/dev/davinci8 \
--device=/dev/davinci9:/dev/davinci9 \
--device=/dev/davinci10:/dev/davinci10 \
--device=/dev/davinci11:/dev/davinci11 \
--device=/dev/davinci12:/dev/davinci12 \
--device=/dev/davinci13:/dev/davinci13 \
--device=/dev/davinci14:/dev/davinci14 \
--device=/dev/davinci15:/dev/davinci15 \
--device=/dev/davinci_manager:/dev/davinci_manager \
--device=/dev/hisi_hdc:/dev/hisi_hdc \
--entrypoint=bash \
swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:${TAG}
```
</CodeGroup>
## Deployment
### Single-node Deployment
- Quantized model `qwen35_w8a8` can be deployed on 1 Atlas 800 A3 (64G × 16) .
Run the following script to execute online inference.
<CodeGroup>
```shell Launch Server
# high performance cpu
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000
# bind cpu
export SGLANG_SET_CPU_AFFINITY=1
unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING
# cann
source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh
export STREAMS_PER_DEVICE=32
export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600
export SGLANG_ENABLE_SPEC_V2=1
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
export SGLANG_NPU_USE_MULTI_STREAM=1
export HCCL_BUFFSIZE=1000
export HCCL_OP_EXPANSION_MODE=AIV
export HCCL_SOCKET_IFNAME=lo
export GLOO_SOCKET_IFNAME=lo
python3 -m sglang.launch_server \
--model-path $MODEL_PATH \
--attention-backend ascend \
--device npu \
--tp-size 16 --nnodes 1 --node-rank 0 \
--chunked-prefill-size 16384 --max-prefill-tokens 280000 \
--trust-remote-code \
--host 127.0.0.1 \
--mem-fraction-static 0.7 \
--port 8000 \
--cuda-graph-bs 16 \
--quantization modelslim \
--enable-multimodal \
--mm-attention-backend ascend_attn \
--dtype bfloat16
```
</CodeGroup>
### Prefill-Decode Disaggregation
Not test yet.
### Using Benchmark
Refer to [Benchmark and Profiling](../../developer_guide/benchmark_and_profiling) for details.
docs_new/docs/hardware-platforms/ascend-npus/SGLang-installation-with-NPUs-support.mdx
-318
View File
@@ -1,318 +0,0 @@
---
title: SGLang installation with NPUs support
---
You can install SGLang using any of the methods below. Please go through `System Settings` section to ensure the clusters are roaring at max performance. Feel free to leave an issue [here at sglang](https://github.com/sgl-project/sglang/issues) if you encounter any issues or have any problems.
## Component Version Mapping For SGLang
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
<colgroup>
<col style={{width: "34%"}} />
<col style={{width: "33%"}} />
<col style={{width: "33%"}} />
</colgroup>
<thead>
<tr style={{borderBottom: "2px solid #d55816"}}>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Component</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Version</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Obtain Way</th>
</tr>
</thead>
<tbody>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>HDK</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>25.3.RC1</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>[<Icon icon="download" />](https://hiascend.com/hardware/firmware-drivers/commercial?product=7\&model=33)</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>CANN</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>8.5.0</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>[Obtain Images](#obtain-cann-image)</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Pytorch Adapter</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>7.3.0</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>[<Icon icon="external-link" />](https://gitcode.com/Ascend/pytorch/releases)</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>MemFabric</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1.0.5</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`pip install memfabric-hybrid==1.0.5`</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Triton</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>3.2.0</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`pip install triton-ascend`</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Bisheng</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>20251121</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>[<Icon icon="external-link" />](https://sglang-ascend.obs.cn-east-3.myhuaweicloud.com/sglang/triton_ascend/Ascend-BiSheng-toolkit_aarch64_20251121.run)</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>SGLang NPU Kernel</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>NA</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>[<Icon icon="external-link" />](https://github.com/sgl-project/sgl-kernel-npu/releases)</td>
</tr>
</tbody>
</table>
<Accordion title="Obtain CANN Image" defaultOpen="true">
You can obtain the dependency of a specified version of CANN through an image.
```bash
# for Atlas 800I A3 and Ubuntu OS
docker pull quay.io/ascend/cann:8.5.0-a3-ubuntu22.04-py3.11
# for Atlas 800I A2 and Ubuntu OS
docker pull quay.io/ascend/cann:8.5.0-910b-ubuntu22.04-py3.11
```
</Accordion>
## Preparing the Running Environment
<Tabs>
<Tab title="Source">
<AccordionGroup>
<Accordion title="Python Version">
Only `python==3.11` is supported currently. If you don't want to break system pre-installed python, try installing with [conda](https://github.com/conda/conda).
```bash
conda create --name sglang_npu python=3.11
conda activate sglang_npu
```
</Accordion>
<Accordion title="CANN">
Prior to start work with SGLang on Ascend you need to install CANN Toolkit, Kernels operator package and NNAL version 8.3.RC2 or higher, check the [installation guide](https://www.hiascend.com/document/detail/zh/CANNCommunityEdition/83RC1/softwareinst/instg/instg_0008.html?Mode=PmIns\&InstallType=local\&OS=openEuler\&Software=cannToolKit)
</Accordion>
<Accordion title="MemFabric-Hybrid">
If you want to use PD disaggregation mode, you need to install MemFabric-Hybrid. MemFabric-Hybrid is a drop-in replacement of Mooncake Transfer Engine that enables KV cache transfer on Ascend NPU clusters.
```bash
pip install memfabric-hybrid==1.0.5
```
</Accordion>
<Accordion title="Pytorch and Pytorch Framework Adaptor on Ascend">
```bash
PYTORCH_VERSION=2.8.0
TORCHVISION_VERSION=0.23.0
TORCH_NPU_VERSION=2.8.0
pip install torch==$PYTORCH_VERSION torchvision==$TORCHVISION_VERSION --index-url https://download.pytorch.org/whl/cpu
pip install torch_npu==$TORCH_NPU_VERSION
```
If you are using other versions of `torch` and install `torch_npu`, check [installation guide](https://github.com/Ascend/pytorch/blob/master/README)
</Accordion>
<Accordion title="Triton on Ascend">
We provide our own implementation of Triton for Ascend.
```bash
BISHENG_NAME="Ascend-BiSheng-toolkit_aarch64_20251121.run"
BISHENG_URL="https://sglang-ascend.obs.cn-east-3.myhuaweicloud.com/sglang/triton_ascend/${BISHENG_NAME}"
wget -O "${BISHENG_NAME}" "${BISHENG_URL}" && chmod a+x "${BISHENG_NAME}" && "./${BISHENG_NAME}" --install && rm "${BISHENG_NAME}"
```
```bash
pip install triton-ascend
```
For installation of Triton on Ascend nightly builds or from sources, follow [installation guide](https://gitcode.com/Ascend/triton-ascend/blob/master/docs/sources/getting-started/installation)
</Accordion>
<Accordion title="SGLang Kernels NPU">
We provide SGL kernels for Ascend NPU, check [installation guide](https://github.com/sgl-project/sgl-kernel-npu/blob/main/python/sgl_kernel_npu/README).
</Accordion>
<Accordion title="DeepEP-compatible Library">
We provide a DeepEP-compatible Library as a drop-in replacement of deepseek-ai's DeepEP library, check the [installation guide](https://github.com/sgl-project/sgl-kernel-npu/blob/main/python/deep_ep/README).
</Accordion>
<Accordion title="Installing SGLang from source">
```bash
# Use the last release branch
git clone https://github.com/sgl-project/sglang.git
cd sglang
mv python/pyproject_npu.toml python/pyproject.toml
pip install -e python[all_npu]
```
</Accordion>
</AccordionGroup>
</Tab>
<Tab title="Docker">
### Obtain Image
You can download the SGLang image or build an image based on Dockerfile to obtain the Ascend NPU image.
1. **Download SGLang image**
```bash
dockerhub: docker.io/lmsysorg/sglang:$tag
# Main-based tag, change main to specific version like v0.5.6,
# you can get image for specific version
Atlas 800I A3 : {main}-cann8.5.0-a3
Atlas 800I A2: {main}-cann8.5.0-910b
```
2. **Build an image based on Dockerfile**
```bash
# Clone the SGLang repository
git clone https://github.com/sgl-project/sglang.git
cd sglang/docker
# Build the docker image
# If there are network errors, please modify the Dockerfile to use offline dependencies or use a proxy
docker build -t <image_name> -f npu.Dockerfile .
```
### Create Docker
<Info>`--privileged` and `--network=host` are required by RDMA, which is typically needed by Ascend NPU clusters.</Info>
<Note>The following docker command is based on Atlas 800I A3 machines. If you are using Atlas 800I A2, make sure only `davinci[0-7]` are mapped into container.</Note>
```bash
alias drun='docker run -it --rm --privileged --network=host --ipc=host --shm-size=16g \
--device=/dev/davinci0 --device=/dev/davinci1 --device=/dev/davinci2 --device=/dev/davinci3 \
--device=/dev/davinci4 --device=/dev/davinci5 --device=/dev/davinci6 --device=/dev/davinci7 \
--device=/dev/davinci8 --device=/dev/davinci9 --device=/dev/davinci10 --device=/dev/davinci11 \
--device=/dev/davinci12 --device=/dev/davinci13 --device=/dev/davinci14 --device=/dev/davinci15 \
--device=/dev/davinci_manager --device=/dev/hisi_hdc \
--volume /usr/local/sbin:/usr/local/sbin --volume /usr/local/Ascend/driver:/usr/local/Ascend/driver \
--volume /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \
--volume /etc/ascend_install.info:/etc/ascend_install.info \
--volume /var/queue_schedule:/var/queue_schedule --volume ~/.cache/:/root/.cache/'
# Add HF_TOKEN env for download model by SGLang.
drun --env "HF_TOKEN=<secret>" \
<image_name> \
python3 -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct --attention-backend ascend
```
</Tab>
</Tabs>
## System Settings
<AccordionGroup>
<Accordion title="CPU performance power scheme" defaultOpen="true">
The default power scheme on Ascend hardware is `ondemand` which could affect performance, changing it to `performance` is recommended.
```bash
echo performance | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
# Make sure changes are applied successfully
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor # shows performance
```
</Accordion>
<Accordion title="Disable NUMA balancing" defaultOpen="true">
```bash
sudo sysctl -w kernel.numa_balancing=0
# Check
cat /proc/sys/kernel/numa_balancing # shows 0
```
</Accordion>
<Accordion title="Prevent swapping out system memory" defaultOpen="true">
```bash
sudo sysctl -w vm.swappiness=10
# Check
cat /proc/sys/vm/swappiness # shows 10
```
</Accordion>
</AccordionGroup>
## Running SGLang Service
<Tabs>
<Tab title="For Large Language Models">
### PD Mixed Scene
```bash
# Enabling CPU Affinity
export SGLANG_SET_CPU_AFFINITY=1
python3 -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct --attention-backend ascend
```
### PD Separation Scene
1. **Launch Prefill Server**
```bash
# Enabling CPU Affinity
export SGLANG_SET_CPU_AFFINITY=1
# PIP: recommended to config first Prefill Server IP
# PORT: one free port
# all sglang servers need to be config the same PIP and PORT,
export ASCEND_MF_STORE_URL="tcp://PIP:PORT"
# if you are Atlas 800I A2 hardware and use rdma for kv cache transfer, add this parameter
export ASCEND_MF_TRANSFER_PROTOCOL="device_rdma"
python3 -m sglang.launch_server \
--model-path meta-llama/Llama-3.1-8B-Instruct \
--disaggregation-mode prefill \
--disaggregation-transfer-backend ascend \
--disaggregation-bootstrap-port 8995 \
--attention-backend ascend \
--device npu \
--base-gpu-id 0 \
--tp-size 1 \
--host 127.0.0.1 \
--port 8000
```
2. **Launch Decode Server**
```bash
# PIP: recommended to config first Prefill Server IP
# PORT: one free port
# all sglang servers need to be config the same PIP and PORT,
export ASCEND_MF_STORE_URL="tcp://PIP:PORT"
# if you are Atlas 800I A2 hardware and use rdma for kv cache transfer, add this parameter
export ASCEND_MF_TRANSFER_PROTOCOL="device_rdma"
python3 -m sglang.launch_server \
--model-path meta-llama/Llama-3.1-8B-Instruct \
--disaggregation-mode decode \
--disaggregation-transfer-backend ascend \
--attention-backend ascend \
--device npu \
--base-gpu-id 1 \
--tp-size 1 \
--host 127.0.0.1 \
--port 8001
```
3. **Launch Router**
```bash
python3 -m sglang_router.launch_router \
--pd-disaggregation \
--policy cache_aware \
--prefill http://127.0.0.1:8000 8995 \
--decode http://127.0.0.1:8001 \
--host 127.0.0.1 \
--port 6688
```
</Tab>
<Tab title="For Multimodal Language Models">
### PD Mixed Scene
```bash
python3 -m sglang.launch_server \
--model-path Qwen3-VL-30B-A3B-Instruct \
--host 127.0.0.1 \
--port 8000 \
--tp 4 \
--device npu \
--attention-backend ascend \
--mm-attention-backend ascend_attn \
--disable-radix-cache \
--trust-remote-code \
--enable-multimodal \
--sampling-backend ascend
```
</Tab>
</Tabs>
docs_new/docs/hardware-platforms/ascend-npus/ascend_contribution_guide.mdx
+167
View File
@@ -0,0 +1,167 @@
---
title: "Contribution Guide"
metatags:
description: "Set up the Ascend NPU development environment, run tests, build documentation, and open SGLang pull requests."
---
Welcome to **SGLang**! We appreciate your interest in contributing. This guide provides a concise overview of how to set up your environment, run tests, build documentation, and open a Pull Request (PR). Whether you’re fixing a small bug or developing a major feature, we encourage following these steps for a smooth contribution process.
## Install SGLang from Source
### Prepare Environment
Before contributing, please ensure that your environment is set up correctly. Follow the steps in the [Installation Guide](./ascend_npu) to install the necessary dependencies. We recommend [using docker](./ascend_npu#method-2-using-docker-image) to build the environment.
### Fork and clone the repository
**Note**: New contributors do **not** have the write permission to push to the official SGLang repo. Please fork the repository under your GitHub account, then clone your fork locally.
```bash
git clone https://github.com/<your_user_name>/sglang.git
# if you are using docker, the environment is already set up.
cd sglang
export PYTHONPATH=$PWD/python:$PYTHONPATH
```
## Format code with pre-commit
We use [pre-commit](https://pre-commit.com/) to maintain consistent code style checks. Before pushing your changes, please run:
```bash
pip3 install pre-commit
pre-commit install
pre-commit run --all-files
```
- **`pre-commit run --all-files`** manually runs all configured checks, applying fixes if possible. If it fails the first time, re-run it to ensure lint errors are fully resolved. Make sure your code passes all checks **before** creating a Pull Request.
- **Do not commit** directly to the `main` branch. Always create a new branch (e.g., `feature/my-new-feature`), push your changes, and open a PR from that branch.
## Run and add unit tests
If you add a new feature or fix a bug, please add corresponding unit tests to ensure coverage and prevent regression.
SGLang uses Python's built-in [unittest](https://docs.python.org/3/library/unittest.html) framework.
For detailed instructions on running tests and integrating them into CI, refer to [test/README.md](https://github.com/sgl-project/sglang/tree/main/test/README.md).
If you need to use model which is not in `python/sglang/test/ascend/test_ascend_utils.py` list. Follow these steps:
1. Register account and upload your model to [modelscope](https://modelscope.cn/models).
2. Make sure your model is pre-cached on the CI server and is on the way "/data/ascend-ci-share-pkking-sglang/modelscope/hub/models/{your_model_repo}/{your_model}".
If this is not the case, use following command on CI server:
```bash
modelscope download
--model {your_model_repo}/{your_model}
--local_dir /data/ascend-ci-share-pkking-sglang/modelscope/hub/models/{your_model_repo}/{your_model}
```
> Note: If you don’t have access to CI server, please ask maintainers (zl19940307@163.com) to download your model.
4. Add model to ```python/sglang/test/ascend/test_ascend_utils.py``` (use docker ```"/root/.cache/modelscope/hub/models/{your_model_repo}/{your_model}"``` path).
## Write documentations
We recommend new contributors start from writing documentation, which helps you quickly understand SGLang codebase.
For more details, please refer to [docs/README.md](https://github.com/sgl-project/sglang/tree/main/docs/README.md).
## Test the accuracy
If your code changes the model output, please run the accuracy tests. A quick sanity check is the few-shot GSM8K.
```
# Launch a server
python3 -m sglang.launch_server --model Qwen/Qwen2-7B-Instruct
# Evaluate
python3 -m sglang.test.few_shot_gsm8k --num-questions 200
```
Please note that the above script is primarily a sanity check, not a rigorous accuracy or speed test.
This test can have significant variance (1%–5%) in accuracy due to batching and the non-deterministic nature of the inference engine.
Also, do not rely on the "Latency/Output throughput" from this script, as it is not a proper speed test.
GSM8K is too easy for state-of-the-art models nowadays. Please try your own more challenging accuracy tests.
You can find additional accuracy eval examples in:
- [test_eval_accuracy_large.py](https://github.com/sgl-project/sglang/blob/main/test/registered/eval/test_eval_accuracy_large.py)
- [test_gpt_oss_1gpu.py](https://github.com/sgl-project/sglang/blob/main/test/registered/core/test_gpt_oss_1gpu.py)
## Benchmark the speed
Refer to [Benchmark and Profiling](../../developer_guide/benchmark_and_profiling).
## Requesting a review for merge
You can follow the pull request merge process described in [MAINTAINER.md](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md).
You will need to work with the Merge Oncall, Codeowner, and other reviewers to get their approvals.
Then your PR can be merged.
## How to Trigger CI Tests
We have a lot of open PRs but limited CI machines, so only top and trusted contributors have permission to trigger CI tests.
Users with permission are listed in the [CI_PERMISSIONS.json](https://github.com/sgl-project/sglang/blob/main/.github/CI_PERMISSIONS.json)
For CI to run on a pull request, it must have the "run-ci" label. Authorized users can add the label or rerun failed tests by commenting on the PR with one of these commands:
- `/tag-run-ci-label`: Adds the "run-ci" label. Every future commit will trigger CI.
- `/rerun-failed-ci`: Reruns the failed or flaky tests from the most recent commit.
- `/tag-and-rerun-ci`: A single command that performs both `/tag-run-ci-label` and `/rerun-failed-ci`.
- `/rerun-stage <stage-name>`: Reruns a specific test stage without waiting for its dependencies. This is useful when you want to quickly validate a fix for a specific test failure instead of waiting ~30 minutes for preceding stages to complete.
If you have permission, the [Slash Command Handler](https://github.com/sgl-project/sglang/actions/workflows/slash-command-handler.yml) will run your command and react with a 👍 to your comment. It may take up to a few minutes for the reaction to appear. Here’s a usage [example](https://github.com/sgl-project/sglang/pull/14253#issuecomment-3599509302).
To avoid spamming a PR with too many `/rerun-failed-ci` comments, you can also trigger the command by editing an existing comment and adding any suffix (e.g., `/rerun-failed-ci try again`).
Example of rerunning a single test stage: `/rerun-stage unit-test-backend-4-gpu`.
If you don’t have permission, please ask maintainers to trigger CI for you.
### CI rate limits
Due to CI scheduling and limited resources, higher-priority PRs may preempt running jobs. In such cases, you may need to rerun the tests.
We apply CI rate limits to prevent abuse and ensure fair usage of our CI resources.
Each CI workflow has a default limit defined in its workflow configuration file. For example, in [pr-gate.yml](https://github.com/sgl-project/sglang/blob/main/.github/workflows/pr-gate.yml), the default cooldown period is 120 minutes, and each workflow can override it via the `cool-down-minutes` input parameter:
```yaml
cool-down-minutes:
description: "Cooldown period in minutes for low-permission users; 0 disables rate limiting"
type: number
default: 120
```
Users listed in [CI_PERMISSIONS.json](https://github.com/sgl-project/sglang/blob/main/.github/CI_PERMISSIONS.json) may have a per-user cooldown interval. In practice, we use the minimum of the workflow’s default window and the user-specific interval.
## Code style guidance
- Avoid code duplication. If the same code snippet (more than five lines) appears multiple times, extract it into a shared function.
- Minimize device synchronization. Reduce expensive CPU-GPU synchronization operations, such as `tensor.item()` or `tensor.cpu()`, whenever possible. Use vectorized code.
- Prioritize extreme efficiency. SGLang is a runtime, and most of your code runs on the critical path for every request. Optimize all minor overheads as much as possible, especially in the model forward code.
- A common pattern is some runtime checks in the model forward pass (e.g., [this](https://github.com/sgl-project/sglang/blob/f1b0eda55c2c4838e8ab90a0fac7fb1e3d7064ab/python/sglang/srt/models/deepseek_v2.py#L486-L491)). These are very likely the same for every layer. Please cache the result as a single boolean value whenever possible.
- Make functions as pure as possible. Avoid in-place modification of arguments.
- Keep files concise. If a file exceeds 2,000 lines of code, split it into multiple smaller files. (e.g., `scheduler.py`, `scheduler_output_processor_mixin.py`)
- Keep tests run fast.
- If a single test file run longer than 500 seconds, split it into multiple smaller files (e.g., `test_eagle_infer_a.py`, `test_eagle_infer_b.py`).
- If a single job in a github workflow runs longer than 30 mins, split it into smaller jobs/steps.
- Reuse server launches in your unit tests to make tests run faster.
- When supporting new hardware or features, follow these guidelines:
- Do not drastically change existing code.
- Always prefer new files to introduce specific components for your new hardware (e.g., `allocator_npu.py`).
- If you write multiple if/else blocks for new features, ensure the common path (e.g., NVIDIA hardware or the existing code path) is the first branch.
## How to update sgl-kernel
Since sglang and sgl-kernel are separate Python packages, our current GitHub CI infrastructure does not support updating a kernel and using it immediately within the same pull request (PR).
To add a new kernel or modify an existing one in the `sgl-kernel/` source tree, you must use multiple PRs.
Follow these steps:
1. Submit a PR to update the sgl-kernel source code without using it in sglang python package (e.g., [#8884](https://github.com/sgl-project/sglang/pull/8884/files)).
2. Bump the version of the kernel package (e.g., [#9220](https://github.com/sgl-project/sglang/pull/9220/files)).
- Once merged, this will trigger an automatic release of the `sglang-kernel` wheel to PyPI.
- If not urgent, you can wait for other people to release the wheel. A new version will typically be released within one week.
3. Apply the changes:
- Update the `sglang-kernel` version in `sglang/python/pyproject.toml` to use the modified kernels.
- Update the related caller code in the sglang to use the new kernel.
## How to update sgl-kernel-npu
Sgl-kernel-npu is the kernel package for Ascend NPU and is maintained in the [sgl-kernel-npu](https://github.com/sgl-project/sgl-kernel-npu) repository. if you want to add a new kernel and want to use it in sglang, please follow the steps in [Contribution Guide](https://github.com/sgl-project/sgl-kernel-npu/blob/main/docs/developer_guide/contribution_guide.md).
## Tips for newcomers
If you want to contribute but don’t have a specific idea in mind, pick issues labeled [“good first issue” or “help wanted”](https://github.com/sgl-project/sglang/issues?q=is%3Aissue+label%3A%22good+first+issue%22%2C%22help+wanted%22). These tasks typically have lower complexity and provide an excellent introduction to the codebase. Also check out this [code walk-through](https://github.com/zhaochenyang20/Awesome-ML-SYS-Tutorial/tree/main/sglang/code-walk-through) for a deeper look into SGLang’s workflow.
If you have any questions or want to start a discussion, please feel free to ask in our [Slack channel](https://slack.sglang.io).
Thank you for your interest in SGLang. Happy coding!
docs_new/docs/hardware-platforms/ascend-npus/ascend_npu.mdx
+294
View File
@@ -0,0 +1,294 @@
---
title: SGLang installation with NPUs support
---
You can install SGLang using any of the methods below. Please go through `System Settings` section to ensure the clusters are roaring at max performance. Feel free to leave an issue [here at sglang](https://github.com/sgl-project/sglang/issues) if you encounter any issues or have any problems.
## Component Version Mapping For SGLang
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
<colgroup>
<col style={{width: "34%"}} />
<col style={{width: "33%"}} />
<col style={{width: "33%"}} />
</colgroup>
<thead>
<tr style={{borderBottom: "2px solid #d55816"}}>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Component</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Version</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Obtain Way</th>
</tr>
</thead>
<tbody>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>HDK</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>25.5.2</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><a href="https://www.hiascend.com/hardware/firmware-drivers/commercial?product=7&amp;model=33">link</a></td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>CANN</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>8.5.0</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><a href="#obtain-cann-image">Obtain Images</a></td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Pytorch Adapter</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>7.3.0</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><a href="https://gitcode.com/Ascend/pytorch/releases">link</a></td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>MemFabric</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1.0.5</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`pip install memfabric-hybrid==1.0.5`</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Triton</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>3.2.0</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`pip install triton-ascend`</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>SGLang NPU Kernel</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>NA</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><a href="https://github.com/sgl-project/sgl-kernel-npu/releases">link</a></td>
</tr>
</tbody>
</table>
<a id="obtain-cann-image"></a>
### Obtain CANN Image
You can obtain the dependency of a specified version of CANN through an image.
```bash Command
# for Atlas 800I A3 and Ubuntu OS
docker pull quay.io/ascend/cann:8.5.0-a3-ubuntu22.04-py3.11
# for Atlas 800I A2 and Ubuntu OS
docker pull quay.io/ascend/cann:8.5.0-910b-ubuntu22.04-py3.11
```
## Preparing the Running Environment
### Method 1: Installing from source with prerequisites
#### Python Version
Only `python==3.11` is supported currently. If you don't want to break system pre-installed python, try installing with [conda](https://github.com/conda/conda).
```bash Command
conda create --name sglang_npu python=3.11
conda activate sglang_npu
```
#### CANN
Prior to start work with SGLang on Ascend you need to install CANN Toolkit, Kernels operator package and NNAL version 8.5.0, check the [installation guide](https://www.hiascend.com/document/detail/zh/CANNCommunityEdition/850/softwareinst/instg/instg_0008.html?Mode=PmIns&InstallType=local&OS=openEuler&Software=cannToolKit)
#### MemFabric-Hybrid
If you want to use PD disaggregation mode, you need to install MemFabric-Hybrid. MemFabric-Hybrid is a drop-in replacement of Mooncake Transfer Engine that enables KV cache transfer on Ascend NPU clusters.
```bash Command
pip install memfabric-hybrid==1.0.5
```
#### Pytorch and Pytorch Framework Adaptor on Ascend
```bash Command
PYTORCH_VERSION=2.8.0
TORCHVISION_VERSION=0.23.0
TORCH_NPU_VERSION=2.8.0.post2
pip install torch==$PYTORCH_VERSION torchvision==$TORCHVISION_VERSION --index-url https://download.pytorch.org/whl/cpu
pip install torch_npu==$TORCH_NPU_VERSION
```
If you are using other versions of `torch` and install `torch_npu`, check [installation guide](https://github.com/Ascend/pytorch/blob/master/README.md)
#### Triton on Ascend
We provide our own implementation of Triton for Ascend.
```bash Command
pip install triton-ascend
```
For installation of Triton on Ascend nightly builds or from sources, follow [installation guide](https://gitcode.com/Ascend/triton-ascend/blob/master/docs/sources/getting-started/installation.md)
#### SGLang Kernels NPU
We provide SGL kernels for Ascend NPU, check [installation guide](https://github.com/sgl-project/sgl-kernel-npu/blob/main/python/sgl_kernel_npu/README.md).
#### DeepEP-compatible Library
We provide a DeepEP-compatible Library as a drop-in replacement of deepseek-ai's DeepEP library, check the [installation guide](https://github.com/sgl-project/sgl-kernel-npu/blob/main/python/deep_ep/README.md).
#### Some other dependencies
```bash Command
# libGL
apt update
apt install libgl1 libglib2.0-0
# ensure setuptools contains pkg_resources module
pip install "setuptools<80"
```
#### Installing SGLang from source
```bash Command
# Use the last release branch
git clone https://github.com/sgl-project/sglang.git
cd sglang
mv python/pyproject_npu.toml python/pyproject.toml
pip install -e python[all_npu]
```
### Method 2: Using Docker Image
#### Obtain Image
You can download the SGLang image or build an image based on Dockerfile to obtain the Ascend NPU image.
1. Download SGLang image
```angular2html
dockerhub: docker.io/lmsysorg/sglang:$tag
# Main-based tag, change main to specific version like v0.5.6,
# you can get image for specific version
Atlas 800I A3 : {main}-cann8.5.0-a3
Atlas 800I A2: {main}-cann8.5.0-910b
```
2. Build an image based on Dockerfile
```bash Command
# Clone the SGLang repository
git clone https://github.com/sgl-project/sglang.git
cd sglang/docker
# Build the docker image
# If there are network errors, please modify the Dockerfile to use offline dependencies or use a proxy
# <arch_tag> is the target architecture of the image, e.g. amd64, arm64
docker build --build-arg TARGETARCH=<arch_tag> -t <image_name> -f npu.Dockerfile .
```
#### Create Docker
__Notice:__ `--privileged` and `--network=host` are required by RDMA, which is typically needed by Ascend NPU clusters.
__Notice:__ The following docker command is based on Atlas 800I A3 machines. If you are using Atlas 800I A2, make sure only `davinci[0-7]` are mapped into container.
```bash Command
alias drun='docker run -it --rm --privileged --network=host --ipc=host --shm-size=16g \
--device=/dev/davinci0 --device=/dev/davinci1 --device=/dev/davinci2 --device=/dev/davinci3 \
--device=/dev/davinci4 --device=/dev/davinci5 --device=/dev/davinci6 --device=/dev/davinci7 \
--device=/dev/davinci8 --device=/dev/davinci9 --device=/dev/davinci10 --device=/dev/davinci11 \
--device=/dev/davinci12 --device=/dev/davinci13 --device=/dev/davinci14 --device=/dev/davinci15 \
--device=/dev/davinci_manager --device=/dev/hisi_hdc \
--volume /usr/local/sbin:/usr/local/sbin --volume /usr/local/Ascend/driver:/usr/local/Ascend/driver \
--volume /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \
--volume /etc/ascend_install.info:/etc/ascend_install.info \
--volume /var/queue_schedule:/var/queue_schedule --volume ~/.cache/:/root/.cache/'
# Add HF_TOKEN env for download model by SGLang.
drun --env "HF_TOKEN=<secret>" \
<image_name> \
python3 -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct --attention-backend ascend
```
## System Settings
### CPU performance power scheme
The default power scheme on Ascend hardware is `ondemand` which could affect performance, changing it to `performance` is recommended.
```bash Command
echo performance | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
# Make sure changes are applied successfully
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor # shows performance
```
### Disable NUMA balancing
```bash Command
sudo sysctl -w kernel.numa_balancing=0
# Check
cat /proc/sys/kernel/numa_balancing # shows 0
```
### Prevent swapping out system memory
```bash Command
sudo sysctl -w vm.swappiness=10
# Check
cat /proc/sys/vm/swappiness # shows 10
```
## Running SGLang Service
### Running Service For Large Language Models
#### PD Mixed Scene
```bash Command
# Enabling CPU Affinity
export SGLANG_SET_CPU_AFFINITY=1
python3 -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct --attention-backend ascend
```
#### PD Disaggregation Scene
1. Launch Prefill Server
```bash Command
# Enabling CPU Affinity
export SGLANG_SET_CPU_AFFINITY=1
# PIP: recommended to config first Prefill Server IP
# PORT: one free port
# all sglang servers need to be config the same PIP and PORT,
export ASCEND_MF_STORE_URL="tcp://PIP:PORT"
# if you are Atlas 800I A2 hardware and use rdma for kv cache transfer, add this parameter
export ASCEND_MF_TRANSFER_PROTOCOL="device_rdma"
python3 -m sglang.launch_server \
--model-path meta-llama/Llama-3.1-8B-Instruct \
--disaggregation-mode prefill \
--disaggregation-transfer-backend ascend \
--disaggregation-bootstrap-port 8995 \
--attention-backend ascend \
--device npu \
--base-gpu-id 0 \
--tp-size 1 \
--host 127.0.0.1 \
--port 8000
```
2. Launch Decode Server
```bash Command
# PIP: recommended to config first Prefill Server IP
# PORT: one free port
# all sglang servers need to be config the same PIP and PORT,
export ASCEND_MF_STORE_URL="tcp://PIP:PORT"
# if you are Atlas 800I A2 hardware and use rdma for kv cache transfer, add this parameter
export ASCEND_MF_TRANSFER_PROTOCOL="device_rdma"
python3 -m sglang.launch_server \
--model-path meta-llama/Llama-3.1-8B-Instruct \
--disaggregation-mode decode \
--disaggregation-transfer-backend ascend \
--attention-backend ascend \
--device npu \
--base-gpu-id 1 \
--tp-size 1 \
--host 127.0.0.1 \
--port 8001
```
3. Launch Router
```bash Command
python3 -m sglang_router.launch_router \
--pd-disaggregation \
--policy cache_aware \
--prefill http://127.0.0.1:8000 8995 \
--decode http://127.0.0.1:8001 \
--host 127.0.0.1 \
--port 6688
```
### Running Service For Multimodal Language Models
#### PD Mixed Scene
```bash Command
python3 -m sglang.launch_server \
--model-path Qwen3-VL-30B-A3B-Instruct \
--host 127.0.0.1 \
--port 8000 \
--tp 4 \
--device npu \
--attention-backend ascend \
--mm-attention-backend ascend_attn \
--disable-radix-cache \
--trust-remote-code \
--enable-multimodal \
--sampling-backend ascend
```
docs_new/docs/hardware-platforms/ascend-npus/Best-Practice-on-Ascend-NPU.mdx → docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_best_practice.mdx
+1769 -423
View File
File diff suppressed because it is too large Load Diff
docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_deepseek_example.mdx
+301
View File
@@ -0,0 +1,301 @@
---
title: "DeepSeek Examples"
metatags:
description: "Examples for running DeepSeek models on Ascend NPUs, including PD mixed mode, PD disaggregation, and SGLang Model Gateway."
---
## Running DeepSeek-V3
### Running DeepSeek in PD mixed mode on 1 x Atlas 800I A3.
W4A8 Model weights could be found [here](https://modelers.cn/models/Modelers_Park/DeepSeek-R1-0528-w4a8).
```shell Launch Server
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export STREAMS_PER_DEVICE=32
#Deepep communication settings
export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=32
export HCCL_BUFFSIZE=1600
#spec overlap
export SGLANG_ENABLE_SPEC_V2=1
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
#npu acceleration operator
export SGLANG_NPU_USE_MLAPO=1
export SGLANG_USE_FIA_NZ=1
python3 -m sglang.launch_server \
--model-path ${MODEL_PATH} \
--tp 16 \
--trust-remote-code \
--attention-backend ascend \
--device npu \
--quantization modelslim \
--watchdog-timeout 9000 \
--cuda-graph-bs 8 16 24 28 32 \
--mem-fraction-static 0.68 \
--max-running-requests 128 \
--context-length 8188 \
--disable-radix-cache \
--chunked-prefill-size -1 \
--max-prefill-tokens 16384 \
--moe-a2a-backend deepep \
--deepep-mode auto \
--enable-dp-attention \
--dp-size 4 \
--enable-dp-lm-head \
--speculative-algorithm NEXTN \
--speculative-num-steps 3 \
--speculative-eagle-topk 1 \
--speculative-num-draft-tokens 4 \
--dtype bfloat16
```
### Running DeepSeek with PD disaggregation mode on 2 x Atlas 800I A3.
W4A8 Model weights could be found [here](https://modelers.cn/models/Modelers_Park/DeepSeek-R1-0528-w4a8).
1. Prefill:
```bash Command
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export STREAMS_PER_DEVICE=32
#memfabric config store
export ASCEND_MF_STORE_URL="tcp://<PREFILL_HOST_IP>:<PORT>"
#Deepep communication settings
export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
export HCCL_BUFFSIZE=1536
#npu acceleration operator
export SGLANG_NPU_USE_MLAPO=1
export SGLANG_USE_FIA_NZ=1
export TASK_QUEUE_ENABLE=2
python -m sglang.launch_server \
--model-path ${MODEL_PATH} \
--host $PREFILL_HOST_IP \
--port 8000 \
--disaggregation-mode prefill \
--disaggregation-bootstrap-port 8996 \
--disaggregation-transfer-backend ascend \
--trust-remote-code \
--nnodes 1 \
--node-rank 0 \
--tp-size 16 \
--mem-fraction-static 0.6 \
--attention-backend ascend \
--device npu \
--quantization modelslim \
--load-balance-method round_robin \
--max-running-requests 8 \
--context-length 8192 \
--disable-radix-cache \
--chunked-prefill-size -1 \
--max-prefill-tokens 28680 \
--moe-a2a-backend deepep \
--deepep-mode normal \
--speculative-algorithm NEXTN \
--speculative-num-steps 3 \
--speculative-eagle-topk 1 \
--speculative-num-draft-tokens 4 \
--dp-size 2 \
--enable-dp-attention \
--disable-shared-experts-fusion \
--dtype bfloat16
```
2. Decode:
```bash Command
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export STREAMS_PER_DEVICE=32
#memfabric config store
export ASCEND_MF_STORE_URL="tcp://<PREFILL_HOST_IP>:<PORT>"
#Deepep communication settings
export HCCL_BUFFSIZE=720
export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=88
#spec overlap
export SGLANG_ENABLE_SPEC_V2=1
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
#npu acceleration operator
unset TASK_QUEUE_ENABLE
export SGLANG_NPU_USE_MLAPO=1
export SGLANG_USE_FIA_NZ=1
# suggest max-running-requests <= max-cuda-graph-bs * dp_size, Because when this value is exceeded, performance will significantly degrade.
python -m sglang.launch_server \
--model-path ${MODEL_PATH} \
--disaggregation-mode decode \
--host $DECODE_HOST_IP \
--port 8001 \
--trust-remote-code \
--nnodes 1 \
--node-rank 0 \
--tp-size 16 \
--dp-size 16 \
--mem-fraction-static 0.8 \
--max-running-requests 352 \
--attention-backend ascend \
--device npu \
--quantization modelslim \
--moe-a2a-backend deepep \
--enable-dp-attention \
--deepep-mode low_latency \
--enable-dp-lm-head \
--cuda-graph-bs 8 10 12 14 16 18 20 22 \
--disaggregation-transfer-backend ascend \
--watchdog-timeout 9000 \
--context-length 8192 \
--speculative-algorithm NEXTN \
--speculative-num-steps 3 \
--speculative-eagle-topk 1 \
--speculative-num-draft-tokens 4 \
--disable-shared-experts-fusion \
--dtype bfloat16 \
--tokenizer-worker-num 4
```
3. SGLang Model Gateway (former Router)
```bash Command
python -m sglang_router.launch_router \
--pd-disaggregation \
--policy cache_aware \
--prefill http://<PREFILL_HOST_IP>:8000 8996 \
--decode http://<DECODE_HOST_IP>:8001 \
--host 127.0.0.1 \
--port 6688
```
### Running DeepSeek with PD disaggregation on 4 x Atlas 800I A3.
W8A8 Model weights could be found [here](https://modelers.cn/models/State_Cloud/Deepseek-R1-bf16-hfd-w8a8).
1. Prefill & Decode:
```bash Command
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000
export SGLANG_SET_CPU_AFFINITY=1
unset ASCEND_LAUNCH_BLOCKING
source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh
export PATH=/usr/local/Ascend/8.5.0/compiler/bishengir/bin:$PATH
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export STREAMS_PER_DEVICE=32
export ASCEND_MF_STORE_URL="tcp://your prefill ip1:24669"
P_IP=('your prefill ip1' 'your prefill ip2')
D_IP=('your decode ip1' 'your decode ip2')
MODEL_PATH=xxx
export SGLANG_NPU_USE_MLAPO=1
export SGLANG_USE_FIA_NZ=1
LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'`
LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'`
echo "${LOCAL_HOST1}"
echo "${LOCAL_HOST2}"
# prefill
for i in "${!P_IP[@]}";
do
if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]];
then
echo "${P_IP[$i]}"
export HCCL_BUFFSIZE=1536
export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
export TASK_QUEUE_ENABLE=2
export HCCL_SOCKET_IFNAME=lo
export GLOO_SOCKET_IFNAME=lo
python -m sglang.launch_server --model-path ${MODEL_PATH} --disaggregation-mode prefill --host ${P_IP[$i]} \
--port 8000 --disaggregation-bootstrap-port $((8998+$i)) --trust-remote-code --nnodes 1 --node-rank 0 \
--tp-size 16 --mem-fraction-static 0.81 --attention-backend ascend --device npu --quantization modelslim \
--disaggregation-transfer-backend ascend --max-running-requests 8 --context-length 8192 --disable-radix-cache \
--chunked-prefill-size -1 --max-prefill-tokens 28680 --moe-a2a-backend deepep --deepep-mode normal \
--speculative-algorithm NEXTN --speculative-num-steps 1 --speculative-eagle-topk 1 --speculative-num-draft-tokens 2 \
--dp-size 2 --enable-dp-attention --disable-shared-experts-fusion --dtype bfloat16 --enable-attn-tp-input-scattered
NODE_RANK=$i
break
fi
done
# decode
for i in "${!D_IP[@]}";
do
if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]];
then
echo "${D_IP[$i]}"
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
export SGLANG_ENABLE_SPEC_V2=1
export HCCL_BUFFSIZE=650
export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=78
export TASK_QUEUE_ENABLE=1
export SGLANG_SCHEDULER_SKIP_ALL_GATHER=1
export HCCL_SOCKET_IFNAME=xxx
export GLOO_SOCKET_IFNAME=xxx
python -m sglang.launch_server --model-path ${MODEL_PATH} --disaggregation-mode decode --host ${D_IP[$i]} \
--port 8001 --trust-remote-code --dist-init-addr ${D_IP[0]}:5000 --nnodes 2 --node-rank $i --tp-size 32 --dp-size 32 \
--mem-fraction-static 0.815 --max-running-requests 832 --attention-backend ascend --device npu --quantization modelslim \
--moe-a2a-backend deepep --enable-dp-attention --deepep-mode low_latency --enable-dp-lm-head --moe-dense-tp 1 \
--cuda-graph-bs 12 14 16 18 20 22 24 26 --disaggregation-transfer-backend ascend --watchdog-timeout 9000 --context-length 8192 \
--speculative-algorithm NEXTN --speculative-num-steps 2 --speculative-eagle-topk 1 --speculative-num-draft-tokens 3 \
--tokenizer-worker-num 4 --disable-shared-experts-fusion --dtype bfloat16 \
--load-balance-method decode_round_robin
NODE_RANK=$i
break
fi
done
```
2. SGLang Model Gateway (former Router):
```bash Command
python -m sglang_router.launch_router \
--pd-disaggregation \
--policy cache_aware \
--prefill http://P_IP:8000 8998 \
--prefill http://P_IP:8000 8999 \
--decode http://D_IP:8001 \
--host 127.0.0.1 \
--port 6688 \
--mini-lb
```
### test gsm8k
```python Test GSM8K
from types import SimpleNamespace
from sglang.test.few_shot_gsm8k import run_eval
def gsm8k():
args = SimpleNamespace(
num_shots=5,
data_path=None,
num_questions=200,
max_new_tokens=512,
parallel=32,
host=f"http://127.0.0.1",
port=6688,
)
metrics = run_eval(args)
print(f"{metrics=}")
print(f"{metrics['accuracy']=}")
if __name__ == "__main__":
gsm8k()
```
docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_environment_variables.mdx
+149
View File
@@ -0,0 +1,149 @@
---
title: "Environment Variables"
metatags:
description: "Reference commonly used Ascend NPU environment variables for configuring SGLang runtime behavior."
---
SGLang supports various environment variables related to Ascend NPU that can be used to configure its runtime behavior.
This document provides a list of commonly used environment variables and aims to stay updated over time.
## Directly Used in SGLang
<table>
<thead>
<tr>
<th>Environment Variable</th>
<th>Description</th>
<th>Default Value</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>SGLANG_NPU_USE_MLAPO</code></td>
<td>Adopts the <code>MLAPO</code> fusion operator in attention &lt;br/&gt; preprocessing stage of the MLA model.</td>
<td><code>false</code></td>
</tr>
<tr>
<td><code>SGLANG_USE_FIA_NZ</code></td>
<td>Reshapes KV Cache for FIA NZ format.&lt;br/&gt; <code>SGLANG_USE_FIA_NZ</code> must be enabled with <code>SGLANG_NPU_USE_MLAPO</code></td>
<td><code>false</code></td>
</tr>
<tr>
<td><code>SGLANG_NPU_USE_MULTI_STREAM</code></td>
<td>Enable dual-stream computation of shared experts &lt;br/&gt; and routing experts in DeepSeek models.&lt;br/&gt; Enable dual-stream computation in DeepSeek NSA Indexer.</td>
<td><code>false</code></td>
</tr>
<tr>
<td><code>SGLANG_NPU_DISABLE_ACL_FORMAT_WEIGHT</code></td>
<td>Disable cast model weight tensor to a specific NPU &lt;br/&gt; ACL format.</td>
<td><code>false</code></td>
</tr>
<tr>
<td><code>SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK</code></td>
<td>The maximum number of dispatched tokens on each rank.</td>
<td><code>128</code></td>
</tr>
</tbody>
</table>
## Used in DeepEP Ascend
<table>
<thead>
<tr>
<th>Environment Variable</th>
<th>Description</th>
<th>Default Value</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS</code></td>
<td>Enable ant-moving function in dispatch stage. Indicates &lt;br/&gt; the number of tokens transmitted per round on each rank.</td>
<td><code>8192</code></td>
</tr>
<tr>
<td><code>DEEPEP_NORMAL_LONG_SEQ_ROUND</code></td>
<td>Enable ant-moving function in dispatch stage. Indicates &lt;br/&gt; the number of rounds transmitted on each rank.</td>
<td><code>1</code></td>
</tr>
<tr>
<td><code>DEEPEP_NORMAL_COMBINE_ENABLE_LONG_SEQ</code></td>
<td>Enable ant-moving function in combine stage. &lt;br/&gt; The value <code>0</code> means disabled.</td>
<td><code>0</code></td>
</tr>
<tr>
<td><code>MOE_ENABLE_TOPK_NEG_ONE</code></td>
<td>Needs to be enabled when the expert ID to be processed by &lt;br/&gt; DEEPEP contains -1.</td>
<td><code>0</code></td>
</tr>
<tr>
<td><code>DEEP_NORMAL_MODE_USE_INT8_QUANT</code></td>
<td>Quantizes x to int8 and returns (tensor, scales) in dispatch operator.</td>
<td><code>0</code></td>
</tr>
</tbody>
</table>
## Others
<table>
<thead>
<tr>
<th>Environment Variable</th>
<th>Description</th>
<th>Default Value</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>TASK_QUEUE_ENABLE</code></td>
<td>Used to control the optimization level of the dispatch queue&lt;br/&gt; about the task_queue operator. <a href="https://www.hiascend.com/document/detail/zh/Pytorch/730/comref/Envvariables/docs/zh/environment_variable_reference/TASK_QUEUE_ENABLE.md">Detail</a></td>
<td><code>1</code></td>
</tr>
<tr>
<td><code>INF_NAN_MODE_ENABLE</code></td>
<td>Controls whether the chip uses saturation mode or INF_NAN mode. <a href="https://www.hiascend.com/document/detail/zh/CANNCommunityEdition/800alpha001/apiref/envref/envref_07_0056.html">Detail</a></td>
<td><code>1</code></td>
</tr>
<tr>
<td><code>STREAMS_PER_DEVICE</code></td>
<td>Configures the maximum number of streams for the stream pool. <a href="https://www.hiascend.com/document/detail/zh/Pytorch/720/comref/Envvariables/Envir_041.html">Detail</a></td>
<td><code>32</code></td>
</tr>
<tr>
<td><code>PYTORCH_NPU_ALLOC_CONF</code></td>
<td>Controls the behavior of the cache allocator. &lt;br/&gt;This variable changes memory usage and may cause performance fluctuations. <a href="https://www.hiascend.com/document/detail/zh/Pytorch/700/comref/Envvariables/Envir_012.html">Detail</a></td>
<td></td>
</tr>
<tr>
<td><code>ASCEND_MF_STORE_URL</code></td>
<td>The address of config store in MemFabric during PD separation, &lt;br/&gt;which is generally set to the IP address of the P primary node&lt;br/&gt; with an arbitrary port number.</td>
<td></td>
</tr>
<tr>
<td><code>ASCEND_LAUNCH_BLOCKING</code></td>
<td>Controls whether synchronous mode is enabled during operator execution. <a href="https://www.hiascend.com/document/detail/zh/Pytorch/710/comref/Envvariables/Envir_006.html">Detail</a></td>
<td><code>0</code></td>
</tr>
<tr>
<td><code>HCCL_OP_EXPANSION_MODE</code></td>
<td>Configures the expansion position for communication algorithm scheduling. <a href="https://www.hiascend.com/document/detail/zh/CANNCommunityEdition/800alpha001/apiref/envref/envref_07_0094.html">Detail</a></td>
<td></td>
</tr>
<tr>
<td><code>HCCL_BUFFSIZE</code></td>
<td>Controls the size of the buffer area for shared data between two NPUs. &lt;br/&gt;The unit is MB, and the value must be greater than or equal to 1. <a href="https://www.hiascend.com/document/detail/zh/Pytorch/60RC3/ptmoddevg/trainingmigrguide/performance_tuning_0047.html">Detail</a></td>
<td><code>200</code></td>
</tr>
<tr>
<td><code>HCCL_SOCKET_IFNAME</code></td>
<td>Configures the name of the network card used by the Host &lt;br/&gt;during HCCL initialization. <a href="https://www.hiascend.com/document/detail/zh/canncommercial/81RC1/apiref/envvar/envref_07_0075.html">Detail</a></td>
<td></td>
</tr>
<tr>
<td><code>GLOO_SOCKET_IFNAME</code></td>
<td>Configures the network interface name for GLOO communication.</td>
<td></td>
</tr>
</tbody>
</table>
docs_new/docs/hardware-platforms/ascend-npus/GLM-5.mdx → docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_glm5_examples.mdx
+17 -14
View File
@@ -1,3 +1,8 @@
---
title: "GLM-5 examples"
metatags:
description: "Documentation for GLM-5 examples"
---
## Introduction
The GLM (General Language Model) series is an open-source bilingual large language model family jointly developed by the KEG Laboratory of Tsinghua University and Zhipu AI. This series of models has performed outstandingly in the field of Chinese NLP with its unique unified pre-training framework and bilingual capabilities. [GLM-5](https://huggingface.co/zai-org/GLM-5) adopts the DeepSeek-V3/V3.2 architecture, including the sparse attention (DSA) and multi-token prediction (MTP). Ascend supports GLM-5 with 0Day based on the SGLang inference framework, achieving low-code seamless enablement and compatibility with the mainstream distributed parallel capabilities within the current SGLang framework. We welcome developers to download and experience it.
@@ -13,10 +18,9 @@ The GLM (General Language Model) series is an open-source bilingual large langua
### Installation
The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the quay.io platform. You can directly pull it.
The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the online platform. You can directly pull it.
<CodeGroup>
```bash Pull and Start Container
```bash Command
#Atlas 800 A3
docker pull swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:cann8.5.0-a3-glm5
#Atlas 800 A2
@@ -31,7 +35,7 @@ docker run -itd --shm-size=16g --privileged=true --name ${NAME} \
-v /usr/local/Ascend/driver:/usr/local/Ascend/driver \
-v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \
--device=/dev/davinci0:/dev/davinci0 \
--device=/dev/davinci1:/dev/avinci1 \
--device=/dev/davinci1:/dev/davinci1 \
--device=/dev/davinci2:/dev/davinci2 \
--device=/dev/davinci3:/dev/davinci3 \
--device=/dev/davinci4:/dev/davinci4 \
@@ -51,15 +55,18 @@ docker run -itd --shm-size=16g --privileged=true --name ${NAME} \
--entrypoint=bash \
swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:${TAG}
```
</CodeGroup>
Note: Using this image, you need to update transformers to main branch
<CodeGroup>
```shell Update Transformers
### Best Practices
Note: Using this image for **best practices**, you need to update transformers to version 5.3.0
```
# reinstall transformers
pip install git+https://github.com/huggingface/transformers.git
# Install transformers version 5.3.0 from PyPI
pip install transformers==5.3.0
# Install from GitHub v5.3.0 tag from GitHub
pip install git+https://github.com/huggingface/transformers.git@v5.3.0
```
</CodeGroup>
## Deployment
@@ -69,7 +76,6 @@ pip install git+https://github.com/huggingface/transformers.git
Run the following script to execute online inference.
<CodeGroup>
```shell Launch Server
# high performance cpu
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
@@ -113,7 +119,6 @@ python3 -m sglang.launch_server \
--quantization modelslim \
--moe-a2a-backend deepep --deepep-mode auto
```
</CodeGroup>
### Multi-node Deployment
@@ -125,7 +130,6 @@ Modify the IP of 2 nodes, then run the same scripts on two nodes.
**node 0/1**
<CodeGroup>
```shell Launch Multi-node Server
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
@@ -189,7 +193,6 @@ do
done
```
</CodeGroup>
### Prefill-Decode Disaggregation
docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_quantization.mdx
+257
View File
@@ -0,0 +1,257 @@
---
title: "Quantization on Ascend"
metatags:
description: "Load, export, and serve quantized models on Ascend NPUs with SGLang."
---
To load already quantized models, simply load the model weights and config. Again, if the model has been quantized offline, there's no need to add `--quantization` argument when starting the engine. The quantization method will be automatically parsed from the downloaded `quant_model_description.json` or `config.json` config.
SGLang support **mix-bits** quantization (independently defines and loads each layer depending on the type of quantification specified in the `quant_model_description'.json`). [Advanced mix-bits for MoE](https://github.com/sgl-project/sglang/pull/17361) in progress, will add independent quantization determination for the w13 (up-gate) and w2 (down) layers.
[ModelSlim on Ascend support](https://github.com/sgl-project/sglang/pull/14504)
<table>
<thead>
<tr>
<th>Quantization scheme</th>
<th>Layer type</th>
<th>A2 Supported</th>
<th>A3 Supported</th>
<th>A5 Supported</th>
<th>Diffusion models</th>
</tr>
</thead>
<tbody>
<tr>
<td>W4A4 dynamic</td>
<td>Linear</td>
<td><strong>&lt;span style="color: green;"&gt;√&lt;/span&gt;</strong></td>
<td><strong>&lt;span style="color: green;"&gt;√&lt;/span&gt;</strong></td>
<td><strong>&lt;span style="color: yellow;"&gt;TBD&lt;/span&gt;</strong></td>
<td><strong>&lt;span style="color: green;"&gt;√&lt;/span&gt;</strong></td>
</tr>
<tr>
<td>W8A8 static</td>
<td>Linear</td>
<td><strong>&lt;span style="color: green;"&gt;√&lt;/span&gt;</strong></td>
<td><strong>&lt;span style="color: green;"&gt;√&lt;/span&gt;</strong></td>
<td><strong>&lt;span style="color: yellow;"&gt;TBD&lt;/span&gt;</strong></td>
<td><strong>&lt;span style="color: green;"&gt;√&lt;/span&gt;</strong></td>
</tr>
<tr>
<td>W8A8 dynamic</td>
<td>Linear</td>
<td><strong>&lt;span style="color: green;"&gt;√&lt;/span&gt;</strong></td>
<td><strong>&lt;span style="color: green;"&gt;√&lt;/span&gt;</strong></td>
<td><strong>&lt;span style="color: yellow;"&gt;TBD&lt;/span&gt;</strong></td>
<td><strong>&lt;span style="color: green;"&gt;√&lt;/span&gt;</strong></td>
</tr>
<tr>
<td><a href="https://github.com/sgl-project/sglang/pull/20922">MXFP8</a></td>
<td>Linear</td>
<td><strong>&lt;span style="color: red;"&gt;x&lt;/span&gt;</strong></td>
<td><strong>&lt;span style="color: red;"&gt;x&lt;/span&gt;</strong></td>
<td><strong>&lt;span style="color: blue;"&gt;WIP&lt;/span&gt;</strong></td>
<td><strong>&lt;span style="color: blue;"&gt;WIP&lt;/span&gt;</strong></td>
</tr>
<tr>
<td>W4A4 dynamic</td>
<td>MoE</td>
<td><strong>&lt;span style="color: green;"&gt;√&lt;/span&gt;</strong></td>
<td><strong>&lt;span style="color: green;"&gt;√&lt;/span&gt;</strong></td>
<td><strong>&lt;span style="color: yellow;"&gt;TBD&lt;/span&gt;</strong></td>
<td><strong>&lt;span style="color: red;"&gt;x&lt;/span&gt;</strong></td>
</tr>
<tr>
<td>W4A8 dynamic</td>
<td>MoE</td>
<td><strong>&lt;span style="color: green;"&gt;√&lt;/span&gt;</strong></td>
<td><strong>&lt;span style="color: green;"&gt;√&lt;/span&gt;</strong></td>
<td><strong>&lt;span style="color: yellow;"&gt;TBD&lt;/span&gt;</strong></td>
<td><strong>&lt;span style="color: red;"&gt;x&lt;/span&gt;</strong></td>
</tr>
<tr>
<td>W8A8 dynamic</td>
<td>MoE</td>
<td><strong>&lt;span style="color: green;"&gt;√&lt;/span&gt;</strong></td>
<td><strong>&lt;span style="color: green;"&gt;√&lt;/span&gt;</strong></td>
<td><strong>&lt;span style="color: yellow;"&gt;TBD&lt;/span&gt;</strong></td>
<td><strong>&lt;span style="color: red;"&gt;x&lt;/span&gt;</strong></td>
</tr>
<tr>
<td><a href="https://github.com/sgl-project/sglang/pull/20922">MXFP8</a></td>
<td>MoE</td>
<td><strong>&lt;span style="color: red;"&gt;x&lt;/span&gt;</strong></td>
<td><strong>&lt;span style="color: red;"&gt;x&lt;/span&gt;</strong></td>
<td><strong>&lt;span style="color: blue;"&gt;WIP&lt;/span&gt;</strong></td>
<td><strong>&lt;span style="color: red;"&gt;x&lt;/span&gt;</strong></td>
</tr>
</tbody>
</table>
[AWQ on Ascend support](https://github.com/sgl-project/sglang/pull/10158):
<table>
<thead>
<tr>
<th>Quantization scheme</th>
<th>Layer type</th>
<th>A2 Supported</th>
<th>A3 Supported</th>
<th>A5 Supported</th>
</tr>
</thead>
<tbody>
<tr>
<td>W4A16</td>
<td>Linear</td>
<td><strong>&lt;span style="color: green;"&gt;√&lt;/span&gt;</strong></td>
<td><strong>&lt;span style="color: green;"&gt;√&lt;/span&gt;</strong></td>
<td><strong>&lt;span style="color: yellow;"&gt;TBD&lt;/span&gt;</strong></td>
</tr>
<tr>
<td>W8A16</td>
<td>Linear</td>
<td><strong>&lt;span style="color: green;"&gt;√&lt;/span&gt;</strong></td>
<td><strong>&lt;span style="color: green;"&gt;√&lt;/span&gt;</strong></td>
<td><strong>&lt;span style="color: yellow;"&gt;TBD&lt;/span&gt;</strong></td>
</tr>
<tr>
<td>W4A16</td>
<td>MoE</td>
<td><strong>&lt;span style="color: green;"&gt;√&lt;/span&gt;</strong></td>
<td><strong>&lt;span style="color: green;"&gt;√&lt;/span&gt;</strong></td>
<td><strong>&lt;span style="color: yellow;"&gt;TBD&lt;/span&gt;</strong></td>
</tr>
</tbody>
</table>
GPTQ on Ascend support
<table>
<thead>
<tr>
<th>Quantization scheme</th>
<th>Layer type</th>
<th>A2 Supported</th>
<th>A3 Supported</th>
<th>A5 Supported</th>
</tr>
</thead>
<tbody>
<tr>
<td><a href="https://github.com/sgl-project/sglang/pull/15203">W4A16</a></td>
<td>Linear</td>
<td><strong>&lt;span style="color: green;"&gt;√&lt;/span&gt;</strong></td>
<td><strong>&lt;span style="color: green;"&gt;√&lt;/span&gt;</strong></td>
<td><strong>&lt;span style="color: yellow;"&gt;TBD&lt;/span&gt;</strong></td>
</tr>
<tr>
<td><a href="https://github.com/sgl-project/sglang/pull/15203">W8A16</a></td>
<td>Linear</td>
<td><strong>&lt;span style="color: green;"&gt;√&lt;/span&gt;</strong></td>
<td><strong>&lt;span style="color: green;"&gt;√&lt;/span&gt;</strong></td>
<td><strong>&lt;span style="color: yellow;"&gt;TBD&lt;/span&gt;</strong></td>
</tr>
<tr>
<td><a href="https://github.com/sgl-project/sglang/pull/16364">W4A16 MOE</a></td>
<td>MoE</td>
<td><strong>&lt;span style="color: green;"&gt;√&lt;/span&gt;</strong></td>
<td><strong>&lt;span style="color: green;"&gt;√&lt;/span&gt;</strong></td>
<td><strong>&lt;span style="color: yellow;"&gt;TBD&lt;/span&gt;</strong></td>
</tr>
<tr>
<td><a href="https://github.com/sgl-project/sglang/pull/16364">W8A16 MOE</a></td>
<td>MoE</td>
<td><strong>&lt;span style="color: green;"&gt;√&lt;/span&gt;</strong></td>
<td><strong>&lt;span style="color: green;"&gt;√&lt;/span&gt;</strong></td>
<td><strong>&lt;span style="color: yellow;"&gt;TBD&lt;/span&gt;</strong></td>
</tr>
</tbody>
</table>
[Auto-round on Ascend support](https://github.com/sgl-project/sglang/pull/16699)
<table>
<thead>
<tr>
<th>Quantization scheme</th>
<th>Layer type</th>
<th>A2 Supported</th>
<th>A3 Supported</th>
<th>A5 Supported</th>
</tr>
</thead>
<tbody>
<tr>
<td>W4A16</td>
<td>Linear</td>
<td><strong>&lt;span style="color: green;"&gt;√&lt;/span&gt;</strong></td>
<td><strong>&lt;span style="color: green;"&gt;√&lt;/span&gt;</strong></td>
<td><strong>&lt;span style="color: yellow;"&gt;TBD&lt;/span&gt;</strong></td>
</tr>
<tr>
<td>W8A16</td>
<td>Linear</td>
<td><strong>&lt;span style="color: green;"&gt;√&lt;/span&gt;</strong></td>
<td><strong>&lt;span style="color: green;"&gt;√&lt;/span&gt;</strong></td>
<td><strong>&lt;span style="color: yellow;"&gt;TBD&lt;/span&gt;</strong></td>
</tr>
<tr>
<td>W4A16</td>
<td>MoE</td>
<td><strong>&lt;span style="color: green;"&gt;√&lt;/span&gt;</strong></td>
<td><strong>&lt;span style="color: green;"&gt;√&lt;/span&gt;</strong></td>
<td><strong>&lt;span style="color: yellow;"&gt;TBD&lt;/span&gt;</strong></td>
</tr>
<tr>
<td>W8A16</td>
<td>MoE</td>
<td><strong>&lt;span style="color: green;"&gt;√&lt;/span&gt;</strong></td>
<td><strong>&lt;span style="color: green;"&gt;√&lt;/span&gt;</strong></td>
<td><strong>&lt;span style="color: yellow;"&gt;TBD&lt;/span&gt;</strong></td>
</tr>
</tbody>
</table>
Compressed-tensors (LLM Compressor) on Ascend support:
<table>
<thead>
<tr>
<th>Quantization scheme</th>
<th>Layer type</th>
<th>A2 Supported</th>
<th>A3 Supported</th>
<th>A5 Supported</th>
</tr>
</thead>
<tbody>
<tr>
<td><a href="https://github.com/sgl-project/sglang/pull/14504">W8A8 dynamic</a></td>
<td>Linear</td>
<td><strong>&lt;span style="color: green;"&gt;√&lt;/span&gt;</strong></td>
<td><strong>&lt;span style="color: green;"&gt;√&lt;/span&gt;</strong></td>
<td><strong>&lt;span style="color: yellow;"&gt;TBD&lt;/span&gt;</strong></td>
</tr>
<tr>
<td><a href="https://github.com/sgl-project/sglang/pull/14736">W4A8 dynamic with/without activation clip</a></td>
<td>MoE</td>
<td><strong>&lt;span style="color: green;"&gt;√&lt;/span&gt;</strong></td>
<td><strong>&lt;span style="color: green;"&gt;√&lt;/span&gt;</strong></td>
<td><strong>&lt;span style="color: yellow;"&gt;TBD&lt;/span&gt;</strong></td>
</tr>
<tr>
<td><a href="https://github.com/sgl-project/sglang/pull/12759">W4A16 MOE</a></td>
<td>MoE</td>
<td><strong>&lt;span style="color: green;"&gt;√&lt;/span&gt;</strong></td>
<td><strong>&lt;span style="color: green;"&gt;√&lt;/span&gt;</strong></td>
<td><strong>&lt;span style="color: yellow;"&gt;TBD&lt;/span&gt;</strong></td>
</tr>
<tr>
<td><a href="https://github.com/sgl-project/sglang/pull/14504">W8A8 dynamic</a></td>
<td>MoE</td>
<td><strong>&lt;span style="color: green;"&gt;√&lt;/span&gt;</strong></td>
<td><strong>&lt;span style="color: green;"&gt;√&lt;/span&gt;</strong></td>
<td><strong>&lt;span style="color: yellow;"&gt;TBD&lt;/span&gt;</strong></td>
</tr>
</tbody>
</table>
[GGUF on Ascend support](https://github.com/sgl-project/sglang/pull/17883)
in progress
docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_quick_start.mdx
+107
View File
@@ -0,0 +1,107 @@
---
title: "Ascend NPU Quickstart"
metatags:
description: "Quickstart for running SGLang on Ascend NPUs with the official container image, including server launch and test request examples."
---
## Prerequisites
### Supported Devices
- Atlas 800I A2 inference series (Atlas 800I A2)
- Atlas 800I A3 inference series (Atlas 800I A3)
## Setup environment using container
__Notice:__ The following commands are based on Atlas 800I A3 machines. If you are using Atlas 800I A2, some changes are needed.
- The image tag needs to be `main-cann8.5.0-a3` for Atlas 800I A3 and `main-cann8.5.0-910b` for Atlas 800I A2.
- The device mapping in `docker run` command needs to be changed to `davinci[0-7]` for Atlas 800I A2.
```shell Command
# For Atlas 800I A3
export IMAGE=quay.io/ascend/sglang:main-cann8.5.0-a3
docker run -it --rm --privileged --network=host --ipc=host --shm-size=16g \
--device=/dev/davinci0 --device=/dev/davinci1 --device=/dev/davinci2 --device=/dev/davinci3 \
--device=/dev/davinci4 --device=/dev/davinci5 --device=/dev/davinci6 --device=/dev/davinci7 \
--device=/dev/davinci8 --device=/dev/davinci9 --device=/dev/davinci10 --device=/dev/davinci11 \
--device=/dev/davinci12 --device=/dev/davinci13 --device=/dev/davinci14 --device=/dev/davinci15 \
--device=/dev/davinci_manager \
--device=/dev/hisi_hdc \
--volume /usr/local/sbin:/usr/local/sbin \
--volume /usr/local/Ascend/driver:/usr/local/Ascend/driver \
--volume /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \
--volume /etc/ascend_install.info:/etc/ascend_install.info \
--volume /var/queue_schedule:/var/queue_schedule \
--volume ~/.cache/:/root/.cache/ \
--entrypoint=bash \
$IMAGE
```
## Usage
The SGLang server is installed in the container by default. You can use `pip show sglang` to check the version.
### Start SGLang server
SGLang will automatically download the model from Hugging Face.
```shell Command
# Set HF_ENDPOINT to a mirror site if network is not available
export HF_ENDPOINT=https://hf-mirror.com
# Set your own HF_TOKEN to download restricted models
export HF_TOKEN=<secret>
# Start SGLang server
# It may take several minutes to download the model on the first run
sglang serve --model-path Qwen/Qwen2.5-7B-Instruct --attention-backend ascend &
```
If you see output like the following, the server is running.
```log Output
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on http://127.0.0.1:30000 (Press CTRL+C to quit)
The server is fired up and ready to roll!
```
### Send a test request
You can do inference using the server:
```shell Command
curl -X POST http://localhost:30000/generate \
-H "Content-Type: application/json" \
-d '{
"text": "The capital of France is",
"sampling_params": {
"temperature": 0,
"max_new_tokens": 16
}
}'
```
If the "text" field in the response contains "Paris", the server is working as expected.
### Stop server and exit container
The SGLang server is running as a background process. You can send a `SIGINT` signal to stop it.
```shell Command
SGLANG_PID=$(pgrep -f "sglang serve")
kill -SIGINT $SGLANG_PID
```
The output should be like the following:
```log Output
INFO: Shutting down
INFO: Waiting for application shutdown.
INFO: Application shutdown complete.
INFO: Finished server process [25310]
```
The server has now stopped. You can verify it with `ps -ef | grep sglang`, then exit the container by pressing `Ctrl+D`.
docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_qwen3_5_examples.mdx
+234
View File
@@ -0,0 +1,234 @@
---
title: "Qwen3.5 examples"
metatags:
description: "Documentation for Qwen3.5 examples"
---
## Environment Preparation
### Installation
The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the quay.io platform. You can directly pull it.
```bash Command
#Atlas 800 A3
docker pull quay.io/ascend/sglang:main-cann8.5.0-a3
#Atlas 800 A2
docker pull quay.io/ascend/sglang:main-cann8.5.0-910b
#start container
docker run -itd --shm-size=16g --privileged=true --name ${NAME} \
--privileged=true --net=host \
-v /var/queue_schedule:/var/queue_schedule \
-v /etc/ascend_install.info:/etc/ascend_install.info \
-v /usr/local/sbin:/usr/local/sbin \
-v /usr/local/Ascend/driver:/usr/local/Ascend/driver \
-v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \
--device=/dev/davinci0:/dev/davinci0 \
--device=/dev/davinci1:/dev/davinci1 \
--device=/dev/davinci2:/dev/davinci2 \
--device=/dev/davinci3:/dev/davinci3 \
--device=/dev/davinci4:/dev/davinci4 \
--device=/dev/davinci5:/dev/davinci5 \
--device=/dev/davinci6:/dev/davinci6 \
--device=/dev/davinci7:/dev/davinci7 \
--device=/dev/davinci8:/dev/davinci8 \
--device=/dev/davinci9:/dev/davinci9 \
--device=/dev/davinci10:/dev/davinci10 \
--device=/dev/davinci11:/dev/davinci11 \
--device=/dev/davinci12:/dev/davinci12 \
--device=/dev/davinci13:/dev/davinci13 \
--device=/dev/davinci14:/dev/davinci14 \
--device=/dev/davinci15:/dev/davinci15 \
--device=/dev/davinci_manager:/dev/davinci_manager \
--device=/dev/hisi_hdc:/dev/hisi_hdc \
--entrypoint=bash \
quay.io/ascend/sglang:${tag}
```
## Deployment
### Single-node Deployment
Run the following script to execute online inference.
#### Qwen3.5 397B
```bash Command
# high performance cpu
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000
# bind cpu
export SGLANG_SET_CPU_AFFINITY=1
unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING
# cann
source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh
export STREAMS_PER_DEVICE=32
export HCCL_BUFFSIZE=1000
export HCCL_OP_EXPANSION_MODE=AIV
export HCCL_SOCKET_IFNAME=lo
export GLOO_SOCKET_IFNAME=lo
python3 -m sglang.launch_server \
--model-path $MODEL_PATH \
--attention-backend ascend \
--device npu \
--tp-size 16 --nnodes 1 --node-rank 0 \
--chunked-prefill-size 4096 --max-prefill-tokens 280000 \
--disable-radix-cache \
--trust-remote-code \
--host 127.0.0.1 \
--mem-fraction-static 0.7 \
--port 8000 \
--cuda-graph-bs 16 \
--quantization modelslim \
--enable-multimodal \
--mm-attention-backend ascend_attn \
--dtype bfloat16
```
#### Qwen3.5 122B
```bash Command
# high performance cpu
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000
# bind cpu
export SGLANG_SET_CPU_AFFINITY=1
unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING
# cann
source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh
export STREAMS_PER_DEVICE=32
export HCCL_BUFFSIZE=1000
export HCCL_OP_EXPANSION_MODE=AIV
export HCCL_SOCKET_IFNAME=lo
export GLOO_SOCKET_IFNAME=lo
python3 -m sglang.launch_server \
--model-path $MODEL_PATH \
--attention-backend ascend \
--device npu \
--tp-size 8 --nnodes 1 --node-rank 0 \
--chunked-prefill-size 4096 --max-prefill-tokens 280000 \
--disable-radix-cache \
--trust-remote-code \
--host 127.0.0.1 \
--mem-fraction-static 0.7 \
--port 8000 \
--cuda-graph-bs 16 \
--quantization modelslim \
--enable-multimodal \
--mm-attention-backend ascend_attn \
--dtype bfloat16
```
#### Qwen3.5 35B
```bash Command
# high performance cpu
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000
# bind cpu
export SGLANG_SET_CPU_AFFINITY=1
unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING
# cann
source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh
export STREAMS_PER_DEVICE=32
export HCCL_BUFFSIZE=1000
export HCCL_OP_EXPANSION_MODE=AIV
export HCCL_SOCKET_IFNAME=lo
export GLOO_SOCKET_IFNAME=lo
python3 -m sglang.launch_server \
--model-path $MODEL_PATH \
--attention-backend ascend \
--device npu \
--tp-size 2 --nnodes 1 --node-rank 0 \
--chunked-prefill-size 4096 --max-prefill-tokens 280000 \
--disable-radix-cache \
--trust-remote-code \
--host 127.0.0.1 \
--mem-fraction-static 0.7 \
--port 8000 \
--cuda-graph-bs 16 \
--quantization modelslim \
--enable-multimodal \
--mm-attention-backend ascend_attn \
--dtype bfloat16
```
#### Qwen3.5 27B
```bash Command
# high performance cpu
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
sysctl -w vm.swappiness=0
sysctl -w kernel.numa_balancing=0
sysctl -w kernel.sched_migration_cost_ns=50000
# bind cpu
export SGLANG_SET_CPU_AFFINITY=1
unset https_proxy
unset http_proxy
unset HTTPS_PROXY
unset HTTP_PROXY
unset ASCEND_LAUNCH_BLOCKING
# cann
source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh
export STREAMS_PER_DEVICE=32
export HCCL_BUFFSIZE=1000
export HCCL_OP_EXPANSION_MODE=AIV
export HCCL_SOCKET_IFNAME=lo
export GLOO_SOCKET_IFNAME=lo
python3 -m sglang.launch_server \
--model-path $MODEL_PATH \
--attention-backend ascend \
--device npu \
--tp-size 2 \
--chunked-prefill-size -1 --max-prefill-tokens 120000 \
--disable-radix-cache \
--trust-remote-code \
--host 127.0.0.1 \
--mem-fraction-static 0.8 \
--port 8000 \
--cuda-graph-bs 32 \
--enable-multimodal \
--mm-attention-backend ascend_attn
```
### Prefill-Decode Disaggregation
Not test yet.
### Using Benchmark
Refer to [Benchmark and Profiling](../../developer_guide/benchmark_and_profiling) for details.
docs_new/docs/hardware-platforms/ascend-npus/Qwen3-Examples.mdx → docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_qwen3_examples.mdx
+104 -16
View File
@@ -1,10 +1,16 @@
## Running Qwen3
---
title: "Qwen3 Examples"
metatags:
description: "Documentation for Qwen3 Examples"
---
## Qwen3 examples
### Running Qwen3-32B on 1 x Atlas 800I A3
### Running Qwen3
#### Running Qwen3-32B on 1 x Atlas 800I A3.
Model weights could be found [here](https://huggingface.co/Qwen/Qwen3-32B)
<CodeGroup>
```shell Launch Server
export SGLANG_SET_CPU_AFFINITY=1
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
@@ -20,15 +26,13 @@ python -m sglang.launch_server \
--model-path Qwen/Qwen3-32B \
--mem-fraction-static 0.8
```
</CodeGroup>
### Running Qwen3-32B on 1 x Atlas 800I A3 with Qwen3-32B-Eagle3
#### Running Qwen3-32B on 1 x Atlas 800I A3 with Qwen3-32B-Eagle3.
Model weights could be found [here](https://huggingface.co/Qwen/Qwen3-32B)
Speculative model weights could be found [here](https://huggingface.co/Zhihu-ai/Zhi-Create-Qwen3-32B-Eagle3)
<CodeGroup>
```shell Launch Server with Eagle3
export SGLANG_SET_CPU_AFFINITY=1
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
@@ -50,13 +54,11 @@ python -m sglang.launch_server \
--speculative-eagle-topk 1 \
--speculative-num-draft-tokens 2
```
</CodeGroup>
### Running Qwen3-30B-A3B MOE on 1 x Atlas 800I A3
#### Running Qwen3-30B-A3B MOE on 1 x Atlas 800I A3.
Model weights could be found [here](https://huggingface.co/Qwen/Qwen3-30B-A3B)
<CodeGroup>
```shell Launch Server
export SGLANG_SET_CPU_AFFINITY=1
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
@@ -74,13 +76,11 @@ python -m sglang.launch_server \
--model-path Qwen/Qwen3-30B-A3B \
--mem-fraction-static 0.8
```
</CodeGroup>
### Running Qwen3-235B-A22B-Instruct-2507 MOE on 1 x Atlas 800I A3
#### Running Qwen3-235B-A22B-Instruct-2507 MOE on 1 x Atlas 800I A3.
Model weights could be found [here](https://huggingface.co/Qwen/Qwen3-235B-A22B-Instruct-2507)
<CodeGroup>
```shell Launch Server
export SGLANG_SET_CPU_AFFINITY=1
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
@@ -98,13 +98,102 @@ python -m sglang.launch_server \
--watchdog-timeout 9000 \
--mem-fraction-static 0.8
```
</CodeGroup>
### Running Qwen3-VL-8B-Instruct on 1 x Atlas 800I A3
#### Running Qwen3-235B-A22B-Instruct-2507 with 256K long sequence on 2 x Atlas 800I A3 without CP
This example uses **PD disaggregation** for long-sequence inference and keeps **context parallel disabled**.
Set the shared environment variables on both nodes first:
```bash Command
export ASCEND_USE_FIA=1
export SGLANG_SET_CPU_AFFINITY=1
export ASCEND_MF_STORE_URL="tcp://<PREFILL_HOST_IP>:12345"
export HCCL_SOCKET_IFNAME=<NETWORK_IFACE>
export GLOO_SOCKET_IFNAME=<NETWORK_IFACE>
MODEL_PATH=/root/.cache/modelscope/hub/models/zcgy26/Qwen3-235B-A22B-Instruct-2507-w8a8
```
**Prefill node:**
```bash Command
export ASCEND_LAUNCH_BLOCKING=1
export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
export HCCL_BUFFSIZE=1500
export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=1024
export DEEPEP_NORMAL_LONG_SEQ_ROUND=128
export DEEPEP_NORMAL_COMBINE_ENABLE_LONG_SEQ=1
python3 -m sglang.launch_server \
--model-path ${MODEL_PATH} \
--disaggregation-mode prefill \
--disaggregation-transfer-backend ascend \
--disaggregation-bootstrap-port 8995 \
--attention-backend ascend \
--disable-radix-cache \
--quantization modelslim \
--chunked-prefill-size -1 \
--skip-server-warmup \
--device npu \
--tp-size 16 \
--mem-fraction-static 0.45 \
--max-running-requests 1 \
--host <PREFILL_HOST_IP> \
--port 8000 \
--dist-init-addr <PREFILL_HOST_IP>:5000 \
--nnodes 1 \
--node-rank 0 \
--moe-a2a-backend deepep \
--deepep-mode normal
```
**Decode node:**
```bash Command
export SGLANG_DEEPEP_BF16_DISPATCH=0
export HCCL_BUFFSIZE=4000
export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=4096
export DEEPEP_NORMAL_LONG_SEQ_ROUND=16
python3 -m sglang.launch_server \
--model-path ${MODEL_PATH} \
--disaggregation-mode decode \
--disaggregation-transfer-backend ascend \
--attention-backend ascend \
--mem-fraction-static 0.8 \
--disable-cuda-graph \
--device npu \
--disable-radix-cache \
--quantization modelslim \
--chunked-prefill-size 8192 \
--skip-server-warmup \
--tp-size 16 \
--max-running-requests 1 \
--host <DECODE_HOST_IP> \
--port 8232 \
--moe-a2a-backend deepep \
--deepep-mode low_latency \
--disable-overlap-schedule
```
**Router:**
```bash Command
python3 -m sglang_router.launch_router \
--pd-disaggregation \
--policy cache_aware \
--prefill http://<PREFILL_HOST_IP>:8000 8995 \
--decode http://<DECODE_HOST_IP>:8232 \
--host <ROUTER_HOST_IP> \
--port 6689 \
--prometheus-port 29010
```
#### Running Qwen3-VL-8B-Instruct on 1 x Atlas 800I A3.
Model weights could be found [here](https://huggingface.co/Qwen/Qwen3-VL-8B-Instruct)
<CodeGroup>
```shell Launch Server
export SGLANG_SET_CPU_AFFINITY=1
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
@@ -121,4 +210,3 @@ python -m sglang.launch_server \
--model-path Qwen/Qwen3-VL-8B-Instruct \
--mem-fraction-static 0.8
```
</CodeGroup>
docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_ring_sp_performance.mdx
+110
View File
@@ -0,0 +1,110 @@
---
title: "Ascend NPU Ring-SP Performance (Wan2.1-T2V-1.3B)"
metatags:
description: "This page reports Ring-SP performance on Ascend NPU with torchnpu==2.10.0."
---
This page reports Ring-SP performance on Ascend NPU with `torch_npu==2.10.0`.
- Baseline config: `ulysses=1, ring=1` (short: `u1r1`)
- Ring-SP config: `ulysses=1, ring=2` (short: `u1r2`)
## Benchmark Setup
- Model: `Wan2.1-T2V-1.3B-Diffusers`
- Prompt: `"a cat is playing piano"`
- Framework command: `sglang generate`
- Runtime: `torch_npu==2.10.0`
## Generate Commands
### Baseline (`u1r1`)
```bash
sglang generate --model-path /nas/disk1/Wan2.1-T2V-1.3B-Diffusers \
--prompt "a cat is playing piano" --num-gpus 1 --ring-degree 1 \
--save-output
```
### Ring-SP (`u1r2`)
```bash
sglang generate --model-path /nas/disk1/Wan2.1-T2V-1.3B-Diffusers \
--prompt "a cat is playing piano" --num-gpus 2 --ring-degree 2 \
--save-output
```
## Benchmarks
Benchmark Disclaimer
These numbers are from one fixed setup and one prompt case. Actual performance may vary by model settings, environment, and workload.
### Stage Time Breakdown
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
<colgroup>
<col style={{width: "25%"}} />
<col style={{width: "25%"}} />
<col style={{width: "25%"}} />
<col style={{width: "25%"}} />
</colgroup>
<thead>
<tr>
<th>Stage / Metric</th>
<th><code>u1r2</code> (s)</th>
<th><code>u1r1</code> baseline (s)</th>
<th>Speedup</th>
</tr>
</thead>
<tbody>
<tr>
<td>InputValidation</td>
<td>0.0003</td>
<td>0.0002</td>
<td>0.67x</td>
</tr>
<tr>
<td>TextEncoding</td>
<td>3.5936</td>
<td>3.5820</td>
<td>1.00x</td>
</tr>
<tr>
<td>LatentPreparation</td>
<td>0.0007</td>
<td>0.0055</td>
<td>7.86x</td>
</tr>
<tr>
<td>TimestepPreparation</td>
<td>0.0008</td>
<td>0.0007</td>
<td>0.88x</td>
</tr>
<tr>
<td>Denoising</td>
<td>121.2788</td>
<td>239.2580</td>
<td>1.97x</td>
</tr>
<tr>
<td>Decoding</td>
<td>13.8685</td>
<td>16.4969</td>
<td>1.19x</td>
</tr>
<tr>
<td><strong>Total (Pixel data generated)</strong></td>
<td><strong>141.86</strong></td>
<td><strong>266.50</strong></td>
<td><strong>1.88x</strong></td>
</tr>
</tbody>
</table>
## Summary
- With `torch_npu==2.10.0`, Ring-SP (`u1r2`) runs successfully on NPU for this case.
- End-to-end generation time improves from `266.50s` to `141.86s` (`1.88x`).
- The main gain comes from `DenoisingStage` (`1.97x`), while decoding also improves (`1.19x`).
docs_new/docs/hardware-platforms/ascend-npus/Support-Features-on-Ascend-NPU.mdx → docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_support_features.mdx
+703 -1450
View File
File diff suppressed because it is too large Load Diff
docs_new/docs/hardware-platforms/ascend-npus/Support-Models-on-Ascend-NPU.mdx → docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_support_models.mdx
+29 -46
View File
@@ -1,9 +1,13 @@
---
title: "Support Models on Ascend NPU"
metatags:
description: "Documentation for Support Models on Ascend NPU"
---
This section describes the models supported on the Ascend NPU, including Large Language Models, Multimodal Language
Models, Embedding Models, Reward Models and Rerank Models. Mainstream DeepSeek/Qwen/GLM series are included.
You are welcome to enable various models based on your business requirements.
<Accordion title="Large Language Models">
## Large Language Models
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
<colgroup>
@@ -16,8 +20,8 @@ You are welcome to enable various models based on your business requirements.
<tr style={{borderBottom: "2px solid #d55816"}}>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Models</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Model Family</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>A2</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>A3</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>A2 Supported</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>A3 Supported</th>
</tr>
</thead>
<tbody>
@@ -28,19 +32,19 @@ You are welcome to enable various models based on your business requirements.
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>vllm-ascend/DeepSeek-V3.2-Exp-W8A8</td>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>DeepSeek-V3.2-W8A8</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>DeepSeek</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>vllm-ascend/DeepSeek-R1-0528-W8A8</td>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>DeepSeek-R1-0528-W8A8</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>DeepSeek</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>vllm-ascend/DeepSeek-V2-Lite-W8A8</td>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>DeepSeek-V2-Lite-W8A8</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>DeepSeek</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
@@ -64,7 +68,7 @@ You are welcome to enable various models based on your business requirements.
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>vllm-ascend/Qwen3-235B-A22B-W8A8</td>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen3-235B-A22B-W8A8</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Qwen</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
@@ -88,7 +92,7 @@ You are welcome to enable various models based on your business requirements.
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>vllm-ascend/QWQ-32B-W8A8</td>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>QWQ-32B-W8A8</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Qwen</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
@@ -256,13 +260,13 @@ You are welcome to enable various models based on your business requirements.
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Kimi/Kimi-K2-Thinking</td>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>moonshotai/Kimi-K2-Thinking</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Kimi</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>openai/gpt-oss-120b</td>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>eigen-ai-labs/gpt-oss-120b-bf16</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>GPTOSS</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
@@ -274,7 +278,7 @@ You are welcome to enable various models based on your business requirements.
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>minimax/MiniMax-M2</td>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>cyankiwi/MiniMax-M2-BF16</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>MiniMax-M2</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
@@ -300,12 +304,7 @@ You are welcome to enable various models based on your business requirements.
</tbody>
</table>
</Accordion>
<Accordion title="Multimodal Language Models">
## Multimodal Language Models
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
<colgroup>
@@ -318,8 +317,8 @@ You are welcome to enable various models based on your business requirements.
<tr style={{borderBottom: "2px solid #d55816"}}>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Models</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Model Family (Variants)</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>A2</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>A3</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>A2 Supported</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>A3 Supported</th>
</tr>
</thead>
<tbody>
@@ -432,7 +431,7 @@ You are welcome to enable various models based on your business requirements.
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Kimi/Kimi-VL-A3B-Instruct</td>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>moonshotai/Kimi-VL-A3B-Instruct</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Kimi-VL (A3B)</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
@@ -458,12 +457,7 @@ You are welcome to enable various models based on your business requirements.
</tbody>
</table>
</Accordion>
<Accordion title="Embedding Models">
## Embedding Models
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
<colgroup>
@@ -476,8 +470,8 @@ You are welcome to enable various models based on your business requirements.
<tr style={{borderBottom: "2px solid #d55816"}}>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Models</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Model Family</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>A2</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>A3</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>A2 Supported</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>A3 Supported</th>
</tr>
</thead>
<tbody>
@@ -520,12 +514,7 @@ You are welcome to enable various models based on your business requirements.
</tbody>
</table>
</Accordion>
<Accordion title="Reward Models">
## Reward Models
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
<colgroup>
@@ -538,8 +527,8 @@ You are welcome to enable various models based on your business requirements.
<tr style={{borderBottom: "2px solid #d55816"}}>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Models</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Model Family</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>A2</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>A3</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>A2 Supported</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>A3 Supported</th>
</tr>
</thead>
<tbody>
@@ -576,12 +565,7 @@ You are welcome to enable various models based on your business requirements.
</tbody>
</table>
</Accordion>
<Accordion title="Rerank Models">
## Rerank Models
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
<colgroup>
@@ -594,8 +578,8 @@ You are welcome to enable various models based on your business requirements.
<tr style={{borderBottom: "2px solid #d55816"}}>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Models</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Model Family</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>A2</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>A3</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>A2 Supported</th>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>A3 Supported</th>
</tr>
</thead>
<tbody>
@@ -607,4 +591,3 @@ You are welcome to enable various models based on your business requirements.
</tr>
</tbody>
</table>
</Accordion>
docs_new/docs/hardware-platforms/ascend-npus/MindSpore-Models.mdx → docs_new/docs/hardware-platforms/ascend-npus/mindspore_backend.mdx
+1 -1
View File
@@ -18,7 +18,7 @@ Currently, the following models are supported:
## Installation
<Note>
Currently, MindSpore models are provided by an independent package `sgl-mindspore`. Support for MindSpore is built upon current SGLang support for Ascend NPU platform. Please first [install SGLang for Ascend NPU](./SGLang-installation-with-NPUs-support) and then install `sgl-mindspore`:
Currently, MindSpore models are provided by an independent package `sgl-mindspore`. Support for MindSpore is built upon current SGLang support for Ascend NPU platform. Please first [install SGLang for Ascend NPU](./ascend_npu) and then install `sgl-mindspore`:
</Note>
<CodeGroup>
docs_new/docs/hardware-platforms/cpu-server.mdx
-355
View File
@@ -1,355 +0,0 @@
---
title: "CPU Servers"
---
The document addresses how to set up the [SGLang](https://github.com/sgl-project/sglang) environment and run LLM inference on CPU servers.
SGLang is enabled and optimized on the CPUs equipped with Intel® AMX® Instructions,
which are 4th generation or newer Intel® Xeon® Scalable Processors.
## Optimized Model List
A list of popular LLMs are optimized and run efficiently on CPU,
including the most notable open-source models like Llama series, Qwen series,
and DeepSeek series like DeepSeek-R1 and DeepSeek-V3.1-Terminus.
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
<colgroup>
<col style={{width: "22%"}} />
<col style={{width: "26%"}} />
<col style={{width: "30%"}} />
<col style={{width: "22%"}} />
</colgroup>
<thead>
<tr style={{borderBottom: "2px solid #d55816"}}>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Model Name</th>
<th style={{textAlign: "center", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>BF16</th>
<th style={{textAlign: "center", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>W8A8_INT8</th>
<th style={{textAlign: "center", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>FP8</th>
</tr>
</thead>
<tbody>
<tr>
<td style={{padding: "9px 12px", whiteSpace: "nowrap", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>DeepSeek-R1</td>
<td style={{padding: "9px 12px", textAlign: "center", color: "gray", backgroundColor: "rgba(255,255,255,0.05)"}}>—</td>
<td style={{padding: "9px 12px", textAlign: "center", backgroundColor: "rgba(255,255,255,0.02)"}}><a href="https://huggingface.co/meituan/DeepSeek-R1-Channel-INT8">DeepSeek-R1-Channel-INT8</a></td>
<td style={{padding: "9px 12px", textAlign: "center", backgroundColor: "rgba(255,255,255,0.05)"}}><a href="https://huggingface.co/deepseek-ai/DeepSeek-R1">DeepSeek-R1</a></td>
</tr>
<tr>
<td style={{padding: "9px 12px", whiteSpace: "nowrap", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>DeepSeek-V3.1-Terminus</td>
<td style={{padding: "9px 12px", textAlign: "center", color: "gray", backgroundColor: "rgba(255,255,255,0.05)"}}>—</td>
<td style={{padding: "9px 12px", textAlign: "center", backgroundColor: "rgba(255,255,255,0.02)"}}><a href="https://huggingface.co/IntervitensInc/DeepSeek-V3.1-Terminus-Channel-int8">DeepSeek-V3.1-Terminus-Channel-int8</a></td>
<td style={{padding: "9px 12px", textAlign: "center", backgroundColor: "rgba(255,255,255,0.05)"}}><a href="https://huggingface.co/deepseek-ai/DeepSeek-V3.1-Terminus">DeepSeek-V3.1-Terminus</a></td>
</tr>
<tr>
<td style={{padding: "9px 12px", whiteSpace: "nowrap", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Llama-3.2-3B</td>
<td style={{padding: "9px 12px", textAlign: "center", backgroundColor: "rgba(255,255,255,0.05)"}}><a href="https://huggingface.co/meta-llama/Llama-3.2-3B-Instruct">Llama-3.2-3B-Instruct</a></td>
<td style={{padding: "9px 12px", textAlign: "center", backgroundColor: "rgba(255,255,255,0.02)"}}><a href="https://huggingface.co/RedHatAI/Llama-3.2-3B-Instruct-quantized.w8a8">Llama-3.2-3B-quantized.w8a8</a></td>
<td style={{padding: "9px 12px", textAlign: "center", color: "gray", backgroundColor: "rgba(255,255,255,0.05)"}}>—</td>
</tr>
<tr>
<td style={{padding: "9px 12px", whiteSpace: "nowrap", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Llama-3.1-8B</td>
<td style={{padding: "9px 12px", textAlign: "center", backgroundColor: "rgba(255,255,255,0.05)"}}><a href="https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct">Llama-3.1-8B-Instruct</a></td>
<td style={{padding: "9px 12px", textAlign: "center", backgroundColor: "rgba(255,255,255,0.02)"}}><a href="https://huggingface.co/RedHatAI/Meta-Llama-3.1-8B-quantized.w8a8">Llama-3.1-8B-quantized.w8a8</a></td>
<td style={{padding: "9px 12px", textAlign: "center", color: "gray", backgroundColor: "rgba(255,255,255,0.05)"}}>—</td>
</tr>
<tr>
<td style={{padding: "9px 12px", whiteSpace: "nowrap", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>QwQ-32B</td>
<td style={{padding: "9px 12px", textAlign: "center", color: "gray", backgroundColor: "rgba(255,255,255,0.05)"}}>—</td>
<td style={{padding: "9px 12px", textAlign: "center", color: "gray", backgroundColor: "rgba(255,255,255,0.02)"}}><a href="https://huggingface.co/RedHatAI/QwQ-32B-quantized.w8a8">QwQ-32B-quantized.w8a8</a></td>
<td style={{padding: "9px 12px", textAlign: "center", color: "gray", backgroundColor: "rgba(255,255,255,0.05)"}}>—</td>
</tr>
<tr>
<td style={{padding: "9px 12px", whiteSpace: "nowrap", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>DeepSeek-Distilled-Llama</td>
<td style={{padding: "9px 12px", textAlign: "center", color: "gray", backgroundColor: "rgba(255,255,255,0.05)"}}>—</td>
<td style={{padding: "9px 12px", textAlign: "center", backgroundColor: "rgba(255,255,255,0.02)"}}><a href="https://huggingface.co/RedHatAI/DeepSeek-R1-Distill-Llama-70B-quantized.w8a8">DeepSeek-R1-Distill-Llama-70B-quantized.w8a8</a></td>
<td style={{padding: "9px 12px", textAlign: "center", color: "gray", backgroundColor: "rgba(255,255,255,0.05)"}}>—</td>
</tr>
<tr>
<td style={{padding: "9px 12px", whiteSpace: "nowrap", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen3-235B</td>
<td style={{padding: "9px 12px", textAlign: "center", color: "gray", backgroundColor: "rgba(255,255,255,0.05)"}}>—</td>
<td style={{padding: "9px 12px", textAlign: "center", color: "gray", backgroundColor: "rgba(255,255,255,0.02)"}}>—</td>
<td style={{padding: "9px 12px", textAlign: "center", backgroundColor: "rgba(255,255,255,0.05)"}}><a href="https://huggingface.co/Qwen/Qwen3-235B-A22B-FP8">Qwen3-235B-A22B-FP8</a></td>
</tr>
</tbody>
</table>
> **Note:** The model identifiers listed in the table above have been verified on 6th Gen Intel® Xeon® P-core platforms.
## Installation
<Tabs>
<Tab title="Docker (Recommended)">
It is recommended to use Docker for setting up the SGLang environment.
A [Dockerfile](https://github.com/sgl-project/sglang/blob/main/docker/xeon.Dockerfile) is provided to facilitate the installation.
> **Note:** Replace `<secret>` below with your [HuggingFace access token](https://huggingface.co/docs/hub/en/security-tokens).
<CodeGroup>
```bash Clone, Build and Run
# Clone the SGLang repository
git clone https://github.com/sgl-project/sglang.git
cd sglang/docker
# Build the docker image
docker build -t sglang-cpu:latest -f xeon.Dockerfile .
# Initiate a docker container
docker run \
-it \
--privileged \
--ipc=host \
--network=host \
-v /dev/shm:/dev/shm \
-v ~/.cache/huggingface:/root/.cache/huggingface \
-p 30000:30000 \
-e "HF_TOKEN=<secret>" \
sglang-cpu:latest /bin/bash
```
</CodeGroup>
</Tab>
<Tab title="From Source">
If you prefer to install SGLang in a bare metal environment, the setup process is as follows.
Please install the required packages and libraries beforehand if they are not already present on your system.
You can refer to the Ubuntu-based installation commands in
[the Dockerfile](https://github.com/sgl-project/sglang/blob/main/docker/xeon.Dockerfile#L11) for guidance.
1. **Install uv and create a virtual environment**
<CodeGroup>
```bash Create Virtual Environment
# Taking '/opt' as the example uv env folder, feel free to change it as needed
cd /opt
curl -LsSf https://astral.sh/uv/install.sh | sh
source $HOME/.local/bin/env
uv venv --python 3.12
source .venv/bin/activate
```
</CodeGroup>
2. **Create a config file for torch package indexes**
Create the `uv.toml` config file:
<CodeGroup>
```bash Open Config File
vim .venv/uv.toml
```
</CodeGroup>
Press `a` to enter insert mode in `vim`, then paste the following content:
<CodeGroup>
```toml
[[index]]
name = "torch"
url = "https://download.pytorch.org/whl/cpu"
[[index]]
name = "torchvision"
url = "https://download.pytorch.org/whl/cpu"
[[index]]
name = "torchaudio"
url = "https://download.pytorch.org/whl/cpu"
[[index]]
name = "triton"
url = "https://download.pytorch.org/whl/cpu"
```
</CodeGroup>
Save the file (press `Esc`, then type `:x` and hit `Enter`), then set it as the default `uv` config:
<CodeGroup>
```bash Set Config Path
export UV_CONFIG_FILE=/opt/.venv/uv.toml
```
</CodeGroup>
3. **Clone SGLang and build packages**
<CodeGroup>
```bash Build SGLang
# Clone the SGLang code
git clone https://github.com/sgl-project/sglang.git
cd sglang
git checkout <YOUR-DESIRED-VERSION>
# Use dedicated toml file
cd python
cp pyproject_cpu.toml pyproject.toml
# Install SGLang dependent libs, and build SGLang main package
uv pip install --upgrade pip setuptools
uv pip install .
# Build the CPU backend kernels
cd ../sgl-kernel
cp pyproject_cpu.toml pyproject.toml
uv pip install .
```
</CodeGroup>
4. **Set required environment variables**
<CodeGroup>
```bash Set Environment Variables
export SGLANG_USE_CPU_ENGINE=1
# Set 'LD_LIBRARY_PATH' and 'LD_PRELOAD' to ensure the libs can be loaded by sglang processes
export LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu
export LD_PRELOAD=${LD_PRELOAD}:/opt/.venv/lib/libiomp5.so:${LD_LIBRARY_PATH}/libtcmalloc.so.4:${LD_LIBRARY_PATH}/libtbbmalloc.so.2
```
</CodeGroup>
> **Note:** The environment variable `SGLANG_USE_CPU_ENGINE=1` is required to enable the SGLang service with the CPU engine.
> **Note:** If you encounter code compilation issues during the `sgl-kernel` building process, please check your `gcc` and `g++` versions and upgrade them if they are outdated. It is recommended to use `gcc-13` and `g++-13` as they have been verified in the official Docker container.
> **Note:** The system library path is typically located in one of the following directories: `~/.local/lib/`, `/usr/local/lib/`, `/usr/local/lib64/`, `/usr/lib/`, `/usr/lib64/`, and `/usr/lib/x86_64-linux-gnu/`. In the above example commands, `/usr/lib/x86_64-linux-gnu` is used. Please adjust the path according to your server configuration.
It is recommended to add the following to your `~/.bashrc` file to avoid setting these variables every time you open a new terminal:
<CodeGroup>
```bash Persist in ~/.bashrc
source .venv/bin/activate
export SGLANG_USE_CPU_ENGINE=1
export LD_LIBRARY_PATH=<YOUR-SYSTEM-LIBRARY-FOLDER>
export LD_PRELOAD=<YOUR-LIBS-PATHS>
```
</CodeGroup>
</Tab>
</Tabs>
## Launch of the Serving Engine
Example command to launch SGLang serving:
<CodeGroup>
```bash Launch Server
python -m sglang.launch_server \
--model <MODEL_ID_OR_PATH> \
--trust-remote-code \
--disable-overlap-schedule \
--device cpu \
--host 0.0.0.0 \
--tp 6
```
</CodeGroup>
> **Note:** For running W8A8 quantized models, please add the flag `--quantization w8a8_int8`.
> **Note:** The flag `--tp 6` specifies that tensor parallelism will be applied using 6 ranks (TP6). On a CPU platform, a TP rank means a sub-NUMA cluster (SNC). You can get the SNC count using `lscpu`. If the specified TP rank number differs from the total SNC count, the system will automatically utilize the first `n` SNCs — but `n` cannot exceed the total SNC number.
>
> To specify the cores to be used, set the environment variable `SGLANG_CPU_OMP_THREADS_BIND`. For example, to use the first 40 cores of each SNC on a Xeon® 6980P server (which has 43-43-42 cores on the 3 SNCs of a socket):
<CodeGroup>
```bash Set Thread Binding
export SGLANG_CPU_OMP_THREADS_BIND="0-39|43-82|86-125|128-167|171-210|214-253"
```
</CodeGroup>
> Please beware that with `SGLANG_CPU_OMP_THREADS_BIND` set, the available memory amounts of the ranks may not be determined in advance. You may need to set `--max-total-tokens` to avoid out-of-memory errors.
> **Note:** For optimizing decoding with `torch.compile`, add the flag `--enable-torch-compile`. To specify the maximum batch size, set `--torch-compile-max-bs`. For example, `--enable-torch-compile --torch-compile-max-bs 4` uses `torch.compile` with a maximum batch size of 4. The maximum applicable batch size is 16.
> **Note:** A warmup step is automatically triggered when the service is started. The server is ready when you see the log `The server is fired up and ready to roll!`.
## Benchmarking with Requests
You can benchmark the performance via the `bench_serving` script.
Run the command in another terminal. An example command would be:
<CodeGroup>
```bash Run Benchmark
python -m sglang.bench_serving \
--dataset-name random \
--random-input-len 1024 \
--random-output-len 1024 \
--num-prompts 1 \
--request-rate inf \
--random-range-ratio 1.0
```
</CodeGroup>
Detailed parameter descriptions are available via the command:
<CodeGroup>
```bash Benchmark Help
python -m sglang.bench_serving -h
```
</CodeGroup>
Additionally, requests can be formatted using
[the OpenAI Completions API](../basic_usage/openai_api_completions)
and sent via the command line (e.g., using `curl`) or through your own scripts.
## Example Usage Commands
Large Language Models can range from fewer than 1 billion to several hundred billion parameters.
Dense models larger than 20B are expected to run on flagship 6th Gen Intel® Xeon® processors
with dual sockets and a total of 6 sub-NUMA clusters. Dense models of approximately 10B parameters or fewer,
or MoE (Mixture of Experts) models with fewer than 10B activated parameters, can run on more common
4th generation or newer Intel® Xeon® processors, or utilize a single socket of the flagship 6th Gen Intel® Xeon® processors.
### Example: Running DeepSeek-V3.1-Terminus
<CodeGroup>
```bash W8A8_INT8
python -m sglang.launch_server \
--model IntervitensInc/DeepSeek-V3.1-Terminus-Channel-int8 \
--trust-remote-code \
--disable-overlap-schedule \
--device cpu \
--quantization w8a8_int8 \
--host 0.0.0.0 \
--enable-torch-compile \
--torch-compile-max-bs 4 \
--tp 6
```
```bash FP8
python -m sglang.launch_server \
--model deepseek-ai/DeepSeek-V3.1-Terminus \
--trust-remote-code \
--disable-overlap-schedule \
--device cpu \
--host 0.0.0.0 \
--enable-torch-compile \
--torch-compile-max-bs 4 \
--tp 6
```
</CodeGroup>
> **Note:** Please set `--torch-compile-max-bs` to the maximum desired batch size for your deployment, which can be up to 16. The value `4` in the examples is illustrative.
### Example: Running Llama-3.2-3B
<CodeGroup>
```bash BF16
python -m sglang.launch_server \
--model meta-llama/Llama-3.2-3B-Instruct \
--trust-remote-code \
--disable-overlap-schedule \
--device cpu \
--host 0.0.0.0 \
--enable-torch-compile \
--torch-compile-max-bs 16 \
--tp 2
```
```bash W8A8_INT8
python -m sglang.launch_server \
--model RedHatAI/Llama-3.2-3B-quantized.w8a8 \
--trust-remote-code \
--disable-overlap-schedule \
--device cpu \
--quantization w8a8_int8 \
--host 0.0.0.0 \
--enable-torch-compile \
--torch-compile-max-bs 16 \
--tp 2
```
</CodeGroup>
> **Note:** The `--torch-compile-max-bs` and `--tp` settings are examples that should be adjusted for your setup. For instance, use `--tp 3` to utilize 1 socket with 3 sub-NUMA clusters on an Intel® Xeon® 6980P server.
Once the server has been launched, you can test it using the `bench_serving` command or create
your own commands or scripts following [the benchmarking example](#benchmarking-with-requests).
docs_new/docs/hardware-platforms/cpu_server.mdx
+387
View File
@@ -0,0 +1,387 @@
---
title: "CPU Servers"
---
The document addresses how to set up the [SGLang](https://github.com/sgl-project/sglang) environment and run LLM inference on CPU servers.
SGLang is enabled and optimized on the CPUs equipped with Intel® AMX® Instructions,
which are 4th generation or newer Intel® Xeon® Scalable Processors.
## Optimized Model List
A list of popular LLMs are optimized and run efficiently on CPU,
including the most notable open-source models like Llama series, Qwen series,
and DeepSeek series like DeepSeek-R1 and DeepSeek-V3.1-Terminus.
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
<colgroup>
<col style={{width: "22%"}} />
<col style={{width: "26%"}} />
<col style={{width: "30%"}} />
<col style={{width: "22%"}} />
</colgroup>
<thead>
<tr style={{borderBottom: "2px solid #d55816"}}>
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Model Name</th>
<th style={{textAlign: "center", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>BF16</th>
<th style={{textAlign: "center", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>W8A8_INT8</th>
<th style={{textAlign: "center", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>FP8</th>
</tr>
</thead>
<tbody>
<tr>
<td style={{padding: "9px 12px", whiteSpace: "nowrap", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>DeepSeek-R1</td>
<td style={{padding: "9px 12px", textAlign: "center", color: "gray", backgroundColor: "rgba(255,255,255,0.05)"}}></td>
<td style={{padding: "9px 12px", textAlign: "center", backgroundColor: "rgba(255,255,255,0.02)"}}><a href="https://huggingface.co/meituan/DeepSeek-R1-Channel-INT8">meituan/DeepSeek-R1-Channel-INT8</a></td>
<td style={{padding: "9px 12px", textAlign: "center", backgroundColor: "rgba(255,255,255,0.05)"}}><a href="https://huggingface.co/deepseek-ai/DeepSeek-R1">deepseek-ai/DeepSeek-R1</a></td>
</tr>
<tr>
<td style={{padding: "9px 12px", whiteSpace: "nowrap", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>DeepSeek-V3.1-Terminus</td>
<td style={{padding: "9px 12px", textAlign: "center", color: "gray", backgroundColor: "rgba(255,255,255,0.05)"}}></td>
<td style={{padding: "9px 12px", textAlign: "center", backgroundColor: "rgba(255,255,255,0.02)"}}><a href="https://huggingface.co/IntervitensInc/DeepSeek-V3.1-Terminus-Channel-int8">IntervitensInc/DeepSeek-V3.1-Terminus-Channel-int8</a></td>
<td style={{padding: "9px 12px", textAlign: "center", backgroundColor: "rgba(255,255,255,0.05)"}}><a href="https://huggingface.co/deepseek-ai/DeepSeek-V3.1-Terminus">deepseek-ai/DeepSeek-V3.1-Terminus</a></td>
</tr>
<tr>
<td style={{padding: "9px 12px", whiteSpace: "nowrap", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Llama-3.2-3B</td>
<td style={{padding: "9px 12px", textAlign: "center", backgroundColor: "rgba(255,255,255,0.05)"}}><a href="https://huggingface.co/meta-llama/Llama-3.2-3B-Instruct">meta-llama/Llama-3.2-3B-Instruct</a></td>
<td style={{padding: "9px 12px", textAlign: "center", backgroundColor: "rgba(255,255,255,0.02)"}}><a href="https://huggingface.co/RedHatAI/Llama-3.2-3B-Instruct-quantized.w8a8">RedHatAI/Llama-3.2-3B-quantized.w8a8</a></td>
<td style={{padding: "9px 12px", textAlign: "center", color: "gray", backgroundColor: "rgba(255,255,255,0.05)"}}></td>
</tr>
<tr>
<td style={{padding: "9px 12px", whiteSpace: "nowrap", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Llama-3.1-8B</td>
<td style={{padding: "9px 12px", textAlign: "center", backgroundColor: "rgba(255,255,255,0.05)"}}><a href="https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct">meta-llama/Llama-3.1-8B-Instruct</a></td>
<td style={{padding: "9px 12px", textAlign: "center", backgroundColor: "rgba(255,255,255,0.02)"}}><a href="https://huggingface.co/RedHatAI/Meta-Llama-3.1-8B-quantized.w8a8">RedHatAI/Meta-Llama-3.1-8B-quantized.w8a8</a></td>
<td style={{padding: "9px 12px", textAlign: "center", color: "gray", backgroundColor: "rgba(255,255,255,0.05)"}}></td>
</tr>
<tr>
<td style={{padding: "9px 12px", whiteSpace: "nowrap", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>QwQ-32B</td>
<td style={{padding: "9px 12px", textAlign: "center", color: "gray", backgroundColor: "rgba(255,255,255,0.05)"}}></td>
<td style={{padding: "9px 12px", textAlign: "center", color: "gray", backgroundColor: "rgba(255,255,255,0.02)"}}><a href="https://huggingface.co/RedHatAI/QwQ-32B-quantized.w8a8">RedHatAI/QwQ-32B-quantized.w8a8</a></td>
<td style={{padding: "9px 12px", textAlign: "center", color: "gray", backgroundColor: "rgba(255,255,255,0.05)"}}></td>
</tr>
<tr>
<td style={{padding: "9px 12px", whiteSpace: "nowrap", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>DeepSeek-Distilled-Llama</td>
<td style={{padding: "9px 12px", textAlign: "center", color: "gray", backgroundColor: "rgba(255,255,255,0.05)"}}></td>
<td style={{padding: "9px 12px", textAlign: "center", backgroundColor: "rgba(255,255,255,0.02)"}}><a href="https://huggingface.co/RedHatAI/DeepSeek-R1-Distill-Llama-70B-quantized.w8a8">RedHatAI/DeepSeek-R1-Distill-Llama-70B-quantized.w8a8</a></td>
<td style={{padding: "9px 12px", textAlign: "center", color: "gray", backgroundColor: "rgba(255,255,255,0.05)"}}></td>
</tr>
<tr>
<td style={{padding: "9px 12px", whiteSpace: "nowrap", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen3-235B</td>
<td style={{padding: "9px 12px", textAlign: "center", color: "gray", backgroundColor: "rgba(255,255,255,0.05)"}}></td>
<td style={{padding: "9px 12px", textAlign: "center", color: "gray", backgroundColor: "rgba(255,255,255,0.02)"}}></td>
<td style={{padding: "9px 12px", textAlign: "center", backgroundColor: "rgba(255,255,255,0.05)"}}><a href="https://huggingface.co/Qwen/Qwen3-235B-A22B-FP8">Qwen/Qwen3-235B-A22B-FP8</a></td>
</tr>
</tbody>
</table>
**Note:** The model identifiers listed in the table above
have been verified on 6th Gen Intel® Xeon® P-core platforms.
## Installation
### Install Using Docker
It is recommended to use Docker for setting up the SGLang environment.
A [Dockerfile](https://github.com/sgl-project/sglang/blob/main/docker/xeon.Dockerfile) is provided to facilitate the installation.
Replace `<secret>` below with your [HuggingFace access token](https://huggingface.co/docs/hub/en/security-tokens).
```bash Command
# Clone the SGLang repository
git clone https://github.com/sgl-project/sglang.git
cd sglang/docker
# Build the docker image
docker build -t sglang-cpu:latest -f xeon.Dockerfile .
# Initiate a docker container
docker run \
-it \
--privileged \
--ipc=host \
--network=host \
-v /dev/shm:/dev/shm \
-v ~/.cache/huggingface:/root/.cache/huggingface \
-p 30000:30000 \
-e "HF_TOKEN=<secret>" \
sglang-cpu:latest /bin/bash
```
### Install From Source
If you prefer to install SGLang in a bare metal environment,
the setup process is as follows:
Please install the required packages and libraries beforehand if
they are not already present on your system.
You can refer to the Ubuntu-based installation commands in
[the Dockerfile](https://github.com/sgl-project/sglang/blob/main/docker/xeon.Dockerfile#L11)
for guidance.
1. Install `uv` package manager, then create and activate a virtual environment:
```bash Command
# Taking '/opt' as the example uv env folder, feel free to change it as needed
cd /opt
curl -LsSf https://astral.sh/uv/install.sh | sh
source $HOME/.local/bin/env
uv venv --python 3.12
source .venv/bin/activate
```
2. Create a config file to direct the installation channel
(a.k.a. index-url) of `torch` related packages:
```bash Command
vim .venv/uv.toml
```
Press 'a' to enter insert mode of `vim`, paste the following content into the created file
```file
[[index]]
name = "torch"
url = "https://download.pytorch.org/whl/cpu"
[[index]]
name = "torchvision"
url = "https://download.pytorch.org/whl/cpu"
[[index]]
name = "torchaudio"
url = "https://download.pytorch.org/whl/cpu"
[[index]]
name = "triton"
url = "https://download.pytorch.org/whl/cpu"
```
Save the file (in `vim`, press 'esc' to exit insert mode, then ':x+Enter'),
and set it as the default `uv` config.
```bash Command
export UV_CONFIG_FILE=/opt/.venv/uv.toml
```
3. Clone the `sglang` source code and build the packages
```bash Command
# Clone the SGLang code
git clone https://github.com/sgl-project/sglang.git
cd sglang
git checkout <YOUR-DESIRED-VERSION>
# Use dedicated toml file
cd python
cp pyproject_cpu.toml pyproject.toml
# Install SGLang dependent libs, and build SGLang main package
uv pip install --upgrade pip setuptools
uv pip install .
# Build the CPU backend kernels
cd ../sgl-kernel
cp pyproject_cpu.toml pyproject.toml
uv pip install .
```
4. Set the required environment variables
```bash Command
export SGLANG_USE_CPU_ENGINE=1
# Set 'LD_LIBRARY_PATH' and 'LD_PRELOAD' to ensure the libs can be loaded by sglang processes
export LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu
export LD_PRELOAD=${LD_PRELOAD}:/opt/.venv/lib/libiomp5.so:${LD_LIBRARY_PATH}/libtcmalloc.so.4:${LD_LIBRARY_PATH}/libtbbmalloc.so.2
```
Notes:
- Note that the environment variable `SGLANG_USE_CPU_ENGINE=1`
is required to enable the SGLang service with the CPU engine.
- If you encounter code compilation issues during the `sgl-kernel` building process,
please check your `gcc` and `g++` versions and upgrade them if they are outdated.
It is recommended to use `gcc-13` and `g++-13` as they have been verified
in the official Docker container.
- The system library path is typically located in one of the following directories:
`~/.local/lib/`, `/usr/local/lib/`, `/usr/local/lib64/`, `/usr/lib/`, `/usr/lib64/`
and `/usr/lib/x86_64-linux-gnu/`. In the above example commands, `/usr/lib/x86_64-linux-gnu`
is used. Please adjust the path according to your server configuration.
- It is recommended to add the following to your `~/.bashrc` file to
avoid setting these variables every time you open a new terminal:
```bash Command
source .venv/bin/activate
export SGLANG_USE_CPU_ENGINE=1
export LD_LIBRARY_PATH=<YOUR-SYSTEM-LIBRARY-FOLDER>
export LD_PRELOAD=<YOUR-LIBS-PATHS>
```
## Launch of the Serving Engine
Example command to launch SGLang serving:
```bash Launch Server
python -m sglang.launch_server \
--model <MODEL_ID_OR_PATH> \
--trust-remote-code \
--disable-overlap-schedule \
--device cpu \
--host 0.0.0.0 \
--tp 6
```
Notes:
1. For running W8A8 quantized models, please add the flag `--quantization w8a8_int8`.
2. The flag `--tp 6` specifies that tensor parallelism will be applied using 6 ranks (TP6).
The number of TP specified is how many TP ranks will be used during the execution.
On a CPU platform, a TP rank means a sub-NUMA cluster (SNC).
Usually we can get the SNC information (How many available) from the Operating System with e.g. `lscpu` command.
If the specified TP rank number differs from the total SNC count,
the system will automatically utilize the first `n` SNCs.
Note that `n` cannot exceed the total SNC number, doing so will result in an error.
`SGLANG_CPU_OMP_THREADS_BIND` allows explicit control of CPU cores for each tensor parallel (TP) rank.
**example 1**: Run SGLang service with TP=6, using the first 40 cores of each SNC on a Xeon® 6980P server,
which has 43-43-42 cores on the 3 SNCs of a socket, we should set:
```bash Command
export SGLANG_CPU_OMP_THREADS_BIND="0-39|43-82|86-125|128-167|171-210|214-253"
```
This configuration is equivalent to:
- rank 0: `numactl -C 0-39 -m 0`
- rank 1: `numactl -C 43-82 -m 1`
- rank 2: `numactl -C 86-125 -m 2`
- rank 3: `numactl -C 128-167 -m 3`
- rank 4: `numactl -C 171-210 -m 4`
- rank 5: `numactl -C 214-253 -m 5`
**example 2**: Run SGLang service with TP=2, using 96 cores cross 3 SNCs on a Xeon® 6972P server,
which has 32-32-32 cores on the 3 SNCs in a socket, we should set:
```bash Command
export SGLANG_CPU_OMP_THREADS_BIND="0-95|96-191"
```
This configuration is equivalent to:
- rank 0: `numactl -C 0-95 -m 0-2`
- rank 1: `numactl -C 96-191 -m 3-5`
Please beware that with SGLANG_CPU_OMP_THREADS_BIND set,
the available memory amounts of the ranks may not be determined in prior.
You may need to set proper `--max-total-tokens` to avoid the out-of-memory error.
3. For optimizing decoding with torch.compile, please add the flag `--enable-torch-compile`.
To specify the maximum batch size when using `torch.compile`, set the flag `--torch-compile-max-bs`.
For example, `--enable-torch-compile --torch-compile-max-bs 4` means using `torch.compile`
and setting the maximum batch size to 4.
4. A warmup step is automatically triggered when the service is started.
The server is ready when you see the log `The server is fired up and ready to roll!`.
## Benchmarking with Requests
You can benchmark the performance via the `bench_serving` script.
Run the command in another terminal. An example command would be:
```bash Run Benchmark
python -m sglang.bench_serving \
--dataset-name random \
--random-input-len 1024 \
--random-output-len 1024 \
--num-prompts 1 \
--request-rate inf \
--random-range-ratio 1.0
```
Detailed parameter descriptions are available via the command:
```bash Benchmark Help
python -m sglang.bench_serving -h
```
Additionally, requests can be formatted using
[the OpenAI Completions API](../basic_usage/openai_api_completions)
and sent via the command line (e.g., using `curl`) or through your own scripts.
## Example Usage Commands
Large Language Models can range from fewer than 1 billion to several hundred billion parameters.
Dense models larger than 20B are expected to run on flagship 6th Gen Intel® Xeon® processors
with dual sockets and a total of 6 sub-NUMA clusters. Dense models of approximately 10B parameters or fewer,
or MoE (Mixture of Experts) models with fewer than 10B activated parameters, can run on more common
4th generation or newer Intel® Xeon® processors, or utilize a single socket of the flagship 6th Gen Intel® Xeon® processors.
### Example: Running DeepSeek-V3.1-Terminus
An example command to launch service of W8A8_INT8 DeepSeek-V3.1-Terminus on a Xeon® 6980P server:
```bash W8A8_INT8
python -m sglang.launch_server \
--model IntervitensInc/DeepSeek-V3.1-Terminus-Channel-int8 \
--trust-remote-code \
--disable-overlap-schedule \
--device cpu \
--quantization w8a8_int8 \
--host 0.0.0.0 \
--enable-torch-compile \
--torch-compile-max-bs 4 \
--tp 6
```
Similarly, an example command to launch service of FP8 DeepSeek-V3.1-Terminus would be:
```bash FP8
python -m sglang.launch_server \
--model deepseek-ai/DeepSeek-V3.1-Terminus \
--trust-remote-code \
--disable-overlap-schedule \
--device cpu \
--host 0.0.0.0 \
--enable-torch-compile \
--torch-compile-max-bs 4 \
--tp 6
```
Note: Please set `--torch-compile-max-bs` to the maximum desired batch size for your deployment,
which can be up to 16. The value `4` in the examples is illustrative.
### Example: Running Llama-3.2-3B
An example command to launch service of Llama-3.2-3B with BF16 precision:
```bash BF16
python -m sglang.launch_server \
--model meta-llama/Llama-3.2-3B-Instruct \
--trust-remote-code \
--disable-overlap-schedule \
--device cpu \
--host 0.0.0.0 \
--enable-torch-compile \
--torch-compile-max-bs 16 \
--tp 2
```
The example command to launch service of W8A8_INT8 version of Llama-3.2-3B:
```bash W8A8_INT8
python -m sglang.launch_server \
--model RedHatAI/Llama-3.2-3B-quantized.w8a8 \
--trust-remote-code \
--disable-overlap-schedule \
--device cpu \
--quantization w8a8_int8 \
--host 0.0.0.0 \
--enable-torch-compile \
--torch-compile-max-bs 16 \
--tp 2
```
Note: The `--torch-compile-max-bs` and `--tp` settings are examples that should be adjusted for your setup.
For instance, use `--tp 3` to utilize 1 socket with 3 sub-NUMA clusters on an Intel® Xeon® 6980P server.
Once the server have been launched, you can test it using the `bench_serving` command or create
your own commands or scripts following [the benchmarking example](#benchmarking-with-requests).
docs_new/docs/hardware-platforms/mthreads_gpu.mdx
+29
View File
@@ -0,0 +1,29 @@
---
title: "Moore Threads GPUs"
metatags:
description: "Run SGLang on Moore Threads GPUs."
---
This document describes how run SGLang on Moore Threads GPUs. If you encounter issues or have questions, please [open an issue](https://github.com/sgl-project/sglang/issues).
## Install SGLang
You can install SGLang using one of the methods below.
### Install from Source
```bash
# Use the default branch
git clone https://github.com/sgl-project/sglang.git
cd sglang
# Compile sgl-kernel
pip install --upgrade pip
cd sgl-kernel
python setup_musa.py install
# Install sglang python package
cd ..
rm -f python/pyproject.toml && mv python/pyproject_other.toml python/pyproject.toml
pip install -e "python[all_musa]"
```
docs_new/docs/hardware-platforms/nvidia-gpus.mdx
+1 -1
View File
@@ -2,4 +2,4 @@
title: NVIDIA GPUs
---
Please refer to the [Installation Guide](/docs/get-started/installation) to get started with SGLang on NVIDIA GPUs.
Please refer to the [Installation Guide](../get-started/install) to get started with SGLang on NVIDIA GPUs.
docs_new/docs/hardware-platforms/nvidia.mdx
-102
View File
@@ -1,102 +0,0 @@
---
title: NVIDIA Jetson Orin
description: Guide for installing and running SGLang on NVIDIA Jetson Orin devices.
---
## Prerequisites
Before starting, ensure the following:
- [NVIDIA Jetson AGX Orin Devkit](https://www.nvidia.com/en-us/autonomous-machines/embedded-systems/jetson-orin/) is set up with JetPack 6.1 or later.
- CUDA Toolkit and cuDNN are installed.
- Verify that the Jetson AGX Orin is in high-performance mode:
<CodeGroup>
```bash
sudo nvpmodel -m 0
```
</CodeGroup>
## Installing and Running SGLang with Jetson Containers
1. **Clone the jetson-containers repository**
```bash
git clone https://github.com/dusty-nv/jetson-containers.git
```
2. **Run the installation script**
```bash
bash jetson-containers/install.sh
```
3. **Build the container image**
```bash
jetson-containers build sglang
```
4. **Run the container**
<Tabs>
<Tab title="Using jetson-containers">
```bash
jetson-containers run $(autotag sglang)
```
</Tab>
<Tab title="Using Docker manually">
```bash
docker run --runtime nvidia -it --rm --network=host IMAGE_NAME
```
</Tab>
</Tabs>
## Running Inference
Launch the server:
<CodeGroup>
```bash
python -m sglang.launch_server \
--model-path deepseek-ai/DeepSeek-R1-Distill-Llama-8B \
--device cuda \
--dtype half \
--attention-backend flashinfer \
--mem-fraction-static 0.8 \
--context-length 8192
```
</CodeGroup>
The quantization and limited context length (`--dtype half` `--context-length 8192`) are due to the limited computational resources in [Nvidia jetson kit](https://www.nvidia.com/en-us/autonomous-machines/embedded-systems/jetson-orin/). A detailed explanation can be found in [Server Arguments](../advanced_features/server_arguments).
After launching the engine, refer to [Chat completions](../basic_usage/openai_api_completions#Usage) to test the usability.
## Running Quantization with TorchAO
TorchAO is suggested to NVIDIA Jetson Orin.
<CodeGroup>
```bash
python -m sglang.launch_server \
--model-path meta-llama/Meta-Llama-3.1-8B-Instruct \
--device cuda \
--dtype bfloat16 \
--attention-backend flashinfer \
--mem-fraction-static 0.8 \
--context-length 8192 \
--torchao-config int4wo-128
```
</CodeGroup>
This enables TorchAO's int4 weight-only quantization with a 128-group size. The usage of `--torchao-config int4wo-128` is also for memory efficiency.
## Structured Output with XGrammar
Please refer to [SGLang doc structured output](../advanced_features/structured_outputs).
Thanks to the support from [Nurgaliyev Shakhizat](https://github.com/shahizat), [Dustin Franklin](https://github.com/dusty-nv) and [Johnny Núñez Cano](https://github.com/johnnynunez).
## References
- [NVIDIA Jetson AGX Orin Documentation](https://developer.nvidia.com/embedded/jetson-agx-orin)
docs_new/docs/hardware-platforms/nvidia_jetson.mdx
+82
View File
@@ -0,0 +1,82 @@
---
title: NVIDIA Jetson Orin
description: Guide for installing and running SGLang on NVIDIA Jetson Orin devices.
---
## Prerequisites
Before starting, ensure the following:
- [**NVIDIA Jetson AGX Orin Devkit**](https://www.nvidia.com/en-us/autonomous-machines/embedded-systems/jetson-orin/) is set up with **JetPack 6.1** or later.
- **CUDA Toolkit** and **cuDNN** are installed.
- Verify that the Jetson AGX Orin is in **high-performance mode**:
```bash
sudo nvpmodel -m 0
```
* * * * *
## Installing and running SGLang with Jetson Containers
Clone the jetson-containers github repository:
```bash
git clone https://github.com/dusty-nv/jetson-containers.git
```
Run the installation script:
```bash
bash jetson-containers/install.sh
```
Build the container image:
```bash
jetson-containers build sglang
```
Run the container:
```
jetson-containers run $(autotag sglang)
```
Or you can also manually run a container with this command:
```
docker run --runtime nvidia -it --rm --network=host IMAGE_NAME
```
* * * * *
Running Inference
-----------------------------------------
Launch the server:
```bash
python -m sglang.launch_server \
--model-path deepseek-ai/DeepSeek-R1-Distill-Llama-8B \
--device cuda \
--dtype half \
--attention-backend flashinfer \
--mem-fraction-static 0.8 \
--context-length 8192
```
The quantization and limited context length (`--dtype half --context-length 8192`) are due to the limited computational resources in [Nvidia jetson kit](https://www.nvidia.com/en-us/autonomous-machines/embedded-systems/jetson-orin/). A detailed explanation can be found in [Server Arguments](../advanced_features/server_arguments).
After launching the engine, refer to [Chat completions](../basic_usage/openai_api_completions#Usage) to test the usability.
* * * * *
Running quantization with TorchAO
-------------------------------------
TorchAO is suggested to NVIDIA Jetson Orin.
```bash Command
python -m sglang.launch_server \
--model-path meta-llama/Meta-Llama-3.1-8B-Instruct \
--device cuda \
--dtype bfloat16 \
--attention-backend flashinfer \
--mem-fraction-static 0.8 \
--context-length 8192 \
--torchao-config int4wo-128
```
This enables TorchAO's int4 weight-only quantization with a 128-group size. The usage of `--torchao-config int4wo-128` is also for memory efficiency.
* * * * *
Structured output with XGrammar
-------------------------------
Please refer to [SGLang doc structured output](../advanced_features/structured_outputs).
* * * * *
Thanks to the support from [Nurgaliyev Shakhizat](https://github.com/shahizat), [Dustin Franklin](https://github.com/dusty-nv) and [Johnny Núñez Cano](https://github.com/johnnynunez).
References
----------
- [NVIDIA Jetson AGX Orin Documentation](https://developer.nvidia.com/embedded/jetson-agx-orin)
docs_new/docs/hardware-platforms/overview.mdx
+4 -4
View File
@@ -4,9 +4,9 @@ description: Platform-specific guides for running SGLang on GPUs, TPUs, NPUs, CP
---
- [NVIDIA GPUs](./nvidia-gpus)
- [AMD GPUs](./amd-gpus)
- [Ascend NPUs](./ascend-npus/SGLang-installation-with-NPUs-support)
- [CPU Server](./cpu-server)
- [NVIDIA (Edge & Embedded)](./nvidia)
- [AMD GPUs](./amd_gpu)
- [Ascend NPUs](./ascend-npus/ascend_npu)
- [CPU Server](./cpu_server)
- [NVIDIA Jetson Orin](./nvidia_jetson)
- [TPU](./tpu)
- [XPU](./xpu)
docs_new/docs/hardware-platforms/plugin.mdx
+849
View File
@@ -0,0 +1,849 @@
---
title: "SGLang Plugin System"
metatags:
description: "Allows hardware vendors and developers to extend SGLang without modifying the main repository code."
---
## Overview
Allows hardware vendors and developers to extend SGLang **without modifying the main repository code**.
The framework provides two plugin types, both discovered via Python's standard `setuptools` entry_points:
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
<colgroup>
<col style={{width: "33.33%"}} />
<col style={{width: "33.33%"}} />
<col style={{width: "33.33%"}} />
</colgroup>
<thead>
<tr>
<th>Plugin Type</th>
<th>Entry Point Group</th>
<th>Purpose</th>
</tr>
</thead>
<tbody>
<tr>
<td><strong>Hardware Platform Plugin</strong></td>
<td><code>sglang.srt.platforms</code></td>
<td>Register a custom hardware platform (device operations, KV cache pools, attention backends, graph capture, compilation backends, etc.)</td>
</tr>
<tr>
<td><strong>General Plugin</strong></td>
<td><code>sglang.srt.plugins</code></td>
<td>Inject hooks (before/after/around/replace) into any function/method, or replace entire classes</td>
</tr>
</tbody>
</table>
### Principles
- **Non-intrusive**: Existing CUDA/ROCm/NPU/XPU code remains unchanged. OOT code paths are added alongside existing hardware-specific logic.
- **Zero configuration**: Plugins are automatically discovered after `pip install`, no sglang code changes required.
- **Environment variable control**: `SGLANG_PLATFORM` selects or validates the active platform plugin; `SGLANG_PLUGINS` (comma-separated) controls which general plugins to load.
### Current Scope & Future Direction
The plugin system currently targets **out-of-tree (OOT) hardware platforms** — enabling new devices to integrate with SGLang without any changes to the main repository. The main-repo hardware paths (CUDA, ROCm, NPU, XPU, etc.) continue to use the existing `is_cuda()`/`is_npu()`/… utility functions.
As the plugin interfaces mature and stabilize, in-tree hardware backends can be gradually migrated to the same plugin architecture. This would replace the scattered `if device == "cuda" … elif device == "npu" …` branches throughout the codebase with a single polymorphic dispatch through the platform interface, making each hardware backend self-contained and the core engine hardware-agnostic.
## Architecture
### Platform Hierarchy
The platform hierarchy uses a DeviceMixin pattern to share device operations between SRT (LLM inference) and Multimodal subsystems:
```
DeviceMixin (shared device identity + operations)
├── SRTPlatform(DeviceMixin) # + graph runner, KV pool, …
│ └── MySRTPlatform(SRTPlatform, MyDeviceMixin) # OOT plugin
└── MMPlatform(DeviceMixin) # + attention backend, VAE, … (future)
└── MyMMPlatform(MMPlatform, MyDeviceMixin) # OOT plugin
```
Key design points:
- **DeviceMixin** provides platform identity queries (`is_cuda()`, `is_npu()`, etc.) and device operations (`set_device()`, `get_device_name()`, etc.)
- **SRTPlatform** adds SRT-specific factory methods, capability flags, and lifecycle hooks
- OOT plugins implement a **device mixin** (vendor-specific operations) and compose it with **SRTPlatform** via multiple inheritance
- All methods are **instance methods** (not classmethods), called through the `current_platform` singleton
- Device operations and factory methods raise `NotImplementedError` by default (fail-fast)
- Capability flags use safe conservative defaults (`False`/`pass`)
- Methods are annotated `[Active]` (called by SGLang core) or `[Planned]` (reserved for future migration)
### Platform Discovery (`current_platform`)
`current_platform` is a **lazy singleton** in `sglang.srt.platforms`. On first access it resolves the active platform through the following priority chain:
```
entry_points("sglang.srt.platforms") → Enumerate ALL plugins by name (metadata only)
│
├─ SGLANG_PLATFORM set (front-loading filter):
│ ├─ Name not found in discovered → RuntimeError
│ ├─ activate() returns non-None → load that platform
│ └─ activate() returns None → RuntimeError (hardware unavailable)
│
└─ SGLANG_PLATFORM unset (auto-discover, activate all):
├─ 0 activated → fallback base SRTPlatform
├─ 1 activated → use it
└─ N activated → RuntimeError (must set SGLANG_PLATFORM)
```
### Plugin Loading Flow
`load_plugins()` discovers and executes general plugins, then applies all registered hooks. It is called at four points:
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
<colgroup>
<col style={{width: "33.33%"}} />
<col style={{width: "33.33%"}} />
<col style={{width: "33.33%"}} />
</colgroup>
<thead>
<tr>
<th>Call Site</th>
<th>Process</th>
<th>Timing</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>cli/serve.py</code> serve()</td>
<td>Main</td>
<td>Before <code>prepare_server_args()</code></td>
</tr>
<tr>
<td><code>launch_server.py</code> <code>__main__</code></td>
<td>Main</td>
<td>Before <code>prepare_server_args()</code></td>
</tr>
<tr>
<td><code>engine.py</code> <code>_launch_subprocesses()</code></td>
<td>Main</td>
<td>Before <code>server_args.check_server_args()</code></td>
</tr>
<tr>
<td><code>scheduler.py</code> <code>run_scheduler_process()</code></td>
<td>Subprocess</td>
<td>Before <code>Scheduler()</code> construction</td>
</tr>
</tbody>
</table>
> **Note**: `load_plugins()` is idempotent (guarded by `_plugins_loaded` flag). In spawn'd subprocesses the flag resets, so plugins are correctly re-loaded.
```
load_plugins()
├── _get_excluded_dists() → compute dists to skip (via SGLANG_PLATFORM)
├── load_plugins_by_group("sglang.srt.plugins", → discover entry_points, filter by SGLANG_PLUGINS
│ excluded_dists=...) skip plugins from unselected platform packages
├── for each plugin: → set _current_plugin_source context var
│ func() side effects (register hooks with source tracking)
└── HookRegistry.apply_hooks() → monkey-patch targets
```
---
## Plugin Type 1: Hardware Platform Plugin
### Description
A hardware platform plugin registers an `SRTPlatform` subclass that tells SGLang how to interact with a specific hardware backend.
### Quick Start
**1. Create a minimal package:**
```
my_platform_plugin/
├── pyproject.toml
└── my_platform_plugin/
├── __init__.py # activate() function
├── device.py # MyDeviceMixin
└── platform.py # MySRTPlatform
```
**2. `pyproject.toml`:**
```toml
[build-system]
requires = ["setuptools"]
build-backend = "setuptools.build_meta"
[project]
name = "my-platform-plugin"
version = "0.1.0"
[project.entry-points."sglang.srt.platforms"]
my_device = "my_platform_plugin:activate"
```
**3. `__init__.py`** — activation function:
```python
def activate():
"""Return fully-qualified class name to activate, or None to skip."""
if _my_device_is_available():
return "my_platform_plugin.platform.MySRTPlatform"
return None
```
**4. `device.py`** — device mixin:
```python
from sglang.srt.platforms.device_mixin import DeviceMixin, PlatformEnum
class MyDeviceMixin(DeviceMixin):
_enum = PlatformEnum.OOT
device_name = "my_device"
device_type = "my_device" # torch device type
def set_device(self, device) -> None: ...
def get_device_name(self, device_id=0) -> str: ...
def get_device_total_memory(self, device_id=0) -> int: ...
def get_current_memory_usage(self, device=None) -> float: ...
def get_device_capability(self, device_id=0): ...
def get_torch_distributed_backend_str(self) -> str: ...
```
**5. `platform.py`** — SRT platform:
```python
from sglang.srt.platforms.interface import SRTPlatform
from my_platform_plugin.device import MyDeviceMixin
class MySRTPlatform(SRTPlatform, MyDeviceMixin):
def get_default_attention_backend(self) -> str: ...
def support_cuda_graph(self) -> bool: ...
# ... override other methods as needed
```
**6. Install and verify:**
```bash
pip install -e my_platform_plugin/
python -c "from sglang.srt.platforms import current_platform; print(current_platform)"
```
### Platform Interface Reference
#### Identity Queries (from DeviceMixin)
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
<colgroup>
<col style={{width: "33.33%"}} />
<col style={{width: "33.33%"}} />
<col style={{width: "33.33%"}} />
</colgroup>
<thead>
<tr>
<th>Method</th>
<th>Default</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>is_cuda()</code></td>
<td>Based on <code>_enum</code></td>
<td>Whether this is an NVIDIA CUDA platform</td>
</tr>
<tr>
<td><code>is_rocm()</code></td>
<td>Based on <code>_enum</code></td>
<td>Whether this is an AMD ROCm platform</td>
</tr>
<tr>
<td><code>is_npu()</code></td>
<td>Based on <code>_enum</code></td>
<td>Whether this is a Huawei NPU platform</td>
</tr>
<tr>
<td><code>is_cpu()</code></td>
<td>Based on <code>_enum</code></td>
<td>Whether this is a CPU-only platform</td>
</tr>
<tr>
<td><code>is_xpu()</code></td>
<td>Based on <code>_enum</code></td>
<td>Whether this is an Intel XPU platform</td>
</tr>
<tr>
<td><code>is_musa()</code></td>
<td>Based on <code>_enum</code></td>
<td>Whether this is a Moore Threads MUSA platform</td>
</tr>
<tr>
<td><code>is_cuda_alike()</code></td>
<td>CUDA+ROCM+MUSA</td>
<td>True if the hardware supports CUDA-like APIs</td>
</tr>
<tr>
<td><code>is_out_of_tree()</code></td>
<td><code>True</code> for OOT</td>
<td>Automatically detected based on <code>_enum = PlatformEnum.OOT</code></td>
</tr>
</tbody>
</table>
#### Device Operations (from DeviceMixin)
> Methods annotated **[Active]** are called by SGLang core through `current_platform` — OOT implementations take effect immediately.
> Methods annotated **[Planned]** are reserved interfaces — SGLang core still uses hardcoded calls (e.g. `torch.cuda.empty_cache()`). OOT implementations will NOT take effect until the core is migrated in a future PR.
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
<colgroup>
<col style={{width: "25%"}} />
<col style={{width: "25%"}} />
<col style={{width: "25%"}} />
<col style={{width: "25%"}} />
</colgroup>
<thead>
<tr>
<th>Method</th>
<th>Default</th>
<th>Status</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>get_device(local_rank)</code></td>
<td><code>raise NotImplementedError</code></td>
<td>Planned</td>
<td>Return <code>torch.device</code> for a given local rank</td>
</tr>
<tr>
<td><code>set_device(device)</code></td>
<td><code>raise NotImplementedError</code></td>
<td>Planned</td>
<td>Set the current device</td>
</tr>
<tr>
<td><code>get_device_name(device_id)</code></td>
<td><code>raise NotImplementedError</code></td>
<td>Planned</td>
<td>Get human-readable device name</td>
</tr>
<tr>
<td><code>get_device_uuid(device_id)</code></td>
<td><code>raise NotImplementedError</code></td>
<td>Planned</td>
<td>Get unique device identifier</td>
</tr>
<tr>
<td><code>get_device_capability(device_id)</code></td>
<td><code>raise NotImplementedError</code></td>
<td>Planned</td>
<td>Get <code>DeviceCapability(major, minor)</code>. None if N/A</td>
</tr>
<tr>
<td><code>empty_cache()</code></td>
<td><code>pass</code></td>
<td>Planned</td>
<td>Release cached device memory</td>
</tr>
<tr>
<td><code>synchronize()</code></td>
<td><code>pass</code></td>
<td>Planned</td>
<td>Synchronize device operations</td>
</tr>
<tr>
<td><code>get_device_total_memory(device_id)</code></td>
<td><code>raise NotImplementedError</code></td>
<td><strong>Active</strong></td>
<td>Get total device memory in bytes</td>
</tr>
<tr>
<td><code>get_available_memory(device_id)</code></td>
<td><code>raise NotImplementedError</code></td>
<td>Planned</td>
<td>Return <code>(free_bytes, total_bytes)</code></td>
</tr>
<tr>
<td><code>get_current_memory_usage(device)</code></td>
<td><code>raise NotImplementedError</code></td>
<td><strong>Active</strong></td>
<td>Get current peak memory usage in bytes</td>
</tr>
<tr>
<td><code>get_torch_distributed_backend_str()</code></td>
<td><code>raise NotImplementedError</code></td>
<td>Planned</td>
<td>Distributed backend string (e.g. "nccl", "hccl")</td>
</tr>
<tr>
<td><code>get_communicator_class()</code></td>
<td><code>None</code></td>
<td>Planned</td>
<td>Platform-specific communicator class</td>
</tr>
<tr>
<td><code>inference_mode()</code></td>
<td><code>torch.inference_mode(True)</code></td>
<td>Planned</td>
<td>Return inference mode context manager</td>
</tr>
<tr>
<td><code>seed_everything(seed)</code></td>
<td>Set random/np/torch seeds</td>
<td>Planned</td>
<td>Set random seeds for reproducibility</td>
</tr>
<tr>
<td><code>verify_quantization(quant)</code></td>
<td><code>pass</code></td>
<td>Planned</td>
<td>Validate quantization method support</td>
</tr>
<tr>
<td><code>get_cpu_architecture()</code></td>
<td>Auto-detect x86/arm</td>
<td>Planned</td>
<td>Detect CPU architecture (<code>CpuArchEnum</code>)</td>
</tr>
</tbody>
</table>
#### Types (from DeviceMixin)
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
<colgroup>
<col style={{width: "50%"}} />
<col style={{width: "50%"}} />
</colgroup>
<thead>
<tr>
<th>Type</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>PlatformEnum</code></td>
<td>Enumeration of platform types: CUDA, ROCM, CPU, XPU, MUSA, NPU, TPU, MPS, OOT, UNSPECIFIED</td>
</tr>
<tr>
<td><code>CpuArchEnum</code></td>
<td>CPU architecture: X86, ARM, UNSPECIFIED</td>
</tr>
<tr>
<td><code>DeviceCapability</code></td>
<td><code>NamedTuple(major, minor)</code> with comparison support. Methods: <code>as_version_str()</code>, <code>to_int()</code></td>
</tr>
</tbody>
</table>
#### Capability Flags (from SRTPlatform)
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
<colgroup>
<col style={{width: "33.33%"}} />
<col style={{width: "33.33%"}} />
<col style={{width: "33.33%"}} />
</colgroup>
<thead>
<tr>
<th>Method</th>
<th>Default</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>support_cuda_graph()</code></td>
<td><code>False</code></td>
<td>Whether device graph capture is supported (plain CUDA graph)</td>
</tr>
<tr>
<td><code>support_piecewise_cuda_graph()</code></td>
<td><code>False</code></td>
<td>Whether piecewise CUDA graph (torch.compile backend) is supported</td>
</tr>
<tr>
<td><code>supports_fp8()</code></td>
<td><code>False</code></td>
<td>Whether FP8 quantization is supported</td>
</tr>
<tr>
<td><code>is_pin_memory_available()</code></td>
<td><code>True</code></td>
<td>Whether pinned memory is available</td>
</tr>
</tbody>
</table>
#### Subsystem Factory Methods (from SRTPlatform)
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
<colgroup>
<col style={{width: "33.33%"}} />
<col style={{width: "33.33%"}} />
<col style={{width: "33.33%"}} />
</colgroup>
<thead>
<tr>
<th>Method</th>
<th>Default</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>get_default_attention_backend()</code></td>
<td><code>raise NotImplementedError</code></td>
<td>Default attention backend name</td>
</tr>
<tr>
<td><code>get_graph_runner_cls()</code></td>
<td><code>raise NotImplementedError</code></td>
<td>Graph Runner class</td>
</tr>
<tr>
<td><code>get_mha_kv_pool_cls()</code></td>
<td><code>raise NotImplementedError</code></td>
<td>MHA KV cache pool class</td>
</tr>
<tr>
<td><code>get_mla_kv_pool_cls()</code></td>
<td><code>raise NotImplementedError</code></td>
<td>MLA KV cache pool class</td>
</tr>
<tr>
<td><code>get_nsa_kv_pool_cls()</code></td>
<td><code>raise NotImplementedError</code></td>
<td>NSA KV cache pool class (DeepSeek V3.2)</td>
</tr>
<tr>
<td><code>get_paged_allocator_cls()</code></td>
<td><code>raise NotImplementedError</code></td>
<td>Paged allocator class</td>
</tr>
<tr>
<td><code>get_piecewise_backend_cls()</code></td>
<td><code>raise NotImplementedError</code></td>
<td>Piecewise compilation backend class</td>
</tr>
<tr>
<td><code>get_compile_backend(mode)</code></td>
<td><code>"inductor"</code></td>
<td>Compilation backend string</td>
</tr>
<tr>
<td><code>get_dispatch_key_name()</code></td>
<td><code>"native"</code></td>
<td>MultiPlatformOp dispatch key name</td>
</tr>
</tbody>
</table>
#### Lifecycle Hooks (from SRTPlatform)
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
<colgroup>
<col style={{width: "33.33%"}} />
<col style={{width: "33.33%"}} />
<col style={{width: "33.33%"}} />
</colgroup>
<thead>
<tr>
<th>Method</th>
<th>Invocation Timing</th>
<th>Purpose</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>apply_server_args_defaults(server_args)</code></td>
<td>After ServerArgs parsing, in <code>__post_init__</code></td>
<td>Set platform-specific defaults</td>
</tr>
<tr>
<td><code>init_backend()</code></td>
<td>In each worker, before model construction</td>
<td>One-time backend initialization</td>
</tr>
</tbody>
</table>
### Environment Variables
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
<colgroup>
<col style={{width: "50%"}} />
<col style={{width: "50%"}} />
</colgroup>
<thead>
<tr>
<th>Variable</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>SGLANG_PLATFORM</code></td>
<td>Select the platform plugin by entry_point name (e.g. <code>kunlun</code>, <code>demo_cuda</code>). When set, <strong>only</strong> the named plugin's <code>activate()</code> is called (front-loading filter) — other plugins are not touched. Additionally, general plugins (<code>sglang.srt.plugins</code>) from unselected platform packages are automatically skipped to avoid importing their dependencies. Required when multiple plugins would activate. Errors if the name is not found or if the plugin's hardware is unavailable.</td>
</tr>
<tr>
<td><code>SGLANG_PLUGINS</code></td>
<td>Comma-separated whitelist of general plugin names to load (group: <code>sglang.srt.plugins</code>). If unset, all discovered general plugins are loaded.</td>
</tr>
</tbody>
</table>
---
## Plugin Type 2: General Plugin
### Description
General function plugins inject behavior into sglang **without requiring a custom platform**. Use cases include:
- **Observability**: Add logging, metrics, and tracing to any function
- **Behavior modification**: Modify function arguments or return values
- **Performance profiling**: Add timing to critical functions
- **A/B testing**: Replace implementations at runtime
### Quick Start
**1. Create a minimal package:**
```
my_general_plugin/
├── pyproject.toml
└── my_general_plugin/
└── __init__.py # register() function
```
**2. `pyproject.toml`:**
```toml
[build-system]
requires = ["setuptools"]
build-backend = "setuptools.build_meta"
[project]
name = "my-general-plugin"
version = "0.1.0"
[project.entry-points."sglang.srt.plugins"]
my_plugin = "my_general_plugin:register"
```
**3. `__init__.py`** — register hooks:
```python
from sglang.srt.plugins.hook_registry import HookRegistry, HookType
def register():
"""Entry point called by load_plugins()."""
HookRegistry.register(
"sglang.srt.managers.scheduler.Scheduler.__init__",
my_hook,
HookType.AROUND,
)
def my_hook(original_fn, self, *args, **kwargs):
result = original_fn(self, *args, **kwargs)
print(f"Scheduler initialized! gpu_id={self.gpu_id}")
return result
```
**4. Install and run:**
```bash
pip install -e my_general_plugin/
sglang serve --model-path <model> [options]
# Look for "Scheduler initialized!" in logs
```
### Hook Types
`HookRegistry` supports four hook types:
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
<colgroup>
<col style={{width: "33.33%"}} />
<col style={{width: "33.33%"}} />
<col style={{width: "33.33%"}} />
</colgroup>
<thead>
<tr>
<th>Hook Type</th>
<th>Signature</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td><strong>BEFORE</strong></td>
<td><code>fn(*args, **kwargs) -&gt; (args, kwargs) \| None</code></td>
<td>Runs before the original. Return <code>None</code> to keep args unchanged, or <code>(args, kwargs)</code> to modify.</td>
</tr>
<tr>
<td><strong>AFTER</strong></td>
<td><code>fn(result, *args, **kwargs) -&gt; new_result \| None</code></td>
<td>Runs after the original. Return <code>None</code> to keep result, or a new value to replace.</td>
</tr>
<tr>
<td><strong>AROUND</strong></td>
<td><code>fn(original_fn, *args, **kwargs) -&gt; result</code></td>
<td>Wraps the original. You must call <code>original_fn</code> yourself. Full control over execution.</td>
</tr>
<tr>
<td><strong>REPLACE</strong></td>
<td><code>fn(*args, **kwargs) -&gt; result</code> or <code>class</code></td>
<td>Replace the original function or class entirely. For class targets, pass a replacement class directly — it is substituted via <code>setattr</code> preserving <code>isinstance()</code>/<code>issubclass()</code> semantics.</td>
</tr>
</tbody>
</table>
> **Note**: Only `REPLACE` accepts a class as the hook. Passing a class to `BEFORE`/`AFTER`/`AROUND` raises `TypeError` at registration time.
### Registration API
Hooks can be registered using the **imperative API** or the **decorator API**:
```python
# --- Imperative API ---
from sglang.srt.plugins.hook_registry import HookRegistry, HookType
def my_timer(original_fn, *args, **kwargs):
start = time.perf_counter()
result = original_fn(*args, **kwargs)
print(f"Elapsed: {time.perf_counter() - start:.3f}s")
return result
HookRegistry.register(
"sglang.srt.managers.scheduler.Scheduler.get_next_batch_to_run",
my_timer,
HookType.AROUND,
)
# --- Decorator API ---
from sglang.srt.plugins.hook_registry import plugin_hook, HookType
@plugin_hook(
"sglang.srt.managers.scheduler.Scheduler.get_next_batch_to_run",
type=HookType.AROUND,
)
def my_timer(original_fn, *args, **kwargs):
start = time.perf_counter()
result = original_fn(*args, **kwargs)
print(f"Elapsed: {time.perf_counter() - start:.3f}s")
return result
# --- Class replacement (REPLACE) ---
from sglang.srt.plugins.hook_registry import plugin_hook, HookType
from sglang.srt.managers.scheduler import Scheduler
@plugin_hook(
"sglang.srt.managers.scheduler.Scheduler",
type=HookType.REPLACE,
)
class MyScheduler(Scheduler):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
print("Enhanced scheduler initialized!")
```
### Hook Target Resolution
Target paths use fully-qualified dotted notation. Both formats are supported:
- **Dotted**: `sglang.srt.managers.scheduler.Scheduler.__init__`
- **Entry-points style**: `sglang.srt.managers.scheduler:Scheduler.__init__` (colon treated as dot)
### Common Hook Targets
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
<colgroup>
<col style={{width: "50%"}} />
<col style={{width: "50%"}} />
</colgroup>
<thead>
<tr>
<th>Target</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>sglang.srt.server_args.ServerArgs.add_cli_args</code></td>
<td>Add custom CLI arguments</td>
</tr>
<tr>
<td><code>sglang.srt.server_args.ServerArgs.__post_init__</code></td>
<td>Modify ServerArgs after parsing</td>
</tr>
<tr>
<td><code>sglang.srt.server_args.ServerArgs.check_server_args</code></td>
<td>Add/relax validation</td>
</tr>
<tr>
<td><code>sglang.srt.managers.scheduler.Scheduler.__init__</code></td>
<td>Custom scheduler state</td>
</tr>
<tr>
<td><code>sglang.srt.managers.scheduler.Scheduler.get_next_batch_to_run</code></td>
<td>Custom scheduling policy</td>
</tr>
<tr>
<td><code>sglang.srt.managers.scheduler.Scheduler.run_batch</code></td>
<td>Profiling / inspection</td>
</tr>
<tr>
<td><code>sglang.srt.managers.scheduler.Scheduler.process_batch_result</code></td>
<td>Custom metrics</td>
</tr>
<tr>
<td><code>sglang.srt.managers.tp_worker.TpModelWorker.__init__</code></td>
<td>Custom worker state</td>
</tr>
<tr>
<td><code>sglang.srt.managers.tp_worker.TpModelWorker.forward_batch_generation</code></td>
<td>Forward pass wrapping</td>
</tr>
</tbody>
</table>
---
## File Reference
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
<colgroup>
<col style={{width: "50%"}} />
<col style={{width: "50%"}} />
</colgroup>
<thead>
<tr>
<th>File</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>sglang/srt/platforms/device_mixin.py</code></td>
<td><code>PlatformEnum</code> + <code>DeviceMixin</code> base class</td>
</tr>
<tr>
<td><code>sglang/srt/platforms/interface.py</code></td>
<td><code>SRTPlatform</code> base class (extends DeviceMixin)</td>
</tr>
<tr>
<td><code>sglang/srt/platforms/__init__.py</code></td>
<td><code>current_platform</code> lazy singleton + discovery logic</td>
</tr>
<tr>
<td><code>sglang/srt/plugins/__init__.py</code></td>
<td><code>load_plugins()</code> + <code>load_plugins_by_group()</code></td>
</tr>
<tr>
<td><code>sglang/srt/plugins/hook_registry.py</code></td>
<td><code>HookRegistry</code>, <code>HookType</code>, <code>plugin_hook</code> decorator</td>
</tr>
</tbody>
</table>
docs_new/docs/hardware-platforms/tpu.mdx
+398 -441
View File
File diff suppressed because it is too large Load Diff
docs_new/docs/hardware-platforms/xpu.mdx
+31 -62
View File
@@ -2,12 +2,10 @@
title: XPU
sidebarTitle: Intel GPUs (XPU)
---
The document addresses how to set up the [SGLang](https://github.com/sgl-project/sglang) environment and run LLM inference on Intel GPU, [see more context about Intel GPU support within PyTorch ecosystem](https://docs.pytorch.org/docs/stable/notes/get_start_xpu.html).
Specifically, SGLang is optimized for:
- [Intel® Arc™ Pro B-Series Graphics](https://www.intel.com/content/www/us/en/ark/products/series/242616/intel-arc-pro-b-series-graphics.html)
- [Intel® Arc™ B-Series Graphics](https://www.intel.com/content/www/us/en/ark/products/series/240391/intel-arc-b-series-graphics.html).
Specifically, SGLang is optimized for [Intel® Arc™ Pro B-Series Graphics](https://www.intel.com/content/www/us/en/ark/products/series/242616/intel-arc-pro-b-series-graphics.html) and [
Intel® Arc™ B-Series Graphics](https://www.intel.com/content/www/us/en/ark/products/series/240391/intel-arc-b-series-graphics.html).
## Optimized Model List
@@ -40,72 +38,45 @@ A list of LLMs have been optimized on Intel GPU, and more are on the way:
</tbody>
</table>
<Note>The model identifiers listed in the table above have been verified on [Intel® Arc™ B580 Graphics](https://www.intel.com/content/www/us/en/products/sku/241598/intel-arc-b580-graphics/specifications.html).</Note>
**Note:** The model identifiers listed in the table above
have been verified on [Intel® Arc™ B580 Graphics](https://www.intel.com/content/www/us/en/products/sku/241598/intel-arc-b580-graphics/specifications.html).
## Installation
<Tabs>
### Install From Source
<Tab title="Source">
Currently SGLang XPU only supports installation from source. Please refer to ["Getting Started on Intel GPU"](https://docs.pytorch.org/docs/stable/notes/get_start_xpu.html) to install XPU dependency.
Currently SGLang XPU only supports installation from source. Please refer to [“Getting Started on Intel GPU”](https://docs.pytorch.org/docs/stable/notes/get_start_xpu.html) to install XPU dependency.
```bash Command
# Create and activate a conda environment
conda create -n sgl-xpu python=3.12 -y
conda activate sgl-xpu
1. **Creation & Activation**
# Set PyTorch XPU as primary pip install channel to avoid installing the larger CUDA-enabled version and prevent potential runtime issues.
pip3 install torch==2.11.0+xpu torchao torchvision torchaudio==2.11.0+xpu --index-url https://download.pytorch.org/whl/xpu
pip3 install xgrammar --no-deps # xgrammar will introduce CUDA-enabled triton which might conflict with XPU
Create and activate a conda environment.
# Clone the SGLang code
git clone https://github.com/sgl-project/sglang.git
cd sglang
git checkout <YOUR-DESIRED-VERSION>
```bash
conda create -n sgl-xpu python=3.12 -y
conda activate sgl-xpu
```
# Use dedicated toml file
cd python
cp pyproject_xpu.toml pyproject.toml
# Install SGLang dependent libs, and build SGLang main package
pip install --upgrade pip setuptools
pip install -v . --extra-index-url https://download.pytorch.org/whl/xpu
```
2. **Install PyTorch and Dependencies**
### Install Using Docker
Set PyTorch XPU as primary pip install channel to avoid installing the larger CUDA-enabled version and prevent potential runtime issues.
The docker for XPU is under active development. Please stay tuned.
```bash
pip3 install torch==2.9.0+xpu torchao torchvision torchaudio pytorch-triton-xpu==3.5.0 --index-url https://download.pytorch.org/whl/xpu
pip3 install xgrammar --no-deps # xgrammar will introduce CUDA-enabled triton which might conflict with XPU
```
3. **Cloning**
Clone the SGLang code
```bash
git clone https://github.com/sgl-project/sglang.git
cd sglang
git checkout <YOUR-DESIRED-VERSION>
```
4. **Configure Build File**
Use dedicated toml file
```bash
cd python
cp pyproject_xpu.toml pyproject.toml
```
5. **Build and Install**
Install SGLang dependent libs, and build SGLang main package
```bash
pip install --upgrade pip setuptools
pip install -v .
```
</Tab>
<Tab title="Docker">
<Info>The docker for XPU is under active development. Please stay tuned.</Info>
</Tab>
</Tabs>
## Launch of the Serving Engine
Example command to launch SGLang serving:
<CodeGroup>
```bash
python -m sglang.launch_server \
--model <MODEL_ID_OR_PATH> \
@@ -117,13 +88,12 @@ python -m sglang.launch_server \
--attention-backend intel_xpu \ # using intel optimized XPU attention backend
--page-size \ # intel_xpu attention backend supports [32, 64, 128]
```
</CodeGroup>
## Benchmarking with Requests
You can benchmark the performance via the `bench_serving` script. Run the command in another terminal.
You can benchmark the performance via the `bench_serving` script.
Run the command in another terminal.
<CodeGroup>
```bash
python -m sglang.bench_serving \
--dataset-name random \
@@ -133,14 +103,13 @@ python -m sglang.bench_serving \
--request-rate inf \
--random-range-ratio 1.0
```
</CodeGroup>
The detail explanations of the parameters can be looked up by the command:
<CodeGroup>
```bash
python -m sglang.bench_serving -h
```
</CodeGroup>
Additionally, the requests can be formed with [OpenAI Completions API](../basic_usage/openai_api_completions) and sent via the command line (e.g. using `curl`) or via your own script.
Additionally, the requests can be formed with
[OpenAI Completions API](../basic_usage/openai_api_completions)
and sent via the command line (e.g. using `curl`) or via your own script.