Add new Mintlify documentation site (docs_new/) (#23001)
Co-authored-by: AdityaVKochar <adityavardhankochar@gmail.com> Co-authored-by: mintlify[bot] <109931778+mintlify[bot]@users.noreply.github.com> Co-authored-by: adhyan-jain <adhyanjain2006@gmail.com> Co-authored-by: Adhyan Jain <71976554+adhyan-jain@users.noreply.github.com> Co-authored-by: Maitri-shah29 <maitrirajivshah@gmail.com> Co-authored-by: Adarsh Shirawalmath <114558126+adarshxs@users.noreply.github.com> Co-authored-by: Maitri Shah <shah29maitri@gmail.com> Co-authored-by: Aditya Vardhan Kochar <80113212+AdityaVKochar@users.noreply.github.com> Co-authored-by: Rishit Shivam <164783543+pokymono@users.noreply.github.com> Co-authored-by: Rishitshivam <164783543+Rishitshivam@users.noreply.github.com> Co-authored-by: IshhanKheria <ishhankheria06@gmail.com> Co-authored-by: Ishita Joshi <ishitata.joshi@gmail.com> Co-authored-by: Richard Chen <104477092+Richardczl98@users.noreply.github.com> Co-authored-by: longGGGGGG <553746008@qq.com> Co-authored-by: Richard <richardchen@radixark.ai> Co-authored-by: Nakul Sinha <nakul.new4socials@gmail.com> Co-authored-by: Divyam Agrawal <ludicrouslytrue@gmail.com> Co-authored-by: Richardczl98 <Zhenlinc@stanford.edu> Co-authored-by: Krishang Zinzuwadia <krishangzinzuwadia@gmail.com> Co-authored-by: nimeshas <nimesha.s106@gmail.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Jignas Paturu <86356085+JignasP@users.noreply.github.com> Co-authored-by: zijiexia <37504505+zijiexia@users.noreply.github.com>
This commit is contained in:
Mingyi
co-authored by
AdityaVKochar
mintlify[bot]
adhyan-jain
Adhyan Jain
Maitri-shah29
Adarsh Shirawalmath
Maitri Shah
Aditya Vardhan Kochar
Rishit Shivam
Rishitshivam
IshhanKheria
Ishita Joshi
Richard Chen
longGGGGGG
Richard
Nakul Sinha
Divyam Agrawal
Richardczl98
Krishang Zinzuwadia
nimeshas
Claude Opus 4.6
github-actions[bot]
Jignas Paturu
zijiexia
AdityaVKochar
mintlify[bot]
adhyan-jain
Adhyan Jain
Maitri-shah29
Adarsh Shirawalmath
Maitri Shah
Aditya Vardhan Kochar
Rishit Shivam
Rishitshivam
IshhanKheria
Ishita Joshi
Richard Chen
longGGGGGG
Richard
Nakul Sinha
Divyam Agrawal
Richardczl98
Krishang Zinzuwadia
nimeshas
Claude Opus 4.6
github-actions[bot]
Jignas Paturu
zijiexia
parent
575fdc2c4c
commit
a3291b5654
@@ -0,0 +1,194 @@
|
||||
---
|
||||
title: "AMD GPUs"
|
||||
---
|
||||
This document describes how run SGLang on AMD GPUs. If you encounter issues or have questions, please [open an issue](https://github.com/sgl-project/sglang/issues).
|
||||
|
||||
## System Configuration
|
||||
|
||||
When using AMD GPUs (such as MI300X), certain system-level optimizations help ensure stable performance. Here we take MI300X as an example. AMD provides official documentation for MI300X optimization and system tuning:
|
||||
|
||||
* [AMD MI300X Tuning Guides](https://rocm.docs.amd.com/en/latest/how-to/tuning-guides/mi300x/index.html)
|
||||
* [LLM inference performance validation on AMD Instinct MI300X](https://rocm.docs.amd.com/en/latest/how-to/rocm-for-ai/inference/vllm-benchmark.html)
|
||||
* [AMD Instinct MI300X System Optimization](https://rocm.docs.amd.com/en/latest/how-to/system-optimization/mi300x.html)
|
||||
* [AMD Instinct MI300X Workload Optimization](https://rocm.docs.amd.com/en/latest/how-to/rocm-for-ai/inference-optimization/workload.html)
|
||||
* [Supercharge DeepSeek-R1 Inference on AMD Instinct MI300X](https://rocm.blogs.amd.com/artificial-intelligence/DeepSeekR1-Part2/README.html)
|
||||
|
||||
<Note>
|
||||
We strongly recommend reading these docs and guides entirely to fully utilize your system.
|
||||
</Note>
|
||||
|
||||
Below are a few key settings to confirm or enable for SGLang:
|
||||
|
||||
### Update GRUB Settings
|
||||
|
||||
In `/etc/default/grub`, append the following to `GRUB_CMDLINE_LINUX`:
|
||||
|
||||
<CodeGroup>
|
||||
```text GRUB Configuration
|
||||
pci=realloc=off iommu=pt
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
Afterward, run `sudo update-grub` (or your distro's equivalent) and reboot.
|
||||
|
||||
### Disable NUMA Auto-Balancing
|
||||
|
||||
<CodeGroup>
|
||||
```bash Disable NUMA
|
||||
sudo sh -c 'echo 0 > /proc/sys/kernel/numa_balancing'
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
You can automate or verify this change using [this helpful script](https://github.com/ROCm/triton/blob/rocm_env/scripts/amd/env_check.sh).
|
||||
|
||||
Again, please go through the entire documentation to confirm your system is using the recommended configuration.
|
||||
|
||||
## Install SGLang
|
||||
|
||||
<Tabs>
|
||||
<Tab title="Docker (Recommended)">
|
||||
The docker images are available on Docker Hub at [lmsysorg/sglang](https://hub.docker.com/r/lmsysorg/sglang/tags), built from [rocm.Dockerfile](https://github.com/sgl-project/sglang/tree/main/docker).
|
||||
|
||||
1. **Build the docker image**
|
||||
If you use pre-built images, you can skip this step and replace `sglang_image` with the pre-built image names in the steps below.
|
||||
|
||||
<CodeGroup>
|
||||
```bash Build Image
|
||||
docker build -t sglang_image -f rocm.Dockerfile .
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
2. **Create a convenient alias**
|
||||
<CodeGroup>
|
||||
```bash Create Alias
|
||||
alias drun='docker run -it --rm --network=host --privileged --device=/dev/kfd --device=/dev/dri \
|
||||
--ipc=host --shm-size 16G --group-add video --cap-add=SYS_PTRACE \
|
||||
--security-opt seccomp=unconfined \
|
||||
-v $HOME/dockerx:/dockerx \
|
||||
-v /data:/data'
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
If you are using RDMA, please note that:
|
||||
|
||||
* `--network host` and `--privileged` are required by RDMA. If you don't need RDMA, you can remove them.
|
||||
* You may need to set `NCCL_IB_GID_INDEX` if you are using RoCE, for example: `export NCCL_IB_GID_INDEX=3`.
|
||||
|
||||
3. **Launch the server**
|
||||
<Note>
|
||||
Replace `<secret>` below with your [huggingface hub token](https://huggingface.co/docs/hub/en/security-tokens).
|
||||
</Note>
|
||||
|
||||
<CodeGroup>
|
||||
```bash Launch Server
|
||||
drun -p 30000:30000 \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
--env "HF_TOKEN=<secret>" \
|
||||
sglang_image \
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path NousResearch/Meta-Llama-3.1-8B \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
4. **Verify the installation**
|
||||
You can run a benchmark in another terminal or refer to [other docs](../basic_usage/openai_api_completions) to send requests to the engine.
|
||||
|
||||
<CodeGroup>
|
||||
```bash Run Benchmark
|
||||
drun sglang_image \
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--dataset-name random \
|
||||
--num-prompts 4000 \
|
||||
--random-input 128 \
|
||||
--random-output 128
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
With your AMD system properly configured and SGLang installed, you can now fully leverage AMD hardware to power SGLang's machine learning capabilities.
|
||||
</Tab>
|
||||
|
||||
<Tab title="From Source">
|
||||
1. **Clone the repository**
|
||||
Clone the SGLang repository.
|
||||
|
||||
<CodeGroup>
|
||||
```bash
|
||||
git clone https://github.com/sgl-project/sglang.git
|
||||
cd sglang
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
2. **Compile sgl-kernel**
|
||||
Upgrade pip and compile the sgl-kernel for ROCm support.
|
||||
|
||||
<CodeGroup>
|
||||
```bash
|
||||
pip install --upgrade pip
|
||||
cd sgl-kernel
|
||||
python setup_rocm.py install
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
3. **Install sglang package**
|
||||
Install the SGLang Python package with HIP and diffusion support.
|
||||
|
||||
<CodeGroup>
|
||||
```bash
|
||||
cd ..
|
||||
rm -rf python/pyproject.toml && mv python/pyproject_other.toml python/pyproject.toml
|
||||
pip install -e "python[all_hip]"
|
||||
```
|
||||
</CodeGroup>
|
||||
</Tab>
|
||||
</Tabs>
|
||||
|
||||
## Examples
|
||||
|
||||
### Running DeepSeek-V3
|
||||
|
||||
The only difference when running DeepSeek-V3 is in how you start the server.
|
||||
|
||||
<CodeGroup>
|
||||
```bash DeepSeek-V3
|
||||
drun -p 30000:30000 \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
--ipc=host \
|
||||
--env "HF_TOKEN=<secret>" \
|
||||
sglang_image \
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path deepseek-ai/DeepSeek-V3 \
|
||||
--tp 8 \
|
||||
--trust-remote-code \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
[Running DeepSeek-R1 on a single NDv5 MI300X VM](https://techcommunity.microsoft.com/blog/azurehighperformancecomputingblog/running-deepseek-r1-on-a-single-ndv5-mi300x-vm/4372726) could also be a good reference.
|
||||
|
||||
### Running Llama3.1
|
||||
|
||||
Running Llama3.1 is nearly identical to running DeepSeek-V3. The only difference is in the model specified when starting the server.
|
||||
|
||||
<CodeGroup>
|
||||
```bash Llama3.1
|
||||
drun -p 30000:30000 \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
--ipc=host \
|
||||
--env "HF_TOKEN=<secret>" \
|
||||
sglang_image \
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path meta-llama/Meta-Llama-3.1-8B-Instruct \
|
||||
--tp 8 \
|
||||
--trust-remote-code \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
### Warmup Step
|
||||
|
||||
When the server displays `The server is fired up and ready to roll!`, it means the startup is successful.
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,309 @@
|
||||
## Running DeepSeek-V3
|
||||
|
||||
### Running DeepSeek in PD mixed mode on 1 x Atlas 800I A3
|
||||
|
||||
W4A8 Model weights could be found [here](https://modelers.cn/models/Modelers_Park/DeepSeek-R1-0528-w4a8).
|
||||
|
||||
<CodeGroup>
|
||||
```shell Launch Server
|
||||
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
|
||||
export STREAMS_PER_DEVICE=32
|
||||
|
||||
#Deepep communication settings
|
||||
export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
|
||||
export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=32
|
||||
export HCCL_BUFFSIZE=1600
|
||||
|
||||
#spec overlap
|
||||
export SGLANG_ENABLE_SPEC_V2=1
|
||||
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
|
||||
|
||||
#npu acceleration operator
|
||||
export SGLANG_NPU_USE_MLAPO=1
|
||||
export SGLANG_USE_FIA_NZ=1
|
||||
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path ${MODEL_PATH} \
|
||||
--tp 16 \
|
||||
--trust-remote-code \
|
||||
--attention-backend ascend \
|
||||
--device npu \
|
||||
--quantization modelslim \
|
||||
--watchdog-timeout 9000 \
|
||||
--cuda-graph-bs 8 16 24 28 32 \
|
||||
--mem-fraction-static 0.68 \
|
||||
--max-running-requests 128 \
|
||||
--context-length 8188 \
|
||||
--disable-radix-cache \
|
||||
--chunked-prefill-size -1 \
|
||||
--max-prefill-tokens 16384 \
|
||||
--moe-a2a-backend deepep \
|
||||
--deepep-mode auto \
|
||||
--enable-dp-attention \
|
||||
--dp-size 4 \
|
||||
--enable-dp-lm-head \
|
||||
--speculative-algorithm NEXTN \
|
||||
--speculative-num-steps 3 \
|
||||
--speculative-eagle-topk 1 \
|
||||
--speculative-num-draft-tokens 4 \
|
||||
--dtype bfloat16
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
### Running DeepSeek with PD disaggregation mode on 2 x Atlas 800I A3
|
||||
|
||||
W4A8 Model weights could be found [here](https://modelers.cn/models/Modelers_Park/DeepSeek-R1-0528-w4a8).
|
||||
|
||||
<Tabs>
|
||||
<Tab title="Prefill">
|
||||
```shell Command
|
||||
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
|
||||
export STREAMS_PER_DEVICE=32
|
||||
|
||||
#memfabric config store
|
||||
export ASCEND_MF_STORE_URL="tcp://<PREFILL_HOST_IP>:<PORT>"
|
||||
|
||||
#Deepep communication settings
|
||||
export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
|
||||
export HCCL_BUFFSIZE=1536
|
||||
|
||||
#npu acceleration operator
|
||||
export SGLANG_NPU_USE_MLAPO=1
|
||||
export SGLANG_USE_FIA_NZ=1
|
||||
export TASK_QUEUE_ENABLE=2
|
||||
|
||||
python -m sglang.launch_server \
|
||||
--model-path ${MODEL_PATH} \
|
||||
--host $PREFILL_HOST_IP \
|
||||
--port 8000 \
|
||||
--disaggregation-mode prefill \
|
||||
--disaggregation-bootstrap-port 8996 \
|
||||
--disaggregation-transfer-backend ascend \
|
||||
--trust-remote-code \
|
||||
--nnodes 1 \
|
||||
--node-rank 0 \
|
||||
--tp-size 16 \
|
||||
--mem-fraction-static 0.6 \
|
||||
--attention-backend ascend \
|
||||
--device npu \
|
||||
--quantization modelslim \
|
||||
--load-balance-method round_robin \
|
||||
--max-running-requests 8 \
|
||||
--context-length 8192 \
|
||||
--disable-radix-cache \
|
||||
--chunked-prefill-size -1 \
|
||||
--max-prefill-tokens 28680 \
|
||||
--moe-a2a-backend deepep \
|
||||
--deepep-mode normal \
|
||||
--speculative-algorithm NEXTN \
|
||||
--speculative-num-steps 3 \
|
||||
--speculative-eagle-topk 1 \
|
||||
--speculative-num-draft-tokens 4 \
|
||||
--dp-size 2 \
|
||||
--enable-dp-attention \
|
||||
--disable-shared-experts-fusion \
|
||||
--dtype bfloat16
|
||||
```
|
||||
|
||||
</Tab>
|
||||
|
||||
<Tab title="Decode">
|
||||
```shell Command
|
||||
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
|
||||
export STREAMS_PER_DEVICE=32
|
||||
|
||||
#memfabric config store
|
||||
export ASCEND_MF_STORE_URL="tcp://<PREFILL_HOST_IP>:<PORT>"
|
||||
|
||||
#Deepep communication settings
|
||||
export HCCL_BUFFSIZE=720
|
||||
export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=88
|
||||
|
||||
#spec overlap
|
||||
export SGLANG_ENABLE_SPEC_V2=1
|
||||
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
|
||||
|
||||
#npu acceleration operator
|
||||
unset TASK_QUEUE_ENABLE
|
||||
export SGLANG_NPU_USE_MLAPO=1
|
||||
export SGLANG_USE_FIA_NZ=1
|
||||
export ENABLE_MOE_NZ=1
|
||||
|
||||
# suggest max-running-requests <= max-cuda-graph-bs * dp_size, Because when this value is exceeded, performance will significantly degrade.
|
||||
python -m sglang.launch_server \
|
||||
--model-path ${MODEL_PATH} \
|
||||
--disaggregation-mode decode \
|
||||
--host $DECODE_HOST_IP \
|
||||
--port 8001 \
|
||||
--trust-remote-code \
|
||||
--nnodes 1 \
|
||||
--node-rank 0 \
|
||||
--tp-size 16 \
|
||||
--dp-size 16 \
|
||||
--mem-fraction-static 0.8 \
|
||||
--max-running-requests 352 \
|
||||
--attention-backend ascend \
|
||||
--device npu \
|
||||
--quantization modelslim \
|
||||
--prefill-round-robin-balance \
|
||||
--moe-a2a-backend deepep \
|
||||
--enable-dp-attention \
|
||||
--deepep-mode low_latency \
|
||||
--enable-dp-lm-head \
|
||||
--cuda-graph-bs 8 10 12 14 16 18 20 22 \
|
||||
--disaggregation-transfer-backend ascend \
|
||||
--watchdog-timeout 9000 \
|
||||
--context-length 8192 \
|
||||
--speculative-algorithm NEXTN \
|
||||
--speculative-num-steps 3 \
|
||||
--speculative-eagle-topk 1 \
|
||||
--speculative-num-draft-tokens 4 \
|
||||
--disable-shared-experts-fusion \
|
||||
--dtype bfloat16 \
|
||||
--tokenizer-worker-num 4
|
||||
```
|
||||
|
||||
</Tab>
|
||||
|
||||
<Tab title="Router">
|
||||
```shell Command
|
||||
python -m sglang_router.launch_router \
|
||||
--pd-disaggregation \
|
||||
--policy cache_aware \
|
||||
--prefill http://<PREFILL_HOST_IP>:8000 8996 \
|
||||
--decode http://<DECODE_HOST_IP>:8001 \
|
||||
--host 127.0.0.1 \
|
||||
--port 6688
|
||||
```
|
||||
</Tab>
|
||||
</Tabs>
|
||||
|
||||
### Running DeepSeek with PD disaggregation on 4 x Atlas 800I A3
|
||||
|
||||
W8A8 Model weights could be found [here](https://modelers.cn/models/State_Cloud/Deepseek-R1-bf16-hfd-w8a8).
|
||||
|
||||
<Tabs>
|
||||
<Tab title="Prefill & Decode">
|
||||
```shell Command
|
||||
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
|
||||
sysctl -w vm.swappiness=0
|
||||
sysctl -w kernel.numa_balancing=0
|
||||
sysctl -w kernel.sched_migration_cost_ns=50000
|
||||
export SGLANG_SET_CPU_AFFINITY=1
|
||||
unset ASCEND_LAUNCH_BLOCKING
|
||||
source /usr/local/Ascend/ascend-toolkit/set_env.sh
|
||||
source /usr/local/Ascend/nnal/atb/set_env.sh
|
||||
export PATH=/usr/local/Ascend/8.5.0/compiler/bishengir/bin:$PATH
|
||||
|
||||
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
|
||||
export STREAMS_PER_DEVICE=32
|
||||
|
||||
export ASCEND_MF_STORE_URL="tcp://your prefill ip1:24669"
|
||||
|
||||
P_IP=('your prefill ip1' 'your prefill ip2')
|
||||
|
||||
D_IP=('your decode ip1' 'your decode ip2')
|
||||
|
||||
MODEL_PATH=xxx
|
||||
|
||||
export SGLANG_NPU_USE_MLAPO=1
|
||||
export SGLANG_USE_FIA_NZ=1
|
||||
|
||||
LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'`
|
||||
LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'`
|
||||
echo "${LOCAL_HOST1}"
|
||||
echo "${LOCAL_HOST2}"
|
||||
# prefill
|
||||
for i in "${!P_IP[@]}";
|
||||
do
|
||||
if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]];
|
||||
then
|
||||
echo "${P_IP[$i]}"
|
||||
export HCCL_BUFFSIZE=1536
|
||||
export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
|
||||
export TASK_QUEUE_ENABLE=2
|
||||
|
||||
export HCCL_SOCKET_IFNAME=lo
|
||||
export GLOO_SOCKET_IFNAME=lo
|
||||
python -m sglang.launch_server --model-path ${MODEL_PATH} --disaggregation-mode prefill --host ${P_IP[$i]} \
|
||||
--port 8000 --disaggregation-bootstrap-port $((8998+$i)) --trust-remote-code --nnodes 1 --node-rank 0 \
|
||||
--tp-size 16 --mem-fraction-static 0.81 --attention-backend ascend --device npu --quantization modelslim \
|
||||
--disaggregation-transfer-backend ascend --max-running-requests 8 --context-length 8192 --disable-radix-cache \
|
||||
--chunked-prefill-size -1 --max-prefill-tokens 28680 --moe-a2a-backend deepep --deepep-mode normal \
|
||||
--speculative-algorithm NEXTN --speculative-num-steps 1 --speculative-eagle-topk 1 --speculative-num-draft-tokens 2 \
|
||||
--dp-size 2 --enable-dp-attention --disable-shared-experts-fusion --dtype bfloat16 --enable-attn-tp-input-scattered
|
||||
NODE_RANK=$i
|
||||
break
|
||||
fi
|
||||
done
|
||||
|
||||
# decode
|
||||
for i in "${!D_IP[@]}";
|
||||
do
|
||||
if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]];
|
||||
then
|
||||
echo "${D_IP[$i]}"
|
||||
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
|
||||
export SGLANG_ENABLE_SPEC_V2=1
|
||||
export HCCL_BUFFSIZE=650
|
||||
export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=78
|
||||
export TASK_QUEUE_ENABLE=1
|
||||
export SGLANG_SCHEDULER_SKIP_ALL_GATHER=1
|
||||
export HCCL_SOCKET_IFNAME=xxx
|
||||
export GLOO_SOCKET_IFNAME=xxx
|
||||
python -m sglang.launch_server --model-path ${MODEL_PATH} --disaggregation-mode decode --host ${D_IP[$i]} \
|
||||
--port 8001 --trust-remote-code --dist-init-addr ${D_IP[0]}:5000 --nnodes 2 --node-rank $i --tp-size 32 --dp-size 32 \
|
||||
--mem-fraction-static 0.815 --max-running-requests 832 --attention-backend ascend --device npu --quantization modelslim \
|
||||
--moe-a2a-backend deepep --enable-dp-attention --deepep-mode low_latency --enable-dp-lm-head --moe-dense-tp 1 \
|
||||
--cuda-graph-bs 12 14 16 18 20 22 24 26 --disaggregation-transfer-backend ascend --watchdog-timeout 9000 --context-length 8192 \
|
||||
--speculative-algorithm NEXTN --speculative-num-steps 2 --speculative-eagle-topk 1 --speculative-num-draft-tokens 3 \
|
||||
--tokenizer-worker-num 4 --prefill-round-robin-balance --disable-shared-experts-fusion --dtype bfloat16 \
|
||||
--load-balance-method decode_round_robin
|
||||
NODE_RANK=$i
|
||||
break
|
||||
fi
|
||||
done
|
||||
```
|
||||
|
||||
</Tab>
|
||||
|
||||
<Tab title="Router">
|
||||
```shell Command
|
||||
export SGLANG_DP_ROUND_ROBIN=1
|
||||
python -m sglang_router.launch_router \
|
||||
--pd-disaggregation \
|
||||
--policy cache_aware \
|
||||
--prefill http://P_IP:8000 8998 \
|
||||
--prefill http://P_IP:8000 8999 \
|
||||
--decode http://D_IP:8001 \
|
||||
--host 127.0.0.1 \
|
||||
--port 6688 \
|
||||
--mini-lb
|
||||
```
|
||||
</Tab>
|
||||
</Tabs>
|
||||
|
||||
### Test GSM8K
|
||||
|
||||
<CodeGroup>
|
||||
```python Test GSM8K
|
||||
from types import SimpleNamespace
|
||||
from sglang.test.few_shot_gsm8k import run_eval
|
||||
|
||||
def gsm8k():
|
||||
args = SimpleNamespace(
|
||||
num_shots=5,
|
||||
data_path=None,
|
||||
num_questions=200,
|
||||
max_new_tokens=512,
|
||||
parallel=32,
|
||||
host=f"http://127.0.0.1",
|
||||
port=6688,
|
||||
)
|
||||
metrics = run_eval(args)
|
||||
print(f"{metrics=}")
|
||||
print(f"{metrics['accuracy']=}")
|
||||
if __name__ == "__main__":
|
||||
gsm8k()
|
||||
```
|
||||
</CodeGroup>
|
||||
@@ -0,0 +1,200 @@
|
||||
## Introduction
|
||||
|
||||
The GLM (General Language Model) series is an open-source bilingual large language model family jointly developed by the KEG Laboratory of Tsinghua University and Zhipu AI. This series of models has performed outstandingly in the field of Chinese NLP with its unique unified pre-training framework and bilingual capabilities. [GLM-5](https://huggingface.co/zai-org/GLM-5) adopts the DeepSeek-V3/V3.2 architecture, including the sparse attention (DSA) and multi-token prediction (MTP). Ascend supports GLM-5 with 0Day based on the SGLang inference framework, achieving low-code seamless enablement and compatibility with the mainstream distributed parallel capabilities within the current SGLang framework. We welcome developers to download and experience it.
|
||||
|
||||
## Environment Preparation
|
||||
|
||||
### Model Weight
|
||||
|
||||
- `GLM-5.0`(BF16 version): [Download model weight](https://www.modelscope.cn/models/ZhipuAI/GLM-5).
|
||||
- `GLM-5.0-w4a8`(Quantized version without mtp): [Download model weight](https://modelers.cn/models/Eco-Tech/GLM-5-w4a8).
|
||||
- You can use [msmodelslim](https://gitcode.com/Ascend/msmodelslim) to quantify the model naively.
|
||||
|
||||
|
||||
### Installation
|
||||
|
||||
The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the quay.io platform. You can directly pull it.
|
||||
|
||||
<CodeGroup>
|
||||
```bash Pull and Start Container
|
||||
#Atlas 800 A3
|
||||
docker pull swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:cann8.5.0-a3-glm5
|
||||
#Atlas 800 A2
|
||||
docker pull swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:cann8.5.0-910b-glm5
|
||||
|
||||
#start container
|
||||
docker run -itd --shm-size=16g --privileged=true --name ${NAME} \
|
||||
--privileged=true --net=host \
|
||||
-v /var/queue_schedule:/var/queue_schedule \
|
||||
-v /etc/ascend_install.info:/etc/ascend_install.info \
|
||||
-v /usr/local/sbin:/usr/local/sbin \
|
||||
-v /usr/local/Ascend/driver:/usr/local/Ascend/driver \
|
||||
-v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \
|
||||
--device=/dev/davinci0:/dev/davinci0 \
|
||||
--device=/dev/davinci1:/dev/avinci1 \
|
||||
--device=/dev/davinci2:/dev/davinci2 \
|
||||
--device=/dev/davinci3:/dev/davinci3 \
|
||||
--device=/dev/davinci4:/dev/davinci4 \
|
||||
--device=/dev/davinci5:/dev/davinci5 \
|
||||
--device=/dev/davinci6:/dev/davinci6 \
|
||||
--device=/dev/davinci7:/dev/davinci7 \
|
||||
--device=/dev/davinci8:/dev/davinci8 \
|
||||
--device=/dev/davinci9:/dev/davinci9 \
|
||||
--device=/dev/davinci10:/dev/davinci10 \
|
||||
--device=/dev/davinci11:/dev/davinci11 \
|
||||
--device=/dev/davinci12:/dev/davinci12 \
|
||||
--device=/dev/davinci13:/dev/davinci13 \
|
||||
--device=/dev/davinci14:/dev/davinci14 \
|
||||
--device=/dev/davinci15:/dev/davinci15 \
|
||||
--device=/dev/davinci_manager:/dev/davinci_manager \
|
||||
--device=/dev/hisi_hdc:/dev/hisi_hdc \
|
||||
--entrypoint=bash \
|
||||
swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:${TAG}
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
Note: Using this image, you need to update transformers to main branch
|
||||
<CodeGroup>
|
||||
```shell Update Transformers
|
||||
# reinstall transformers
|
||||
pip install git+https://github.com/huggingface/transformers.git
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
## Deployment
|
||||
|
||||
### Single-node Deployment
|
||||
|
||||
- Quantized model `glm5_w4a8` can be deployed on 1 Atlas 800 A3 (64G × 16) .
|
||||
|
||||
Run the following script to execute online inference.
|
||||
|
||||
<CodeGroup>
|
||||
```shell Launch Server
|
||||
# high performance cpu
|
||||
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
|
||||
sysctl -w vm.swappiness=0
|
||||
sysctl -w kernel.numa_balancing=0
|
||||
sysctl -w kernel.sched_migration_cost_ns=50000
|
||||
# bind cpu
|
||||
export SGLANG_SET_CPU_AFFINITY=1
|
||||
|
||||
unset https_proxy
|
||||
unset http_proxy
|
||||
unset HTTPS_PROXY
|
||||
unset HTTP_PROXY
|
||||
unset ASCEND_LAUNCH_BLOCKING
|
||||
# cann
|
||||
source /usr/local/Ascend/ascend-toolkit/set_env.sh
|
||||
source /usr/local/Ascend/nnal/atb/set_env.sh
|
||||
|
||||
export STREAMS_PER_DEVICE=32
|
||||
export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600
|
||||
export SGLANG_ENABLE_SPEC_V2=1
|
||||
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
|
||||
export SGLANG_NPU_USE_MULTI_STREAM=1
|
||||
export HCCL_BUFFSIZE=1000
|
||||
export HCCL_OP_EXPANSION_MODE=AIV
|
||||
export HCCL_SOCKET_IFNAME=lo
|
||||
export GLOO_SOCKET_IFNAME=lo
|
||||
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path $MODEL_PATH \
|
||||
--attention-backend ascend \
|
||||
--device npu \
|
||||
--tp-size 16 --nnodes 1 --node-rank 0 \
|
||||
--chunked-prefill-size 16384 --max-prefill-tokens 280000 \
|
||||
--trust-remote-code \
|
||||
--host 127.0.0.1 \
|
||||
--mem-fraction-static 0.7 \
|
||||
--port 8000 \
|
||||
--served-model-name glm-5 \
|
||||
--cuda-graph-bs 16 \
|
||||
--quantization modelslim \
|
||||
--moe-a2a-backend deepep --deepep-mode auto
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
### Multi-node Deployment
|
||||
|
||||
- `GLM-5-bf16`: require at least 2 Atlas 800 A3 (64G × 16).
|
||||
|
||||
**A3 series**
|
||||
|
||||
Modify the IP of 2 nodes, then run the same scripts on two nodes.
|
||||
|
||||
**node 0/1**
|
||||
|
||||
<CodeGroup>
|
||||
```shell Launch Multi-node Server
|
||||
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
|
||||
sysctl -w vm.swappiness=0
|
||||
sysctl -w kernel.numa_balancing=0
|
||||
sysctl -w kernel.sched_migration_cost_ns=50000
|
||||
# bind cpu
|
||||
export SGLANG_SET_CPU_AFFINITY=1
|
||||
|
||||
unset https_proxy
|
||||
unset http_proxy
|
||||
unset HTTPS_PROXY
|
||||
unset HTTP_PROXY
|
||||
unset ASCEND_LAUNCH_BLOCKING
|
||||
# cann
|
||||
source /usr/local/Ascend/ascend-toolkit/set_env.sh
|
||||
source /usr/local/Ascend/nnal/atb/set_env.sh
|
||||
|
||||
export STREAMS_PER_DEVICE=32
|
||||
export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600
|
||||
export SGLANG_ENABLE_SPEC_V2=1
|
||||
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
|
||||
export SGLANG_NPU_USE_MULTI_STREAM=1
|
||||
export HCCL_BUFFSIZE=1000
|
||||
export HCCL_OP_EXPANSION_MODE=AIV
|
||||
|
||||
# Run command ifconfig on two nodes, find out which inet addr has same IP with your node IP. That is your public interface, which should be added here
|
||||
export HCCL_SOCKET_IFNAME=lo
|
||||
export GLOO_SOCKET_IFNAME=lo
|
||||
|
||||
|
||||
P_IP=('your ip1' 'your ip2')
|
||||
P_MASTER="${P_IP[0]}:your port"
|
||||
export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600
|
||||
|
||||
export SGLANG_ENABLE_SPEC_V2=1
|
||||
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
|
||||
|
||||
LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'`
|
||||
LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'`
|
||||
for i in "${!P_IP[@]}";
|
||||
do
|
||||
if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]];
|
||||
then
|
||||
echo "${P_IP[$i]}"
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path $MODEL_PATH \
|
||||
--attention-backend ascend \
|
||||
--device npu \
|
||||
--tp-size 32 --nnodes 2 --node-rank $i --dist-init-addr $P_MASTER \
|
||||
--chunked-prefill-size 16384 --max-prefill-tokens 131072 \
|
||||
--trust-remote-code \
|
||||
--host 127.0.0.1 \
|
||||
--mem-fraction-static 0.8\
|
||||
--port 8000 \
|
||||
--served-model-name glm-5 \
|
||||
--cuda-graph-max-bs 16 \
|
||||
--disable-radix-cache
|
||||
NODE_RANK=$i
|
||||
break
|
||||
fi
|
||||
done
|
||||
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
### Prefill-Decode Disaggregation
|
||||
|
||||
Not test yet.
|
||||
|
||||
### Using Benchmark
|
||||
|
||||
Refer to [Benchmark and Profiling](../../developer_guide/benchmark_and_profiling) for details.
|
||||
@@ -0,0 +1,169 @@
|
||||
## Introduction
|
||||
|
||||
MindSpore is a high-performance AI framework optimized for Ascend NPUs. This doc guides users to run MindSpore models in SGLang.
|
||||
|
||||
## Requirements
|
||||
|
||||
MindSpore currently only supports Ascend NPU devices. Users need to first install Ascend CANN software packages.
|
||||
The CANN software packages can be downloaded from the [Ascend Official Website](https://www.hiascend.com). The recommended version is 8.3.RC2.
|
||||
|
||||
## Supported Models
|
||||
|
||||
Currently, the following models are supported:
|
||||
|
||||
- **Qwen3**: Dense and MoE models
|
||||
- **DeepSeek V3/R1**
|
||||
- *More models coming soon...*
|
||||
|
||||
## Installation
|
||||
|
||||
<Note>
|
||||
Currently, MindSpore models are provided by an independent package `sgl-mindspore`. Support for MindSpore is built upon current SGLang support for Ascend NPU platform. Please first [install SGLang for Ascend NPU](./SGLang-installation-with-NPUs-support) and then install `sgl-mindspore`:
|
||||
</Note>
|
||||
|
||||
<CodeGroup>
|
||||
```shell Install
|
||||
git clone https://github.com/mindspore-lab/sgl-mindspore.git
|
||||
cd sgl-mindspore
|
||||
pip install -e .
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
|
||||
## Run Model
|
||||
|
||||
Current SGLang-MindSpore supports Qwen3 and DeepSeek V3/R1 models. This doc uses Qwen3-8B as an example.
|
||||
|
||||
### Offline infer
|
||||
|
||||
Use the following script for offline infer:
|
||||
|
||||
<CodeGroup>
|
||||
```python Offline Inference
|
||||
import sglang as sgl
|
||||
|
||||
# Initialize the engine with MindSpore backend
|
||||
llm = sgl.Engine(
|
||||
model_path="/path/to/your/model", # Local model path
|
||||
device="npu", # Use NPU device
|
||||
model_impl="mindspore", # MindSpore implementation
|
||||
attention_backend="ascend", # Attention backend
|
||||
tp_size=1, # Tensor parallelism size
|
||||
dp_size=1 # Data parallelism size
|
||||
)
|
||||
|
||||
# Generate text
|
||||
prompts = [
|
||||
"Hello, my name is",
|
||||
"The capital of France is",
|
||||
"The future of AI is"
|
||||
]
|
||||
|
||||
sampling_params = {"temperature": 0, "top_p": 0.9}
|
||||
outputs = llm.generate(prompts, sampling_params)
|
||||
|
||||
for prompt, output in zip(prompts, outputs):
|
||||
print(f"Prompt: {prompt}")
|
||||
print(f"Generated: {output['text']}")
|
||||
print("---")
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
### Start server
|
||||
|
||||
Launch a server with MindSpore backend:
|
||||
|
||||
<CodeGroup>
|
||||
```bash Launch Server
|
||||
# Basic server startup
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path /path/to/your/model \
|
||||
--host 0.0.0.0 \
|
||||
--device npu \
|
||||
--model-impl mindspore \
|
||||
--attention-backend ascend \
|
||||
--tp-size 1 \
|
||||
--dp-size 1
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
For distributed server with multiple nodes:
|
||||
|
||||
<CodeGroup>
|
||||
```bash Multi-node Distributed
|
||||
# Multi-node distributed server
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path /path/to/your/model \
|
||||
--host 0.0.0.0 \
|
||||
--device npu \
|
||||
--model-impl mindspore \
|
||||
--attention-backend ascend \
|
||||
--dist-init-addr 127.0.0.1:29500 \
|
||||
--nnodes 2 \
|
||||
--node-rank 0 \
|
||||
--tp-size 4 \
|
||||
--dp-size 2
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
#### Debug Mode
|
||||
|
||||
Enable sglang debug logging by log-level argument.
|
||||
|
||||
<CodeGroup>
|
||||
```bash Debug Mode
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path /path/to/your/model \
|
||||
--host 0.0.0.0 \
|
||||
--device npu \
|
||||
--model-impl mindspore \
|
||||
--attention-backend ascend \
|
||||
--log-level DEBUG
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
Enable mindspore info and debug logging by setting environments.
|
||||
|
||||
<CodeGroup>
|
||||
```bash Set Log Level
|
||||
export GLOG_v=1 # INFO
|
||||
export GLOG_v=0 # DEBUG
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
#### Explicitly select devices
|
||||
|
||||
Use the following environment variable to explicitly select the devices to use.
|
||||
|
||||
<CodeGroup>
|
||||
```shell Select Devices
|
||||
export ASCEND_RT_VISIBLE_DEVICES=4,5,6,7 # to set device
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
#### Some communication environment issues
|
||||
|
||||
In case of some environment with special communication environment, users need set some environment variables.
|
||||
|
||||
<CodeGroup>
|
||||
```shell Disable LCCL
|
||||
export MS_ENABLE_LCCL=off # current not support LCCL communication mode in SGLang-MindSpore
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
#### Some dependencies of protobuf
|
||||
|
||||
In case of some environment with special protobuf version, users need set some environment variables to avoid binary version mismatch.
|
||||
|
||||
<CodeGroup>
|
||||
```shell Fix Protobuf
|
||||
export PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python # to avoid protobuf binary version mismatch
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
## Support
|
||||
For MindSpore-specific issues:
|
||||
|
||||
- Refer to the [MindSpore documentation](https://www.mindspore.cn/)
|
||||
@@ -0,0 +1,124 @@
|
||||
## Running Qwen3
|
||||
|
||||
### Running Qwen3-32B on 1 x Atlas 800I A3
|
||||
|
||||
Model weights could be found [here](https://huggingface.co/Qwen/Qwen3-32B)
|
||||
|
||||
<CodeGroup>
|
||||
```shell Launch Server
|
||||
export SGLANG_SET_CPU_AFFINITY=1
|
||||
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
|
||||
export STREAMS_PER_DEVICE=32
|
||||
export HCCL_BUFFSIZE=1536
|
||||
export HCCL_OP_EXPANSION_MODE=AIV
|
||||
|
||||
python -m sglang.launch_server \
|
||||
--device npu \
|
||||
--attention-backend ascend \
|
||||
--trust-remote-code \
|
||||
--tp-size 4 \
|
||||
--model-path Qwen/Qwen3-32B \
|
||||
--mem-fraction-static 0.8
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
### Running Qwen3-32B on 1 x Atlas 800I A3 with Qwen3-32B-Eagle3
|
||||
|
||||
Model weights could be found [here](https://huggingface.co/Qwen/Qwen3-32B)
|
||||
|
||||
Speculative model weights could be found [here](https://huggingface.co/Zhihu-ai/Zhi-Create-Qwen3-32B-Eagle3)
|
||||
|
||||
<CodeGroup>
|
||||
```shell Launch Server with Eagle3
|
||||
export SGLANG_SET_CPU_AFFINITY=1
|
||||
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
|
||||
export STREAMS_PER_DEVICE=32
|
||||
export HCCL_OP_EXPANSION_MODE=AIV
|
||||
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
|
||||
export SGLANG_ENABLE_SPEC_V2=1
|
||||
|
||||
python -m sglang.launch_server \
|
||||
--device npu \
|
||||
--attention-backend ascend \
|
||||
--trust-remote-code \
|
||||
--tp-size 4 \
|
||||
--model-path Qwen/Qwen3-32B \
|
||||
--mem-fraction-static 0.8 \
|
||||
--speculative-algorithm EAGLE3 \
|
||||
--speculative-draft-model-path Qwen/Qwen3-32B-Eagle3 \
|
||||
--speculative-num-steps 1 \
|
||||
--speculative-eagle-topk 1 \
|
||||
--speculative-num-draft-tokens 2
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
### Running Qwen3-30B-A3B MOE on 1 x Atlas 800I A3
|
||||
|
||||
Model weights could be found [here](https://huggingface.co/Qwen/Qwen3-30B-A3B)
|
||||
|
||||
<CodeGroup>
|
||||
```shell Launch Server
|
||||
export SGLANG_SET_CPU_AFFINITY=1
|
||||
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
|
||||
export STREAMS_PER_DEVICE=32
|
||||
export HCCL_BUFFSIZE=1536
|
||||
export HCCL_OP_EXPANSION_MODE=AIV
|
||||
export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=32
|
||||
export SGLANG_DEEPEP_BF16_DISPATCH=1
|
||||
|
||||
python -m sglang.launch_server \
|
||||
--device npu \
|
||||
--attention-backend ascend \
|
||||
--trust-remote-code \
|
||||
--tp-size 4 \
|
||||
--model-path Qwen/Qwen3-30B-A3B \
|
||||
--mem-fraction-static 0.8
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
### Running Qwen3-235B-A22B-Instruct-2507 MOE on 1 x Atlas 800I A3
|
||||
|
||||
Model weights could be found [here](https://huggingface.co/Qwen/Qwen3-235B-A22B-Instruct-2507)
|
||||
|
||||
<CodeGroup>
|
||||
```shell Launch Server
|
||||
export SGLANG_SET_CPU_AFFINITY=1
|
||||
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
|
||||
export STREAMS_PER_DEVICE=32
|
||||
export HCCL_BUFFSIZE=1536
|
||||
export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=32
|
||||
export SGLANG_DEEPEP_BF16_DISPATCH=1
|
||||
|
||||
python -m sglang.launch_server \
|
||||
--model-path Qwen/Qwen3-235B-A22B-Instruct-2507 \
|
||||
--tp-size 16 \
|
||||
--trust-remote-code \
|
||||
--attention-backend ascend \
|
||||
--device npu \
|
||||
--watchdog-timeout 9000 \
|
||||
--mem-fraction-static 0.8
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
### Running Qwen3-VL-8B-Instruct on 1 x Atlas 800I A3
|
||||
|
||||
Model weights could be found [here](https://huggingface.co/Qwen/Qwen3-VL-8B-Instruct)
|
||||
|
||||
<CodeGroup>
|
||||
```shell Launch Server
|
||||
export SGLANG_SET_CPU_AFFINITY=1
|
||||
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
|
||||
export STREAMS_PER_DEVICE=32
|
||||
export HCCL_BUFFSIZE=1536
|
||||
export HCCL_OP_EXPANSION_MODE=AIV
|
||||
|
||||
python -m sglang.launch_server \
|
||||
--enable-multimodal \
|
||||
--attention-backend ascend \
|
||||
--mm-attention-backend ascend_attn \
|
||||
--trust-remote-code \
|
||||
--tp-size 4 \
|
||||
--model-path Qwen/Qwen3-VL-8B-Instruct \
|
||||
--mem-fraction-static 0.8
|
||||
```
|
||||
</CodeGroup>
|
||||
@@ -0,0 +1,106 @@
|
||||
## Environment Preparation
|
||||
|
||||
### Installation
|
||||
|
||||
The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the quay.io platform. You can directly pull it.
|
||||
|
||||
<CodeGroup>
|
||||
```bash Pull and Start Container
|
||||
#Atlas 800 A3
|
||||
docker pull swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:cann8.5.0-a3-qwen3.5
|
||||
#Atlas 800 A2
|
||||
docker pull swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:cann8.5.0-910b-qwen3.5
|
||||
|
||||
#start container
|
||||
docker run -itd --shm-size=16g --privileged=true --name ${NAME} \
|
||||
--privileged=true --net=host \
|
||||
-v /var/queue_schedule:/var/queue_schedule \
|
||||
-v /etc/ascend_install.info:/etc/ascend_install.info \
|
||||
-v /usr/local/sbin:/usr/local/sbin \
|
||||
-v /usr/local/Ascend/driver:/usr/local/Ascend/driver \
|
||||
-v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \
|
||||
--device=/dev/davinci0:/dev/davinci0 \
|
||||
--device=/dev/davinci1:/dev/davinci1 \
|
||||
--device=/dev/davinci2:/dev/davinci2 \
|
||||
--device=/dev/davinci3:/dev/davinci3 \
|
||||
--device=/dev/davinci4:/dev/davinci4 \
|
||||
--device=/dev/davinci5:/dev/davinci5 \
|
||||
--device=/dev/davinci6:/dev/davinci6 \
|
||||
--device=/dev/davinci7:/dev/davinci7 \
|
||||
--device=/dev/davinci8:/dev/davinci8 \
|
||||
--device=/dev/davinci9:/dev/davinci9 \
|
||||
--device=/dev/davinci10:/dev/davinci10 \
|
||||
--device=/dev/davinci11:/dev/davinci11 \
|
||||
--device=/dev/davinci12:/dev/davinci12 \
|
||||
--device=/dev/davinci13:/dev/davinci13 \
|
||||
--device=/dev/davinci14:/dev/davinci14 \
|
||||
--device=/dev/davinci15:/dev/davinci15 \
|
||||
--device=/dev/davinci_manager:/dev/davinci_manager \
|
||||
--device=/dev/hisi_hdc:/dev/hisi_hdc \
|
||||
--entrypoint=bash \
|
||||
swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:${TAG}
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
## Deployment
|
||||
|
||||
### Single-node Deployment
|
||||
|
||||
- Quantized model `qwen35_w8a8` can be deployed on 1 Atlas 800 A3 (64G × 16) .
|
||||
|
||||
Run the following script to execute online inference.
|
||||
|
||||
<CodeGroup>
|
||||
```shell Launch Server
|
||||
# high performance cpu
|
||||
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
|
||||
sysctl -w vm.swappiness=0
|
||||
sysctl -w kernel.numa_balancing=0
|
||||
sysctl -w kernel.sched_migration_cost_ns=50000
|
||||
# bind cpu
|
||||
export SGLANG_SET_CPU_AFFINITY=1
|
||||
|
||||
unset https_proxy
|
||||
unset http_proxy
|
||||
unset HTTPS_PROXY
|
||||
unset HTTP_PROXY
|
||||
unset ASCEND_LAUNCH_BLOCKING
|
||||
# cann
|
||||
source /usr/local/Ascend/ascend-toolkit/set_env.sh
|
||||
source /usr/local/Ascend/nnal/atb/set_env.sh
|
||||
|
||||
export STREAMS_PER_DEVICE=32
|
||||
export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600
|
||||
export SGLANG_ENABLE_SPEC_V2=1
|
||||
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
|
||||
export SGLANG_NPU_USE_MULTI_STREAM=1
|
||||
export HCCL_BUFFSIZE=1000
|
||||
export HCCL_OP_EXPANSION_MODE=AIV
|
||||
export HCCL_SOCKET_IFNAME=lo
|
||||
export GLOO_SOCKET_IFNAME=lo
|
||||
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path $MODEL_PATH \
|
||||
--attention-backend ascend \
|
||||
--device npu \
|
||||
--tp-size 16 --nnodes 1 --node-rank 0 \
|
||||
--chunked-prefill-size 16384 --max-prefill-tokens 280000 \
|
||||
--trust-remote-code \
|
||||
--host 127.0.0.1 \
|
||||
--mem-fraction-static 0.7 \
|
||||
--port 8000 \
|
||||
--cuda-graph-bs 16 \
|
||||
--quantization modelslim \
|
||||
--enable-multimodal \
|
||||
--mm-attention-backend ascend_attn \
|
||||
--dtype bfloat16
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
### Prefill-Decode Disaggregation
|
||||
|
||||
Not test yet.
|
||||
|
||||
### Using Benchmark
|
||||
|
||||
Refer to [Benchmark and Profiling](../../developer_guide/benchmark_and_profiling) for details.
|
||||
+318
@@ -0,0 +1,318 @@
|
||||
---
|
||||
title: SGLang installation with NPUs support
|
||||
---
|
||||
|
||||
You can install SGLang using any of the methods below. Please go through `System Settings` section to ensure the clusters are roaring at max performance. Feel free to leave an issue [here at sglang](https://github.com/sgl-project/sglang/issues) if you encounter any issues or have any problems.
|
||||
|
||||
## Component Version Mapping For SGLang
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "34%"}} />
|
||||
<col style={{width: "33%"}} />
|
||||
<col style={{width: "33%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Component</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Version</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Obtain Way</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>HDK</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>25.3.RC1</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>[<Icon icon="download" />](https://hiascend.com/hardware/firmware-drivers/commercial?product=7\&model=33)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>CANN</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>8.5.0</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>[Obtain Images](#obtain-cann-image)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Pytorch Adapter</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>7.3.0</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>[<Icon icon="external-link" />](https://gitcode.com/Ascend/pytorch/releases)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>MemFabric</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>1.0.5</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`pip install memfabric-hybrid==1.0.5`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Triton</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>3.2.0</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`pip install triton-ascend`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Bisheng</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>20251121</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>[<Icon icon="external-link" />](https://sglang-ascend.obs.cn-east-3.myhuaweicloud.com/sglang/triton_ascend/Ascend-BiSheng-toolkit_aarch64_20251121.run)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>SGLang NPU Kernel</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>NA</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>[<Icon icon="external-link" />](https://github.com/sgl-project/sgl-kernel-npu/releases)</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
<Accordion title="Obtain CANN Image" defaultOpen="true">
|
||||
|
||||
You can obtain the dependency of a specified version of CANN through an image.
|
||||
|
||||
```bash
|
||||
# for Atlas 800I A3 and Ubuntu OS
|
||||
docker pull quay.io/ascend/cann:8.5.0-a3-ubuntu22.04-py3.11
|
||||
# for Atlas 800I A2 and Ubuntu OS
|
||||
docker pull quay.io/ascend/cann:8.5.0-910b-ubuntu22.04-py3.11
|
||||
```
|
||||
</Accordion>
|
||||
|
||||
## Preparing the Running Environment
|
||||
|
||||
<Tabs>
|
||||
<Tab title="Source">
|
||||
<AccordionGroup>
|
||||
<Accordion title="Python Version">
|
||||
Only `python==3.11` is supported currently. If you don't want to break system pre-installed python, try installing with [conda](https://github.com/conda/conda).
|
||||
|
||||
```bash
|
||||
conda create --name sglang_npu python=3.11
|
||||
conda activate sglang_npu
|
||||
```
|
||||
</Accordion>
|
||||
<Accordion title="CANN">
|
||||
Prior to start work with SGLang on Ascend you need to install CANN Toolkit, Kernels operator package and NNAL version 8.3.RC2 or higher, check the [installation guide](https://www.hiascend.com/document/detail/zh/CANNCommunityEdition/83RC1/softwareinst/instg/instg_0008.html?Mode=PmIns\&InstallType=local\&OS=openEuler\&Software=cannToolKit)
|
||||
</Accordion>
|
||||
<Accordion title="MemFabric-Hybrid">
|
||||
If you want to use PD disaggregation mode, you need to install MemFabric-Hybrid. MemFabric-Hybrid is a drop-in replacement of Mooncake Transfer Engine that enables KV cache transfer on Ascend NPU clusters.
|
||||
|
||||
```bash
|
||||
pip install memfabric-hybrid==1.0.5
|
||||
```
|
||||
</Accordion>
|
||||
<Accordion title="Pytorch and Pytorch Framework Adaptor on Ascend">
|
||||
```bash
|
||||
PYTORCH_VERSION=2.8.0
|
||||
TORCHVISION_VERSION=0.23.0
|
||||
TORCH_NPU_VERSION=2.8.0
|
||||
pip install torch==$PYTORCH_VERSION torchvision==$TORCHVISION_VERSION --index-url https://download.pytorch.org/whl/cpu
|
||||
pip install torch_npu==$TORCH_NPU_VERSION
|
||||
```
|
||||
|
||||
If you are using other versions of `torch` and install `torch_npu`, check [installation guide](https://github.com/Ascend/pytorch/blob/master/README)
|
||||
</Accordion>
|
||||
<Accordion title="Triton on Ascend">
|
||||
We provide our own implementation of Triton for Ascend.
|
||||
|
||||
```bash
|
||||
BISHENG_NAME="Ascend-BiSheng-toolkit_aarch64_20251121.run"
|
||||
BISHENG_URL="https://sglang-ascend.obs.cn-east-3.myhuaweicloud.com/sglang/triton_ascend/${BISHENG_NAME}"
|
||||
wget -O "${BISHENG_NAME}" "${BISHENG_URL}" && chmod a+x "${BISHENG_NAME}" && "./${BISHENG_NAME}" --install && rm "${BISHENG_NAME}"
|
||||
```
|
||||
|
||||
```bash
|
||||
pip install triton-ascend
|
||||
```
|
||||
|
||||
For installation of Triton on Ascend nightly builds or from sources, follow [installation guide](https://gitcode.com/Ascend/triton-ascend/blob/master/docs/sources/getting-started/installation)
|
||||
|
||||
</Accordion>
|
||||
<Accordion title="SGLang Kernels NPU">
|
||||
We provide SGL kernels for Ascend NPU, check [installation guide](https://github.com/sgl-project/sgl-kernel-npu/blob/main/python/sgl_kernel_npu/README).
|
||||
</Accordion>
|
||||
<Accordion title="DeepEP-compatible Library">
|
||||
We provide a DeepEP-compatible Library as a drop-in replacement of deepseek-ai's DeepEP library, check the [installation guide](https://github.com/sgl-project/sgl-kernel-npu/blob/main/python/deep_ep/README).
|
||||
</Accordion>
|
||||
<Accordion title="Installing SGLang from source">
|
||||
```bash
|
||||
# Use the last release branch
|
||||
git clone https://github.com/sgl-project/sglang.git
|
||||
cd sglang
|
||||
mv python/pyproject_npu.toml python/pyproject.toml
|
||||
pip install -e python[all_npu]
|
||||
```
|
||||
</Accordion>
|
||||
</AccordionGroup>
|
||||
</Tab>
|
||||
|
||||
<Tab title="Docker">
|
||||
|
||||
### Obtain Image
|
||||
|
||||
You can download the SGLang image or build an image based on Dockerfile to obtain the Ascend NPU image.
|
||||
|
||||
1. **Download SGLang image**
|
||||
|
||||
```bash
|
||||
dockerhub: docker.io/lmsysorg/sglang:$tag
|
||||
# Main-based tag, change main to specific version like v0.5.6,
|
||||
# you can get image for specific version
|
||||
Atlas 800I A3 : {main}-cann8.5.0-a3
|
||||
Atlas 800I A2: {main}-cann8.5.0-910b
|
||||
```
|
||||
|
||||
2. **Build an image based on Dockerfile**
|
||||
|
||||
```bash
|
||||
# Clone the SGLang repository
|
||||
git clone https://github.com/sgl-project/sglang.git
|
||||
cd sglang/docker
|
||||
|
||||
# Build the docker image
|
||||
# If there are network errors, please modify the Dockerfile to use offline dependencies or use a proxy
|
||||
docker build -t <image_name> -f npu.Dockerfile .
|
||||
```
|
||||
|
||||
### Create Docker
|
||||
|
||||
<Info>`--privileged` and `--network=host` are required by RDMA, which is typically needed by Ascend NPU clusters.</Info>
|
||||
<Note>The following docker command is based on Atlas 800I A3 machines. If you are using Atlas 800I A2, make sure only `davinci[0-7]` are mapped into container.</Note>
|
||||
|
||||
```bash
|
||||
alias drun='docker run -it --rm --privileged --network=host --ipc=host --shm-size=16g \
|
||||
--device=/dev/davinci0 --device=/dev/davinci1 --device=/dev/davinci2 --device=/dev/davinci3 \
|
||||
--device=/dev/davinci4 --device=/dev/davinci5 --device=/dev/davinci6 --device=/dev/davinci7 \
|
||||
--device=/dev/davinci8 --device=/dev/davinci9 --device=/dev/davinci10 --device=/dev/davinci11 \
|
||||
--device=/dev/davinci12 --device=/dev/davinci13 --device=/dev/davinci14 --device=/dev/davinci15 \
|
||||
--device=/dev/davinci_manager --device=/dev/hisi_hdc \
|
||||
--volume /usr/local/sbin:/usr/local/sbin --volume /usr/local/Ascend/driver:/usr/local/Ascend/driver \
|
||||
--volume /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \
|
||||
--volume /etc/ascend_install.info:/etc/ascend_install.info \
|
||||
--volume /var/queue_schedule:/var/queue_schedule --volume ~/.cache/:/root/.cache/'
|
||||
|
||||
# Add HF_TOKEN env for download model by SGLang.
|
||||
drun --env "HF_TOKEN=<secret>" \
|
||||
<image_name> \
|
||||
python3 -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct --attention-backend ascend
|
||||
```
|
||||
|
||||
</Tab>
|
||||
</Tabs>
|
||||
|
||||
## System Settings
|
||||
|
||||
<AccordionGroup>
|
||||
<Accordion title="CPU performance power scheme" defaultOpen="true">
|
||||
The default power scheme on Ascend hardware is `ondemand` which could affect performance, changing it to `performance` is recommended.
|
||||
|
||||
```bash
|
||||
echo performance | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
|
||||
|
||||
# Make sure changes are applied successfully
|
||||
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor # shows performance
|
||||
```
|
||||
</Accordion>
|
||||
<Accordion title="Disable NUMA balancing" defaultOpen="true">
|
||||
```bash
|
||||
sudo sysctl -w kernel.numa_balancing=0
|
||||
# Check
|
||||
cat /proc/sys/kernel/numa_balancing # shows 0
|
||||
```
|
||||
</Accordion>
|
||||
<Accordion title="Prevent swapping out system memory" defaultOpen="true">
|
||||
```bash
|
||||
sudo sysctl -w vm.swappiness=10
|
||||
|
||||
# Check
|
||||
cat /proc/sys/vm/swappiness # shows 10
|
||||
```
|
||||
</Accordion>
|
||||
</AccordionGroup>
|
||||
|
||||
## Running SGLang Service
|
||||
|
||||
<Tabs>
|
||||
<Tab title="For Large Language Models">
|
||||
|
||||
### PD Mixed Scene
|
||||
|
||||
```bash
|
||||
# Enabling CPU Affinity
|
||||
export SGLANG_SET_CPU_AFFINITY=1
|
||||
python3 -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct --attention-backend ascend
|
||||
```
|
||||
|
||||
### PD Separation Scene
|
||||
|
||||
1. **Launch Prefill Server**
|
||||
|
||||
```bash
|
||||
# Enabling CPU Affinity
|
||||
export SGLANG_SET_CPU_AFFINITY=1
|
||||
|
||||
# PIP: recommended to config first Prefill Server IP
|
||||
# PORT: one free port
|
||||
# all sglang servers need to be config the same PIP and PORT,
|
||||
export ASCEND_MF_STORE_URL="tcp://PIP:PORT"
|
||||
# if you are Atlas 800I A2 hardware and use rdma for kv cache transfer, add this parameter
|
||||
export ASCEND_MF_TRANSFER_PROTOCOL="device_rdma"
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path meta-llama/Llama-3.1-8B-Instruct \
|
||||
--disaggregation-mode prefill \
|
||||
--disaggregation-transfer-backend ascend \
|
||||
--disaggregation-bootstrap-port 8995 \
|
||||
--attention-backend ascend \
|
||||
--device npu \
|
||||
--base-gpu-id 0 \
|
||||
--tp-size 1 \
|
||||
--host 127.0.0.1 \
|
||||
--port 8000
|
||||
```
|
||||
|
||||
2. **Launch Decode Server**
|
||||
|
||||
```bash
|
||||
# PIP: recommended to config first Prefill Server IP
|
||||
# PORT: one free port
|
||||
# all sglang servers need to be config the same PIP and PORT,
|
||||
export ASCEND_MF_STORE_URL="tcp://PIP:PORT"
|
||||
# if you are Atlas 800I A2 hardware and use rdma for kv cache transfer, add this parameter
|
||||
export ASCEND_MF_TRANSFER_PROTOCOL="device_rdma"
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path meta-llama/Llama-3.1-8B-Instruct \
|
||||
--disaggregation-mode decode \
|
||||
--disaggregation-transfer-backend ascend \
|
||||
--attention-backend ascend \
|
||||
--device npu \
|
||||
--base-gpu-id 1 \
|
||||
--tp-size 1 \
|
||||
--host 127.0.0.1 \
|
||||
--port 8001
|
||||
```
|
||||
|
||||
3. **Launch Router**
|
||||
|
||||
```bash
|
||||
python3 -m sglang_router.launch_router \
|
||||
--pd-disaggregation \
|
||||
--policy cache_aware \
|
||||
--prefill http://127.0.0.1:8000 8995 \
|
||||
--decode http://127.0.0.1:8001 \
|
||||
--host 127.0.0.1 \
|
||||
--port 6688
|
||||
```
|
||||
</Tab>
|
||||
<Tab title="For Multimodal Language Models">
|
||||
|
||||
### PD Mixed Scene
|
||||
|
||||
```bash
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path Qwen3-VL-30B-A3B-Instruct \
|
||||
--host 127.0.0.1 \
|
||||
--port 8000 \
|
||||
--tp 4 \
|
||||
--device npu \
|
||||
--attention-backend ascend \
|
||||
--mm-attention-backend ascend_attn \
|
||||
--disable-radix-cache \
|
||||
--trust-remote-code \
|
||||
--enable-multimodal \
|
||||
--sampling-backend ascend
|
||||
```
|
||||
|
||||
</Tab>
|
||||
</Tabs>
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,610 @@
|
||||
This section describes the models supported on the Ascend NPU, including Large Language Models, Multimodal Language
|
||||
Models, Embedding Models, Reward Models and Rerank Models. Mainstream DeepSeek/Qwen/GLM series are included.
|
||||
You are welcome to enable various models based on your business requirements.
|
||||
|
||||
<Accordion title="Large Language Models">
|
||||
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "25.0%"}} />
|
||||
<col style={{width: "25.0%"}} />
|
||||
<col style={{width: "25.0%"}} />
|
||||
<col style={{width: "25.0%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Models</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Model Family</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>A2</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>A3</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>DeepSeek V3/V3.1</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>DeepSeek</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>vllm-ascend/DeepSeek-V3.2-Exp-W8A8</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>DeepSeek</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>vllm-ascend/DeepSeek-R1-0528-W8A8</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>DeepSeek</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>vllm-ascend/DeepSeek-V2-Lite-W8A8</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>DeepSeek</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen/Qwen3-30B-A3B-Instruct-2507</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Qwen</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen/Qwen3-32B</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Qwen</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen/Qwen3-0.6B</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Qwen</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>vllm-ascend/Qwen3-235B-A22B-W8A8</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Qwen</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen/Qwen3-Next-80B-A3B-Instruct</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Qwen</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen3-Coder-480B-A35B-Instruct-w8a8-QuaRot</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Qwen</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen/Qwen2.5-7B-Instruct</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Qwen</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>vllm-ascend/QWQ-32B-W8A8</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Qwen</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>meta-llama/Llama-4-Scout-17B-16E-Instruct</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Llama</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>AI-ModelScope/Llama-3.1-8B-Instruct</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Llama</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>LLM-Research/llama-2-7b</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Llama</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>LLM-Research/Llama-3.2-1B-Instruct</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Llama</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>mistralai/Mistral-7B-Instruct-v0.2</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Mistral</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>google/gemma-3-4b-it</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Gemma</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>microsoft/Phi-4-multimodal-instruct</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Phi</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>allenai/OLMoE-1B-7B-0924</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>OLMoE</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>stabilityai/stablelm-2-1_6b</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>StableLM</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>CohereForAI/c4ai-command-r-v01</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Command-R</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>huihui-ai/grok-2</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Grok</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>ZhipuAI/chatglm2-6b</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>ChatGLM</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Shanghai_AI_Laboratory/internlm2-7b</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>InternLM 2</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>LGAI-EXAONE/EXAONE-3.5-7.8B-Instruct</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>ExaONE 3</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>xverse/XVERSE-MoE-A36B</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>XVERSE</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>HuggingFaceTB/SmolLM-1.7B</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>SmolLM</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>ZhipuAI/glm-4-9b-chat</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>GLM-4</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>XiaomiMiMo/MiMo-7B-RL</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>MiMo</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>arcee-ai/AFM-4.5B-Base</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Arcee AFM-4.5B</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Howeee/persimmon-8b-chat</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Persimmon</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>inclusionAI/Ling-lite</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Ling</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>ibm-granite/granite-3.1-8b-instruct</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Granite</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>ibm-granite/granite-3.0-3b-a800m-instruct</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Granite MoE</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>AI-ModelScope/dbrx-instruct</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>DBRX (Databricks)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>baichuan-inc/Baichuan2-13B-Chat</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Baichuan 2 (7B, 13B)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>baidu/ERNIE-4.5-21B-A3B-PT</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>ERNIE-4.5 (4.5, 4.5MoE series)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>OpenBMB/MiniCPM3-4B</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>MiniCPM (v3, 4B)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Kimi/Kimi-K2-Thinking</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Kimi</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>openai/gpt-oss-120b</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>GPTOSS</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>allenai/OLMo-2-1124-7B-Instruct</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>OLMo</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>minimax/MiniMax-M2</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>MiniMax-M2</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>upstage/SOLAR-10.7B-Instruct-v1.0</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Solar</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>bigcode/starcoder2-7b</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>StarCoder2</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>arcee-ai/Trinity-Mini</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Trinity (Nano, Mini)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
|
||||
</Accordion>
|
||||
|
||||
|
||||
<Accordion title="Multimodal Language Models">
|
||||
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "25.0%"}} />
|
||||
<col style={{width: "25.0%"}} />
|
||||
<col style={{width: "25.0%"}} />
|
||||
<col style={{width: "25.0%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Models</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Model Family (Variants)</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>A2</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>A3</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen/Qwen2.5-VL-3B-Instruct</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Qwen-VL</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen/Qwen2.5-VL-72B-Instruct</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Qwen-VL</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen/Qwen3-VL-30B-A3B-Instruct</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Qwen-VL</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen/Qwen3-VL-8B-Instruct</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Qwen-VL</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen/Qwen3-VL-4B-Instruct</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Qwen-VL</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen/Qwen3-VL-235B-A22B-Instruct</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Qwen-VL</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>deepseek-ai/deepseek-vl2</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>DeepSeek-VL2</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>deepseek-ai/Janus-Pro-1B</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Janus-Pro (1B, 7B)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>deepseek-ai/Janus-Pro-7B</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Janus-Pro (1B, 7B)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>openbmb/MiniCPM-V-2_6</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>MiniCPM-V / MiniCPM-o</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>openbmb/MiniCPM-o-2_6</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>MiniCPM-V / MiniCPM-o</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>google/gemma-3-4b-it</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Gemma 3 (Multimodal)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>mistralai/Mistral-Small-3.1-24B-Instruct-2503</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Mistral-Small-3.1-24B</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>microsoft/Phi-4-multimodal-instruct</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Phi-4-multimodal-instruct</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>XiaomiMiMo/MiMo-VL-7B-RL</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>MiMo-VL (7B)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>AI-ModelScope/llava-v1.6-34b</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>LLaVA (v1.5 & v1.6)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>lmms-lab/llava-next-72b</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>LLaVA-NeXT (8B, 72B)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>lmms-lab/llava-onevision-qwen2-7b-ov</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>LLaVA-OneVision</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Kimi/Kimi-VL-A3B-Instruct</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Kimi-VL (A3B)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>ZhipuAI/GLM-4.5V</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>GLM-4.5V (106B)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>LLM-Research/Llama-3.2-11B-Vision-Instruct</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Llama 3.2 Vision (11B)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>rednote-hilab/dots.ocr</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>DotsVLM-OCR</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
|
||||
</Accordion>
|
||||
|
||||
|
||||
<Accordion title="Embedding Models">
|
||||
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "25.0%"}} />
|
||||
<col style={{width: "25.0%"}} />
|
||||
<col style={{width: "25.0%"}} />
|
||||
<col style={{width: "25.0%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Models</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Model Family</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>A2</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>A3</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>intfloat/e5-mistral-7b-instruct</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>E5 (Llama/Mistral based)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>iic/gte_Qwen2-1.5B-instruct</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>GTE-Qwen2</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen/Qwen3-Embedding-8B</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Qwen3-Embedding</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Alibaba-NLP/gme-Qwen2-VL-2B-Instruct</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>GME (Multimodal)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>AI-ModelScope/clip-vit-large-patch14-336</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>CLIP</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>BAAI/bge-large-en-v1.5</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>BGE</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
|
||||
</Accordion>
|
||||
|
||||
|
||||
<Accordion title="Reward Models">
|
||||
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "25.0%"}} />
|
||||
<col style={{width: "25.0%"}} />
|
||||
<col style={{width: "25.0%"}} />
|
||||
<col style={{width: "25.0%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Models</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Model Family</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>A2</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>A3</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Skywork/Skywork-Reward-Llama-3.1-8B-v0.2</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Llama3.1 Reward</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Shanghai_AI_Laboratory/internlm2-7b-reward</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>InternLM 2 Reward</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen/Qwen2.5-Math-RM-72B</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Qwen2.5 Reward - Math</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Howeee/Qwen2.5-1.5B-apeach</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Qwen2.5 Reward - Sequence</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>AI-ModelScope/Skywork-Reward-Gemma-2-27B-v0.2</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Gemma 2-27B Reward</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
|
||||
</Accordion>
|
||||
|
||||
|
||||
<Accordion title="Rerank Models">
|
||||
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "25.0%"}} />
|
||||
<col style={{width: "25.0%"}} />
|
||||
<col style={{width: "25.0%"}} />
|
||||
<col style={{width: "25.0%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Models</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Model Family</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>A2</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>A3</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>BAAI/bge-reranker-v2-m3</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>BGE-Reranker</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
</Accordion>
|
||||
@@ -0,0 +1,355 @@
|
||||
---
|
||||
title: "CPU Servers"
|
||||
---
|
||||
|
||||
The document addresses how to set up the [SGLang](https://github.com/sgl-project/sglang) environment and run LLM inference on CPU servers.
|
||||
SGLang is enabled and optimized on the CPUs equipped with Intel® AMX® Instructions,
|
||||
which are 4th generation or newer Intel® Xeon® Scalable Processors.
|
||||
|
||||
## Optimized Model List
|
||||
|
||||
A list of popular LLMs are optimized and run efficiently on CPU,
|
||||
including the most notable open-source models like Llama series, Qwen series,
|
||||
and DeepSeek series like DeepSeek-R1 and DeepSeek-V3.1-Terminus.
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "22%"}} />
|
||||
<col style={{width: "26%"}} />
|
||||
<col style={{width: "30%"}} />
|
||||
<col style={{width: "22%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Model Name</th>
|
||||
<th style={{textAlign: "center", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>BF16</th>
|
||||
<th style={{textAlign: "center", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>W8A8_INT8</th>
|
||||
<th style={{textAlign: "center", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>FP8</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", whiteSpace: "nowrap", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>DeepSeek-R1</td>
|
||||
<td style={{padding: "9px 12px", textAlign: "center", color: "gray", backgroundColor: "rgba(255,255,255,0.05)"}}>—</td>
|
||||
<td style={{padding: "9px 12px", textAlign: "center", backgroundColor: "rgba(255,255,255,0.02)"}}><a href="https://huggingface.co/meituan/DeepSeek-R1-Channel-INT8">DeepSeek-R1-Channel-INT8</a></td>
|
||||
<td style={{padding: "9px 12px", textAlign: "center", backgroundColor: "rgba(255,255,255,0.05)"}}><a href="https://huggingface.co/deepseek-ai/DeepSeek-R1">DeepSeek-R1</a></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", whiteSpace: "nowrap", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>DeepSeek-V3.1-Terminus</td>
|
||||
<td style={{padding: "9px 12px", textAlign: "center", color: "gray", backgroundColor: "rgba(255,255,255,0.05)"}}>—</td>
|
||||
<td style={{padding: "9px 12px", textAlign: "center", backgroundColor: "rgba(255,255,255,0.02)"}}><a href="https://huggingface.co/IntervitensInc/DeepSeek-V3.1-Terminus-Channel-int8">DeepSeek-V3.1-Terminus-Channel-int8</a></td>
|
||||
<td style={{padding: "9px 12px", textAlign: "center", backgroundColor: "rgba(255,255,255,0.05)"}}><a href="https://huggingface.co/deepseek-ai/DeepSeek-V3.1-Terminus">DeepSeek-V3.1-Terminus</a></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", whiteSpace: "nowrap", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Llama-3.2-3B</td>
|
||||
<td style={{padding: "9px 12px", textAlign: "center", backgroundColor: "rgba(255,255,255,0.05)"}}><a href="https://huggingface.co/meta-llama/Llama-3.2-3B-Instruct">Llama-3.2-3B-Instruct</a></td>
|
||||
<td style={{padding: "9px 12px", textAlign: "center", backgroundColor: "rgba(255,255,255,0.02)"}}><a href="https://huggingface.co/RedHatAI/Llama-3.2-3B-Instruct-quantized.w8a8">Llama-3.2-3B-quantized.w8a8</a></td>
|
||||
<td style={{padding: "9px 12px", textAlign: "center", color: "gray", backgroundColor: "rgba(255,255,255,0.05)"}}>—</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", whiteSpace: "nowrap", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Llama-3.1-8B</td>
|
||||
<td style={{padding: "9px 12px", textAlign: "center", backgroundColor: "rgba(255,255,255,0.05)"}}><a href="https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct">Llama-3.1-8B-Instruct</a></td>
|
||||
<td style={{padding: "9px 12px", textAlign: "center", backgroundColor: "rgba(255,255,255,0.02)"}}><a href="https://huggingface.co/RedHatAI/Meta-Llama-3.1-8B-quantized.w8a8">Llama-3.1-8B-quantized.w8a8</a></td>
|
||||
<td style={{padding: "9px 12px", textAlign: "center", color: "gray", backgroundColor: "rgba(255,255,255,0.05)"}}>—</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", whiteSpace: "nowrap", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>QwQ-32B</td>
|
||||
<td style={{padding: "9px 12px", textAlign: "center", color: "gray", backgroundColor: "rgba(255,255,255,0.05)"}}>—</td>
|
||||
<td style={{padding: "9px 12px", textAlign: "center", color: "gray", backgroundColor: "rgba(255,255,255,0.02)"}}><a href="https://huggingface.co/RedHatAI/QwQ-32B-quantized.w8a8">QwQ-32B-quantized.w8a8</a></td>
|
||||
<td style={{padding: "9px 12px", textAlign: "center", color: "gray", backgroundColor: "rgba(255,255,255,0.05)"}}>—</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", whiteSpace: "nowrap", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>DeepSeek-Distilled-Llama</td>
|
||||
<td style={{padding: "9px 12px", textAlign: "center", color: "gray", backgroundColor: "rgba(255,255,255,0.05)"}}>—</td>
|
||||
<td style={{padding: "9px 12px", textAlign: "center", backgroundColor: "rgba(255,255,255,0.02)"}}><a href="https://huggingface.co/RedHatAI/DeepSeek-R1-Distill-Llama-70B-quantized.w8a8">DeepSeek-R1-Distill-Llama-70B-quantized.w8a8</a></td>
|
||||
<td style={{padding: "9px 12px", textAlign: "center", color: "gray", backgroundColor: "rgba(255,255,255,0.05)"}}>—</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", whiteSpace: "nowrap", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen3-235B</td>
|
||||
<td style={{padding: "9px 12px", textAlign: "center", color: "gray", backgroundColor: "rgba(255,255,255,0.05)"}}>—</td>
|
||||
<td style={{padding: "9px 12px", textAlign: "center", color: "gray", backgroundColor: "rgba(255,255,255,0.02)"}}>—</td>
|
||||
<td style={{padding: "9px 12px", textAlign: "center", backgroundColor: "rgba(255,255,255,0.05)"}}><a href="https://huggingface.co/Qwen/Qwen3-235B-A22B-FP8">Qwen3-235B-A22B-FP8</a></td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
> **Note:** The model identifiers listed in the table above have been verified on 6th Gen Intel® Xeon® P-core platforms.
|
||||
|
||||
## Installation
|
||||
|
||||
<Tabs>
|
||||
<Tab title="Docker (Recommended)">
|
||||
It is recommended to use Docker for setting up the SGLang environment.
|
||||
A [Dockerfile](https://github.com/sgl-project/sglang/blob/main/docker/xeon.Dockerfile) is provided to facilitate the installation.
|
||||
|
||||
> **Note:** Replace `<secret>` below with your [HuggingFace access token](https://huggingface.co/docs/hub/en/security-tokens).
|
||||
|
||||
<CodeGroup>
|
||||
```bash Clone, Build and Run
|
||||
# Clone the SGLang repository
|
||||
git clone https://github.com/sgl-project/sglang.git
|
||||
cd sglang/docker
|
||||
|
||||
# Build the docker image
|
||||
docker build -t sglang-cpu:latest -f xeon.Dockerfile .
|
||||
|
||||
# Initiate a docker container
|
||||
docker run \
|
||||
-it \
|
||||
--privileged \
|
||||
--ipc=host \
|
||||
--network=host \
|
||||
-v /dev/shm:/dev/shm \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
-p 30000:30000 \
|
||||
-e "HF_TOKEN=<secret>" \
|
||||
sglang-cpu:latest /bin/bash
|
||||
```
|
||||
</CodeGroup>
|
||||
</Tab>
|
||||
|
||||
<Tab title="From Source">
|
||||
If you prefer to install SGLang in a bare metal environment, the setup process is as follows.
|
||||
|
||||
Please install the required packages and libraries beforehand if they are not already present on your system.
|
||||
You can refer to the Ubuntu-based installation commands in
|
||||
[the Dockerfile](https://github.com/sgl-project/sglang/blob/main/docker/xeon.Dockerfile#L11) for guidance.
|
||||
|
||||
1. **Install uv and create a virtual environment**
|
||||
<CodeGroup>
|
||||
```bash Create Virtual Environment
|
||||
# Taking '/opt' as the example uv env folder, feel free to change it as needed
|
||||
cd /opt
|
||||
curl -LsSf https://astral.sh/uv/install.sh | sh
|
||||
source $HOME/.local/bin/env
|
||||
uv venv --python 3.12
|
||||
source .venv/bin/activate
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
2. **Create a config file for torch package indexes**
|
||||
Create the `uv.toml` config file:
|
||||
|
||||
<CodeGroup>
|
||||
```bash Open Config File
|
||||
vim .venv/uv.toml
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
Press `a` to enter insert mode in `vim`, then paste the following content:
|
||||
|
||||
<CodeGroup>
|
||||
```toml
|
||||
[[index]]
|
||||
name = "torch"
|
||||
url = "https://download.pytorch.org/whl/cpu"
|
||||
|
||||
[[index]]
|
||||
name = "torchvision"
|
||||
url = "https://download.pytorch.org/whl/cpu"
|
||||
|
||||
[[index]]
|
||||
name = "torchaudio"
|
||||
url = "https://download.pytorch.org/whl/cpu"
|
||||
|
||||
[[index]]
|
||||
name = "triton"
|
||||
url = "https://download.pytorch.org/whl/cpu"
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
Save the file (press `Esc`, then type `:x` and hit `Enter`), then set it as the default `uv` config:
|
||||
|
||||
<CodeGroup>
|
||||
```bash Set Config Path
|
||||
export UV_CONFIG_FILE=/opt/.venv/uv.toml
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
3. **Clone SGLang and build packages**
|
||||
<CodeGroup>
|
||||
```bash Build SGLang
|
||||
# Clone the SGLang code
|
||||
git clone https://github.com/sgl-project/sglang.git
|
||||
cd sglang
|
||||
git checkout <YOUR-DESIRED-VERSION>
|
||||
|
||||
# Use dedicated toml file
|
||||
cd python
|
||||
cp pyproject_cpu.toml pyproject.toml
|
||||
# Install SGLang dependent libs, and build SGLang main package
|
||||
uv pip install --upgrade pip setuptools
|
||||
uv pip install .
|
||||
|
||||
# Build the CPU backend kernels
|
||||
cd ../sgl-kernel
|
||||
cp pyproject_cpu.toml pyproject.toml
|
||||
uv pip install .
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
4. **Set required environment variables**
|
||||
<CodeGroup>
|
||||
```bash Set Environment Variables
|
||||
export SGLANG_USE_CPU_ENGINE=1
|
||||
|
||||
# Set 'LD_LIBRARY_PATH' and 'LD_PRELOAD' to ensure the libs can be loaded by sglang processes
|
||||
export LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu
|
||||
export LD_PRELOAD=${LD_PRELOAD}:/opt/.venv/lib/libiomp5.so:${LD_LIBRARY_PATH}/libtcmalloc.so.4:${LD_LIBRARY_PATH}/libtbbmalloc.so.2
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
> **Note:** The environment variable `SGLANG_USE_CPU_ENGINE=1` is required to enable the SGLang service with the CPU engine.
|
||||
|
||||
> **Note:** If you encounter code compilation issues during the `sgl-kernel` building process, please check your `gcc` and `g++` versions and upgrade them if they are outdated. It is recommended to use `gcc-13` and `g++-13` as they have been verified in the official Docker container.
|
||||
|
||||
> **Note:** The system library path is typically located in one of the following directories: `~/.local/lib/`, `/usr/local/lib/`, `/usr/local/lib64/`, `/usr/lib/`, `/usr/lib64/`, and `/usr/lib/x86_64-linux-gnu/`. In the above example commands, `/usr/lib/x86_64-linux-gnu` is used. Please adjust the path according to your server configuration.
|
||||
|
||||
It is recommended to add the following to your `~/.bashrc` file to avoid setting these variables every time you open a new terminal:
|
||||
|
||||
<CodeGroup>
|
||||
```bash Persist in ~/.bashrc
|
||||
source .venv/bin/activate
|
||||
export SGLANG_USE_CPU_ENGINE=1
|
||||
export LD_LIBRARY_PATH=<YOUR-SYSTEM-LIBRARY-FOLDER>
|
||||
export LD_PRELOAD=<YOUR-LIBS-PATHS>
|
||||
```
|
||||
</CodeGroup>
|
||||
</Tab>
|
||||
</Tabs>
|
||||
|
||||
## Launch of the Serving Engine
|
||||
|
||||
Example command to launch SGLang serving:
|
||||
|
||||
<CodeGroup>
|
||||
```bash Launch Server
|
||||
python -m sglang.launch_server \
|
||||
--model <MODEL_ID_OR_PATH> \
|
||||
--trust-remote-code \
|
||||
--disable-overlap-schedule \
|
||||
--device cpu \
|
||||
--host 0.0.0.0 \
|
||||
--tp 6
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
> **Note:** For running W8A8 quantized models, please add the flag `--quantization w8a8_int8`.
|
||||
|
||||
> **Note:** The flag `--tp 6` specifies that tensor parallelism will be applied using 6 ranks (TP6). On a CPU platform, a TP rank means a sub-NUMA cluster (SNC). You can get the SNC count using `lscpu`. If the specified TP rank number differs from the total SNC count, the system will automatically utilize the first `n` SNCs — but `n` cannot exceed the total SNC number.
|
||||
>
|
||||
> To specify the cores to be used, set the environment variable `SGLANG_CPU_OMP_THREADS_BIND`. For example, to use the first 40 cores of each SNC on a Xeon® 6980P server (which has 43-43-42 cores on the 3 SNCs of a socket):
|
||||
|
||||
<CodeGroup>
|
||||
```bash Set Thread Binding
|
||||
export SGLANG_CPU_OMP_THREADS_BIND="0-39|43-82|86-125|128-167|171-210|214-253"
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
> Please beware that with `SGLANG_CPU_OMP_THREADS_BIND` set, the available memory amounts of the ranks may not be determined in advance. You may need to set `--max-total-tokens` to avoid out-of-memory errors.
|
||||
|
||||
> **Note:** For optimizing decoding with `torch.compile`, add the flag `--enable-torch-compile`. To specify the maximum batch size, set `--torch-compile-max-bs`. For example, `--enable-torch-compile --torch-compile-max-bs 4` uses `torch.compile` with a maximum batch size of 4. The maximum applicable batch size is 16.
|
||||
|
||||
> **Note:** A warmup step is automatically triggered when the service is started. The server is ready when you see the log `The server is fired up and ready to roll!`.
|
||||
|
||||
## Benchmarking with Requests
|
||||
|
||||
You can benchmark the performance via the `bench_serving` script.
|
||||
Run the command in another terminal. An example command would be:
|
||||
|
||||
<CodeGroup>
|
||||
```bash Run Benchmark
|
||||
python -m sglang.bench_serving \
|
||||
--dataset-name random \
|
||||
--random-input-len 1024 \
|
||||
--random-output-len 1024 \
|
||||
--num-prompts 1 \
|
||||
--request-rate inf \
|
||||
--random-range-ratio 1.0
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
Detailed parameter descriptions are available via the command:
|
||||
|
||||
<CodeGroup>
|
||||
```bash Benchmark Help
|
||||
python -m sglang.bench_serving -h
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
Additionally, requests can be formatted using
|
||||
[the OpenAI Completions API](../basic_usage/openai_api_completions)
|
||||
and sent via the command line (e.g., using `curl`) or through your own scripts.
|
||||
|
||||
## Example Usage Commands
|
||||
|
||||
Large Language Models can range from fewer than 1 billion to several hundred billion parameters.
|
||||
Dense models larger than 20B are expected to run on flagship 6th Gen Intel® Xeon® processors
|
||||
with dual sockets and a total of 6 sub-NUMA clusters. Dense models of approximately 10B parameters or fewer,
|
||||
or MoE (Mixture of Experts) models with fewer than 10B activated parameters, can run on more common
|
||||
4th generation or newer Intel® Xeon® processors, or utilize a single socket of the flagship 6th Gen Intel® Xeon® processors.
|
||||
|
||||
### Example: Running DeepSeek-V3.1-Terminus
|
||||
|
||||
<CodeGroup>
|
||||
```bash W8A8_INT8
|
||||
python -m sglang.launch_server \
|
||||
--model IntervitensInc/DeepSeek-V3.1-Terminus-Channel-int8 \
|
||||
--trust-remote-code \
|
||||
--disable-overlap-schedule \
|
||||
--device cpu \
|
||||
--quantization w8a8_int8 \
|
||||
--host 0.0.0.0 \
|
||||
--enable-torch-compile \
|
||||
--torch-compile-max-bs 4 \
|
||||
--tp 6
|
||||
```
|
||||
|
||||
```bash FP8
|
||||
python -m sglang.launch_server \
|
||||
--model deepseek-ai/DeepSeek-V3.1-Terminus \
|
||||
--trust-remote-code \
|
||||
--disable-overlap-schedule \
|
||||
--device cpu \
|
||||
--host 0.0.0.0 \
|
||||
--enable-torch-compile \
|
||||
--torch-compile-max-bs 4 \
|
||||
--tp 6
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
> **Note:** Please set `--torch-compile-max-bs` to the maximum desired batch size for your deployment, which can be up to 16. The value `4` in the examples is illustrative.
|
||||
|
||||
### Example: Running Llama-3.2-3B
|
||||
|
||||
<CodeGroup>
|
||||
```bash BF16
|
||||
python -m sglang.launch_server \
|
||||
--model meta-llama/Llama-3.2-3B-Instruct \
|
||||
--trust-remote-code \
|
||||
--disable-overlap-schedule \
|
||||
--device cpu \
|
||||
--host 0.0.0.0 \
|
||||
--enable-torch-compile \
|
||||
--torch-compile-max-bs 16 \
|
||||
--tp 2
|
||||
```
|
||||
|
||||
```bash W8A8_INT8
|
||||
python -m sglang.launch_server \
|
||||
--model RedHatAI/Llama-3.2-3B-quantized.w8a8 \
|
||||
--trust-remote-code \
|
||||
--disable-overlap-schedule \
|
||||
--device cpu \
|
||||
--quantization w8a8_int8 \
|
||||
--host 0.0.0.0 \
|
||||
--enable-torch-compile \
|
||||
--torch-compile-max-bs 16 \
|
||||
--tp 2
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
> **Note:** The `--torch-compile-max-bs` and `--tp` settings are examples that should be adjusted for your setup. For instance, use `--tp 3` to utilize 1 socket with 3 sub-NUMA clusters on an Intel® Xeon® 6980P server.
|
||||
|
||||
Once the server has been launched, you can test it using the `bench_serving` command or create
|
||||
your own commands or scripts following [the benchmarking example](#benchmarking-with-requests).
|
||||
@@ -0,0 +1,5 @@
|
||||
---
|
||||
title: NVIDIA GPUs
|
||||
---
|
||||
|
||||
Please refer to the [Installation Guide](/docs/get-started/installation) to get started with SGLang on NVIDIA GPUs.
|
||||
@@ -0,0 +1,102 @@
|
||||
---
|
||||
title: NVIDIA Jetson Orin
|
||||
description: Guide for installing and running SGLang on NVIDIA Jetson Orin devices.
|
||||
---
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Before starting, ensure the following:
|
||||
|
||||
- [NVIDIA Jetson AGX Orin Devkit](https://www.nvidia.com/en-us/autonomous-machines/embedded-systems/jetson-orin/) is set up with JetPack 6.1 or later.
|
||||
- CUDA Toolkit and cuDNN are installed.
|
||||
- Verify that the Jetson AGX Orin is in high-performance mode:
|
||||
|
||||
<CodeGroup>
|
||||
```bash
|
||||
sudo nvpmodel -m 0
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
## Installing and Running SGLang with Jetson Containers
|
||||
|
||||
1. **Clone the jetson-containers repository**
|
||||
|
||||
```bash
|
||||
git clone https://github.com/dusty-nv/jetson-containers.git
|
||||
```
|
||||
|
||||
2. **Run the installation script**
|
||||
|
||||
```bash
|
||||
bash jetson-containers/install.sh
|
||||
```
|
||||
|
||||
3. **Build the container image**
|
||||
|
||||
```bash
|
||||
jetson-containers build sglang
|
||||
```
|
||||
|
||||
4. **Run the container**
|
||||
|
||||
<Tabs>
|
||||
<Tab title="Using jetson-containers">
|
||||
```bash
|
||||
jetson-containers run $(autotag sglang)
|
||||
```
|
||||
</Tab>
|
||||
<Tab title="Using Docker manually">
|
||||
```bash
|
||||
docker run --runtime nvidia -it --rm --network=host IMAGE_NAME
|
||||
```
|
||||
</Tab>
|
||||
</Tabs>
|
||||
|
||||
## Running Inference
|
||||
|
||||
Launch the server:
|
||||
|
||||
<CodeGroup>
|
||||
```bash
|
||||
python -m sglang.launch_server \
|
||||
--model-path deepseek-ai/DeepSeek-R1-Distill-Llama-8B \
|
||||
--device cuda \
|
||||
--dtype half \
|
||||
--attention-backend flashinfer \
|
||||
--mem-fraction-static 0.8 \
|
||||
--context-length 8192
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
The quantization and limited context length (`--dtype half` `--context-length 8192`) are due to the limited computational resources in [Nvidia jetson kit](https://www.nvidia.com/en-us/autonomous-machines/embedded-systems/jetson-orin/). A detailed explanation can be found in [Server Arguments](../advanced_features/server_arguments).
|
||||
|
||||
After launching the engine, refer to [Chat completions](../basic_usage/openai_api_completions#Usage) to test the usability.
|
||||
|
||||
## Running Quantization with TorchAO
|
||||
|
||||
TorchAO is suggested to NVIDIA Jetson Orin.
|
||||
|
||||
<CodeGroup>
|
||||
```bash
|
||||
python -m sglang.launch_server \
|
||||
--model-path meta-llama/Meta-Llama-3.1-8B-Instruct \
|
||||
--device cuda \
|
||||
--dtype bfloat16 \
|
||||
--attention-backend flashinfer \
|
||||
--mem-fraction-static 0.8 \
|
||||
--context-length 8192 \
|
||||
--torchao-config int4wo-128
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
This enables TorchAO's int4 weight-only quantization with a 128-group size. The usage of `--torchao-config int4wo-128` is also for memory efficiency.
|
||||
|
||||
## Structured Output with XGrammar
|
||||
|
||||
Please refer to [SGLang doc structured output](../advanced_features/structured_outputs).
|
||||
|
||||
Thanks to the support from [Nurgaliyev Shakhizat](https://github.com/shahizat), [Dustin Franklin](https://github.com/dusty-nv) and [Johnny Núñez Cano](https://github.com/johnnynunez).
|
||||
|
||||
## References
|
||||
|
||||
- [NVIDIA Jetson AGX Orin Documentation](https://developer.nvidia.com/embedded/jetson-agx-orin)
|
||||
@@ -0,0 +1,12 @@
|
||||
---
|
||||
title: Hardware Platforms
|
||||
description: Platform-specific guides for running SGLang on GPUs, TPUs, NPUs, CPUs, and more.
|
||||
---
|
||||
|
||||
- [NVIDIA GPUs](./nvidia-gpus)
|
||||
- [AMD GPUs](./amd-gpus)
|
||||
- [Ascend NPUs](./ascend-npus/SGLang-installation-with-NPUs-support)
|
||||
- [CPU Server](./cpu-server)
|
||||
- [NVIDIA (Edge & Embedded)](./nvidia)
|
||||
- [TPU](./tpu)
|
||||
- [XPU](./xpu)
|
||||
@@ -0,0 +1,673 @@
|
||||
---
|
||||
title: "TPU"
|
||||
description: "SGLang supports high-performance TPU inference through the SGLang-JAX backend, which is specifically optimized for Google Cloud TPUs. The JAX-based implementation delivers exceptional throughput and low latency for Large Language Model (LLM) serving workloads on TPU hardware."
|
||||
---
|
||||
|
||||
<Note>
|
||||
SGLang TPU support is implemented via the SGLang-JAX backend, a dedicated JAX-based inference engine maintained as a separate repository at [sgl-project/sglang-jax](https://github.com/sgl-project/sglang-jax).
|
||||
</Note>
|
||||
|
||||
For TPU-specific issues or feature requests, please visit the [sglang-jax GitHub issues page](https://github.com/sgl-project/sglang-jax/issues).
|
||||
|
||||
---
|
||||
|
||||
# System Requirements
|
||||
|
||||
---
|
||||
|
||||
## Supported TPU Hardware
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="TPU v6e" icon="server">
|
||||
**32 GB** HBM Memory — Available on Google Cloud
|
||||
</Card>
|
||||
<Card title="TPU v7" icon="server">
|
||||
**96 GB per core** HBM Memory — Available on Google Cloud
|
||||
</Card>
|
||||
</CardGroup>
|
||||
|
||||
---
|
||||
|
||||
## Software Requirements
|
||||
|
||||
<CardGroup cols={3}>
|
||||
<Card title="Python" icon="python">
|
||||
Version **3.12 or higher**
|
||||
</Card>
|
||||
<Card title="JAX" icon="code">
|
||||
**Latest version** with TPU support
|
||||
</Card>
|
||||
<Card title="Environment" icon="cloud">
|
||||
Google Cloud TPU VM or compatible TPU runtime. **Optional:** SkyPilot for simplified cloud deployment.
|
||||
</Card>
|
||||
</CardGroup>
|
||||
|
||||
---
|
||||
|
||||
# Feature Support Matrix
|
||||
|
||||
SGLang-JAX provides comprehensive TPU-optimized features for production LLM serving:
|
||||
|
||||
### Supported Features
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "34%"}} />
|
||||
<col style={{width: "33%"}} />
|
||||
<col style={{width: "33%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>**Feature**</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>**Support Status**</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>**Description**</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>High-Throughput Continuous Batching</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Dynamic request batching for maximum TPU utilization</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Radix Tree KV Cache</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Memory-efficient prefix sharing between requests</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>FlashAttention Backend</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>TPU-optimized attention kernel for long sequences</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Tensor Parallelism</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Distribute models across multiple TPU cores</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Paged Attention</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Flexible KV cache management with paging</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Speculative Decoding (EAGLE/EAGLE3)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>20-40% throughput improvement for compatible models</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Chunked Prefill</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Mixed prefill-decode batching</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>OpenAI-Compatible API</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Drop-in replacement for OpenAI API</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Data Parallel Attention</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>🚧</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>In development — Attention computation with data parallelism</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Quantization</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>🚧</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>In development — Model quantization for reduced memory usage</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Multi-LoRA</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>🚧</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>In development — Serve multiple LoRA adapters simultaneously</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
---
|
||||
|
||||
### Attention Backend Comparison
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "20%"}} />
|
||||
<col style={{width: "20%"}} />
|
||||
<col style={{width: "20%"}} />
|
||||
<col style={{width: "20%"}} />
|
||||
<col style={{width: "20%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>**Backend**</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>**Paged Attention**</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>**Spec Decoding**</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>**MLA**</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>**Sliding Window**</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>FlashAttention (fa)</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>✅</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Native</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>❌</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>❌</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
<Note>
|
||||
FlashAttention backend is recommended for production workloads due to superior memory efficiency and performance.
|
||||
</Note>
|
||||
|
||||
---
|
||||
|
||||
# Optimized Model List
|
||||
|
||||
The following models have been tested and optimized for TPU deployment:
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "50%"}} />
|
||||
<col style={{width: "50%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>**Model Family**</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>**Performance Status**</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen 3</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>⭐ Recommended for production</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen 3 MoE</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>⭐ Best performance</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen 2</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Needs improvement</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen 2 MoE</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Needs improvement</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen 1.5</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Needs improvement</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Llama/LLaMA</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Needs improvement</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Grok-2</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Needs improvement</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Gemma 2</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Verified on TPU</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Bailing MoE</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Needs improvement</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
---
|
||||
|
||||
# Installation
|
||||
|
||||
<Tabs>
|
||||
<Tab title="PyPI (Recommended)">
|
||||
```bash
|
||||
pip install sglang-jax
|
||||
```
|
||||
</Tab>
|
||||
<Tab title="From Source">
|
||||
```bash
|
||||
git clone https://github.com/sgl-project/sglang-jax
|
||||
cd sglang-jax
|
||||
uv venv --python 3.12 && source .venv/bin/activate
|
||||
uv pip install -e "python[all]"
|
||||
```
|
||||
</Tab>
|
||||
<Tab title="Docker">
|
||||
<Warning>
|
||||
Docker support for TPU is currently under development. Please use PyPI or source installation methods.
|
||||
</Warning>
|
||||
</Tab>
|
||||
<Tab title="SkyPilot (Cloud TPU)">
|
||||
SkyPilot provides simplified deployment on Google Cloud TPU:
|
||||
|
||||
1. **Install SkyPilot and configure GCP access**
|
||||
|
||||
See the [SkyPilot documentation](https://docs.skypilot.co/en/latest/) for setup instructions.
|
||||
|
||||
2. **Create a SkyPilot configuration file**
|
||||
|
||||
Create a SkyPilot YAML file: `sglang-jax.sky.yaml`
|
||||
|
||||
3. **Launch your TPU cluster**
|
||||
|
||||
```bash
|
||||
# Standard deployment
|
||||
sky launch -c sglang-jax sglang-jax.sky.yaml --infra=gcp
|
||||
|
||||
# With spot instances for cost savings
|
||||
sky launch -c sglang-jax sglang-jax.sky.yaml --infra=gcp --use-spot
|
||||
```
|
||||
</Tab>
|
||||
</Tabs>
|
||||
|
||||
---
|
||||
|
||||
# Launch the Serving Engine
|
||||
|
||||
<Tabs>
|
||||
<Tab title="Basic: Qwen-7B">
|
||||
```bash
|
||||
JAX_COMPILATION_CACHE_DIR=/tmp/jit_cache python3 -u -m sgl_jax.launch_server \
|
||||
--model-path Qwen/Qwen-7B-Chat \
|
||||
--trust-remote-code \
|
||||
--dist-init-addr=0.0.0.0:10011 \
|
||||
--nnodes=1 \
|
||||
--tp-size=4 \
|
||||
--device=tpu \
|
||||
--random-seed=3 \
|
||||
--node-rank=0 \
|
||||
--mem-fraction-static=0.8 \
|
||||
--max-prefill-tokens=8192 \
|
||||
--download-dir=/tmp \
|
||||
--dtype=bfloat16 \
|
||||
--skip-server-warmup \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
<AccordionGroup>
|
||||
<Accordion title="Key Parameters Explained" icon="sliders">
|
||||
<ParamField path="JAX_COMPILATION_CACHE_DIR" type="string">
|
||||
Enables JIT compilation caching to accelerate server startup on subsequent runs. Recommended: `/tmp/jit_cache`
|
||||
</ParamField>
|
||||
<ParamField path="--tp-size" type="integer" default="1">
|
||||
Tensor parallelism size; match this to your TPU core count (typically `1`, `4`, or `8`).
|
||||
</ParamField>
|
||||
<ParamField path="--device" type="string" default="tpu">
|
||||
Specifies TPU device. This is the default for `sglang-jax`.
|
||||
</ParamField>
|
||||
<ParamField path="--dtype" type="string" default="bfloat16">
|
||||
Uses bfloat16 precision, which TPUs are optimized for.
|
||||
</ParamField>
|
||||
<ParamField path="--mem-fraction-static" type="float" default="0.8">
|
||||
Allocates this fraction of TPU HBM for static memory. Adjustable from `0.2` to `0.9`.
|
||||
</ParamField>
|
||||
<ParamField path="--max-prefill-tokens" type="integer" default="8192">
|
||||
Maximum number of tokens processed in the prefill phase.
|
||||
</ParamField>
|
||||
</Accordion>
|
||||
</AccordionGroup>
|
||||
</Tab>
|
||||
<Tab title="High-Performance: Qwen3-8B">
|
||||
For production workloads with optimal throughput:
|
||||
|
||||
```bash
|
||||
python3 -u -m sgl_jax.launch_server \
|
||||
--model-path Qwen/Qwen3-8B \
|
||||
--trust-remote-code \
|
||||
--tp-size=4 \
|
||||
--device=tpu \
|
||||
--mem-fraction-static=0.8 \
|
||||
--chunked-prefill-size=2048 \
|
||||
--dtype=bfloat16 \
|
||||
--max-running-requests=256 \
|
||||
--page-size=128 \
|
||||
--attention-backend=fa
|
||||
```
|
||||
</Tab>
|
||||
<Tab title="Speculative Decoding (EAGLE3)">
|
||||
Speculative decoding can improve throughput by 20-40% for compatible models:
|
||||
|
||||
```bash
|
||||
python3 -u -m sgl_jax.launch_server \
|
||||
--model-path Qwen/Qwen3-32B \
|
||||
--trust-remote-code \
|
||||
--device=tpu \
|
||||
--tp-size=4 \
|
||||
--mem-fraction-static=0.8 \
|
||||
--max-prefill-tokens=4096 \
|
||||
--attention-backend=fa \
|
||||
--dtype=bfloat16 \
|
||||
--port=30000 \
|
||||
--host=0.0.0.0 \
|
||||
--disable-overlap-schedule \
|
||||
--speculative-algorithm=EAGLE3 \
|
||||
--speculative-draft-model-path=AngelSlim/Qwen3-32B_eagle3 \
|
||||
--page-size=64 \
|
||||
--speculative-eagle-topk=1 \
|
||||
--speculative-num-steps=3 \
|
||||
--speculative-num-draft-tokens=4
|
||||
```
|
||||
|
||||
<Note>
|
||||
Speculative decoding is currently supported for Qwen3 and LLaMA model families. See the [Speculative Decoding documentation](../advanced_features/speculative_decoding) for detailed configuration guidance.
|
||||
</Note>
|
||||
</Tab>
|
||||
<Tab title="Multi-Node Distributed">
|
||||
For large models requiring multiple TPU VMs:
|
||||
|
||||
```bash
|
||||
# Node 0 (coordinator)
|
||||
python3 -m sgl_jax.launch_server \
|
||||
--model-path MODEL_PATH \
|
||||
--dist-init-addr=NODE0_IP:10011 \
|
||||
--nnodes=2 \
|
||||
--node-rank=0 \
|
||||
--tp-size=8 \
|
||||
[other parameters...]
|
||||
|
||||
# Node 1 (worker)
|
||||
python3 -m sgl_jax.launch_server \
|
||||
--model-path MODEL_PATH \
|
||||
--dist-init-addr=NODE0_IP:10011 \
|
||||
--nnodes=2 \
|
||||
--node-rank=1 \
|
||||
--tp-size=8 \
|
||||
[other parameters...]
|
||||
```
|
||||
</Tab>
|
||||
</Tabs>
|
||||
|
||||
---
|
||||
|
||||
# Benchmarking with Requests
|
||||
|
||||
<Tabs>
|
||||
<Tab title="Throughput Testing">
|
||||
Basic throughput benchmark:
|
||||
|
||||
```bash
|
||||
python3 -m sgl_jax.bench_serving \
|
||||
--backend sgl-jax \
|
||||
--dataset-name random \
|
||||
--num-prompts=100 \
|
||||
--random-input=512 \
|
||||
--random-output=128 \
|
||||
--max-concurrency=8 \
|
||||
--random-range-ratio=1 \
|
||||
--warmup-requests=0
|
||||
```
|
||||
</Tab>
|
||||
<Tab title="Latency Testing">
|
||||
Measure single-batch latency:
|
||||
|
||||
```bash
|
||||
python3 -m sgl_jax.bench_one_batch_server \
|
||||
--base-url http://127.0.0.1:30000 \
|
||||
--model-path Qwen/Qwen-7B-Chat \
|
||||
--batch-size=32 \
|
||||
--input-len=256 \
|
||||
--output-len=32
|
||||
```
|
||||
</Tab>
|
||||
<Tab title="Comprehensive Benchmark Script">
|
||||
For systematic performance evaluation across different configurations:
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
set -e
|
||||
|
||||
backend=${1:-sgl-jax}
|
||||
num_prompts_per_concurrency=3
|
||||
input_seq_lens=(1024 4096 8192)
|
||||
output_seq_lens=(1 1024)
|
||||
max_concurrencies=(8 16 32 64 128 256)
|
||||
|
||||
for input_seq_len in "${input_seq_lens[@]}"; do
|
||||
for output_seq_len in "${output_seq_lens[@]}"; do
|
||||
echo "======================================="
|
||||
echo "Testing ISL/OSL: $input_seq_len/$output_seq_len"
|
||||
echo "======================================="
|
||||
for max_concurrency in "${max_concurrencies[@]}"; do
|
||||
num_prompts=$((num_prompts_per_concurrency * max_concurrency))
|
||||
python3 -m sgl_jax.bench_serving \
|
||||
--backend ${backend} \
|
||||
--dataset-name random \
|
||||
--num-prompts ${num_prompts} \
|
||||
--random-input ${input_seq_len} \
|
||||
--random-output ${output_seq_len} \
|
||||
--max-concurrency ${max_concurrency} \
|
||||
--random-range-ratio 1 \
|
||||
--disable-ignore-eos \
|
||||
--warmup-requests 0
|
||||
done
|
||||
done
|
||||
done
|
||||
```
|
||||
|
||||
For detailed help on all benchmark parameters:
|
||||
|
||||
```bash
|
||||
python3 -m sgl_jax.bench_serving --help
|
||||
```
|
||||
|
||||
See the [Benchmark and Profiling Guide](../developer_guide/benchmark_and_profiling) for advanced benchmarking techniques and profiling with JAX Profiler.
|
||||
</Tab>
|
||||
</Tabs>
|
||||
|
||||
---
|
||||
|
||||
# Performance Optimization
|
||||
|
||||
<AccordionGroup>
|
||||
<Accordion title="Memory Optimization" icon="memory">
|
||||
**Reduce memory usage:**
|
||||
- Lower `--mem-fraction-static` (from `0.8` → `0.5` → `0.3`)
|
||||
- Decrease `--max-prefill-tokens` (from `16384` → `8192` → `4096`)
|
||||
- Reduce `--max-running-requests`
|
||||
|
||||
**Handle OOM errors:**
|
||||
- Start with conservative memory settings (`--mem-fraction-static=0.5`)
|
||||
- Gradually increase until you find the optimal balance
|
||||
- Increase `--page-size` for better memory locality (`1` → `16` → `64` → `128`)
|
||||
</Accordion>
|
||||
<Accordion title="Throughput Optimization" icon="chart-line">
|
||||
To maximize tokens per second:
|
||||
- Use FlashAttention backend: `--attention-backend=fa`
|
||||
- Enable speculative decoding (EAGLE3) for Qwen3 models (20-40% improvement)
|
||||
- Increase `--max-running-requests` to `256+`
|
||||
- Set `--mem-fraction-static` to `0.8+` (if memory allows)
|
||||
- Use larger page sizes (`64-128`)
|
||||
- Enable chunked prefill: `--chunked-prefill-size=2048`
|
||||
</Accordion>
|
||||
<Accordion title="Latency Optimization" icon="gauge-high">
|
||||
To minimize time-to-first-token (TTFT) and inter-token latency:
|
||||
- Reduce `--page-size` to `1-4`
|
||||
- Lower `--max-running-requests` (`16-32`) for smaller batches
|
||||
- Reduce `--chunked-prefill-size`
|
||||
- Use conservative memory settings to avoid GC pauses
|
||||
</Accordion>
|
||||
<Accordion title="TPU-Specific Optimizations" icon="microchip">
|
||||
**JIT Compilation Cache:**
|
||||
|
||||
```bash
|
||||
export JAX_COMPILATION_CACHE_DIR=/tmp/jit_cache
|
||||
```
|
||||
|
||||
Always set this environment variable to cache compiled kernels and accelerate server startup.
|
||||
|
||||
**Data Type Optimization:** Use `--dtype=bfloat16` for TPU native optimization. TPUs are specifically designed for bfloat16 computations.
|
||||
|
||||
**Tensor Parallelism:** Match `--tp-size` to your TPU core configuration (`1`, `4`, or `8`) for optimal model distribution.
|
||||
|
||||
**Attention Backend:** Always use `--attention-backend=fa` (FlashAttention) for production workloads.
|
||||
</Accordion>
|
||||
</AccordionGroup>
|
||||
|
||||
---
|
||||
|
||||
# Troubleshooting
|
||||
|
||||
<AccordionGroup>
|
||||
<Accordion title="OOM (Out of Memory) Errors" icon="triangle-exclamation">
|
||||
If you encounter out-of-memory errors:
|
||||
|
||||
1. **Reduce mem-fraction-static**
|
||||
|
||||
Lower `--mem-fraction-static` from `0.8` to `0.5` or lower.
|
||||
|
||||
2. **Decrease max-prefill-tokens**
|
||||
|
||||
Decrease `--max-prefill-tokens` from `8192` to `4096` or `2048`.
|
||||
|
||||
3. **Lower max-running-requests**
|
||||
|
||||
Lower `--max-running-requests` to reduce concurrent batch size.
|
||||
|
||||
4. **Increase page-size**
|
||||
|
||||
Increase `--page-size` for better memory layout efficiency.
|
||||
</Accordion>
|
||||
<Accordion title="Slow Compilation / Long Startup" icon="clock">
|
||||
If the server takes too long to start:
|
||||
|
||||
<Check>Ensure `JAX_COMPILATION_CACHE_DIR` is properly set</Check>
|
||||
<Check>Understand that the first run requires JIT compilation — this is normal</Check>
|
||||
<Check>Subsequent runs will be significantly faster with cached compilations</Check>
|
||||
<Check>Consider using `--skip-server-warmup` to defer compilation until first request</Check>
|
||||
</Accordion>
|
||||
<Accordion title="Low Throughput" icon="gauge">
|
||||
If you're not achieving expected throughput:
|
||||
|
||||
<Check>Verify `--tp-size` matches your TPU core configuration</Check>
|
||||
<Check>Check that `--attention-backend=fa` is enabled</Check>
|
||||
<Check>Increase `--max-running-requests` to enable larger batch formation</Check>
|
||||
<Check>Consider enabling speculative decoding for compatible models</Check>
|
||||
<Check>Ensure memory settings allow for sufficient batch sizes</Check>
|
||||
</Accordion>
|
||||
<Accordion title="Connection Issues" icon="network-wired">
|
||||
If clients cannot connect to the server:
|
||||
|
||||
<Check>Ensure `--host=0.0.0.0` for external access (not just `127.0.0.1`)</Check>
|
||||
<Check>Verify firewall rules allow traffic on the specified port (default: `30000`)</Check>
|
||||
<Check>Check that the server process is running: `curl http://localhost:30000/health`</Check>
|
||||
</Accordion>
|
||||
</AccordionGroup>
|
||||
|
||||
---
|
||||
|
||||
# Advanced Features
|
||||
|
||||
<AccordionGroup>
|
||||
<Accordion title="Speculative Decoding" icon="bolt">
|
||||
SGLang-JAX supports EAGLE and EAGLE3 speculative decoding algorithms for Qwen3 and LLaMA model families. Speculative decoding can improve throughput by 20-40% without affecting output quality.
|
||||
|
||||
See the [Speculative Decoding documentation](../advanced_features/speculative_decoding) for detailed configuration and supported model combinations.
|
||||
</Accordion>
|
||||
<Accordion title="Chunked Prefill" icon="layer-group">
|
||||
Enable mixed prefill-decode batching for better TPU utilization:
|
||||
|
||||
```bash
|
||||
--chunked-prefill-size=2048 --enable-mixed-chunk
|
||||
```
|
||||
|
||||
This allows the scheduler to mix prefill operations with decode operations in the same batch, improving overall throughput.
|
||||
</Accordion>
|
||||
<Accordion title="Custom Attention Backends" icon="puzzle-piece">
|
||||
SGLang-JAX supports a plugin-based attention backend system. You can implement custom attention kernels optimized for specific use cases.
|
||||
|
||||
See the [Attention Backend documentation](https://github.com/sgl-project/sglang-jax/tree/main/docs) for implementation details.
|
||||
</Accordion>
|
||||
<Accordion title="Environment Verification" icon="circle-check">
|
||||
Verify your TPU setup before deploying:
|
||||
|
||||
```bash
|
||||
python -c "from sgl_jax import check_env; check_env.check_env()"
|
||||
```
|
||||
|
||||
This command checks:
|
||||
- Installed package versions
|
||||
- TPU device availability and specifications
|
||||
- System resources and configuration
|
||||
- Compatibility of settings
|
||||
</Accordion>
|
||||
</AccordionGroup>
|
||||
|
||||
---
|
||||
|
||||
# Contributing
|
||||
|
||||
We welcome contributions to improve TPU support in SGLang-JAX!
|
||||
|
||||
<Note>
|
||||
Check the [Development Roadmap](https://github.com/sgl-project/sglang-jax) to see planned features and find opportunities to contribute new functionality.
|
||||
</Note>
|
||||
|
||||
Current contribution areas include:
|
||||
|
||||
- Performance optimizations for specific TPU generations
|
||||
- Support for additional model architectures
|
||||
- Documentation improvements and examples
|
||||
- Bug reports and fixes
|
||||
- Benchmark results and performance analysis
|
||||
|
||||
<CardGroup cols={3}>
|
||||
<Card title="Repository" icon="github" href="https://github.com/sgl-project/sglang-jax">
|
||||
Visit the sglang-jax repository
|
||||
</Card>
|
||||
<Card title="Contribution Guide" icon="book" href="https://github.com/sgl-project/sglang-jax/blob/main/CONTRIBUTING.md">
|
||||
Read the Contribution Guide
|
||||
</Card>
|
||||
<Card title="Slack Community" icon="slack" href="https://slack.sglang.io/">
|
||||
Join the SGL-JAX Slack community for discussions
|
||||
</Card>
|
||||
</CardGroup>
|
||||
|
||||
---
|
||||
|
||||
## Testing on TPU
|
||||
|
||||
For contributors who need TPU access for testing:
|
||||
|
||||
- Refer to the [TPU Resources Guide](https://cloud.google.com/tpu/docs/managing-tpus-tpu-vm) for information on accessing TPU hardware
|
||||
- Use SkyPilot with spot instances for cost-effective testing
|
||||
- Follow the [Benchmark and Profiling Guide](../developer_guide/benchmark_and_profiling) for performance validation
|
||||
|
||||
---
|
||||
|
||||
# References
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="SGLang-JAX Repository" icon="github" href="https://github.com/sgl-project/sglang-jax">
|
||||
Source code and issue tracker for the JAX TPU backend.
|
||||
</Card>
|
||||
<Card title="SGLang-JAX Installation Guide" icon="book-open" href="https://github.com/sgl-project/sglang-jax?tab=readme-ov-file#installation">
|
||||
Step-by-step installation instructions.
|
||||
</Card>
|
||||
<Card title="Qwen Models Quick Start" icon="rocket" href="https://github.com/sgl-project/sglang-jax/tree/main/docs">
|
||||
Get up and running quickly with the Qwen model family.
|
||||
</Card>
|
||||
<Card title="Benchmark and Profiling Guide" icon="chart-bar" href="../developer_guide/benchmark_and_profiling">
|
||||
Advanced benchmarking techniques and JAX Profiler usage.
|
||||
</Card>
|
||||
<Card title="Speculative Decoding" icon="forward" href="../advanced_features/speculative_decoding">
|
||||
EAGLE and EAGLE3 speculative decoding configuration.
|
||||
</Card>
|
||||
<Card title="JAX Documentation" icon="code" href="https://jax.readthedocs.io">
|
||||
Official JAX documentation and API reference.
|
||||
</Card>
|
||||
<Card title="Google Cloud TPU Docs" icon="cloud" href="https://cloud.google.com/tpu/docs">
|
||||
Google Cloud TPU product documentation.
|
||||
</Card>
|
||||
<Card title="SkyPilot Documentation" icon="paper-plane" href="https://docs.skypilot.co/en/latest/">
|
||||
Simplified cloud deployment with SkyPilot.
|
||||
</Card>
|
||||
</CardGroup>
|
||||
@@ -0,0 +1,146 @@
|
||||
---
|
||||
title: XPU
|
||||
sidebarTitle: Intel GPUs (XPU)
|
||||
---
|
||||
|
||||
The document addresses how to set up the [SGLang](https://github.com/sgl-project/sglang) environment and run LLM inference on Intel GPU, [see more context about Intel GPU support within PyTorch ecosystem](https://docs.pytorch.org/docs/stable/notes/get_start_xpu.html).
|
||||
|
||||
Specifically, SGLang is optimized for:
|
||||
- [Intel® Arc™ Pro B-Series Graphics](https://www.intel.com/content/www/us/en/ark/products/series/242616/intel-arc-pro-b-series-graphics.html)
|
||||
- [Intel® Arc™ B-Series Graphics](https://www.intel.com/content/www/us/en/ark/products/series/240391/intel-arc-b-series-graphics.html).
|
||||
|
||||
## Optimized Model List
|
||||
|
||||
A list of LLMs have been optimized on Intel GPU, and more are on the way:
|
||||
|
||||
<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
|
||||
<colgroup>
|
||||
<col style={{width: "50%"}} />
|
||||
<col style={{width: "50%"}} />
|
||||
</colgroup>
|
||||
<thead>
|
||||
<tr style={{borderBottom: "2px solid #d55816"}}>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Model Name</th>
|
||||
<th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>BF16</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Llama-3.2-3B</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>[meta-llama/Llama-3.2-3B-Instruct](https://huggingface.co/meta-llama/Llama-3.2-3B-Instruct)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Llama-3.1-8B</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>[meta-llama/Llama-3.1-8B-Instruct](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct)</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Qwen2.5-1.5B</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>[Qwen/Qwen2.5-1.5B](https://huggingface.co/Qwen/Qwen2.5-1.5B)</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
<Note>The model identifiers listed in the table above have been verified on [Intel® Arc™ B580 Graphics](https://www.intel.com/content/www/us/en/products/sku/241598/intel-arc-b580-graphics/specifications.html).</Note>
|
||||
|
||||
## Installation
|
||||
|
||||
<Tabs>
|
||||
|
||||
<Tab title="Source">
|
||||
|
||||
Currently SGLang XPU only supports installation from source. Please refer to [“Getting Started on Intel GPU”](https://docs.pytorch.org/docs/stable/notes/get_start_xpu.html) to install XPU dependency.
|
||||
|
||||
1. **Creation & Activation**
|
||||
|
||||
Create and activate a conda environment.
|
||||
|
||||
```bash
|
||||
conda create -n sgl-xpu python=3.12 -y
|
||||
conda activate sgl-xpu
|
||||
```
|
||||
|
||||
2. **Install PyTorch and Dependencies**
|
||||
|
||||
Set PyTorch XPU as primary pip install channel to avoid installing the larger CUDA-enabled version and prevent potential runtime issues.
|
||||
|
||||
```bash
|
||||
pip3 install torch==2.9.0+xpu torchao torchvision torchaudio pytorch-triton-xpu==3.5.0 --index-url https://download.pytorch.org/whl/xpu
|
||||
pip3 install xgrammar --no-deps # xgrammar will introduce CUDA-enabled triton which might conflict with XPU
|
||||
```
|
||||
|
||||
3. **Cloning**
|
||||
|
||||
Clone the SGLang code
|
||||
|
||||
```bash
|
||||
git clone https://github.com/sgl-project/sglang.git
|
||||
cd sglang
|
||||
git checkout <YOUR-DESIRED-VERSION>
|
||||
```
|
||||
|
||||
4. **Configure Build File**
|
||||
|
||||
Use dedicated toml file
|
||||
|
||||
```bash
|
||||
cd python
|
||||
cp pyproject_xpu.toml pyproject.toml
|
||||
```
|
||||
|
||||
5. **Build and Install**
|
||||
|
||||
Install SGLang dependent libs, and build SGLang main package
|
||||
|
||||
```bash
|
||||
pip install --upgrade pip setuptools
|
||||
pip install -v .
|
||||
```
|
||||
</Tab>
|
||||
|
||||
<Tab title="Docker">
|
||||
<Info>The docker for XPU is under active development. Please stay tuned.</Info>
|
||||
</Tab>
|
||||
</Tabs>
|
||||
## Launch of the Serving Engine
|
||||
|
||||
Example command to launch SGLang serving:
|
||||
|
||||
<CodeGroup>
|
||||
```bash
|
||||
python -m sglang.launch_server \
|
||||
--model <MODEL_ID_OR_PATH> \
|
||||
--trust-remote-code \
|
||||
--disable-overlap-schedule \
|
||||
--device xpu \
|
||||
--host 0.0.0.0 \
|
||||
--tp 2 \ # using multi GPUs
|
||||
--attention-backend intel_xpu \ # using intel optimized XPU attention backend
|
||||
--page-size \ # intel_xpu attention backend supports [32, 64, 128]
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
## Benchmarking with Requests
|
||||
|
||||
You can benchmark the performance via the `bench_serving` script. Run the command in another terminal.
|
||||
|
||||
<CodeGroup>
|
||||
```bash
|
||||
python -m sglang.bench_serving \
|
||||
--dataset-name random \
|
||||
--random-input-len 1024 \
|
||||
--random-output-len 1024 \
|
||||
--num-prompts 1 \
|
||||
--request-rate inf \
|
||||
--random-range-ratio 1.0
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
The detail explanations of the parameters can be looked up by the command:
|
||||
|
||||
<CodeGroup>
|
||||
```bash
|
||||
python -m sglang.bench_serving -h
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
Additionally, the requests can be formed with [OpenAI Completions API](../basic_usage/openai_api_completions) and sent via the command line (e.g. using `curl`) or via your own script.
|
||||
Reference in New Issue
Block a user