diff --git a/docs_new/docs/hardware-platforms/cpu_server.mdx b/docs_new/docs/hardware-platforms/cpu_server.mdx index 8d2d36b61..743766656 100644 --- a/docs_new/docs/hardware-platforms/cpu_server.mdx +++ b/docs_new/docs/hardware-platforms/cpu_server.mdx @@ -222,12 +222,12 @@ Notes: Example command to launch SGLang serving: ```bash Launch Server -python -m sglang.launch_server \ - --model \ - --trust-remote-code \ - --disable-overlap-schedule \ - --device cpu \ - --host 0.0.0.0 \ +sglang serve \ + --model-path \ + --trust-remote-code \ + --disable-overlap-schedule \ + --device cpu \ + --host 0.0.0.0 \ --tp 6 ``` @@ -320,64 +320,64 @@ or MoE (Mixture of Experts) models with fewer than 10B activated parameters, can An example command to launch service of W8A8_INT8 DeepSeek-V3.1-Terminus on a Xeon® 6980P server: ```bash W8A8_INT8 -python -m sglang.launch_server \ - --model IntervitensInc/DeepSeek-V3.1-Terminus-Channel-int8 \ - --trust-remote-code \ - --disable-overlap-schedule \ - --device cpu \ - --quantization w8a8_int8 \ - --host 0.0.0.0 \ - --enable-torch-compile \ - --torch-compile-max-bs 4 \ +sglang serve \ + --model-path IntervitensInc/DeepSeek-V3.1-Terminus-Channel-int8 \ + --trust-remote-code \ + --disable-overlap-schedule \ + --device cpu \ + --quantization w8a8_int8 \ + --enable-torch-compile \ + --torch-compile-max-bs 4 \ + --host 0.0.0.0 \ --tp 6 ``` Similarly, an example command to launch service of FP8 DeepSeek-V3.1-Terminus would be: ```bash FP8 -python -m sglang.launch_server \ - --model deepseek-ai/DeepSeek-V3.1-Terminus \ - --trust-remote-code \ - --disable-overlap-schedule \ - --device cpu \ - --host 0.0.0.0 \ - --enable-torch-compile \ - --torch-compile-max-bs 4 \ +sglang serve \ + --model-path deepseek-ai/DeepSeek-V3.1-Terminus \ + --trust-remote-code \ + --disable-overlap-schedule \ + --device cpu \ + --enable-torch-compile \ + --torch-compile-max-bs 4 \ + --host 0.0.0.0 \ --tp 6 ``` -Note: Please set `--torch-compile-max-bs` to the maximum desired batch size for your deployment, -which can be up to 16. The value `4` in the examples is illustrative. +Note: Please set `--torch-compile-max-bs` to the maximum desired batch size for your deployment. +The value `4` in the examples is illustrative. ### Example: Running Llama-3.2-3B An example command to launch service of Llama-3.2-3B with BF16 precision: ```bash BF16 -python -m sglang.launch_server \ - --model meta-llama/Llama-3.2-3B-Instruct \ - --trust-remote-code \ - --disable-overlap-schedule \ - --device cpu \ - --host 0.0.0.0 \ - --enable-torch-compile \ - --torch-compile-max-bs 16 \ - --tp 2 +sglang serve \ + --model-path meta-llama/Llama-3.2-3B-Instruct \ + --trust-remote-code \ + --disable-overlap-schedule \ + --device cpu \ + --enable-torch-compile \ + --torch-compile-max-bs 16 \ + --host 0.0.0.0 \ + --tp 3 ``` The example command to launch service of W8A8_INT8 version of Llama-3.2-3B: ```bash W8A8_INT8 -python -m sglang.launch_server \ - --model RedHatAI/Llama-3.2-3B-quantized.w8a8 \ - --trust-remote-code \ - --disable-overlap-schedule \ - --device cpu \ - --quantization w8a8_int8 \ - --host 0.0.0.0 \ - --enable-torch-compile \ - --torch-compile-max-bs 16 \ - --tp 2 +sglang serve \ + --model-path RedHatAI/Llama-3.2-3B-quantized.w8a8 \ + --trust-remote-code \ + --disable-overlap-schedule \ + --device cpu \ + --quantization w8a8_int8 \ + --enable-torch-compile \ + --torch-compile-max-bs 16 \ + --host 0.0.0.0 \ + --tp 3 ``` Note: The `--torch-compile-max-bs` and `--tp` settings are examples that should be adjusted for your setup. diff --git a/docs_new/docs/hardware-platforms/xpu.mdx b/docs_new/docs/hardware-platforms/xpu.mdx index 46a8d1fb2..0c9f0ac83 100644 --- a/docs_new/docs/hardware-platforms/xpu.mdx +++ b/docs_new/docs/hardware-platforms/xpu.mdx @@ -71,15 +71,40 @@ pip install -v . --extra-index-url https://download.pytorch.org/whl/xpu ### Install Using Docker -The docker for XPU is under active development. Please stay tuned. +[The SGLang XPU Dockerfile](https://github.com/sgl-project/sglang/blob/main/docker/xpu.Dockerfile) is provided to facilitate the installation. +Replace `` below with your [HuggingFace access token](https://huggingface.co/docs/hub/en/security-tokens). + +```bash Command +# Clone the SGLang repository +git clone https://github.com/sgl-project/sglang.git +cd sglang/docker + +# Build the docker image +docker build -t sglang-xpu:latest -f xpu.Dockerfile . + +# Initiate a docker container +docker run \ + -it \ + --privileged \ + --ipc=host \ + --network=host \ + --group-add $(getent group video | cut -d: -f3) \ + --device /dev/dri \ + -v /dev/dri/by-path:/dev/dri/by-path \ + -v /dev/shm:/dev/shm \ + -v ~/.cache/huggingface:/root/.cache/huggingface \ + -p 30000:30000 \ + -e "HF_TOKEN=" \ + sglang-xpu:latest /bin/bash +``` ## Launch of the Serving Engine Example command to launch SGLang serving: ```bash -python -m sglang.launch_server \ - --model \ +sglang serve \ + --model-path \ --trust-remote-code \ --disable-overlap-schedule \ --device xpu \