From 27071e0a4314546e39ff03349594075d546fc279 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=D0=90=D1=80=D1=82=D0=B5=D0=BC=20=D0=A1=D0=B0=D0=B2=D0=BA?= =?UTF-8?q?=D0=B8=D0=BD?= <58187114+OrangeRedeng@users.noreply.github.com> Date: Sat, 28 Mar 2026 21:42:21 +0300 Subject: [PATCH] [NPU] Update quantization&CI documentation (#21100) Co-authored-by: Tamir Baydasov <41994229+TamirBaydasov@users.noreply.github.com> --- .claude/skills/write-sglang-test/SKILL.md | 20 ++- docs/advanced_features/quantization.md | 119 ++++++++++++++---- docs/basic_usage/deepseek_v3.md | 2 +- docs/diffusion/installation.md | 2 +- docs/diffusion/quantization.md | 68 +++++++++- docs/get_started/install.md | 2 +- docs/index.rst | 2 +- .../{ => ascend}/ascend_contribution_guide.md | 17 ++- docs/platforms/{ => ascend}/ascend_npu.md | 0 .../{ => ascend}/ascend_npu_best_practice.md | 0 .../ascend_npu_deepseek_example.md | 0 .../ascend_npu_environment_variables.md | 0 .../{ => ascend}/ascend_npu_glm5_examples.md | 2 +- .../ascend/ascend_npu_quantization.md | 52 ++++++++ .../ascend_npu_qwen3_5_examples.md | 2 +- .../{ => ascend}/ascend_npu_qwen3_examples.md | 0 .../{ => ascend}/ascend_npu_support.rst | 1 + .../ascend_npu_support_features.md | 0 .../{ => ascend}/ascend_npu_support_models.md | 0 .../{ => ascend}/mindspore_backend.md | 0 docs/platforms/ascend_npu_quantization.md | 27 ---- .../extending/mindspore_models.md | 2 +- 22 files changed, 255 insertions(+), 63 deletions(-) rename docs/platforms/{ => ascend}/ascend_contribution_guide.md (89%) rename docs/platforms/{ => ascend}/ascend_npu.md (100%) rename docs/platforms/{ => ascend}/ascend_npu_best_practice.md (100%) rename docs/platforms/{ => ascend}/ascend_npu_deepseek_example.md (100%) rename docs/platforms/{ => ascend}/ascend_npu_environment_variables.md (100%) rename docs/platforms/{ => ascend}/ascend_npu_glm5_examples.md (98%) create mode 100644 docs/platforms/ascend/ascend_npu_quantization.md rename docs/platforms/{ => ascend}/ascend_npu_qwen3_5_examples.md (98%) rename docs/platforms/{ => ascend}/ascend_npu_qwen3_examples.md (100%) rename docs/platforms/{ => ascend}/ascend_npu_support.rst (93%) rename docs/platforms/{ => ascend}/ascend_npu_support_features.md (100%) rename docs/platforms/{ => ascend}/ascend_npu_support_models.md (100%) rename docs/platforms/{ => ascend}/mindspore_backend.md (100%) delete mode 100644 docs/platforms/ascend_npu_quantization.md diff --git a/.claude/skills/write-sglang-test/SKILL.md b/.claude/skills/write-sglang-test/SKILL.md index d24524468..af547ffa3 100644 --- a/.claude/skills/write-sglang-test/SKILL.md +++ b/.claude/skills/write-sglang-test/SKILL.md @@ -92,9 +92,22 @@ Defined in `python/sglang/test/test_utils.py`: | `stage-c-test-large-8-gpu-amd` | `linux-mi325-8gpu-sglang` | 8-GPU MI325 scaling and integration | | `stage-c-test-large-8-gpu-amd-mi35x` | `linux-mi35x-gpu-8` | 8-GPU MI35x scaling (2 partitions) | + +### Per-commit (Ascend NPU) + +| Suite | Runner (label) | Description | +| --- | --- | --- | +| `per-commit-1-npu-a2` | `linux-aarch64-a2-1` | 1-NPU LLM CI machine | +| `per-commit-2-npu-a2` | `linux-aarch64-a2-2` | 2-NPU LLM CI machine | +| `per-commit-4-npu-a3` | `linux-aarch64-a3-4` | 4-NPU LLM CI machine | +| `per-commit-16-npu-a3` | `linux-aarch64-a3-16` | 16-NPU LLM CI machine | +| `multimodal-gen-test-1-npu-a3` | `linux-aarch64-a3-2` | 1-NPU multimodal CI machine | +| `multimodal-gen-test-2-npu-a3` | `linux-aarch64-a3-16` | 2-NPU multimodal CI machine | +| `multimodal-gen-test-8-npu-a3` | `linux-aarch64-a3-16` | 8-NPU multimodal CI machine | + #### Nightly -Nightly suites are listed in `NIGHTLY_SUITES` in [`test/run_suite.py`](../../../test/run_suite.py). They run via `nightly-test-nvidia.yml` and `nightly-test-amd.yml`, not `pr-test.yml`. Examples: +Nightly suites are listed in `NIGHTLY_SUITES` in [`test/run_suite.py`](../../../test/run_suite.py). They run via `nightly-test-nvidia.yml`, `nightly-test-amd.yml` amd `nightly-test-npu.yml`, not `pr-test.yml`. Examples: - `nightly-1-gpu` (CUDA) - `nightly-kernel-1-gpu` (CUDA, JIT kernel full grids) @@ -103,6 +116,11 @@ Nightly suites are listed in `NIGHTLY_SUITES` in [`test/run_suite.py`](../../../ - `nightly-eval-vlm-2-gpu` (CUDA) - `nightly-amd` (AMD) - `nightly-amd-8-gpu-mi35x` (AMD) +- `nightly-1-npu-a3` (NPU) +- `nightly-2-npu-a3` (NPU) +- `nightly-4-npu-a3` (NPU) +- `nightly-8-npu-a3` (NPU) +- `nightly-16-npu-a3` (NPU) > **Note**: Multimodal diffusion uses `python/sglang/multimodal_gen/test/run_suite.py`, not `test/run_suite.py`. diff --git a/docs/advanced_features/quantization.md b/docs/advanced_features/quantization.md index 8a30d5084..26ceba091 100644 --- a/docs/advanced_features/quantization.md +++ b/docs/advanced_features/quantization.md @@ -19,32 +19,35 @@ to guard against abnormal quantization loss regressions. ## Platform Compatibility -The following table summarizes quantization method support across NVIDIA and AMD GPUs. +The following table summarizes quantization method support across NVIDIA and AMD GPUs, Ascend NPUs. -| Method | NVIDIA GPUs | AMD GPUs (MI300X/MI325X/MI350X) | Notes | -|--------|:-----------:|:-------------------------------:|-------| -| `fp8` | Yes | Yes | Aiter or Triton backend on AMD | -| `mxfp4` | Yes | Yes | Requires CDNA3/CDNA4 with MXFP support; uses Aiter | -| `blockwise_int8` | Yes | Yes | Triton-based, works on both platforms | -| `w8a8_int8` | Yes | Yes | | -| `w8a8_fp8` | Yes | Yes | Aiter or Triton FP8 on AMD | -| `awq` | Yes | Yes | Uses Triton dequantize on AMD (vs. optimized CUDA kernels on NVIDIA) | -| `gptq` | Yes | Yes | Uses Triton or vLLM kernels on AMD | -| `compressed-tensors` | Yes | Yes | Aiter paths for FP8/MoE on AMD | -| `quark` | Yes | Yes | AMD Quark quantization; Aiter GEMM paths on AMD | -| `auto-round` | Yes | Yes | Platform-agnostic (Intel auto-round) | -| `quark_int4fp8_moe` | No | Yes | AMD-only; online INT4-to-FP8 MoE quantization (CDNA3/CDNA4) | -| `awq_marlin` | Yes | No | Marlin kernels are CUDA-only | -| `gptq_marlin` | Yes | No | Marlin kernels are CUDA-only | -| `gguf` | Yes | No | CUDA-only kernels in sgl-kernel | -| `modelopt` / `modelopt_fp8` | Yes (Hopper/SM90+) | No | [NVIDIA ModelOpt](https://github.com/NVIDIA/Model-Optimizer); requires NVIDIA hardware | -| `modelopt_fp4` | Yes (Blackwell/SM100+) | No | [NVIDIA ModelOpt](https://github.com/NVIDIA/Model-Optimizer); native FP4 on Blackwell (B200, GB200) | -| `petit_nvfp4` | No | Yes (MI250/MI300X/MI325X) | Enables NVFP4 on ROCm via [Petit](https://github.com/causalflow-ai/petit-kernel); use `modelopt_fp4` on NVIDIA Blackwell. Auto-selected when loading NVFP4 models on AMD. See [LMSYS blog](https://lmsys.org/blog/2025-09-21-petit-amdgpu/) and [AMD ROCm blog](https://rocm.blogs.amd.com/artificial-intelligence/fp4-mixed-precision/README.html). | -| `bitsandbytes` | Yes | Experimental | Depends on bitsandbytes ROCm support | -| `torchao` (`int4wo`, etc.) | Yes | Partial | `int4wo` not supported on AMD; other methods may work | +| Method | NVIDIA GPUs | AMD GPUs (MI300X/MI325X/MI350X) | Ascend NPUs (A2/A3) | Notes | +|--------|:-----------:|:-------------------------------:|:-----------------------:|-------| +| `fp8` | Yes | Yes | WIP | Aiter or Triton backend on AMD | +| `mxfp4` | Yes | Yes | WIP | Requires CDNA3/CDNA4 with MXFP support; uses Aiter | +| `blockwise_int8` | Yes | Yes | No | Triton-based, works on both platforms | +| `w8a8_int8` | Yes | Yes | No | | +| `w8a8_fp8` | Yes | Yes | No | Aiter or Triton FP8 on AMD | +| `awq` | Yes | Yes | Yes | Uses Triton dequantize on AMD (vs. optimized CUDA kernels on NVIDIA). Uses CANN kernels on Ascend| +| `gptq` | Yes | Yes | Yes | Uses Triton or vLLM kernels on AMD. Uses CANN kernels on Ascend| +| `compressed-tensors` | Yes | Yes | Partial | Aiter paths for FP8/MoE on AMD. Uses CANN kernels on Ascend, `FP8` not supported yet| +| `quark` | Yes | Yes | No | AMD Quark quantization; Aiter GEMM paths on AMD | +| `auto-round` | Yes | Yes | Partial | Platform-agnostic (Intel auto-round). Uses CANN kernels on Ascend| +| `quark_int4fp8_moe` | No | Yes | No | AMD-only; online INT4-to-FP8 MoE quantization (CDNA3/CDNA4) | +| `awq_marlin` | Yes | No | No | Marlin kernels are CUDA-only | +| `gptq_marlin` | Yes | No | No | Marlin kernels are CUDA-only | +| `gguf` | Yes | No | WIP | CUDA-only kernels in sgl-kernel | +| `modelopt` / `modelopt_fp8` | Yes (Hopper/SM90+) | No | No | [NVIDIA ModelOpt](https://github.com/NVIDIA/Model-Optimizer); requires NVIDIA hardware | +| `modelopt_fp4` | Yes (Blackwell/SM100+) | No | No | [NVIDIA ModelOpt](https://github.com/NVIDIA/Model-Optimizer); native FP4 on Blackwell (B200, GB200) | +| `petit_nvfp4` | No | Yes (MI250/MI300X/MI325X) | No | Enables NVFP4 on ROCm via [Petit](https://github.com/causalflow-ai/petit-kernel); use `modelopt_fp4` on NVIDIA Blackwell. Auto-selected when loading NVFP4 models on AMD. See [LMSYS blog](https://lmsys.org/blog/2025-09-21-petit-amdgpu/) and [AMD ROCm blog](https://rocm.blogs.amd.com/artificial-intelligence/fp4-mixed-precision/README.html). | +| `bitsandbytes` | Yes | Experimental | No | Depends on bitsandbytes ROCm support | +| `torchao` (`int4wo`, etc.) | Yes | Partial | No | `int4wo` not supported on AMD; other methods may work | +| `modelslim` | No | No | Yes | Ascend quantization; Uses CANN kernels | On AMD, several of these methods use [Aiter](https://github.com/ROCm/aiter) for acceleration -- set `SGLANG_USE_AITER=1` where noted. See [AMD GPU setup](../platforms/amd_gpu.md) for installation and configuration details. +On Ascend, various layers quantization configurations are supported, see [Ascend NPU quantization](../platforms/ascend/ascend_npu_quantization.md) for details. + ## GEMM Backends for FP4/FP8 Quantization :::{note} @@ -81,7 +84,7 @@ When FlashInfer is unavailable for NVFP4, sgl-kernel CUTLASS is used as an autom To load already quantized models, simply load the model weights and config. **Again, if the model has been quantized offline, there's no need to add `--quantization` argument when starting the engine. The quantization method will be parsed from the -downloaded Hugging Face config. For example, DeepSeek V3/R1 models are already in FP8, so do not add redundant parameters.** +downloaded Hugging Face or msModelSlim config. For example, DeepSeek V3/R1 models are already in FP8, so do not add redundant parameters.** ```bash python3 -m sglang.launch_server \ @@ -319,7 +322,6 @@ For detailed usage and supported model architectures, see [NVIDIA Model Optimize SGLang includes a streamlined workflow for quantizing models with ModelOpt and automatically exporting them for deployment. - ##### Installation First, install ModelOpt: @@ -477,6 +479,74 @@ model_loader.load_model(model_config=model_config, device_config=DeviceConfig()) - **Calibration-based**: Uses calibration datasets for optimal quantization quality - **Production Ready**: Enterprise-grade quantization with NVIDIA support +#### Using [ModelSlim](https://gitcode.com/Ascend/msmodelslim) +MindStudio-ModelSlim (msModelSlim) is a model offline quantization compression tool launched by MindStudio and optimized for Ascend hardware. + +- **Installation** + + ```bash + # Clone repo and install msmodelslim: + git clone https://gitcode.com/Ascend/msmodelslim.git + cd msmodelslim + bash install.sh + ``` + +- **LLM quantization** + + Download the original floating-point weights of the large model. Taking Qwen3-32B as an example, you can go to [Qwen3-32B](https://huggingface.co/Qwen/Qwen3-32B) to obtain the original model weights. Then install other dependencies (related to the model, refer to the huggingface model card). + > Note: You can find pre-quantized validated models on [modelscope/Eco-Tech](https://modelscope.cn/models/Eco-Tech). + + _Traditional quantification methods require the preparation of calibration data files (```.jsonl``` formats) for calibration in the quantification process._ + ```bash + Qwen3-32B/ # floating-point model downloaded from official HF (or modelscope) repo + msmodelslim/ # msmodelslim repo + |----- lab_calib # calibration date folder (put your dataset here in ```.jsonl``` format or use pre-prepared ones) + |----- some file (such as laos_calib.jsonl) + |----- lab_practice # best practice folder with configs for quantization + |----- model folder (such as qwen3_5_moe folder) # folder with quantization configs + |----- quant_config (such as qwen3_5_moe_w8a8.yaml) # quantization config + |----- another folders + output_folder/ # generated by below command + |----- quant_model_weights-00001-of-0001.safetensors # quantized weights + |----- quant_model_description.json # file with description of the quantization methods for each layer (```W4A4_DYNAMIC```, etc.) + |----- another files (such as config.json, tokenizer.json, etc.) + ``` + Run quantization using one-click quantization (recommended): + ```bash + msmodelslim quant \ + --model_path ${MODEL_PATH} \ + --save_path ${SAVE_PATH} \ + --device npu:0,1 \ + --model_type Qwen3-32B \ + --quant_type w8a8 \ + --trust_remote_code True + ``` + +- **Usage Example** + ```bash + python3 -m sglang.launch_server \ + --model-path $PWD/Qwen3-32B-w8a8 \ + --port 30000 --host 0.0.0.0 + ``` + +- **Available Quantization Methods**: + - [x] ```W4A4_DYNAMIC``` linear with online quantization of activations + - [x] ```W8A8``` linear with offline quantization of activations + - [x] ```W8A8_DYNAMIC``` linear with online quantization of activations + - [x] ```W4A4_DYNAMIC``` MOE with online quantization of activations + - [x] ```W4A8_DYNAMIC``` MOE with online quantization of activations + - [x] ```W8A8_DYNAMIC``` MOE with online quantization of activations + - [ ] ```W4A8``` linear TBD + - [ ] ```W4A16``` linear TBD + - [ ] ```W48A16``` linear TBD + - [ ] ```W4A16``` MoE in progress + - [ ] ```W8A16``` MoE in progress + - [ ] ```KV Cache``` in progress + - [ ] ```Attention``` in progress + + +For more detailed examples of quantization of models, as well as information about their support, see the [examples](https://gitcode.com/Ascend/msmodelslim/blob/master/example/README.md) section in ModelSLim repo. + ## Online Quantization To enable online quantization, you can simply specify `--quantization` in the command line. For example, you can launch the server with the following command to enable `FP8` quantization for model `meta-llama/Meta-Llama-3.1-8B-Instruct`: @@ -529,3 +599,4 @@ Other layers (e.g. projections in the attention layers) have their weights quant - [Torchao: PyTorch Architecture Optimization](https://github.com/pytorch/ao) - [vLLM Quantization](https://docs.vllm.ai/en/latest/quantization/) - [auto-round](https://github.com/intel/auto-round) +- [ModelSlim](https://gitcode.com/Ascend/msmodelslim) diff --git a/docs/basic_usage/deepseek_v3.md b/docs/basic_usage/deepseek_v3.md index b558c2223..9770c2882 100644 --- a/docs/basic_usage/deepseek_v3.md +++ b/docs/basic_usage/deepseek_v3.md @@ -74,7 +74,7 @@ Detailed commands for reference: - [16 x A100 (INT8)](https://github.com/sgl-project/sglang/tree/main/benchmark/deepseek_v3#example-serving-with-16-a100a800-with-int8-quantization) - [32 x L40S (INT8)](https://github.com/sgl-project/sglang/tree/main/benchmark/deepseek_v3#example-serving-with-32-l40s-with-int8-quantization) - [Xeon 6980P CPU](../platforms/cpu_server.md#example-running-deepseek-r1) -- [4 x Atlas 800I A3 (int8)](../platforms/ascend_npu_deepseek_example.md#running-deepseek-with-pd-disaggregation-on-4-x-atlas-800i-a3) +- [4 x Atlas 800I A3 (int8)](../platforms/ascend/ascend_npu_deepseek_example.md#running-deepseek-with-pd-disaggregation-on-4-x-atlas-800i-a3) ### Download Weights If you encounter errors when starting the server, ensure the weights have finished downloading. It's recommended to download them beforehand or restart multiple times until all weights are downloaded. Please refer to [DeepSeek V3](https://huggingface.co/deepseek-ai/DeepSeek-V3-Base#61-inference-with-deepseek-infer-demo-example-only) official guide to download the weights. diff --git a/docs/diffusion/installation.md b/docs/diffusion/installation.md index 9531d70d2..8c0fa1d11 100644 --- a/docs/diffusion/installation.md +++ b/docs/diffusion/installation.md @@ -84,7 +84,7 @@ pip install -e "python[all_musa]" ## Platform-Specific: Ascend NPU -For Ascend NPU, please follow the [NPU installation guide](../platforms/ascend_npu.md). +For Ascend NPU, please follow the [NPU installation guide](../platforms/ascend/ascend_npu.md). Quick test: diff --git a/docs/diffusion/quantization.md b/docs/diffusion/quantization.md index ab1b634a8..04fa798b6 100644 --- a/docs/diffusion/quantization.md +++ b/docs/diffusion/quantization.md @@ -44,7 +44,8 @@ backend. |------------------|--------------------------------------------------------------------------------------------|------------------------------------------------------|--------------------------------------------------------------|---------------------------------------|-----------------------------------------------------------------------------------------------------------------------| | `fp8` | Quantized transformer component folder, or safetensors with `quantization_config` metadata | `--transformer-path` or `--transformer-weights-path` | ALL | None | Component-folder and single-file flows are both supported | | `nvfp4-modelopt` | NVFP4 safetensors file, sharded directory, or repo providing transformer weights | `--transformer-weights-path` | FLUX.2 | `comfy-kitchen` optional on Blackwell | Blackwell can use a best-performance kit when available; otherwise SGLang falls back to the generic ModelOpt FP4 path | -| `nunchaku-svdq` | Pre-quantized Nunchaku transformer weights, usually named `svdq-{int4\|fp4}_r{rank}-...` | `--transformer-weights-path` | Model-specific support such as Qwen-Image, FLUX, and Z-Image | `nunchaku` | SGLang can infer precision and rank from the filename and supports both `int4` and `nvfp4` | +| `nunchaku-svdq` | Pre-quantized Nunchaku transformer weights, usually named `svdq-{int4\|fp4}_r{rank}-...` | `--transformer-weights-path` | Model-specific support such as Qwen-Image, FLUX, and Z-Image | `nunchaku` | SGLang can infer precision and rank from the filename and supports both `int4` and `nvfp4` | +| `msmodelslim` | Pre-quantized msmodelslim transformer weights | `--model-path` | Wan2.2 family | None | Currently only compatible with the Ascend NPU family and supports both `w8a8` and `w4a4` | ## NVFP4 @@ -171,3 +172,68 @@ sglang generate \ as `4` or `8`. - Current runtime validation only allows Nunchaku on NVIDIA CUDA Ampere (SM8x) or SM12x GPUs. Hopper (SM90) is currently rejected. + +## [ModelSlim](https://gitcode.com/Ascend/msmodelslim) +MindStudio-ModelSlim (msModelSlim) is a model offline quantization compression tool launched by MindStudio and optimized for Ascend hardware. + +- **Installation** + + ```bash + # Clone repo and install msmodelslim: + git clone https://gitcode.com/Ascend/msmodelslim.git + cd msmodelslim + bash install.sh + ``` + +- **Multimodal_sd quantization** + + Download the original floating-point weights of the large model. Taking Wan2.2-T2V-A14B as an example, you can go to [Wan2.2-T2V-A14B](https://modelscope.cn/models/Wan-AI/Wan2.2-T2V-A14B) to obtain the original model weights. Then install other dependencies (related to the model, refer to the modelscope model card). + > Note: You can find pre-quantized validated models on [modelscope/Eco-Tech](https://modelscope.cn/models/Eco-Tech). + + Run quantization using one-click quantization (recommended): + + ```bash + msmodelslim quant \ + --model_path /path/to/wan2_2_float_weights \ + --save_path /path/to/wan2_2_quantized_weights \ + --device npu \ + --model_type Wan2_2 \ + --quant_type w8a8 \ + --trust_remote_code True + ``` + + For more detailed examples of quantization of models, as well as information about their support, see the [examples](https://gitcode.com/Ascend/msmodelslim/blob/master/example/multimodal_sd/README.md) section in ModelSLim repo. + + > Note: SGLang does not support quantized embeddings, please disable this option when quantizing using msmodelslim. + +- **Auto-Detection and different formats** + + For msmodelslim checkpoints, it's enough to specify only ```--model-path```, the detection of quantization occurs automatically for each layer using parsing of `quant_model_description.json` config. + + In the case of `Wan2.2` only `Diffusers` weights storage format are supported, whereas modelslim saves the quantized model in the original `Wan2.2` format, + for conversion in use `python/sglang/multimodal_gen/tools/wan_repack.py` script: + + ```bash + python wan_repack.py \ + --input-path {path_to_quantized_model} \ + --output-path {path_to_converted_model} + ``` + + After that, please copy all files from original `Diffusers` checkpoint (instead of `transformer`/`tranfsormer_2` folders) + +- **Usage Example** + + With auto-detected flow: + + ```bash + sglang generate \ + --model-path Eco-Tech/Wan2.2-T2V-A14B-Diffusers-w8a8 \ + --prompt "a beautiful sunset" \ + --save-output + ``` + +- **Available Quantization Methods**: + - [x] ```W4A4_DYNAMIC``` linear with online quantization of activations + - [x] ```W8A8``` linear with offline quantization of activations + - [x] ```W8A8_DYNAMIC``` linear with online quantization of activations + - [ ] ```mxfp8``` linear in progress diff --git a/docs/get_started/install.md b/docs/get_started/install.md index 9306afc95..a8aab8697 100644 --- a/docs/get_started/install.md +++ b/docs/get_started/install.md @@ -2,7 +2,7 @@ You can install SGLang using one of the methods below. This page primarily applies to common NVIDIA GPU platforms. -For other or newer platforms, please refer to the dedicated pages for [AMD GPUs](../platforms/amd_gpu.md), [Intel Xeon CPUs](../platforms/cpu_server.md), [TPU](../platforms/tpu.md), [NVIDIA DGX Spark](https://lmsys.org/blog/2025-11-03-gpt-oss-on-nvidia-dgx-spark/), [NVIDIA Jetson](../platforms/nvidia_jetson.md), [Ascend NPUs](../platforms/ascend_npu.md), and [Intel XPU](../platforms/xpu.md). +For other or newer platforms, please refer to the dedicated pages for [AMD GPUs](../platforms/amd_gpu.md), [Intel Xeon CPUs](../platforms/cpu_server.md), [TPU](../platforms/tpu.md), [NVIDIA DGX Spark](https://lmsys.org/blog/2025-11-03-gpt-oss-on-nvidia-dgx-spark/), [NVIDIA Jetson](../platforms/nvidia_jetson.md), [Ascend NPUs](../platforms/ascend/ascend_npu.md), and [Intel XPU](../platforms/xpu.md). ## Method 1: With pip or uv diff --git a/docs/index.rst b/docs/index.rst index e61b40ef4..a5b49f1ba 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -99,7 +99,7 @@ Its core features include: platforms/cpu_server.md platforms/tpu.md platforms/nvidia_jetson.md - platforms/ascend_npu_support.rst + platforms/ascend/ascend_npu_support.rst platforms/xpu.md .. toctree:: diff --git a/docs/platforms/ascend_contribution_guide.md b/docs/platforms/ascend/ascend_contribution_guide.md similarity index 89% rename from docs/platforms/ascend_contribution_guide.md rename to docs/platforms/ascend/ascend_contribution_guide.md index fa87161ff..5823bd22a 100644 --- a/docs/platforms/ascend_contribution_guide.md +++ b/docs/platforms/ascend/ascend_contribution_guide.md @@ -6,7 +6,7 @@ Welcome to **SGLang**! We appreciate your interest in contributing. This guide p ### Prepare Environment -Before contributing, please ensure that your environment is set up correctly. Follow the steps in the [Installation Guide](../platforms/ascend_npu.md) to install the necessary dependencies. We recommend [using docker](../platforms/ascend_npu.md#method-2-using-docker-image) to build the environment. +Before contributing, please ensure that your environment is set up correctly. Follow the steps in the [Installation Guide](ascend_npu.md) to install the necessary dependencies. We recommend [using docker](ascend_npu.md#method-2-using-docker-image) to build the environment. ### Fork and clone the repository @@ -38,6 +38,18 @@ If you add a new feature or fix a bug, please add corresponding unit tests to en SGLang uses Python's built-in [unittest](https://docs.python.org/3/library/unittest.html) framework. For detailed instructions on running tests and integrating them into CI, refer to [test/README.md](https://github.com/sgl-project/sglang/tree/main/test/README.md). +If you need to use model which is not in ```python/sglang/test/ascend/test_ascend_utils.py`` list. Follow these steps: +1. Register account and upload your model to [modelscope](https://modelscope.cn/models). +2. Make sure your model is pre-cached on the CI server and is on the way "/data/ascend-ci-share-pkking-sglang/modelscope/hub/models/{your_model_repo}/{your_model}". +If this is not the case, use following command on CI server: + ```bash + modelscope download + --model {your_model_repo}/{your_model} + --local_dir /data/ascend-ci-share-pkking-sglang/modelscope/hub/models/{your_model_repo}/{your_model} + ``` + > Note: If you don’t have access to CI server, please ask maintainers (zl19940307@163.com) to download your model. +4. Add model to ```python/sglang/test/ascend/test_ascend_utils.py``` (use docker ```"/root/.cache/modelscope/hub/models/{your_model_repo}/{your_model}"``` path). + ## Write documentations We recommend new contributors start from writing documentation, which helps you quickly understand SGLang codebase. @@ -64,7 +76,7 @@ You can find additional accuracy eval examples in: - [test_moe_eval_accuracy_large.py](https://github.com/sgl-project/sglang/blob/main/test/registered/eval/test_moe_eval_accuracy_large.py) ## Benchmark the speed -Refer to [Benchmark and Profiling](../developer_guide/benchmark_and_profiling.md). +Refer to [Benchmark and Profiling](../../developer_guide/benchmark_and_profiling.md). ## Requesting a review for merge You can follow the pull request merge process described in [MAINTAINER.md](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md). @@ -108,7 +120,6 @@ cool-down-minutes: Users listed in [CI_PERMISSIONS.json](https://github.com/sgl-project/sglang/blob/main/.github/CI_PERMISSIONS.json) may have a per-user cooldown interval. In practice, we use the minimum of the workflow’s default window and the user-specific interval. - ## Code style guidance - Avoid code duplication. If the same code snippet (more than five lines) appears multiple times, extract it into a shared function. - Minimize device synchronization. Reduce expensive CPU-GPU synchronization operations, such as `tensor.item()` or `tensor.cpu()`, whenever possible. Use vectorized code. diff --git a/docs/platforms/ascend_npu.md b/docs/platforms/ascend/ascend_npu.md similarity index 100% rename from docs/platforms/ascend_npu.md rename to docs/platforms/ascend/ascend_npu.md diff --git a/docs/platforms/ascend_npu_best_practice.md b/docs/platforms/ascend/ascend_npu_best_practice.md similarity index 100% rename from docs/platforms/ascend_npu_best_practice.md rename to docs/platforms/ascend/ascend_npu_best_practice.md diff --git a/docs/platforms/ascend_npu_deepseek_example.md b/docs/platforms/ascend/ascend_npu_deepseek_example.md similarity index 100% rename from docs/platforms/ascend_npu_deepseek_example.md rename to docs/platforms/ascend/ascend_npu_deepseek_example.md diff --git a/docs/platforms/ascend_npu_environment_variables.md b/docs/platforms/ascend/ascend_npu_environment_variables.md similarity index 100% rename from docs/platforms/ascend_npu_environment_variables.md rename to docs/platforms/ascend/ascend_npu_environment_variables.md diff --git a/docs/platforms/ascend_npu_glm5_examples.md b/docs/platforms/ascend/ascend_npu_glm5_examples.md similarity index 98% rename from docs/platforms/ascend_npu_glm5_examples.md rename to docs/platforms/ascend/ascend_npu_glm5_examples.md index f748b6408..f613e8956 100644 --- a/docs/platforms/ascend_npu_glm5_examples.md +++ b/docs/platforms/ascend/ascend_npu_glm5_examples.md @@ -191,4 +191,4 @@ Not test yet. ### Using Benchmark -Refer to [Benchmark and Profiling](../developer_guide/benchmark_and_profiling.md) for details. +Refer to [Benchmark and Profiling](../../developer_guide/benchmark_and_profiling.md) for details. diff --git a/docs/platforms/ascend/ascend_npu_quantization.md b/docs/platforms/ascend/ascend_npu_quantization.md new file mode 100644 index 000000000..8b2e30ba1 --- /dev/null +++ b/docs/platforms/ascend/ascend_npu_quantization.md @@ -0,0 +1,52 @@ +# Quantization on Ascend + +To load already quantized models, simply load the model weights and config. Again, if the model has been quantized offline, there's no need to add `--quantization` argument when starting the engine. The quantization method will be automatically parsed from the downloaded `quant_model_description.json` or `config.json` config. + +SGLang support **mix-bits** quantization (independently defines and loads each layer depending on the type of quantification specified in the `quant_model_description'.json`). [Advanced mix-bits for MoE](https://github.com/sgl-project/sglang/pull/17361) in progress, will add independent quantization determination for the w13 (up-gate) and w2 (down) layers). + +[ModelSlim on Ascend support](https://github.com/sgl-project/sglang/pull/14504) +| Quantization scheme | Layer type | A2 Supported | A3 Supported | A5 Supported | Diffusion models | +|-----------------------------------------------------------|--------------------------|:----------------------------------------:|:----------------------------------------:|:------------------------------------------:|:------------------------------------------:| +| W4A4 dynamic | Linear | **√** | **√** | **TBD** | **√** | +| W8A8 static | Linear | **√** | **√** | **TBD** | **√** | +| W8A8 dynamic | Linear | **√** | **√** | **TBD** | **√** | +| [MXFP8](https://github.com/sgl-project/sglang/pull/20922) | Linear | **x** | **x** | **WIP** | **WIP** | +| W4A4 dynamic | MoE | **√** | **√** | **TBD** | **x** | +| W4A8 dynamic | MoE | **√** | **√** | **TBD** | **x** | +| W8A8 dynamic | MoE | **√** | **√** | **TBD** | **x** | +| [MXFP8](https://github.com/sgl-project/sglang/pull/20922) | MoE | **x** | **x** | **WIP** | **x** | + +[AWQ on Ascend support](https://github.com/sgl-project/sglang/pull/10158): +| Quantization scheme | Layer type | A2 Supported | A3 Supported | A5 Supported | +|--------------------------------|--------------------------|:----------------------------------------:|:----------------------------------------:|:------------------------------------------:| +| W4A16 | Linear | **√** | **√** | **TBD** | +| W8A16 | Linear | **√** | **√** | **TBD** | +| W4A16 | MoE | **√** | **√** | **TBD** | + +GPTQ on Ascend support +| Quantization scheme | Layer type | A2 Supported | A3 Supported | A5 Supported | +|----------------------------------------------------------------------------|--------------------------|:----------------------------------------:|:----------------------------------------:|:-----------------------------------------:| +| [W4A16](https://github.com/sgl-project/sglang/pull/15203) | Linear | **√** | **√** | **TBD** | +| [W8A16](https://github.com/sgl-project/sglang/pull/15203) | Linear | **√** | **√** | **TBD** | +| [W4A16 MOE](https://github.com/sgl-project/sglang/pull/16364) | MoE | **√** | **√** | **TBD** | +| [W8A16 MOE](https://github.com/sgl-project/sglang/pull/16364) | MoE | **√** | **√** | **TBD** | + +[Auto-round on Ascend support](https://github.com/sgl-project/sglang/pull/16699) +| Quantization scheme | Layer type | A2 Supported | A3 Supported | A5 Supported | +|--------------------------------|--------------------------|:----------------------------------------:|:----------------------------------------:|:-----------------------------------------:| +| W4A16 | Linear | **√** | **√** | **TBD** | +| W8A16 | Linear | **√** | **√** | **TBD** | +| W4A16 | MoE | **√** | **√** | **TBD** | +| W8A16 | MoE | **√** | **√** | **TBD** | + +Compressed-tensors (LLM Compressor) on Ascend support: +| Quantization scheme | Layer type | A2 Supported | A3 Supported | A5 Supported | +|-----------------------------------------------------------------------------------------------|--------------------------|:----------------------------------------:|:----------------------------------------:|:-----------------------------------------:| +| [W8A8 dynamic](https://github.com/sgl-project/sglang/pull/14504) | Linear | **√** | **√** | **TBD** | +| [W4A8 dynamic with/without activation clip](https://github.com/sgl-project/sglang/pull/14736) | MoE | **√** | **√** | **TBD** | +| [W4A16 MOE](https://github.com/sgl-project/sglang/pull/12759) | MoE | **√** | **√** | **TBD** | +| [W8A8 dynamic](https://github.com/sgl-project/sglang/pull/14504) | MoE | **√** | **√** | **TBD** | + +[GGUF on Ascend support](https://github.com/sgl-project/sglang/pull/17883) + +in progress diff --git a/docs/platforms/ascend_npu_qwen3_5_examples.md b/docs/platforms/ascend/ascend_npu_qwen3_5_examples.md similarity index 98% rename from docs/platforms/ascend_npu_qwen3_5_examples.md rename to docs/platforms/ascend/ascend_npu_qwen3_5_examples.md index b19f7321e..8660f17cc 100644 --- a/docs/platforms/ascend_npu_qwen3_5_examples.md +++ b/docs/platforms/ascend/ascend_npu_qwen3_5_examples.md @@ -228,4 +228,4 @@ Not test yet. ### Using Benchmark -Refer to [Benchmark and Profiling](../developer_guide/benchmark_and_profiling.md) for details. +Refer to [Benchmark and Profiling](../../developer_guide/benchmark_and_profiling.md) for details. diff --git a/docs/platforms/ascend_npu_qwen3_examples.md b/docs/platforms/ascend/ascend_npu_qwen3_examples.md similarity index 100% rename from docs/platforms/ascend_npu_qwen3_examples.md rename to docs/platforms/ascend/ascend_npu_qwen3_examples.md diff --git a/docs/platforms/ascend_npu_support.rst b/docs/platforms/ascend/ascend_npu_support.rst similarity index 93% rename from docs/platforms/ascend_npu_support.rst rename to docs/platforms/ascend/ascend_npu_support.rst index cd64c58f6..3ec28b837 100644 --- a/docs/platforms/ascend_npu_support.rst +++ b/docs/platforms/ascend/ascend_npu_support.rst @@ -7,6 +7,7 @@ Ascend NPUs ascend_npu.md ascend_npu_support_features.md ascend_npu_support_models.md + ascend_npu_quantization.md ascend_npu_deepseek_example.md ascend_npu_qwen3_examples.md mindspore_backend.md diff --git a/docs/platforms/ascend_npu_support_features.md b/docs/platforms/ascend/ascend_npu_support_features.md similarity index 100% rename from docs/platforms/ascend_npu_support_features.md rename to docs/platforms/ascend/ascend_npu_support_features.md diff --git a/docs/platforms/ascend_npu_support_models.md b/docs/platforms/ascend/ascend_npu_support_models.md similarity index 100% rename from docs/platforms/ascend_npu_support_models.md rename to docs/platforms/ascend/ascend_npu_support_models.md diff --git a/docs/platforms/mindspore_backend.md b/docs/platforms/ascend/mindspore_backend.md similarity index 100% rename from docs/platforms/mindspore_backend.md rename to docs/platforms/ascend/mindspore_backend.md diff --git a/docs/platforms/ascend_npu_quantization.md b/docs/platforms/ascend_npu_quantization.md deleted file mode 100644 index fb4adb54f..000000000 --- a/docs/platforms/ascend_npu_quantization.md +++ /dev/null @@ -1,27 +0,0 @@ -Quantization on Ascend. - -To load already quantized models, simply load the model weights and config. Again, if the model has been quantized offline, there's no need to add `--quantization` argument when starting the engine. The quantization method will be automatically parsed from the downloaded `quant_model_description.json` or `config.json` config. - -[ModelSlim on Ascend support](https://github.com/sgl-project/sglang/pull/14504): -- [x] W4A4 dynamic linear -- [x] W8A8 static linear -- [x] W8A8 dynamic linear -- [x] W4A4 dynamic MOE -- [x] W4A8 dynamic MOE -- [x] W8A8 dynamic MOE - -[AWQ on Ascend support](https://github.com/sgl-project/sglang/pull/10158): -- [x] W4A16 linear -- [x] W8A16 linear # Need to test -- [x] W4A16 MOE # Need to test - -Compressed-tensors (LLM Compressor) on Ascend support: -- [x] [W4A8 dynamic MOE with/without activation clip](https://github.com/sgl-project/sglang/pull/14736) # Need to test -- [x] [W4A16 MOE](https://github.com/sgl-project/sglang/pull/12759) -- [x] [W8A8 dynamic linear](https://github.com/sgl-project/sglang/pull/14504) -- [x] [W8A8 dynamic MOE](https://github.com/sgl-project/sglang/pull/14504) - -Diffusion model [modelslim](https://github.com/sgl-project/sglang/pull/17996) quantization on Ascend support: -- [x] W4A4 dynamic linear -- [x] W8A8 static linear -- [x] W8A8 dynamic linear diff --git a/docs/supported_models/extending/mindspore_models.md b/docs/supported_models/extending/mindspore_models.md index 1e5583293..caa5ade9c 100644 --- a/docs/supported_models/extending/mindspore_models.md +++ b/docs/supported_models/extending/mindspore_models.md @@ -19,7 +19,7 @@ Currently, the following models are supported: ## Installation -> **Note**: Currently, MindSpore models are provided by an independent package `sgl-mindspore`. Support for MindSpore is built upon current SGLang support for Ascend NPU platform. Please first [install SGLang for Ascend NPU](../../platforms/ascend_npu.md) and then install `sgl-mindspore`: +> **Note**: Currently, MindSpore models are provided by an independent package `sgl-mindspore`. Support for MindSpore is built upon current SGLang support for Ascend NPU platform. Please first [install SGLang for Ascend NPU](../../platforms/ascend/ascend_npu.md) and then install `sgl-mindspore`: ```shell git clone https://github.com/mindspore-lab/sgl-mindspore.git