[NPU] Update quantization&CI documentation (#21100)
Co-authored-by: Tamir Baydasov <41994229+TamirBaydasov@users.noreply.github.com>
This commit is contained in:
Артем Савкин
co-authored by
Tamir Baydasov
Tamir Baydasov
parent
ced69c9f84
commit
27071e0a43
+14
-3
@@ -6,7 +6,7 @@ Welcome to **SGLang**! We appreciate your interest in contributing. This guide p
|
||||
|
||||
### Prepare Environment
|
||||
|
||||
Before contributing, please ensure that your environment is set up correctly. Follow the steps in the [Installation Guide](../platforms/ascend_npu.md) to install the necessary dependencies. We recommend [using docker](../platforms/ascend_npu.md#method-2-using-docker-image) to build the environment.
|
||||
Before contributing, please ensure that your environment is set up correctly. Follow the steps in the [Installation Guide](ascend_npu.md) to install the necessary dependencies. We recommend [using docker](ascend_npu.md#method-2-using-docker-image) to build the environment.
|
||||
|
||||
### Fork and clone the repository
|
||||
|
||||
@@ -38,6 +38,18 @@ If you add a new feature or fix a bug, please add corresponding unit tests to en
|
||||
SGLang uses Python's built-in [unittest](https://docs.python.org/3/library/unittest.html) framework.
|
||||
For detailed instructions on running tests and integrating them into CI, refer to [test/README.md](https://github.com/sgl-project/sglang/tree/main/test/README.md).
|
||||
|
||||
If you need to use model which is not in ```python/sglang/test/ascend/test_ascend_utils.py`` list. Follow these steps:
|
||||
1. Register account and upload your model to [modelscope](https://modelscope.cn/models).
|
||||
2. Make sure your model is pre-cached on the CI server and is on the way "/data/ascend-ci-share-pkking-sglang/modelscope/hub/models/{your_model_repo}/{your_model}".
|
||||
If this is not the case, use following command on CI server:
|
||||
```bash
|
||||
modelscope download
|
||||
--model {your_model_repo}/{your_model}
|
||||
--local_dir /data/ascend-ci-share-pkking-sglang/modelscope/hub/models/{your_model_repo}/{your_model}
|
||||
```
|
||||
> Note: If you don’t have access to CI server, please ask maintainers (zl19940307@163.com) to download your model.
|
||||
4. Add model to ```python/sglang/test/ascend/test_ascend_utils.py``` (use docker ```"/root/.cache/modelscope/hub/models/{your_model_repo}/{your_model}"``` path).
|
||||
|
||||
## Write documentations
|
||||
|
||||
We recommend new contributors start from writing documentation, which helps you quickly understand SGLang codebase.
|
||||
@@ -64,7 +76,7 @@ You can find additional accuracy eval examples in:
|
||||
- [test_moe_eval_accuracy_large.py](https://github.com/sgl-project/sglang/blob/main/test/registered/eval/test_moe_eval_accuracy_large.py)
|
||||
|
||||
## Benchmark the speed
|
||||
Refer to [Benchmark and Profiling](../developer_guide/benchmark_and_profiling.md).
|
||||
Refer to [Benchmark and Profiling](../../developer_guide/benchmark_and_profiling.md).
|
||||
|
||||
## Requesting a review for merge
|
||||
You can follow the pull request merge process described in [MAINTAINER.md](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md).
|
||||
@@ -108,7 +120,6 @@ cool-down-minutes:
|
||||
|
||||
Users listed in [CI_PERMISSIONS.json](https://github.com/sgl-project/sglang/blob/main/.github/CI_PERMISSIONS.json) may have a per-user cooldown interval. In practice, we use the minimum of the workflow’s default window and the user-specific interval.
|
||||
|
||||
|
||||
## Code style guidance
|
||||
- Avoid code duplication. If the same code snippet (more than five lines) appears multiple times, extract it into a shared function.
|
||||
- Minimize device synchronization. Reduce expensive CPU-GPU synchronization operations, such as `tensor.item()` or `tensor.cpu()`, whenever possible. Use vectorized code.
|
||||
+1
-1
@@ -191,4 +191,4 @@ Not test yet.
|
||||
|
||||
### Using Benchmark
|
||||
|
||||
Refer to [Benchmark and Profiling](../developer_guide/benchmark_and_profiling.md) for details.
|
||||
Refer to [Benchmark and Profiling](../../developer_guide/benchmark_and_profiling.md) for details.
|
||||
@@ -0,0 +1,52 @@
|
||||
# Quantization on Ascend
|
||||
|
||||
To load already quantized models, simply load the model weights and config. Again, if the model has been quantized offline, there's no need to add `--quantization` argument when starting the engine. The quantization method will be automatically parsed from the downloaded `quant_model_description.json` or `config.json` config.
|
||||
|
||||
SGLang support **mix-bits** quantization (independently defines and loads each layer depending on the type of quantification specified in the `quant_model_description'.json`). [Advanced mix-bits for MoE](https://github.com/sgl-project/sglang/pull/17361) in progress, will add independent quantization determination for the w13 (up-gate) and w2 (down) layers).
|
||||
|
||||
[ModelSlim on Ascend support](https://github.com/sgl-project/sglang/pull/14504)
|
||||
| Quantization scheme | Layer type | A2 Supported | A3 Supported | A5 Supported | Diffusion models |
|
||||
|-----------------------------------------------------------|--------------------------|:----------------------------------------:|:----------------------------------------:|:------------------------------------------:|:------------------------------------------:|
|
||||
| W4A4 dynamic | Linear | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | **<span style="color: yellow;">TBD</span>** | **<span style="color: green;">√</span>** |
|
||||
| W8A8 static | Linear | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | **<span style="color: yellow;">TBD</span>** | **<span style="color: green;">√</span>** |
|
||||
| W8A8 dynamic | Linear | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | **<span style="color: yellow;">TBD</span>** | **<span style="color: green;">√</span>** |
|
||||
| [MXFP8](https://github.com/sgl-project/sglang/pull/20922) | Linear | **<span style="color: red;">x</span>** | **<span style="color: red;">x</span>** | **<span style="color: blue;">WIP</span>** | **<span style="color: blue;">WIP</span>** |
|
||||
| W4A4 dynamic | MoE | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | **<span style="color: yellow;">TBD</span>** | **<span style="color: red;">x</span>** |
|
||||
| W4A8 dynamic | MoE | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | **<span style="color: yellow;">TBD</span>** | **<span style="color: red;">x</span>** |
|
||||
| W8A8 dynamic | MoE | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | **<span style="color: yellow;">TBD</span>** | **<span style="color: red;">x</span>** |
|
||||
| [MXFP8](https://github.com/sgl-project/sglang/pull/20922) | MoE | **<span style="color: red;">x</span>** | **<span style="color: red;">x</span>** | **<span style="color: blue;">WIP</span>** | **<span style="color: red;">x</span>** |
|
||||
|
||||
[AWQ on Ascend support](https://github.com/sgl-project/sglang/pull/10158):
|
||||
| Quantization scheme | Layer type | A2 Supported | A3 Supported | A5 Supported |
|
||||
|--------------------------------|--------------------------|:----------------------------------------:|:----------------------------------------:|:------------------------------------------:|
|
||||
| W4A16 | Linear | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | **<span style="color: yellow;">TBD</span>** |
|
||||
| W8A16 | Linear | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | **<span style="color: yellow;">TBD</span>** |
|
||||
| W4A16 | MoE | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | **<span style="color: yellow;">TBD</span>** |
|
||||
|
||||
GPTQ on Ascend support
|
||||
| Quantization scheme | Layer type | A2 Supported | A3 Supported | A5 Supported |
|
||||
|----------------------------------------------------------------------------|--------------------------|:----------------------------------------:|:----------------------------------------:|:-----------------------------------------:|
|
||||
| [W4A16](https://github.com/sgl-project/sglang/pull/15203) | Linear | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | **<span style="color: yellow;">TBD</span>** |
|
||||
| [W8A16](https://github.com/sgl-project/sglang/pull/15203) | Linear | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | **<span style="color: yellow;">TBD</span>** |
|
||||
| [W4A16 MOE](https://github.com/sgl-project/sglang/pull/16364) | MoE | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | **<span style="color: yellow;">TBD</span>** |
|
||||
| [W8A16 MOE](https://github.com/sgl-project/sglang/pull/16364) | MoE | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | **<span style="color: yellow;">TBD</span>** |
|
||||
|
||||
[Auto-round on Ascend support](https://github.com/sgl-project/sglang/pull/16699)
|
||||
| Quantization scheme | Layer type | A2 Supported | A3 Supported | A5 Supported |
|
||||
|--------------------------------|--------------------------|:----------------------------------------:|:----------------------------------------:|:-----------------------------------------:|
|
||||
| W4A16 | Linear | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | **<span style="color: yellow;">TBD</span>** |
|
||||
| W8A16 | Linear | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | **<span style="color: yellow;">TBD</span>** |
|
||||
| W4A16 | MoE | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | **<span style="color: yellow;">TBD</span>** |
|
||||
| W8A16 | MoE | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | **<span style="color: yellow;">TBD</span>** |
|
||||
|
||||
Compressed-tensors (LLM Compressor) on Ascend support:
|
||||
| Quantization scheme | Layer type | A2 Supported | A3 Supported | A5 Supported |
|
||||
|-----------------------------------------------------------------------------------------------|--------------------------|:----------------------------------------:|:----------------------------------------:|:-----------------------------------------:|
|
||||
| [W8A8 dynamic](https://github.com/sgl-project/sglang/pull/14504) | Linear | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | **<span style="color: yellow;">TBD</span>** |
|
||||
| [W4A8 dynamic with/without activation clip](https://github.com/sgl-project/sglang/pull/14736) | MoE | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | **<span style="color: yellow;">TBD</span>** |
|
||||
| [W4A16 MOE](https://github.com/sgl-project/sglang/pull/12759) | MoE | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | **<span style="color: yellow;">TBD</span>** |
|
||||
| [W8A8 dynamic](https://github.com/sgl-project/sglang/pull/14504) | MoE | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | **<span style="color: yellow;">TBD</span>** |
|
||||
|
||||
[GGUF on Ascend support](https://github.com/sgl-project/sglang/pull/17883)
|
||||
|
||||
in progress
|
||||
+1
-1
@@ -228,4 +228,4 @@ Not test yet.
|
||||
|
||||
### Using Benchmark
|
||||
|
||||
Refer to [Benchmark and Profiling](../developer_guide/benchmark_and_profiling.md) for details.
|
||||
Refer to [Benchmark and Profiling](../../developer_guide/benchmark_and_profiling.md) for details.
|
||||
@@ -7,6 +7,7 @@ Ascend NPUs
|
||||
ascend_npu.md
|
||||
ascend_npu_support_features.md
|
||||
ascend_npu_support_models.md
|
||||
ascend_npu_quantization.md
|
||||
ascend_npu_deepseek_example.md
|
||||
ascend_npu_qwen3_examples.md
|
||||
mindspore_backend.md
|
||||
@@ -1,27 +0,0 @@
|
||||
Quantization on Ascend.
|
||||
|
||||
To load already quantized models, simply load the model weights and config. Again, if the model has been quantized offline, there's no need to add `--quantization` argument when starting the engine. The quantization method will be automatically parsed from the downloaded `quant_model_description.json` or `config.json` config.
|
||||
|
||||
[ModelSlim on Ascend support](https://github.com/sgl-project/sglang/pull/14504):
|
||||
- [x] W4A4 dynamic linear
|
||||
- [x] W8A8 static linear
|
||||
- [x] W8A8 dynamic linear
|
||||
- [x] W4A4 dynamic MOE
|
||||
- [x] W4A8 dynamic MOE
|
||||
- [x] W8A8 dynamic MOE
|
||||
|
||||
[AWQ on Ascend support](https://github.com/sgl-project/sglang/pull/10158):
|
||||
- [x] W4A16 linear
|
||||
- [x] W8A16 linear # Need to test
|
||||
- [x] W4A16 MOE # Need to test
|
||||
|
||||
Compressed-tensors (LLM Compressor) on Ascend support:
|
||||
- [x] [W4A8 dynamic MOE with/without activation clip](https://github.com/sgl-project/sglang/pull/14736) # Need to test
|
||||
- [x] [W4A16 MOE](https://github.com/sgl-project/sglang/pull/12759)
|
||||
- [x] [W8A8 dynamic linear](https://github.com/sgl-project/sglang/pull/14504)
|
||||
- [x] [W8A8 dynamic MOE](https://github.com/sgl-project/sglang/pull/14504)
|
||||
|
||||
Diffusion model [modelslim](https://github.com/sgl-project/sglang/pull/17996) quantization on Ascend support:
|
||||
- [x] W4A4 dynamic linear
|
||||
- [x] W8A8 static linear
|
||||
- [x] W8A8 dynamic linear
|
||||
Reference in New Issue
Block a user