diff --git a/docs/platforms/ascend/ascend_contribution_guide.md b/docs/platforms/ascend/ascend_contribution_guide.md index 5823bd22a..e25cb9f42 100644 --- a/docs/platforms/ascend/ascend_contribution_guide.md +++ b/docs/platforms/ascend/ascend_contribution_guide.md @@ -38,7 +38,7 @@ If you add a new feature or fix a bug, please add corresponding unit tests to en SGLang uses Python's built-in [unittest](https://docs.python.org/3/library/unittest.html) framework. For detailed instructions on running tests and integrating them into CI, refer to [test/README.md](https://github.com/sgl-project/sglang/tree/main/test/README.md). -If you need to use model which is not in ```python/sglang/test/ascend/test_ascend_utils.py`` list. Follow these steps: +If you need to use model which is not in `python/sglang/test/ascend/test_ascend_utils.py` list. Follow these steps: 1. Register account and upload your model to [modelscope](https://modelscope.cn/models). 2. Make sure your model is pre-cached on the CI server and is on the way "/data/ascend-ci-share-pkking-sglang/modelscope/hub/models/{your_model_repo}/{your_model}". If this is not the case, use following command on CI server: diff --git a/docs/platforms/ascend/ascend_npu.md b/docs/platforms/ascend/ascend_npu.md index 6a0eef31d..a689a3cfc 100644 --- a/docs/platforms/ascend/ascend_npu.md +++ b/docs/platforms/ascend/ascend_npu.md @@ -38,7 +38,7 @@ conda activate sglang_npu #### CANN -Prior to start work with SGLang on Ascend you need to install CANN Toolkit, Kernels operator package and NNAL version 8.3.RC2 or higher, check the [installation guide](https://www.hiascend.com/document/detail/zh/CANNCommunityEdition/83RC1/softwareinst/instg/instg_0008.html?Mode=PmIns&InstallType=local&OS=openEuler&Software=cannToolKit) +Prior to start work with SGLang on Ascend you need to install CANN Toolkit, Kernels operator package and NNAL version 8.5.0, check the [installation guide](https://www.hiascend.com/document/detail/zh/CANNCommunityEdition/850/softwareinst/instg/instg_0008.html?Mode=PmIns&InstallType=local&OS=openEuler&Software=cannToolKit) #### MemFabric-Hybrid diff --git a/docs/platforms/ascend/ascend_npu_quantization.md b/docs/platforms/ascend/ascend_npu_quantization.md index 8b2e30ba1..ca579156c 100644 --- a/docs/platforms/ascend/ascend_npu_quantization.md +++ b/docs/platforms/ascend/ascend_npu_quantization.md @@ -2,7 +2,7 @@ To load already quantized models, simply load the model weights and config. Again, if the model has been quantized offline, there's no need to add `--quantization` argument when starting the engine. The quantization method will be automatically parsed from the downloaded `quant_model_description.json` or `config.json` config. -SGLang support **mix-bits** quantization (independently defines and loads each layer depending on the type of quantification specified in the `quant_model_description'.json`). [Advanced mix-bits for MoE](https://github.com/sgl-project/sglang/pull/17361) in progress, will add independent quantization determination for the w13 (up-gate) and w2 (down) layers). +SGLang support **mix-bits** quantization (independently defines and loads each layer depending on the type of quantification specified in the `quant_model_description'.json`). [Advanced mix-bits for MoE](https://github.com/sgl-project/sglang/pull/17361) in progress, will add independent quantization determination for the w13 (up-gate) and w2 (down) layers. [ModelSlim on Ascend support](https://github.com/sgl-project/sglang/pull/14504) | Quantization scheme | Layer type | A2 Supported | A3 Supported | A5 Supported | Diffusion models | diff --git a/docs/platforms/ascend/ascend_npu_support_features.md b/docs/platforms/ascend/ascend_npu_support_features.md index 54a9cf813..3d36ca76f 100644 --- a/docs/platforms/ascend/ascend_npu_support_features.md +++ b/docs/platforms/ascend/ascend_npu_support_features.md @@ -86,7 +86,7 @@ click [Server Arguments](https://docs.sglang.io/advanced_features/server_argumen |----------------------------------------------------|----------|---------------------------|:----------------:| | `--device` | `None` | Type: str | A2, A3 | | `--tensor-parallel-size`
`--tp-size` | `1` | Type: int | A2, A3 | -| `--pipeline-parallel-size`
`--pp-size` | `1` | Type: int | A2, A3 | +| `--pipeline-parallel-size`
`--pp-size` | `1` | Type: int; Currently `2` not supported | A2, A3 | | `--attention-context-parallel-size`
`--attn-cp-size` | `1` | Type: int | A2, A3 | | `--moe-data-parallel-size`
`--moe-dp-size` | `1` | Type: int | A2, A3 | | `--pp-max-micro-batch-size` | `None` | Type: int | A2, A3 | @@ -184,6 +184,7 @@ click [Server Arguments](https://docs.sglang.io/advanced_features/server_argumen | Argument | Defaults | Options | Server supported | |--------------------------|----------|-------------------------------------|:----------------:| | `--enable-lora` | `False` | Bool flag
(set to enable) | A2, A3 | +| `--enable-lora-overlap-loading` | `False` | Bool flag
(set to enable) | A2, A3 | | `--max-lora-rank` | `None` | Type: int | A2, A3 | | `--lora-target-modules` | `None` | `all` | A2, A3 | | `--lora-paths` | `None` | Type: List[str] /
JSON objects | A2, A3 | @@ -252,16 +253,17 @@ click [Server Arguments](https://docs.sglang.io/advanced_features/server_argumen | `--deepep-mode` | `auto` | `normal`,
`low_latency`,
`auto` | A2, A3 | | `--deepep-config` | `None` | Type: str | Special for GPU | | `--ep-num-redundant-experts` | `0` | Type: int | A2, A3 | -| `--ep-dispatch-algorithm` | `None` | Type: str | A2, A3 | +| `--ep-dispatch-algorithm` | `None` | `static`,
`dynamic`,
`fake` | A2, A3 | | `--init-expert-location` | `trivial` | Type: str | A2, A3 | | `--enable-eplb` | `False` | bool flag
(set to enable) | A2, A3 | -| `--eplb-algorithm` | `auto` | Type: str | A2, A3 | +| `--eplb-algorithm` | `deepseek`| Type: str | A2, A3 | +| `--eplb-rebalance-num-iterations` | `1000` | Type: int | A2, A3 | | `--eplb-rebalance-layers-`
`per-chunk` | `None` | Type: int | A2, A3 | | `--eplb-min-rebalancing-`
`utilization-threshold` | `1.0` | Type: float | A2, A3 | -| `--expert-distribution-`
`recorder-mode` | `None` | Type: str | A2, A3 | +| `--expert-distribution-`
`recorder-mode` | `None` | `stat`,
`stat_approx`,
`per_pass`,
`per_token` | A2, A3 | | `--expert-distribution-`
`recorder-buffer-size` | `None` | Type: int | A2, A3 | | `--enable-expert-distribution-`
`metrics` | `False` | bool flag (set to enable) | A2, A3 | -| `--moe-dense-tp-size` | `None` | Type: int | A2, A3 | +| `--moe-dense-tp-size` | `None` | `1` | A2, A3 | | `--elastic-ep-backend` | `None` | `none`, `mooncake` | Special for GPU | | `--mooncake-ib-device` | `None` | Type: str | Special for GPU | @@ -357,8 +359,8 @@ click [Server Arguments](https://docs.sglang.io/advanced_features/server_argumen | `--allow-auto-truncate` | `False` | bool flag
(set to enable) | A2, A3 | | `--enable-custom-`
`logit-processor` | `False` | bool flag
(set to enable) | A2, A3 | | `--flashinfer-mla-`
`disable-ragged` | `False` | bool flag
(set to enable) | Special for GPU | -| `--disable-shared-`
`experts-fusion` | `False` | bool flag
(set to enable) | A2, A3 | -| `--disable-chunked-`
`prefix-cache` | `False` | bool flag
(set to enable) | A2, A3 | +| `--disable-shared-`
`experts-fusion` | `True` | bool flag
(set to enable) | A2, A3 | +| `--disable-chunked-`
`prefix-cache` | `True` | bool flag
(set to enable) | A2, A3 | | `--disable-fast-`
`image-processor` | `False` | bool flag
(set to enable) | A2, A3 | | `--keep-mm-feature-`
`on-device` | `False` | bool flag
(set to enable) | A2, A3 | | `--enable-return-`
`hidden-states` | `False` | bool flag
(set to enable) | A2, A3 |