fix issues for npu docs (#22307)
This commit is contained in:
amote-i
@@ -38,7 +38,7 @@ If you add a new feature or fix a bug, please add corresponding unit tests to en
|
||||
SGLang uses Python's built-in [unittest](https://docs.python.org/3/library/unittest.html) framework.
|
||||
For detailed instructions on running tests and integrating them into CI, refer to [test/README.md](https://github.com/sgl-project/sglang/tree/main/test/README.md).
|
||||
|
||||
If you need to use model which is not in ```python/sglang/test/ascend/test_ascend_utils.py`` list. Follow these steps:
|
||||
If you need to use model which is not in `python/sglang/test/ascend/test_ascend_utils.py` list. Follow these steps:
|
||||
1. Register account and upload your model to [modelscope](https://modelscope.cn/models).
|
||||
2. Make sure your model is pre-cached on the CI server and is on the way "/data/ascend-ci-share-pkking-sglang/modelscope/hub/models/{your_model_repo}/{your_model}".
|
||||
If this is not the case, use following command on CI server:
|
||||
|
||||
@@ -38,7 +38,7 @@ conda activate sglang_npu
|
||||
|
||||
#### CANN
|
||||
|
||||
Prior to start work with SGLang on Ascend you need to install CANN Toolkit, Kernels operator package and NNAL version 8.3.RC2 or higher, check the [installation guide](https://www.hiascend.com/document/detail/zh/CANNCommunityEdition/83RC1/softwareinst/instg/instg_0008.html?Mode=PmIns&InstallType=local&OS=openEuler&Software=cannToolKit)
|
||||
Prior to start work with SGLang on Ascend you need to install CANN Toolkit, Kernels operator package and NNAL version 8.5.0, check the [installation guide](https://www.hiascend.com/document/detail/zh/CANNCommunityEdition/850/softwareinst/instg/instg_0008.html?Mode=PmIns&InstallType=local&OS=openEuler&Software=cannToolKit)
|
||||
|
||||
#### MemFabric-Hybrid
|
||||
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
To load already quantized models, simply load the model weights and config. Again, if the model has been quantized offline, there's no need to add `--quantization` argument when starting the engine. The quantization method will be automatically parsed from the downloaded `quant_model_description.json` or `config.json` config.
|
||||
|
||||
SGLang support **mix-bits** quantization (independently defines and loads each layer depending on the type of quantification specified in the `quant_model_description'.json`). [Advanced mix-bits for MoE](https://github.com/sgl-project/sglang/pull/17361) in progress, will add independent quantization determination for the w13 (up-gate) and w2 (down) layers).
|
||||
SGLang support **mix-bits** quantization (independently defines and loads each layer depending on the type of quantification specified in the `quant_model_description'.json`). [Advanced mix-bits for MoE](https://github.com/sgl-project/sglang/pull/17361) in progress, will add independent quantization determination for the w13 (up-gate) and w2 (down) layers.
|
||||
|
||||
[ModelSlim on Ascend support](https://github.com/sgl-project/sglang/pull/14504)
|
||||
| Quantization scheme | Layer type | A2 Supported | A3 Supported | A5 Supported | Diffusion models |
|
||||
|
||||
@@ -86,7 +86,7 @@ click [Server Arguments](https://docs.sglang.io/advanced_features/server_argumen
|
||||
|----------------------------------------------------|----------|---------------------------|:----------------:|
|
||||
| `--device` | `None` | Type: str | A2, A3 |
|
||||
| `--tensor-parallel-size`<br/>`--tp-size` | `1` | Type: int | A2, A3 |
|
||||
| `--pipeline-parallel-size`<br/>`--pp-size` | `1` | Type: int | A2, A3 |
|
||||
| `--pipeline-parallel-size`<br/>`--pp-size` | `1` | Type: int; Currently `2` not supported | A2, A3 |
|
||||
| `--attention-context-parallel-size`<br/>`--attn-cp-size` | `1` | Type: int | A2, A3 |
|
||||
| `--moe-data-parallel-size`<br/>`--moe-dp-size` | `1` | Type: int | A2, A3 |
|
||||
| `--pp-max-micro-batch-size` | `None` | Type: int | A2, A3 |
|
||||
@@ -184,6 +184,7 @@ click [Server Arguments](https://docs.sglang.io/advanced_features/server_argumen
|
||||
| Argument | Defaults | Options | Server supported |
|
||||
|--------------------------|----------|-------------------------------------|:----------------:|
|
||||
| `--enable-lora` | `False` | Bool flag <br/>(set to enable) | A2, A3 |
|
||||
| `--enable-lora-overlap-loading` | `False` | Bool flag <br/>(set to enable) | A2, A3 |
|
||||
| `--max-lora-rank` | `None` | Type: int | A2, A3 |
|
||||
| `--lora-target-modules` | `None` | `all` | A2, A3 |
|
||||
| `--lora-paths` | `None` | Type: List[str] /<br/> JSON objects | A2, A3 |
|
||||
@@ -252,16 +253,17 @@ click [Server Arguments](https://docs.sglang.io/advanced_features/server_argumen
|
||||
| `--deepep-mode` | `auto` | `normal`, <br/>`low_latency`,<br/> `auto` | A2, A3 |
|
||||
| `--deepep-config` | `None` | Type: str | Special for GPU |
|
||||
| `--ep-num-redundant-experts` | `0` | Type: int | A2, A3 |
|
||||
| `--ep-dispatch-algorithm` | `None` | Type: str | A2, A3 |
|
||||
| `--ep-dispatch-algorithm` | `None` | `static`,<br/> `dynamic`,<br/> `fake` | A2, A3 |
|
||||
| `--init-expert-location` | `trivial` | Type: str | A2, A3 |
|
||||
| `--enable-eplb` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--eplb-algorithm` | `auto` | Type: str | A2, A3 |
|
||||
| `--eplb-algorithm` | `deepseek`| Type: str | A2, A3 |
|
||||
| `--eplb-rebalance-num-iterations` | `1000` | Type: int | A2, A3 |
|
||||
| `--eplb-rebalance-layers-`<br/>`per-chunk` | `None` | Type: int | A2, A3 |
|
||||
| `--eplb-min-rebalancing-`<br/>`utilization-threshold` | `1.0` | Type: float | A2, A3 |
|
||||
| `--expert-distribution-`<br/>`recorder-mode` | `None` | Type: str | A2, A3 |
|
||||
| `--expert-distribution-`<br/>`recorder-mode` | `None` | `stat`,<br/> `stat_approx`,<br/> `per_pass`,<br/> `per_token` | A2, A3 |
|
||||
| `--expert-distribution-`<br/>`recorder-buffer-size` | `None` | Type: int | A2, A3 |
|
||||
| `--enable-expert-distribution-`<br/>`metrics` | `False` | bool flag (set to enable) | A2, A3 |
|
||||
| `--moe-dense-tp-size` | `None` | Type: int | A2, A3 |
|
||||
| `--moe-dense-tp-size` | `None` | `1` | A2, A3 |
|
||||
| `--elastic-ep-backend` | `None` | `none`, `mooncake` | Special for GPU |
|
||||
| `--mooncake-ib-device` | `None` | Type: str | Special for GPU |
|
||||
|
||||
@@ -357,8 +359,8 @@ click [Server Arguments](https://docs.sglang.io/advanced_features/server_argumen
|
||||
| `--allow-auto-truncate` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--enable-custom-`<br/>`logit-processor` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--flashinfer-mla-`<br/>`disable-ragged` | `False` | bool flag<br/> (set to enable) | Special for GPU |
|
||||
| `--disable-shared-`<br/>`experts-fusion` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--disable-chunked-`<br/>`prefix-cache` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--disable-shared-`<br/>`experts-fusion` | `True` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--disable-chunked-`<br/>`prefix-cache` | `True` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--disable-fast-`<br/>`image-processor` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--keep-mm-feature-`<br/>`on-device` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--enable-return-`<br/>`hidden-states` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
|
||||
Reference in New Issue
Block a user