Update documentation for Ascend NPU Guide (#28284)
This commit is contained in:
@@ -218,12 +218,12 @@ These arguments and environment variables are critical for tuning decode perform
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td>`--dp-size`</td>
|
||||
<td>`--dp-size` / `--data-parallel-size`</td>
|
||||
<td>Sets the data parallelism degree for the decode server. With DP attention enabled, attention layers are sharded across DP ranks while FFN/MoE layers use tensor parallelism. Higher values create more independent decode instances, increasing throughput through parallel request processing. Choose a value that divides evenly into your total card count, with remaining cards used for TP/EP.</td>
|
||||
<td>`8`</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>`--ep`</td>
|
||||
<td>`--ep` / `--ep-size` / `--expert-parallel-size`</td>
|
||||
<td>Sets the expert parallelism degree. For MoE models, this distributes experts across cards, reducing per-card expert loading overhead and enabling all-to-all dispatch. The code default is `1`; set explicitly for MoE models. The optimal value depends on your model's expert count and architecture. DeepSeek V3.2 with 256 routed experts uses `ep=32`. For models with fewer experts, use a proportionally smaller value.</td>
|
||||
<td>`32`</td>
|
||||
</tr>
|
||||
|
||||
@@ -305,7 +305,12 @@ python -m sglang.bench_serving \
|
||||
--max-concurrency 32
|
||||
```
|
||||
|
||||
<Note>`--dataset-name random` samples token IDs from the ShareGPT dataset to generate realistic input; the first run downloads ShareGPT from Hugging Face automatically. Set `export HF_ENDPOINT=https://hf-mirror.com` if network is not available.</Note>
|
||||
<Note>
|
||||
`--dataset-name random` samples token IDs from the ShareGPT dataset to generate realistic input; the first run downloads ShareGPT from Hugging Face automatically.
|
||||
|
||||
1. If you have network issues, set `export HF_ENDPOINT=https://hf-mirror.com` to use domestic mirror.
|
||||
2. If downloading still fails, manually download the dataset file `ShareGPT_V3_unfiltered_cleaned_split.json` locally, upload it to your server, then specify the file directory via `--dataset-path` to run offline.
|
||||
</Note>
|
||||
|
||||
<Tip>Set `--random-range-ratio 1` for fixed input/output lengths (recommended for consistent comparisons) or `0` (default) for uniform distribution. Add `--request-rate` to control the request rate. For all backends, datasets, and advanced options, see the full [Bench Serving Guide](/docs/developer_guide/bench_serving).</Tip>
|
||||
|
||||
|
||||
@@ -18,7 +18,7 @@ Ensure sufficient disk space before proceeding. The Docker image requires at lea
|
||||
</Warning>
|
||||
|
||||
<Tip>
|
||||
We publish both **stable releases** and **daily builds**. Choose a stable release tag (e.g., `v0.5.10-npu.rc1-a3`) if you prefer a validated version, or a daily build tag (e.g., `main-cann9.0.0-a3`) if you need the latest development changes.
|
||||
We publish both **stable releases** and **daily builds**. Choose a stable release tag (e.g., `v0.5.13.post1-cann9.0.0-a3`) if you prefer a validated version, or a daily build tag (e.g., `main-cann9.0.0-a3`) if you need the latest development changes.
|
||||
</Tip>
|
||||
|
||||
<Tabs>
|
||||
@@ -26,7 +26,7 @@ We publish both **stable releases** and **daily builds**. Choose a stable releas
|
||||
|
||||
```shell Command
|
||||
# Stable release
|
||||
export IMAGE=quay.io/ascend/sglang:v0.5.10-npu.rc1-a3
|
||||
export IMAGE=quay.io/ascend/sglang:v0.5.13.post1-cann9.0.0-a3
|
||||
|
||||
# Daily build
|
||||
export IMAGE=quay.io/ascend/sglang:main-cann9.0.0-a3
|
||||
@@ -53,7 +53,7 @@ docker run -it --rm --privileged --network=host --ipc=host --shm-size=16g \
|
||||
|
||||
```shell Command
|
||||
# Stable release
|
||||
export IMAGE=quay.io/ascend/sglang:v0.5.12-cann8.5.0-910b
|
||||
export IMAGE=quay.io/ascend/sglang:v0.5.13.post1-cann9.0.0-910b
|
||||
|
||||
# Daily build
|
||||
export IMAGE=quay.io/ascend/sglang:main-cann9.0.0-910b
|
||||
|
||||
@@ -1827,7 +1827,7 @@ click [Server Arguments](../../advanced_features/server_arguments).
|
||||
<tr>
|
||||
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--hicache-write-policy`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`write_through`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Currently only `write_back` supported</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`write_back`,<br/> `write_through`,<br/> `write_through_selective`</td>
|
||||
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>A2, A3</td>
|
||||
</tr>
|
||||
<tr>
|
||||
|
||||
Reference in New Issue
Block a user