minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity

[NPU] [DOC] Rename NPU hardware to Ascend A2/A3 Series product (#39389)

Browse Source
This commit is contained in:
amote-i
2026-09-15 10:33:14 +08:00
committed by GitHub
parent 5dde6e8f02
commit bdf8886ad3
66 changed files with 759 additions and 759 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/cookbook/autoregressive/DeepSeek/DeepSeek-R1.mdx
+1 -1
View File
@@ -79,7 +79,7 @@ DeepSeek-R1 shares the same MoE architecture as DeepSeek-V3, so the same hardwar
</tr>
<tr>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><strong>INT8</strong></td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>16× A100/A800, 32× L40S, Xeon 6980P CPU, 4× Atlas 800I A3</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>16× A100/A800, 32× L40S, Xeon 6980P CPU, 4× Ascend A3 Series</td>
</tr>
<tr>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><strong>W4A8 / AWQ / MXFP4 / NVFP4</strong></td>
docs/cookbook/autoregressive/DeepSeek/DeepSeek-V3.mdx
+2 -2
View File
@@ -58,7 +58,7 @@ import { DeepSeekV3Deployment } from "/src/snippets/autoregressive/deepseek-v3-d
</tr>
<tr>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><strong>INT8</strong></td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>16× A100/A800, 32× L40S, Xeon 6980P CPU, 4× Atlas 800I A3</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>16× A100/A800, 32× L40S, Xeon 6980P CPU, 4× Ascend A3 Series</td>
</tr>
<tr>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><strong>W4A8 / AWQ / MXFP4 / NVFP4</strong></td>
@@ -425,7 +425,7 @@ For multi-node serving and hardware-specific examples:
- [16× A100 INT8](https://github.com/sgl-project/sglang/tree/main/benchmark/deepseek_v3#example-serving-with-16-a100a800-with-int8-quantization)
- [32× L40S INT8](https://github.com/sgl-project/sglang/tree/main/benchmark/deepseek_v3#example-serving-with-32-l40s-with-int8-quantization)
- [Xeon 6980P CPU](../../../docs/hardware-platforms/cpu_server#example-running-deepseek-v3-1-terminus)
- [4× Atlas 800I A3 (int8)](../../../docs/hardware-platforms/ascend-npus/model-deployment/tutorials/deepseek_r1#multi-node-pd-disaggregation-deployment)
- [4× Ascend A3 Series (int8)](../../../docs/hardware-platforms/ascend-npus/model-deployment/tutorials/deepseek_r1#multi-node-pd-disaggregation-deployment)
**Blog references for large-scale deployment:**
docs/cookbook/autoregressive/DeepSeek/DeepSeek-V3_1.mdx
+1 -1
View File
@@ -56,7 +56,7 @@ DeepSeek-V3.1 shares the same model architecture as DeepSeek-V3, so the same har
</tr>
<tr>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><strong>INT8</strong></td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>16× A100/A800, 32× L40S, Xeon 6980P CPU, 4× Atlas 800I A3</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>16× A100/A800, 32× L40S, Xeon 6980P CPU, 4× Ascend A3 Series</td>
</tr>
<tr>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><strong>W4A8 / AWQ / MXFP4 / NVFP4</strong></td>
docs/cookbook/autoregressive/DeepSeek/DeepSeek-V3_2.mdx
+1 -1
View File
@@ -40,7 +40,7 @@ docker pull lmsysorg/sglang:v0.5.8-rocm700-mi35x
# Prefer the newest MI30x ROCm image tag from Docker Hub when available, or build from source.
docker pull lmsysorg/sglang:v0.5.8-rocm700-mi30x
# Ascend NPU (Atlas 800I A2 / A3)
# Ascend NPU (A2/A3 Series)
docker pull lmsysorg/sglang:dsv32-a2
docker pull lmsysorg/sglang:dsv32-a3
```
docs/cookbook/autoregressive/Moonshotai/Kimi-K3.mdx
+5 -5
View File
@@ -56,7 +56,7 @@ For host and platform setup, see the
</Accordion>
Pick your hardware, then the deployment shape and operating point. Node count follows the hardware recipe (B200 2×8, GB200 4×4, H100 4×8, B300 1×8, H200 2×8 — 4×8 on Unified High-Throughput, GB300 2×4, MI350X/MI355X 1×8, Atlas 800I A3 4×8 — 32 cards / 64 dies), so it is not a separate choice. If you serve the NVFP4 checkpoint (`nvidia/Kimi-K3-NVFP4`, the **Quantization** row in the panel below), use the `lmsysorg/sglang:dev-dev-kimi-k3-nvfp4` image.
Pick your hardware, then the deployment shape and operating point. Node count follows the hardware recipe (B200 2×8, GB200 4×4, H100 4×8, B300 1×8, H200 2×8 — 4×8 on Unified High-Throughput, GB300 2×4, MI350X/MI355X 1×8, Ascend A3 Series 4×8 — 32 cards / 64 dies), so it is not a separate choice. If you serve the NVFP4 checkpoint (`nvidia/Kimi-K3-NVFP4`, the **Quantization** row in the panel below), use the `lmsysorg/sglang:dev-dev-kimi-k3-nvfp4` image.
**PD Mode** — `Unified` serves prefill and decode together. `Prefill` / `Decode` split them into dedicated pools (see [PD disaggregation](#3-4-pd-disaggregation)); `Prefill` ships two strategies, both chunked at 16k. On the 8-GPU platforms (B300 1×8, GB300 2×4), `Default` is TP8 and `Long-Context` is `--pp-size 8 --tp-size 1`. On the 16-GPU platforms (B200 2×8, GB200 4×4), both are `--pp-size 16 --tp-size 1` and differ only in `--mem-fraction-static` (0.85 vs 0.90) — deep PP is the throughput shape there, not just the long-context one (see [Deep PP](#deep-pp-for-prefill)).
@@ -71,7 +71,7 @@ Pick your hardware, then the deployment shape and operating point. Node count fo
**Spec Decode** — layers onto the strategy without changing it, on every platform except B200. DSPARK proposes 7 draft tokens per step (tune in the Playground) and requires `pp_size == 1`, so on B200 it also drops the pipeline and re-lays the same 16 GPUs flat: PP2 × TP8 → TP16, PP2 × DCPEP8 → DCPEP16. DFLASH has no published draft checkpoint. The win is largest on short interactive traffic and fades as the prompt grows.
<Note>
`--mamba-full-memory-ratio` is the one sizing flag, computed live: set your average request length in the [Mamba ratio calculator](#mamba-ratio-calculator); everything else follows the panels, and the result is pinned into the command. (The Atlas 800I A3 uses `--max-mamba-cache-size` instead.)
`--mamba-full-memory-ratio` is the one sizing flag, computed live: set your average request length in the [Mamba ratio calculator](#mamba-ratio-calculator); everything else follows the panels, and the result is pinned into the command. (The Ascend A3 Series uses `--max-mamba-cache-size` instead.)
</Note>
import { Deployment } from "/src/snippets/_deployment.jsx";
@@ -146,7 +146,7 @@ not been re-measured on any cell — re-measure before you rely on one.
## 2. Configuration Tips
**Memory: two pools, one flag.** K3 splits static memory into a worst-case-reserved **KDA state pool** (it sets the concurrency ceiling) and a paged **MLA KV pool**, divided by `--mamba-full-memory-ratio`. The command panel pins that flag to the [calculator](#mamba-ratio-calculator)'s output — set your average request length there; every other calculator input follows the panels. (On the Atlas 800I A3: `--max-mamba-cache-size`, no calculator.) After boot, read back `max_total_num_tokens` (the KV side) and the admitted-request cap (the state side).
**Memory: two pools, one flag.** K3 splits static memory into a worst-case-reserved **KDA state pool** (it sets the concurrency ceiling) and a paged **MLA KV pool**, divided by `--mamba-full-memory-ratio`. The command panel pins that flag to the [calculator](#mamba-ratio-calculator)'s output — set your average request length there; every other calculator input follows the panels. (On the Ascend A3 Series: `--max-mamba-cache-size`, no calculator.) After boot, read back `max_total_num_tokens` (the KV side) and the admitted-request cap (the state side).
Capacity levers, all in the Playground. Each trades precision or cache behavior for capacity — re-verify accuracy on your workload:
@@ -179,7 +179,7 @@ Speculation: DSPARK holds block size + 1 (= 8) intermediate states per request
| H200 2×8 (4×8 on Unified High-Throughput) | TP16/EP16 + symm-mem, Marlin + FlashMLA; High-Throughput widens to TP32/EP32 over 4 nodes at mem-frac 0.90 with `extra_buffer_lazy` | same block on every node; export the cross-node NIC (`GLOO_SOCKET_IFNAME` / `NCCL_SOCKET_IFNAME`, `SGLANG_HOST_IP`); keep `NCCL_MNNVL_ENABLE=1 NCCL_CUMEM_ENABLE=1` |
| H100 4×8 | TP32/EP32, Marlin + FlashMLA | SM90a build of the K3 image; pin NCCL/Gloo to the same NIC on all nodes; least post-weight headroom (80 GB) |
| MI350X/MI355X 1×8 | TP8 ROCm/AITER | AITER A8W4 FlyDSL MoE, Triton attention (`SGLANG_MLA_DECODE_TUNE=1` for gfx950 MLA decode geometry), graph bs up to 256, fp8 kvcache; DSPARK supported. Activation-quant and fused-KDA-decode knobs: [AMD ROCm/AITER environment](#amd-env) |
| Atlas 800I A3 4×8 (32 cards / 64 dies) | TP64/DP4 + DeepEP | PD-mixed `Unified` only; DSPARK baked in; pin `GLOO`/`HCCL_SOCKET_IFNAME` on every node |
| Ascend A3 Series 4×8 (32 cards / 64 dies) | TP64/DP4 + DeepEP | PD-mixed `Unified` only; DSPARK baked in; pin `GLOO`/`HCCL_SOCKET_IFNAME` on every node |
**DCP notes** — the DCP cells are Balanced and High-Throughput on every Blackwell platform, in both the `Unified` and `Decode` roles:
@@ -239,7 +239,7 @@ Pending update...
### 3.2 Tool Calling
Enable the `kimi_k3` tool-call parser (toggle **Tool Call Parser** in the **Parsers** card of the [Playground above](#playground)) to surface structured tool calls via `message.tool_calls`. Because K3 is a thinking model, the follow-up turn may put text in `reasoning_content` as well as `content` — print both. (Not yet supported on the Atlas 800I A3.)
Enable the `kimi_k3` tool-call parser (toggle **Tool Call Parser** in the **Parsers** card of the [Playground above](#playground)) to surface structured tool calls via `message.tool_calls`. Because K3 is a thinking model, the follow-up turn may put text in `reasoning_content` as well as `content` — print both. (Not yet supported on the Ascend A3 Series.)
<Accordion title="Tool Calling Example (Python)">
docs/cookbook/autoregressive/Qwen/Qwen3.8-27B.mdx
+1 -1
View File
@@ -286,7 +286,7 @@ row back to Auto.
candidates through the target `lm_head`,
including quantized heads, so it runs on the NVFP4 checkpoint (whose head
is NVFP4-packed; the BF16 and FP8 checkpoints keep a dense head).
The Ascend comparison in #35629 used a 910C with BF16 target weights,
The Ascend comparison in #35629 used an A3 Series device with BF16 target weights,
`--tp-size 2 --attention-backend ascend --mamba-ssm-dtype bfloat16
--mamba-radix-cache-strategy extra_buffer`, and disabled RadixCache for both
baseline and DFlash2 to exclude cache warm-up and prefix reuse. The DFlash2
docs/cookbook/diffusion/FLUX/FLUX.mdx
+5 -5
View File
@@ -34,7 +34,7 @@ This section provides deployment configurations optimized for different hardware
FLUX models are optimized for high-quality image generation. The recommended launch configurations vary by hardware and model version.
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform and model version. SGLang supports serving FLUX on NVIDIA B200, H200, H100, and AMD MI355X, MI325X, MI300X GPUs and Ascend A2, A3 NPUs.
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform and model version. SGLang supports serving FLUX on NVIDIA B200, H200, H100, and AMD MI355X, MI325X, MI300X GPUs and Ascend A2/A3 Series NPUs.
<FluxDeployment />
@@ -277,11 +277,11 @@ Test Environment:
============================================================
```
</Tab>
<Tab title="Ascend A3">
<Tab title="Ascend A3 Series">
**Server Command**:
```shell Command
#One A3 card has 2 npu chips
#One A3 Series card has 2 npu chips
sglang serve --tp-size 2 --sp-degree 1 --model-path black-forest-labs/FLUX.1-dev --num-gpus 2
```
@@ -363,11 +363,11 @@ Test Environment:
============================================================
```
</Tab>
<Tab title="Ascend A3">
<Tab title="Ascend A3 Series">
**Server Command** :
```shell Command
#One A3 card has 2 npu chips
#One A3 Series card has 2 npu chips
sglang serve --tp-size 2 --sp-degree 1 --model-path black-forest-labs/FLUX.1-dev --num-gpus 2
```
docs/cookbook/diffusion/Qwen-Image/Qwen-Image.mdx
+4 -4
View File
@@ -27,7 +27,7 @@ This section provides deployment configurations optimized for different hardware
### 3.1 Basic Configuration
Qwen-Image is a text-to-image model. The recommended launch configurations vary by hardware. SGLang supports serving Qwen-Image on NVIDIA B200, B300, H200, H100, AMD MI300X, MI325X, MI355X GPUs and Ascend A2, A3 NPUs.
Qwen-Image is a text-to-image model. The recommended launch configurations vary by hardware. SGLang supports serving Qwen-Image on NVIDIA B200, B300, H200, H100, AMD MI300X, MI325X, MI355X GPUs and Ascend A2/A3 Series NPUs.
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform.
@@ -311,11 +311,11 @@ Test Environment:
============================================================
```
</Tab>
<Tab title="Ascend A3">
<Tab title="Ascend A3 Series">
**Server Command**:
```shell Command
#One A3 card has 2 npu chips
#One A3 Series card has 2 npu chips
sglang serve --tp-size 2 --sp-degree 1 --model-path Qwen/Qwen-Image --num-gpus 2
```
@@ -391,7 +391,7 @@ Test Environment:
============================================================
```
</Tab>
<Tab title="Ascend A3">
<Tab title="Ascend A3 Series">
**Benchmark Command**:
```shell Command
docs/cookbook/diffusion/Wan/Wan2.1.mdx
+5 -5
View File
@@ -27,7 +27,7 @@ This section provides deployment configurations optimized for different hardware
### 3.1 Basic Configuration
The Wan2.1 series offers models in multiple sizes and resolutions. SGLang supports Wan2.1 deployment on NVIDIA B200, B300, H200, H100, and AMD MI300X, MI325X, MI355X GPUs and Ascend A2, A3 NPUs. The recommended launch configurations vary by hardware, model size, and memory headroom.
The Wan2.1 series offers models in multiple sizes and resolutions. SGLang supports Wan2.1 deployment on NVIDIA B200, B300, H200, H100, and AMD MI300X, MI325X, MI355X GPUs and Ascend A2/A3 Series NPUs. The recommended launch configurations vary by hardware, model size, and memory headroom.
**Interactive Command Generator**: Use the configuration selector below to automatically generate an appropriate deployment command for your model variant and options.
@@ -217,11 +217,11 @@ You can use the built-in SGLang diffusion benchmark script to evaluate Wan2.1 pe
```
</Tab>
<Tab title="Ascend A3">
<Tab title="Ascend A3 Series">
**Server Command**:
```bash Command
#One A3 card has 2 npu chips. Benchmark was did with two A3 cards
#One A3 Series card has 2 npu chips. Benchmark was done with two A3 Series cards
sglang serve \
--model-path /models/Wan-AI/Wan2.1-T2V-14B-Diffusers/ \
--tp-size 2 \
@@ -321,11 +321,11 @@ You can use the built-in SGLang diffusion benchmark script to evaluate Wan2.1 pe
```
</Tab>
<Tab title="Ascend A3">
<Tab title="Ascend A3 Series">
**Server Command**:
```bash Command
#One A3 card has 2 npu chips. Benchmark was did with two Atlas 3 cards
#One A3 Series card has 2 npu chips. Benchmark was done with two A3 Series cards
SGLANG_CACHE_DIT_FN=2 \
SGLANG_CACHE_DIT_BN=1 \
SGLANG_CACHE_DIT_WARMUP=4 \
docs/cookbook/diffusion/Wan/Wan2.2.mdx
+5 -5
View File
@@ -35,7 +35,7 @@ This section provides deployment configurations optimized for different hardware
The Wan2.2 series offers models in various sizes, architectures and input types, optimized for different hardware platforms. The recommended launch configurations vary by hardware and model size.
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model size. SGLang supports serving Wan2.2 on NVIDIA B200, H200, AMD MI300X, MI325X, MI355X GPUs and Ascend A2, A3 NPUs.
**Interactive Command Generator**: Use the configuration selector below to automatically generate the appropriate deployment command for your hardware platform, model size. SGLang supports serving Wan2.2 on NVIDIA B200, H200, AMD MI300X, MI325X, MI355X GPUs and Ascend A2/A3 Series NPUs.
<Wan22Deployment />
@@ -297,10 +297,10 @@ Test Environment:
```
</Tab>
<Tab title="Ascend A3">
<Tab title="Ascend A3 Series">
**Server Command**:
```shell Command
#One A3 card has 2 npu chips. Using four A3 cards in benchmarking
#One A3 Series card has 2 npu chips. Using four A3 Series cards in benchmarking
sglang serve \
--model-path /models/Wan-AI/Wan2.2-T2V-A14B-Diffusers/ \
--tp-size 2 \
@@ -399,11 +399,11 @@ Test Environment:
```
</Tab>
<Tab title="Ascend A3">
<Tab title="Ascend A3 Series">
**Server Command**:
```shell Command
#One A3 card has 2 npu chips. Using four A3 cards in benchmarking
#One A3 Series card has 2 npu chips. Using four A3 Series cards in benchmarking
SGLANG_CACHE_DIT_FN=2 \
SGLANG_CACHE_DIT_BN=1 \
SGLANG_CACHE_DIT_WARMUP=4 \
docs/cookbook/diffusion/Z-Image/Z-Image-Turbo.mdx
+3 -3
View File
@@ -249,11 +249,11 @@ Test Environment:
============================================================
```
</Tab>
<Tab title="Ascend A3">
<Tab title="Ascend A3 Series">
**Server Command**:
```shell Command
#One A3 card has 2 npu chips
#One A3 Series card has 2 npu chips
sglang serve --model-path Tongyi-MAI/Z-Image-Turbo --tp-size 2 --sp-degree 1 --num-gpus 2
```
@@ -329,7 +329,7 @@ Test Environment:
============================================================
```
</Tab>
<Tab title="Ascend A3">
<Tab title="Ascend A3 Series">
**Benchmark Command**:
```shell Command