minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity

docs: sync legacy docs/-only updates into docs_new (Mintlify) (#27308)

Browse Source 
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
zijiexia
2026-06-04 19:45:13 -07:00
committed by GitHub
co-authored by Claude Opus 4.8
parent 46c58b5c70
commit c6c1f1a29a
18 changed files with 650 additions and 29 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs_new/docs/hardware-platforms/ascend-npus/ascend_contribution_guide.mdx
+4 -4
View File
@@ -94,10 +94,10 @@ Users with permission are listed in the [CI_PERMISSIONS.json](https://github.com
For CI to run on a pull request, it must have the "run-ci" label. Authorized users can add the label or rerun failed tests by commenting on the PR with one of these commands:
- `/tag-run-ci-label`: Adds the "run-ci" label. Every future commit will trigger CI. Add the `extra` argument (`/tag-run-ci-label extra`) to additionally apply the "run-ci-extra" label, opting the PR into the extra test workflow (`pr-test-extra.yml`).
- `/rerun-failed-ci`: Reruns the failed or flaky tests from the most recent commit.
- `/tag-and-rerun-ci`: A single command that performs both `/tag-run-ci-label` and `/rerun-failed-ci`. Accepts the same `extra` argument (`/tag-and-rerun-ci extra`).
- `/rerun-stage <stage-name>`: Reruns a specific test stage without waiting for its dependencies. This is useful when you want to quickly validate a fix for a specific test failure instead of waiting ~30 minutes for preceding stages to complete.
- `/tag-run-ci-label`: Adds the "run-ci" label. Only **future** commits trigger CI; the current commit is unaffected. Add the `extra` argument (`/tag-run-ci-label extra`) to additionally apply the "run-ci-extra" label, opting the PR into the extra test workflow (`pr-test-extra.yml`).
- `/rerun-failed-ci`: Reruns workflows from the latest commit with conclusion **failed, flaky, or skipped**.
- `/tag-and-rerun-ci`: Runs both. Use this on a fresh PR to kick off CI on the current commit — `/tag-run-ci-label` alone won't. Accepts the same `extra` argument (`/tag-and-rerun-ci extra`).
- `/rerun-stage <stage-name>`: Reruns a single test stage without waiting for its dependencies. Useful for quickly validating a specific test fix instead of waiting ~30 minutes for preceding stages.
If you have permission, the [Slash Command Handler](https://github.com/sgl-project/sglang/actions/workflows/slash-command-handler.yml) will run your command and react with a 👍 to your comment. It may take up to a few minutes for the reaction to appear. Here’s a usage [example](https://github.com/sgl-project/sglang/pull/14253#issuecomment-3599509302).
docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_quantization.mdx
+129
View File
@@ -315,3 +315,132 @@ python3 -m sglang.launch_server \
> - GGUF weights are pre-dequantized to FP16/BF16 during model loading on CPU, then transferred to NPU for inference. This trades higher memory usage for faster runtime performance (no per-forward-pass dequantization overhead).
> - MoE layers use `npu_grouped_matmul` and `npu_moe_init_routing` / `npu_moe_finalize_routing` for high-performance expert computation.
> - TP (tensor parallelism) sharding is supported for both dense and MoE GGUF models.
## Diffusion Model Quantization on Ascend NPU
SGLang-Diffusion supports MXFP8 online and offline quantization for diffusion models (such as Wan2.2) on Ascend NPUs. MXFP8 requires A5; the ModelSlim W8A8/W4A4 schemes work on A2/A3.
**Requirements for MXFP8:** CANN ≥ 8.0.RC3, Ascend A5
<table>
<thead>
<tr>
<th>Quantization method</th>
<th><code>quant_type</code> in JSON</th>
<th>Scheme class</th>
<th>Mode</th>
<th>A2/A3 Supported</th>
<th>A5 Supported</th>
<th>Trigger</th>
</tr>
</thead>
<tbody>
<tr>
<td>MXFP8 (W8A8)</td>
<td>—</td>
<td><code>MXFP8Config</code></td>
<td>Online</td>
<td><strong style={{color: 'red'}}>x</strong></td>
<td><strong style={{color: 'green'}}>√</strong></td>
<td><code>--quantization mxfp8</code></td>
</tr>
<tr>
<td>MXFP8 (W8A8)</td>
<td><code>W8A8_MXFP8</code></td>
<td><code>ModelSlimMXFP8Scheme</code></td>
<td>Offline</td>
<td><strong style={{color: 'red'}}>x</strong></td>
<td><strong style={{color: 'green'}}>√</strong></td>
<td>auto-detected from <code>quant_model_description.json</code></td>
</tr>
<tr>
<td>W8A8 static</td>
<td><code>W8A8</code></td>
<td><code>ModelSlimW8A8Int8</code></td>
<td>Offline</td>
<td><strong style={{color: 'green'}}>√</strong></td>
<td><strong style={{color: 'orange'}}>TBD</strong></td>
<td>auto-detected from <code>quant_model_description.json</code></td>
</tr>
<tr>
<td>W8A8 dynamic</td>
<td><code>W8A8_DYNAMIC</code></td>
<td><code>ModelSlimW8A8Int8</code></td>
<td>Offline</td>
<td><strong style={{color: 'green'}}>√</strong></td>
<td><strong style={{color: 'orange'}}>TBD</strong></td>
<td>auto-detected from <code>quant_model_description.json</code></td>
</tr>
<tr>
<td>W4A4 dynamic</td>
<td><code>W4A4_DYNAMIC</code></td>
<td><code>ModelSlimW4A4Int4</code></td>
<td>Offline</td>
<td><strong style={{color: 'green'}}>√</strong></td>
<td><strong style={{color: 'orange'}}>TBD</strong></td>
<td>auto-detected from <code>quant_model_description.json</code></td>
</tr>
</tbody>
</table>
### Online MXFP8 Quantization
Online quantization dynamically quantizes FP16/BF16 weights to MXFP8 at load time using `npu_dynamic_mx_quant` + `npu_quant_matmul` CANN kernels. Pass `--quantization mxfp8` to override auto-detection.
```bash Command
# Start the diffusion server with online MXFP8 quantization
sglang serve \
--model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers \
--quantization mxfp8 \
--num-gpus 4
```
```bash Command
# One-shot generation
sglang generate \
--model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers \
--quantization mxfp8 \
--prompt "a beautiful sunset over the mountains" \
--save-output
```
### Offline MXFP8 Quantization (ModelSlim)
For offline quantization, pre-quantize the model with msModelSlim and load the resulting checkpoint. The quantization scheme is auto-detected from `quant_model_description.json`, so no extra `--quantization` flag is needed.
**Step 1: Quantize with msModelSlim**
```bash Command
msmodelslim quant \
--model_path /path/to/wan2_2_float_weights \
--save_path /path/to/wan2_2_mxfp8_weights \
--device npu \
--model_type Wan2_2 \
--quant_type mxfp8 \
--trust_remote_code True
```
> Note: SGLang does not support quantized embeddings; disable embedding quantization when using msmodelslim.
**Step 2: Convert to Diffusers format**
msModelSlim saves quantized Wan2.2 weights in the original Wan format. Convert to Diffusers format using the provided repack script:
```bash Command
python python/sglang/multimodal_gen/tools/wan_repack.py \
--input-path /path/to/wan2_2_mxfp8_weights \
--output-path /path/to/wan2_2_mxfp8_diffusers
```
Then copy all files from the original Diffusers checkpoint (except the `transformer`/`transformer_2` folders) into the output directory.
**Step 3: Run inference**
```bash Command
sglang generate \
--model-path /path/to/wan2_2_mxfp8_diffusers \
--prompt "a beautiful sunset over the mountains" \
--save-output
```
For pre-quantized checkpoints available on ModelScope, see [modelscope/Eco-Tech](https://modelscope.cn/models/Eco-Tech).
docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_qwen3_examples.mdx
+80
View File
@@ -188,6 +188,86 @@ python3 -m sglang_router.launch_router \
--prometheus-port 29010
```
#### Running Qwen3-235B-A22B-Instruct-2507-W8A8 with Prefill Context Parallel (CP) on 2 x Atlas 800I A3
This example enables **Prefill Context Parallel** (`--enable-prefill-context-parallel`) to split the context across CP ranks during prefill, reducing per-device memory pressure and improving TTFT for long sequences. PD disaggregation is required.
> **Constraints**
> - Prefill side must set `--max-running-requests 1` (PCP only supports batch_size=1)
> - `--attn-cp-size` must evenly divide `--tp-size`; each CP rank occupies `tp_size / cp_size` NPUs
**Prefill node `<PREFILL_HOST_IP>`:**
```shell Launch Server
export SGLANG_SET_CPU_AFFINITY=1
export ASCEND_MF_STORE_URL="tcp://<PREFILL_HOST_IP>:23456"
export ASCEND_USE_FIA=True
python3 -m sglang.launch_server \
--model-path /mnt/share/weights/Qwen3-235B-A22B-Instruct-2507-W8A8 \
--trust-remote-code \
--disaggregation-mode prefill \
--disaggregation-transfer-backend ascend \
--disaggregation-bootstrap-port 8995 \
--quantization modelslim \
--attention-backend ascend \
--skip-server-warmup \
--mem-fraction-static 0.7 \
--chunked-prefill-size 32768 \
--device npu \
--base-gpu-id 0 \
--tp-size 16 \
--enable-prefill-context-parallel \
--attn-cp-size 2 \
--moe-dp-size 2 \
--max-running-requests 1 \
--host <PREFILL_HOST_IP> \
--port 8000 \
--nnodes 1 \
--node-rank 0 \
--dist-init-addr <PREFILL_HOST_IP>:6688
```
Key parameters for PCP:
| Parameter | Value | Description |
|-----------|-------|-------------|
| `--enable-prefill-context-parallel` | flag | Enable PCP feature |
| `--attn-cp-size` | 2 | Split context across 2 CP ranks (each rank handles half the sequence) |
| `--moe-dp-size` | 2 | MoE DP size, should match `--attn-cp-size` |
| `--max-running-requests` | 1 | Required by PCP (batch_size=1 constraint) |
**Decode node (`<DECODE_HOST_IP>`):**
```shell Launch Server
export ASCEND_MF_STORE_URL="tcp://141.61.39.231:23456"
export ASCEND_USE_FIA=True
python3 -m sglang.launch_server \
--model-path /mnt/share/weights/Qwen3-235B-A22B-Instruct-2507-W8A8 \
--trust-remote-code \
--disaggregation-mode decode \
--disaggregation-transfer-backend ascend \
--quantization modelslim \
--attention-backend ascend \
--disable-radix-cache \
--disable-cuda-graph \
--mem-fraction-static 0.7 \
--chunked-prefill-size 32768 \
--skip-server-warmup \
--device npu \
--base-gpu-id 0 \
--tp-size 8 \
--max-running-requests 32 \
--host <DECODE_HOST_IP> \
--port 8001 \
--nnodes 1 \
--node-rank 0 \
--dist-init-addr <DECODE_HOST_IP>:6688
```
> **Note:** `ASCEND_MF_STORE_URL` on both nodes must point to the same KV store (typically the Prefill node IP). `ASCEND_USE_FIA=True` enables fast interconnect aggregation for KV transfer. PCP is a Prefill-only feature; the Decode side needs no CP-related flags.
#### Running Qwen3-VL-8B-Instruct on 1 x Atlas 800I A3.
Model weights could be found [here](https://huggingface.co/Qwen/Qwen3-VL-8B-Instruct)
docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_support_features.mdx
+1 -1
View File
@@ -1840,7 +1840,7 @@ click [Server Arguments](../../advanced_features/server_arguments).
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--offload-mode`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`cpu`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><code>cpu</code> (DeepSeek only) <br/><code>meta</code> (DeepSeek only) <br/><code>sharded_gpu</code> (DeepSeek only)</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}><code>cpu</code> (DeepSeek only) <br/><code>meta</code> (DeepSeek only) <br/><code>sharded_gpu</code> (DeepSeek only, only support tp=1 dp&gt;1)</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>A2, A3</td>
</tr>
</tbody>
docs_new/docs/hardware-platforms/xpu.mdx
+51
View File
@@ -142,3 +142,54 @@ python -m sglang.bench_serving -h
Additionally, the requests can be formed with
[OpenAI Completions API](../basic_usage/openai_api_completions)
and sent via the command line (e.g. using `curl`) or via your own script.
## Prefill-Decode (P/D) Disaggregation on Intel XPU [Experimental]
SGLang supports prefill-decode disaggregation on Intel XPU using the [NIXL](https://github.com/ai-dynamo/nixl) KV-transfer backend.
**Tested models:**
| Model | Notes |
|:---:|:---:|
| [Qwen/Qwen3-0.6B](https://huggingface.co/Qwen/Qwen3-0.6B) | Used in integration tests; verified on Intel XPU with homogeneous P/D (XPU prefill + XPU decode) |
| [Qwen/Qwen2.5-7B-Instruct](https://huggingface.co/Qwen/Qwen2.5-7B-Instruct) | Verified on Intel XPU with homogeneous P/D (XPU prefill + XPU decode) |
**Prerequisites:** `pip install nixl sglang-router`
**Start the prefill server (GPU 0):**
```bash
ZE_AFFINITY_MASK=0 UCX_POSIX_USE_PROC_LINK=n python -m sglang.launch_server \
--model-path Qwen/Qwen3-0.6B --trust-remote-code --device xpu \
--disaggregation-mode prefill --disaggregation-transfer-backend nixl \
--disaggregation-bootstrap-port 12335 --host 0.0.0.0 --port 30000
```
**Start the decode server (GPU 1):**
```bash
ZE_AFFINITY_MASK=1 UCX_POSIX_USE_PROC_LINK=n python -m sglang.launch_server \
--model-path Qwen/Qwen3-0.6B --trust-remote-code --device xpu \
--disaggregation-mode decode --disaggregation-transfer-backend nixl \
--disaggregation-bootstrap-port 12335 --host 0.0.0.0 --port 30001
```
**Start the router:**
```bash
python -m sglang_router.launch_router \
--pd-disaggregation \
--prefill http://127.0.0.1:30000 \
--decode http://127.0.0.1:30001 \
--host 0.0.0.0 --port 8000
```
**Send a request:**
```bash
curl http://127.0.0.1:8000/v1/completions \
-H "Content-Type: application/json" \
-d '{"model": "Qwen/Qwen3-0.6B", "prompt": "The capital of France is", "max_tokens": 32}'
```
> **Note:** `UCX_POSIX_USE_PROC_LINK=n` is required on Intel XPU to avoid UCX shared-memory transport issues.