minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 10 Packages Projects Releases Wiki Activity

[NPU] [DOC] Update contribution guide of Ascend NPU (#28909)

Browse Source
This commit is contained in:
amote-i
2026-06-23 10:00:12 +08:00
committed by GitHub
parent a17753e449
commit 84338df6f0
3 changed files with 87 additions and 39 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs_new/docs/hardware-platforms/ascend-npus/ascend_contribution_guide.mdx
+83 -35
View File
@@ -1,5 +1,6 @@
---
title: "Contribution Guide"
mode: wide
metatags:
description: "Set up the Ascend NPU development environment, run tests, build documentation, and open SGLang pull requests."
---
@@ -35,39 +36,79 @@ pre-commit run --all-files
- **`pre-commit run --all-files`** manually runs all configured checks, applying fixes if possible. If it fails the first time, re-run it to ensure lint errors are fully resolved. Make sure your code passes all checks **before** creating a Pull Request.
- **Do not commit** directly to the `main` branch. Always create a new branch (e.g., `feature/my-new-feature`), push your changes, and open a PR from that branch.
- Link checking with lychee is **enforced in CI**. By default, it is not blocking local commits.
- To run local link checks manually, use: `pre-commit run --hook-stage manual lychee --all-files`.
## Run and add unit tests
## Run and add tests
If you add a new feature or fix a bug, please add corresponding unit tests to ensure coverage and prevent regression.
SGLang uses Python's built-in [unittest](https://docs.python.org/3/library/unittest.html) framework.
For detailed instructions on running tests and integrating them into CI, refer to [test/README.md](https://github.com/sgl-project/sglang/tree/main/test/README.md).
All NPU tests are end-to-end (E2E) and require launching a server with real model weights.
Tests live under [`test/registered/ascend/`](https://github.com/sgl-project/sglang/tree/main/test/registered/ascend), organized by model type and functionality:
```
ascend/
├── llm_models/ # Per-model inference accuracy
├── vlm_models/ # Vision-language models
├── embedding_models/ # Embedding model tests
├── rerank_models/ # Reranker model tests
├── reward_models/ # Reward model tests
├── interface/ # API correctness, function calling
├── basic_function/ # Cache, sampling, quantization, etc.
└── test_npu_memory_consumption.py
```
### Adding a test
See [`test_npu_sampling_backend.py`](https://github.com/sgl-project/sglang/tree/main/test/registered/ascend/basic_function/backends/test_npu_sampling_backend.py) for a complete example. Key steps:
1. Place your test file in the appropriate directory under `test/registered/ascend/`.
2. Extend `CustomTestCase` (from `sglang.test.test_utils`) for CI retry support.
3. Launch server with `popen_launch_server()` in `setUpClass` and clean up with `kill_process_tree()` in `tearDownClass`.
4. Register your test with `register_npu_ci()`:
```python
from sglang.test.ci.ci_register import register_npu_ci
register_npu_ci(est_time=400, suite="stage-b-test-1-npu-a2", nightly=False)
register_npu_ci(est_time=400, suite="nightly-1-npu-a3", nightly=True)
```
### Running tests locally
```bash
pytest test/registered/ascend/llm_models/test_npu_qwen3_0_6b.py -v
```
For detailed instructions, refer to [test/README.md](https://github.com/sgl-project/sglang/tree/main/test/README.md).
### Register models for CI
If you need to use model which is not in `python/sglang/test/ascend/test_ascend_utils.py` list. Follow these steps:
1. Register account and upload your model to [modelscope](https://modelscope.cn/models).
2. Make sure your model is pre-cached on the CI server and is on the way "/data/ascend-ci-share-pkking-sglang/modelscope/hub/models/{your_model_repo}/{your_model}".
If this is not the case, use following command on CI server:
```bash
modelscope download
--model {your_model_repo}/{your_model}
--local_dir /data/ascend-ci-share-pkking-sglang/modelscope/hub/models/{your_model_repo}/{your_model}
modelscope download \
--model {your_model_repo}/{your_model} \
--local_dir /data/ascend-ci-share-pkking-sglang/modelscope/hub/models/{your_model_repo}/{your_model}
```
> Note: If you don’t have access to CI server, please ask maintainers (zl19940307@163.com) to download your model.
4. Add model to ```python/sglang/test/ascend/test_ascend_utils.py``` (use docker ```"/root/.cache/modelscope/hub/models/{your_model_repo}/{your_model}"``` path).
3. Add model to `python/sglang/test/ascend/test_ascend_utils.py` (use docker `"/root/.cache/modelscope/hub/models/{your_model_repo}/{your_model}"` path).
## Write documentations
## Write documentation
We recommend new contributors start from writing documentation, which helps you quickly understand SGLang codebase.
We recommend new contributors start by writing documentation, which helps you quickly understand SGLang codebase.
For more details, please refer to [docs/README.md](https://github.com/sgl-project/sglang/tree/main/docs/README.md).
## Test the accuracy
If your code changes the model output, please run the accuracy tests. A quick sanity check is the few-shot GSM8K.
```
```bash
# Launch a server
python3 -m sglang.launch_server --model Qwen/Qwen2-7B-Instruct
# Evaluate
python3 -m sglang.test.few_shot_gsm8k --num-questions 200
# Evaluate. --base-url must point at the server launched above.
# The default SGLang server port is 30000; change it if you launched
# the server with a different --port.
python3 -m sglang.test.run_eval --base-url http://localhost:30000 --eval-name gsm8k --num-examples 200
```
Please note that the above script is primarily a sanity check, not a rigorous accuracy or speed test.
@@ -76,8 +117,8 @@ Also, do not rely on the "Latency/Output throughput" from this script, as it is
GSM8K is too easy for state-of-the-art models nowadays. Please try your own more challenging accuracy tests.
You can find additional accuracy eval examples in:
- [test_eval_accuracy_large.py](https://github.com/sgl-project/sglang/blob/main/test/registered/eval/test_eval_accuracy_large.py)
- [test_gpt_oss_1gpu.py](https://github.com/sgl-project/sglang/blob/main/test/registered/core/test_gpt_oss_1gpu.py)
- [test_eval_accuracy_large.py](https://github.com/sgl-project/sglang/blob/main/test/manual/eval/test_eval_accuracy_large.py)
- [test_gpt_oss_1gpu.py](https://github.com/sgl-project/sglang/blob/main/test/manual/core/test_gpt_oss_1gpu.py)
## Benchmark the speed
Refer to [Benchmark and Profiling](../../developer_guide/benchmark_and_profiling).
@@ -92,12 +133,15 @@ Then your PR can be merged.
We have a lot of open PRs but limited CI machines, so only top and trusted contributors have permission to trigger CI tests.
Users with permission are listed in the [CI_PERMISSIONS.json](https://github.com/sgl-project/sglang/blob/main/.github/CI_PERMISSIONS.json)
**PR authors** can always use `/rerun-failed-ci` on their own PRs, even if they are not listed in `CI_PERMISSIONS.json`.
For CI to run on a pull request, it must have the "run-ci" label. Authorized users can add the label or rerun failed tests by commenting on the PR with one of these commands:
- `/tag-run-ci-label`: Adds the "run-ci" label. Only **future** commits trigger CI; the current commit is unaffected. Add the `extra` argument (`/tag-run-ci-label extra`) to additionally apply the "run-ci-extra" label, opting the PR into the extra test workflow (`pr-test-extra.yml`).
- `/rerun-failed-ci`: Reruns workflows from the latest commit with conclusion **failed, flaky, or skipped**.
- `/tag-and-rerun-ci`: Runs both. Use this on a fresh PR to kick off CI on the current commit — `/tag-run-ci-label` alone won't. Accepts the same `extra` argument (`/tag-and-rerun-ci extra`).
- `/rerun-stage <stage-name>`: Reruns a single test stage without waiting for its dependencies. Useful for quickly validating a specific test fix instead of waiting ~30 minutes for preceding stages.
- `/rerun-test <test-spec> [<test-spec> ...]`: Reruns one or more specific tests directly, bypassing stage boundaries. Each `<test-spec>` is pytest-style `<file>::<TestClass>[.<test_method>]` (the `::TestClass` and `.<test_method>` parts are optional). The handler resolves each spec, groups specs by their registered runner-label, and dispatches one [Rerun Test workflow](https://github.com/sgl-project/sglang/actions/workflows/rerun-test.yml) per group. Examples: `/rerun-test test_srt_endpoint.py`, `/rerun-test registered/core/test_srt_endpoint.py::TestSRTEndpoint.test_simple_decode`, `/rerun-test test_a.py test_b.py` (multiple at once).
If you have permission, the [Slash Command Handler](https://github.com/sgl-project/sglang/actions/workflows/slash-command-handler.yml) will run your command and react with a 👍 to your comment. It may take up to a few minutes for the reaction to appear. Here’s a usage [example](https://github.com/sgl-project/sglang/pull/14253#issuecomment-3599509302).
@@ -110,7 +154,6 @@ If you don’t have permission, please ask maintainers to trigger CI for you.
### CI rate limits
Due to CI scheduling and limited resources, higher-priority PRs may preempt running jobs. In such cases, you may need to rerun the tests.
We apply CI rate limits to prevent abuse and ensure fair usage of our CI resources.
Each CI workflow has a default limit defined in its workflow configuration file. For example, in [pr-gate.yml](https://github.com/sgl-project/sglang/blob/main/.github/workflows/pr-gate.yml), the default cooldown period is 120 minutes, and each workflow can override it via the `cool-down-minutes` input parameter:
@@ -126,41 +169,46 @@ Users listed in [CI_PERMISSIONS.json](https://github.com/sgl-project/sglang/blob
## Code style guidance
- Avoid code duplication. If the same code snippet (more than five lines) appears multiple times, extract it into a shared function.
- Minimize device synchronization. Reduce expensive CPU-GPU synchronization operations, such as `tensor.item()` or `tensor.cpu()`, whenever possible. Use vectorized code.
- Minimize device synchronization. Reduce expensive CPU-NPU synchronization operations, such as `tensor.item()` or `tensor.cpu()`, whenever possible. Use vectorized code.
- Prioritize extreme efficiency. SGLang is a runtime, and most of your code runs on the critical path for every request. Optimize all minor overheads as much as possible, especially in the model forward code.
- A common pattern is some runtime checks in the model forward pass (e.g., [this](https://github.com/sgl-project/sglang/blob/f1b0eda55c2c4838e8ab90a0fac7fb1e3d7064ab/python/sglang/srt/models/deepseek_v2.py#L486-L491)). These are very likely the same for every layer. Please cache the result as a single boolean value whenever possible.
- A common pattern is some runtime checks in the model forward pass (e.g., [this](https://github.com/sgl-project/sglang/blob/f1b0eda55c2c4838e8ab90a0fac7fb1e3d7064ab/python/sglang/srt/models/deepseek_v2.py#L486-L491)). These are very likely the same for every layer. Please cache the result as a single boolean value in `__init__` whenever possible.
- Make functions as pure as possible. Avoid in-place modification of arguments.
- Keep files concise. If a file exceeds 2,000 lines of code, split it into multiple smaller files. (e.g., `scheduler.py`, `scheduler_output_processor_mixin.py`)
- Keep files concise. If a file exceeds 2,000 lines of code, split it into multiple smaller files. (e.g., `scheduler.py`, `scheduler_pp_mixin.py`)
- In a file, put core data structures at the top of the file. Put utility functions at the bottom of the file.
- Keep tests run fast.
- If a single test file run longer than 500 seconds, split it into multiple smaller files (e.g., `test_eagle_infer_a.py`, `test_eagle_infer_b.py`).
- If a single job in a github workflow runs longer than 30 mins, split it into smaller jobs/steps.
- Reuse server launches in your unit tests to make tests run faster.
- Never use `pickle.loads()`, `pickle.load()`, or `recv_pyobj()` to deserialize untrusted or network-received data. Python’s [pickle module is not secure](https://docs.python.org/3/library/pickle.html) — it can execute arbitrary code during deserialization. Use safe serialization formats such as [msgpack](https://github.com/jcrist/msgspec) or JSON instead.
- When supporting new hardware or features, follow these guidelines:
- Do not drastically change existing code.
- Always prefer new files to introduce specific components for your new hardware (e.g., `allocator_npu.py`).
- If you write multiple if/else blocks for new features, ensure the common path (e.g., NVIDIA hardware or the existing code path) is the first branch.
## How to update sgl-kernel
Since sglang and sgl-kernel are separate Python packages, our current GitHub CI infrastructure does not support updating a kernel and using it immediately within the same pull request (PR).
To add a new kernel or modify an existing one in the `sgl-kernel/` source tree, you must use multiple PRs.
Follow these steps:
1. Submit a PR to update the sgl-kernel source code without using it in sglang python package (e.g., [#8884](https://github.com/sgl-project/sglang/pull/8884/files)).
2. Bump the version of the kernel package (e.g., [#9220](https://github.com/sgl-project/sglang/pull/9220/files)).
- Once merged, this will trigger an automatic release of the `sglang-kernel` wheel to PyPI.
- If not urgent, you can wait for other people to release the wheel. A new version will typically be released within one week.
3. Apply the changes:
- Update the `sglang-kernel` version in `sglang/python/pyproject.toml` to use the modified kernels.
- Update the related caller code in the sglang to use the new kernel.
## How to update sgl-kernel-npu
Sgl-kernel-npu is the kernel package for Ascend NPU and is maintained in the [sgl-kernel-npu](https://github.com/sgl-project/sgl-kernel-npu) repository. if you want to add a new kernel and want to use it in sglang, please follow the steps in [Contribution Guide](https://github.com/sgl-project/sgl-kernel-npu/blob/main/docs/developer_guide/contribution_guide.md).
Sgl-kernel-npu is the separate kernel package for Ascend NPU, containing both Ascend C and Triton operators. It is maintained in the [sgl-kernel-npu](https://github.com/sgl-project/sgl-kernel-npu) repository.
For detailed guidance on developing and integrating operators (Ascend C directory structure, PyTorch op registration, build, test, and code style), see the [Ascend NPU Operator Development Guide](./ascend_npu_operator_development).
### Multi-PR workflow
Since SGLang and sgl-kernel-npu are separate Python packages, dependency updates require a multi-PR workflow:
1. **Submit sgl-kernel-npu PR**: Add or modify operators in the sgl-kernel-npu repository following the operator development guide. Ensure all tests pass.
2. **Bump sgl-kernel-npu version**: Update the version number. Merging triggers an automatic PyPI release. If not urgent, wait for a regular release (typically within one week).
3. **Reference the new version in SGLang**:
- Update the `SGLANG_KERNEL_NPU_TAG` argument in [`docker/npu.Dockerfile`](https://github.com/sgl-project/sglang/blob/main/docker/npu.Dockerfile) to the new sgl-kernel-npu release tag.
- Use the new operator in SGLang code.
## Tips for newcomers
If you want to contribute but don’t have a specific idea in mind, pick issues labeled [“good first issue” or “help wanted”](https://github.com/sgl-project/sglang/issues?q=is%3Aissue+label%3A%22good+first+issue%22%2C%22help+wanted%22). These tasks typically have lower complexity and provide an excellent introduction to the codebase. Also check out this [code walk-through](https://github.com/zhaochenyang20/Awesome-ML-SYS-Tutorial/tree/main/sglang/code-walk-through) for a deeper look into SGLang’s workflow.
If you want to contribute but don’t have a specific idea in mind, pick issues labeled ["good first issue" or "help wanted"](https://github.com/sgl-project/sglang/issues?q=is%3Aissue+label%3A%22good+first+issue%22%2C%22help+wanted%22). These tasks typically have lower complexity and provide an excellent introduction to the codebase.
Also check out the following materials as startup guide:
- [Mini-SGLang](https://github.com/sgl-project/mini-sglang) for a quick overview on the structure of sglang.
- [Code Walk-through](https://github.com/zhaochenyang20/Awesome-ML-SYS-Tutorial/tree/main/sglang/code-walk-through) for a deeper look into SGLang’s workflow.
- [GTC-2026 Training Lab](https://drive.google.com/file/d/1mwOZEtipNLJzrflCTodj34KhuOZEoEw5/view?usp=drive_link) for hands-on practices of how to do optimization, benchmarking, or profiling on a launched SGLang instance.
If you have any questions or want to start a discussion, please feel free to ask in our [Slack channel](https://slack.sglang.io).
docs_new/docs/hardware-platforms/ascend-npus/ascend_npu.mdx
+2 -2
View File
@@ -41,7 +41,7 @@ You can install SGLang using any of the methods below. Please go through `System
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>Triton</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>3.2.1.dev20260530</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`pip install triton-ascend==3.2.1.dev20260530 --extra-index-url=https://mirrors.huaweicloud.com/ascend/repos/pypi/nightly --trusted-host triton-ascend.osinfra.cn`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`pip install triton-ascend==3.2.1.dev20260530 \`<br/>`--extra-index-url=https://mirrors.huaweicloud.com/ascend/repos/pypi/nightly \`<br/>`--trusted-host triton-ascend.osinfra.cn`</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>SGLang NPU Kernel</td>
@@ -223,7 +223,7 @@ cd sglang/docker
# Replace <arch_tag> with the target architecture, e.g. amd64, arm64.
# Optional build arguments:
# --build-arg DEVICE_TYPE=910b # Required for Atlas 800I A2
# --build-arg APTMIRROR=<mirror_url> # Use a custom APT mirror too improve download speed
# --build-arg APTMIRROR=<mirror_url> # Use a custom APT mirror to improve download speed
# If there are network errors, please modify the Dockerfile to add ARG HTTP_PROXY/HTTPS_PROXY and set them as ENV.
docker build --build-arg TARGETARCH=<arch_tag> -t <image_name> -f npu.Dockerfile .
```
docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_support_features.mdx
+2 -2
View File
@@ -551,7 +551,7 @@ click [Server Arguments](../../advanced_features/server_arguments).
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--pipeline-parallel-size`<br/>`--pp-size`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`1`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>Type: int; Currently `2` not supported</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Experimental</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>A2, A3</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--attention-context-parallel-size`<br/>`--attn-cp-size`</td>
@@ -1852,7 +1852,7 @@ click [Server Arguments](../../advanced_features/server_arguments).
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--hicache-storage-`<br/>`backend`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>`None`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>`file`</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Planned</td>
<td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>A2, A3</td>
</tr>
<tr>
<td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>`--hicache-storage-`<br/>`prefetch-policy`</td>