From 84338df6f053ce020c79034aaf07e404cbcbe473 Mon Sep 17 00:00:00 2001 From: amote-i <49533125+amote-i@users.noreply.github.com> Date: Tue, 23 Jun 2026 10:00:12 +0800 Subject: [PATCH] [NPU] [DOC] Update contribution guide of Ascend NPU (#28909) --- .../ascend-npus/ascend_contribution_guide.mdx | 118 ++++++++++++------ .../ascend-npus/ascend_npu.mdx | 4 +- .../ascend_npu_support_features.mdx | 4 +- 3 files changed, 87 insertions(+), 39 deletions(-) diff --git a/docs_new/docs/hardware-platforms/ascend-npus/ascend_contribution_guide.mdx b/docs_new/docs/hardware-platforms/ascend-npus/ascend_contribution_guide.mdx index 685f899a9..024659009 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/ascend_contribution_guide.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/ascend_contribution_guide.mdx @@ -1,5 +1,6 @@ --- title: "Contribution Guide" +mode: wide metatags: description: "Set up the Ascend NPU development environment, run tests, build documentation, and open SGLang pull requests." --- @@ -35,39 +36,79 @@ pre-commit run --all-files - **`pre-commit run --all-files`** manually runs all configured checks, applying fixes if possible. If it fails the first time, re-run it to ensure lint errors are fully resolved. Make sure your code passes all checks **before** creating a Pull Request. - **Do not commit** directly to the `main` branch. Always create a new branch (e.g., `feature/my-new-feature`), push your changes, and open a PR from that branch. +- Link checking with lychee is **enforced in CI**. By default, it is not blocking local commits. +- To run local link checks manually, use: `pre-commit run --hook-stage manual lychee --all-files`. -## Run and add unit tests +## Run and add tests -If you add a new feature or fix a bug, please add corresponding unit tests to ensure coverage and prevent regression. -SGLang uses Python's built-in [unittest](https://docs.python.org/3/library/unittest.html) framework. -For detailed instructions on running tests and integrating them into CI, refer to [test/README.md](https://github.com/sgl-project/sglang/tree/main/test/README.md). +All NPU tests are end-to-end (E2E) and require launching a server with real model weights. +Tests live under [`test/registered/ascend/`](https://github.com/sgl-project/sglang/tree/main/test/registered/ascend), organized by model type and functionality: + +``` +ascend/ +├── llm_models/ # Per-model inference accuracy +├── vlm_models/ # Vision-language models +├── embedding_models/ # Embedding model tests +├── rerank_models/ # Reranker model tests +├── reward_models/ # Reward model tests +├── interface/ # API correctness, function calling +├── basic_function/ # Cache, sampling, quantization, etc. +└── test_npu_memory_consumption.py +``` + +### Adding a test + +See [`test_npu_sampling_backend.py`](https://github.com/sgl-project/sglang/tree/main/test/registered/ascend/basic_function/backends/test_npu_sampling_backend.py) for a complete example. Key steps: + +1. Place your test file in the appropriate directory under `test/registered/ascend/`. +2. Extend `CustomTestCase` (from `sglang.test.test_utils`) for CI retry support. +3. Launch server with `popen_launch_server()` in `setUpClass` and clean up with `kill_process_tree()` in `tearDownClass`. +4. Register your test with `register_npu_ci()`: + ```python + from sglang.test.ci.ci_register import register_npu_ci + register_npu_ci(est_time=400, suite="stage-b-test-1-npu-a2", nightly=False) + register_npu_ci(est_time=400, suite="nightly-1-npu-a3", nightly=True) + ``` + +### Running tests locally + +```bash +pytest test/registered/ascend/llm_models/test_npu_qwen3_0_6b.py -v +``` + +For detailed instructions, refer to [test/README.md](https://github.com/sgl-project/sglang/tree/main/test/README.md). + +### Register models for CI If you need to use model which is not in `python/sglang/test/ascend/test_ascend_utils.py` list. Follow these steps: + 1. Register account and upload your model to [modelscope](https://modelscope.cn/models). 2. Make sure your model is pre-cached on the CI server and is on the way "/data/ascend-ci-share-pkking-sglang/modelscope/hub/models/{your_model_repo}/{your_model}". If this is not the case, use following command on CI server: ```bash - modelscope download - --model {your_model_repo}/{your_model} - --local_dir /data/ascend-ci-share-pkking-sglang/modelscope/hub/models/{your_model_repo}/{your_model} + modelscope download \ + --model {your_model_repo}/{your_model} \ + --local_dir /data/ascend-ci-share-pkking-sglang/modelscope/hub/models/{your_model_repo}/{your_model} ``` > Note: If you don’t have access to CI server, please ask maintainers (zl19940307@163.com) to download your model. -4. Add model to ```python/sglang/test/ascend/test_ascend_utils.py``` (use docker ```"/root/.cache/modelscope/hub/models/{your_model_repo}/{your_model}"``` path). +3. Add model to `python/sglang/test/ascend/test_ascend_utils.py` (use docker `"/root/.cache/modelscope/hub/models/{your_model_repo}/{your_model}"` path). -## Write documentations +## Write documentation -We recommend new contributors start from writing documentation, which helps you quickly understand SGLang codebase. +We recommend new contributors start by writing documentation, which helps you quickly understand SGLang codebase. For more details, please refer to [docs/README.md](https://github.com/sgl-project/sglang/tree/main/docs/README.md). ## Test the accuracy If your code changes the model output, please run the accuracy tests. A quick sanity check is the few-shot GSM8K. -``` +```bash # Launch a server python3 -m sglang.launch_server --model Qwen/Qwen2-7B-Instruct -# Evaluate -python3 -m sglang.test.few_shot_gsm8k --num-questions 200 +# Evaluate. --base-url must point at the server launched above. +# The default SGLang server port is 30000; change it if you launched +# the server with a different --port. +python3 -m sglang.test.run_eval --base-url http://localhost:30000 --eval-name gsm8k --num-examples 200 ``` Please note that the above script is primarily a sanity check, not a rigorous accuracy or speed test. @@ -76,8 +117,8 @@ Also, do not rely on the "Latency/Output throughput" from this script, as it is GSM8K is too easy for state-of-the-art models nowadays. Please try your own more challenging accuracy tests. You can find additional accuracy eval examples in: -- [test_eval_accuracy_large.py](https://github.com/sgl-project/sglang/blob/main/test/registered/eval/test_eval_accuracy_large.py) -- [test_gpt_oss_1gpu.py](https://github.com/sgl-project/sglang/blob/main/test/registered/core/test_gpt_oss_1gpu.py) +- [test_eval_accuracy_large.py](https://github.com/sgl-project/sglang/blob/main/test/manual/eval/test_eval_accuracy_large.py) +- [test_gpt_oss_1gpu.py](https://github.com/sgl-project/sglang/blob/main/test/manual/core/test_gpt_oss_1gpu.py) ## Benchmark the speed Refer to [Benchmark and Profiling](../../developer_guide/benchmark_and_profiling). @@ -92,12 +133,15 @@ Then your PR can be merged. We have a lot of open PRs but limited CI machines, so only top and trusted contributors have permission to trigger CI tests. Users with permission are listed in the [CI_PERMISSIONS.json](https://github.com/sgl-project/sglang/blob/main/.github/CI_PERMISSIONS.json) +**PR authors** can always use `/rerun-failed-ci` on their own PRs, even if they are not listed in `CI_PERMISSIONS.json`. + For CI to run on a pull request, it must have the "run-ci" label. Authorized users can add the label or rerun failed tests by commenting on the PR with one of these commands: - `/tag-run-ci-label`: Adds the "run-ci" label. Only **future** commits trigger CI; the current commit is unaffected. Add the `extra` argument (`/tag-run-ci-label extra`) to additionally apply the "run-ci-extra" label, opting the PR into the extra test workflow (`pr-test-extra.yml`). - `/rerun-failed-ci`: Reruns workflows from the latest commit with conclusion **failed, flaky, or skipped**. - `/tag-and-rerun-ci`: Runs both. Use this on a fresh PR to kick off CI on the current commit — `/tag-run-ci-label` alone won't. Accepts the same `extra` argument (`/tag-and-rerun-ci extra`). - `/rerun-stage `: Reruns a single test stage without waiting for its dependencies. Useful for quickly validating a specific test fix instead of waiting ~30 minutes for preceding stages. +- `/rerun-test [ ...]`: Reruns one or more specific tests directly, bypassing stage boundaries. Each `` is pytest-style `::[.]` (the `::TestClass` and `.` parts are optional). The handler resolves each spec, groups specs by their registered runner-label, and dispatches one [Rerun Test workflow](https://github.com/sgl-project/sglang/actions/workflows/rerun-test.yml) per group. Examples: `/rerun-test test_srt_endpoint.py`, `/rerun-test registered/core/test_srt_endpoint.py::TestSRTEndpoint.test_simple_decode`, `/rerun-test test_a.py test_b.py` (multiple at once). If you have permission, the [Slash Command Handler](https://github.com/sgl-project/sglang/actions/workflows/slash-command-handler.yml) will run your command and react with a 👍 to your comment. It may take up to a few minutes for the reaction to appear. Here’s a usage [example](https://github.com/sgl-project/sglang/pull/14253#issuecomment-3599509302). @@ -110,7 +154,6 @@ If you don’t have permission, please ask maintainers to trigger CI for you. ### CI rate limits Due to CI scheduling and limited resources, higher-priority PRs may preempt running jobs. In such cases, you may need to rerun the tests. - We apply CI rate limits to prevent abuse and ensure fair usage of our CI resources. Each CI workflow has a default limit defined in its workflow configuration file. For example, in [pr-gate.yml](https://github.com/sgl-project/sglang/blob/main/.github/workflows/pr-gate.yml), the default cooldown period is 120 minutes, and each workflow can override it via the `cool-down-minutes` input parameter: @@ -126,41 +169,46 @@ Users listed in [CI_PERMISSIONS.json](https://github.com/sgl-project/sglang/blob ## Code style guidance - Avoid code duplication. If the same code snippet (more than five lines) appears multiple times, extract it into a shared function. -- Minimize device synchronization. Reduce expensive CPU-GPU synchronization operations, such as `tensor.item()` or `tensor.cpu()`, whenever possible. Use vectorized code. +- Minimize device synchronization. Reduce expensive CPU-NPU synchronization operations, such as `tensor.item()` or `tensor.cpu()`, whenever possible. Use vectorized code. - Prioritize extreme efficiency. SGLang is a runtime, and most of your code runs on the critical path for every request. Optimize all minor overheads as much as possible, especially in the model forward code. - - A common pattern is some runtime checks in the model forward pass (e.g., [this](https://github.com/sgl-project/sglang/blob/f1b0eda55c2c4838e8ab90a0fac7fb1e3d7064ab/python/sglang/srt/models/deepseek_v2.py#L486-L491)). These are very likely the same for every layer. Please cache the result as a single boolean value whenever possible. + - A common pattern is some runtime checks in the model forward pass (e.g., [this](https://github.com/sgl-project/sglang/blob/f1b0eda55c2c4838e8ab90a0fac7fb1e3d7064ab/python/sglang/srt/models/deepseek_v2.py#L486-L491)). These are very likely the same for every layer. Please cache the result as a single boolean value in `__init__` whenever possible. - Make functions as pure as possible. Avoid in-place modification of arguments. -- Keep files concise. If a file exceeds 2,000 lines of code, split it into multiple smaller files. (e.g., `scheduler.py`, `scheduler_output_processor_mixin.py`) +- Keep files concise. If a file exceeds 2,000 lines of code, split it into multiple smaller files. (e.g., `scheduler.py`, `scheduler_pp_mixin.py`) +- In a file, put core data structures at the top of the file. Put utility functions at the bottom of the file. - Keep tests run fast. - If a single test file run longer than 500 seconds, split it into multiple smaller files (e.g., `test_eagle_infer_a.py`, `test_eagle_infer_b.py`). - If a single job in a github workflow runs longer than 30 mins, split it into smaller jobs/steps. - Reuse server launches in your unit tests to make tests run faster. +- Never use `pickle.loads()`, `pickle.load()`, or `recv_pyobj()` to deserialize untrusted or network-received data. Python’s [pickle module is not secure](https://docs.python.org/3/library/pickle.html) — it can execute arbitrary code during deserialization. Use safe serialization formats such as [msgpack](https://github.com/jcrist/msgspec) or JSON instead. - When supporting new hardware or features, follow these guidelines: - Do not drastically change existing code. - Always prefer new files to introduce specific components for your new hardware (e.g., `allocator_npu.py`). - If you write multiple if/else blocks for new features, ensure the common path (e.g., NVIDIA hardware or the existing code path) is the first branch. -## How to update sgl-kernel -Since sglang and sgl-kernel are separate Python packages, our current GitHub CI infrastructure does not support updating a kernel and using it immediately within the same pull request (PR). -To add a new kernel or modify an existing one in the `sgl-kernel/` source tree, you must use multiple PRs. - -Follow these steps: - -1. Submit a PR to update the sgl-kernel source code without using it in sglang python package (e.g., [#8884](https://github.com/sgl-project/sglang/pull/8884/files)). -2. Bump the version of the kernel package (e.g., [#9220](https://github.com/sgl-project/sglang/pull/9220/files)). - - Once merged, this will trigger an automatic release of the `sglang-kernel` wheel to PyPI. - - If not urgent, you can wait for other people to release the wheel. A new version will typically be released within one week. -3. Apply the changes: - - Update the `sglang-kernel` version in `sglang/python/pyproject.toml` to use the modified kernels. - - Update the related caller code in the sglang to use the new kernel. - ## How to update sgl-kernel-npu -Sgl-kernel-npu is the kernel package for Ascend NPU and is maintained in the [sgl-kernel-npu](https://github.com/sgl-project/sgl-kernel-npu) repository. if you want to add a new kernel and want to use it in sglang, please follow the steps in [Contribution Guide](https://github.com/sgl-project/sgl-kernel-npu/blob/main/docs/developer_guide/contribution_guide.md). +Sgl-kernel-npu is the separate kernel package for Ascend NPU, containing both Ascend C and Triton operators. It is maintained in the [sgl-kernel-npu](https://github.com/sgl-project/sgl-kernel-npu) repository. + +For detailed guidance on developing and integrating operators (Ascend C directory structure, PyTorch op registration, build, test, and code style), see the [Ascend NPU Operator Development Guide](./ascend_npu_operator_development). + +### Multi-PR workflow + +Since SGLang and sgl-kernel-npu are separate Python packages, dependency updates require a multi-PR workflow: + +1. **Submit sgl-kernel-npu PR**: Add or modify operators in the sgl-kernel-npu repository following the operator development guide. Ensure all tests pass. +2. **Bump sgl-kernel-npu version**: Update the version number. Merging triggers an automatic PyPI release. If not urgent, wait for a regular release (typically within one week). +3. **Reference the new version in SGLang**: + - Update the `SGLANG_KERNEL_NPU_TAG` argument in [`docker/npu.Dockerfile`](https://github.com/sgl-project/sglang/blob/main/docker/npu.Dockerfile) to the new sgl-kernel-npu release tag. + - Use the new operator in SGLang code. ## Tips for newcomers -If you want to contribute but don’t have a specific idea in mind, pick issues labeled [“good first issue” or “help wanted”](https://github.com/sgl-project/sglang/issues?q=is%3Aissue+label%3A%22good+first+issue%22%2C%22help+wanted%22). These tasks typically have lower complexity and provide an excellent introduction to the codebase. Also check out this [code walk-through](https://github.com/zhaochenyang20/Awesome-ML-SYS-Tutorial/tree/main/sglang/code-walk-through) for a deeper look into SGLang’s workflow. +If you want to contribute but don’t have a specific idea in mind, pick issues labeled ["good first issue" or "help wanted"](https://github.com/sgl-project/sglang/issues?q=is%3Aissue+label%3A%22good+first+issue%22%2C%22help+wanted%22). These tasks typically have lower complexity and provide an excellent introduction to the codebase. + +Also check out the following materials as startup guide: +- [Mini-SGLang](https://github.com/sgl-project/mini-sglang) for a quick overview on the structure of sglang. +- [Code Walk-through](https://github.com/zhaochenyang20/Awesome-ML-SYS-Tutorial/tree/main/sglang/code-walk-through) for a deeper look into SGLang’s workflow. +- [GTC-2026 Training Lab](https://drive.google.com/file/d/1mwOZEtipNLJzrflCTodj34KhuOZEoEw5/view?usp=drive_link) for hands-on practices of how to do optimization, benchmarking, or profiling on a launched SGLang instance. If you have any questions or want to start a discussion, please feel free to ask in our [Slack channel](https://slack.sglang.io). diff --git a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu.mdx b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu.mdx index 2fcd5afa2..1616c87ce 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu.mdx @@ -41,7 +41,7 @@ You can install SGLang using any of the methods below. Please go through `System Triton 3.2.1.dev20260530 - `pip install triton-ascend==3.2.1.dev20260530 --extra-index-url=https://mirrors.huaweicloud.com/ascend/repos/pypi/nightly --trusted-host triton-ascend.osinfra.cn` + `pip install triton-ascend==3.2.1.dev20260530 \`
`--extra-index-url=https://mirrors.huaweicloud.com/ascend/repos/pypi/nightly \`
`--trusted-host triton-ascend.osinfra.cn` SGLang NPU Kernel @@ -223,7 +223,7 @@ cd sglang/docker # Replace with the target architecture, e.g. amd64, arm64. # Optional build arguments: # --build-arg DEVICE_TYPE=910b # Required for Atlas 800I A2 -# --build-arg APTMIRROR= # Use a custom APT mirror too improve download speed +# --build-arg APTMIRROR= # Use a custom APT mirror to improve download speed # If there are network errors, please modify the Dockerfile to add ARG HTTP_PROXY/HTTPS_PROXY and set them as ENV. docker build --build-arg TARGETARCH= -t -f npu.Dockerfile . ``` diff --git a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_support_features.mdx b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_support_features.mdx index 2c032ec41..24f592454 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_support_features.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_support_features.mdx @@ -551,7 +551,7 @@ click [Server Arguments](../../advanced_features/server_arguments). `--pipeline-parallel-size`
`--pp-size` `1` Type: int; Currently `2` not supported - Experimental + A2, A3 `--attention-context-parallel-size`
`--attn-cp-size` @@ -1852,7 +1852,7 @@ click [Server Arguments](../../advanced_features/server_arguments). `--hicache-storage-`
`backend` `None` `file` - Planned + A2, A3 `--hicache-storage-`
`prefetch-policy`