From 97fc4dfd73c3f907b25885887196676b0dc071e3 Mon Sep 17 00:00:00 2001 From: a60124901 Date: Wed, 1 Jul 2026 01:27:38 +0800 Subject: [PATCH] [Doc]Checking and modifying Markdown formatting issues and link validity (#28586) Signed-off-by: a60124901 Co-authored-by: sglang-npu-bot Co-authored-by: ronnie_zheng --- .../ascend-npus/ascend_contribution_guide.mdx | 19 +- .../ascend-npus/ascend_npu.mdx | 19 + .../ascend_npu_accuracy_evaluation.mdx | 17 +- .../ascend-npus/ascend_npu_best_practice.mdx | 6929 +++++++++++++++++ .../ascend-npus/ascend_npu_faq.mdx | 28 +- .../ascend_npu_operator_development.mdx | 7 + ...nd_npu_operator_performance_optimizing.mdx | 10 +- .../ascend-npus/ascend_npu_optimization.mdx | 2 +- .../ascend_npu_performance_testing.mdx | 2 +- .../ascend-npus/ascend_npu_profiling.mdx | 12 +- .../ascend-npus/ascend_npu_quantization.mdx | 1 + .../ascend_npu_support_new_models.mdx | 10 +- .../ascend-npus/diffusion/disaggregation.mdx | 1 + .../ascend-npus/mindspore_backend.mdx | 2 +- 14 files changed, 7024 insertions(+), 35 deletions(-) create mode 100644 docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_best_practice.mdx diff --git a/docs_new/docs/hardware-platforms/ascend-npus/ascend_contribution_guide.mdx b/docs_new/docs/hardware-platforms/ascend-npus/ascend_contribution_guide.mdx index 024659009..995189617 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/ascend_contribution_guide.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/ascend_contribution_guide.mdx @@ -84,13 +84,16 @@ If you need to use model which is not in `python/sglang/test/ascend/test_ascend_ 1. Register account and upload your model to [modelscope](https://modelscope.cn/models). 2. Make sure your model is pre-cached on the CI server and is on the way "/data/ascend-ci-share-pkking-sglang/modelscope/hub/models/{your_model_repo}/{your_model}". -If this is not the case, use following command on CI server: - ```bash - modelscope download \ - --model {your_model_repo}/{your_model} \ - --local_dir /data/ascend-ci-share-pkking-sglang/modelscope/hub/models/{your_model_repo}/{your_model} - ``` + If this is not the case, use following command on CI server: + + ```bash + modelscope download \ + --model {your_model_repo}/{your_model} \ + --local_dir /data/ascend-ci-share-pkking-sglang/modelscope/hub/models/{your_model_repo}/{your_model} + ``` + > Note: If you don’t have access to CI server, please ask maintainers (zl19940307@163.com) to download your model. + 3. Add model to `python/sglang/test/ascend/test_ascend_utils.py` (use docker `"/root/.cache/modelscope/hub/models/{your_model_repo}/{your_model}"` path). ## Write documentation @@ -99,6 +102,7 @@ We recommend new contributors start by writing documentation, which helps you qu For more details, please refer to [docs/README.md](https://github.com/sgl-project/sglang/tree/main/docs/README.md). ## Test the accuracy + If your code changes the model output, please run the accuracy tests. A quick sanity check is the few-shot GSM8K. ```bash @@ -121,9 +125,11 @@ You can find additional accuracy eval examples in: - [test_gpt_oss_1gpu.py](https://github.com/sgl-project/sglang/blob/main/test/manual/core/test_gpt_oss_1gpu.py) ## Benchmark the speed + Refer to [Benchmark and Profiling](../../developer_guide/benchmark_and_profiling). ## Requesting a review for merge + You can follow the pull request merge process described in [MAINTAINER.md](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md). You will need to work with the Merge Oncall, Codeowner, and other reviewers to get their approvals. Then your PR can be merged. @@ -168,6 +174,7 @@ cool-down-minutes: Users listed in [CI_PERMISSIONS.json](https://github.com/sgl-project/sglang/blob/main/.github/CI_PERMISSIONS.json) may have a per-user cooldown interval. In practice, we use the minimum of the workflow’s default window and the user-specific interval. ## Code style guidance + - Avoid code duplication. If the same code snippet (more than five lines) appears multiple times, extract it into a shared function. - Minimize device synchronization. Reduce expensive CPU-NPU synchronization operations, such as `tensor.item()` or `tensor.cpu()`, whenever possible. Use vectorized code. - Prioritize extreme efficiency. SGLang is a runtime, and most of your code runs on the critical path for every request. Optimize all minor overheads as much as possible, especially in the model forward code. diff --git a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu.mdx b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu.mdx index b4b5ef1b8..b41884b57 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu.mdx @@ -5,6 +5,7 @@ description: "Complete installation guide for SGLang on Ascend NPUs, including c You can install SGLang using any of the methods below. Please go through `System Settings` section to ensure the clusters are operating at optimal performance. Feel free to leave an issue [here at sglang](https://github.com/sgl-project/sglang/issues) if you encounter any issues or have any problems. ## Component Version Mapping For SGLang + @@ -97,6 +98,7 @@ conda activate sglang_npu Note on Anaconda repository restrictions If you encounter an error like “Terms of Service have not been accepted” during the conda create step, the default Anaconda repository is blocking package downloads. To resolve this, configure a mirror (e.g., Tsinghua Open Source Mirror): + ```bash Command # Add Tsinghua mirrors conda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/ @@ -104,8 +106,10 @@ conda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/ conda config --set show_channel_urls yes conda config --remove channels defaults ``` + Edit the system-level conda config to remove any hardcoded defaults, e.g. vi ~/miniconda3/.condarc Then remove the failed environment and recreate it: + ```bash Command conda clean -i conda env remove -n sglang_npu @@ -157,12 +161,15 @@ pip install triton-ascend==3.2.1.dev20260530 \ --extra-index-url=https://mirrors.huaweicloud.com/ascend/repos/pypi/nightly \ --trusted-host mirrors.huaweicloud.com ``` + For installation of Triton on Ascend nightly builds or from sources, follow [installation guide](https://gitcode.com/Ascend/triton-ascend/blob/master/docs/sources/getting-started/installation.md) #### SGLang Kernels NPU + We provide SGL kernels for Ascend NPU, check [installation guide](https://github.com/sgl-project/sgl-kernel-npu/blob/main/python/sgl_kernel_npu/README.md). #### DeepEP-compatible Library + We provide a DeepEP-compatible Library as a drop-in replacement of deepseek-ai's DeepEP library, check the [installation guide](https://github.com/sgl-project/sgl-kernel-npu/blob/main/python/deep_ep/README.md). #### Some other dependencies @@ -187,7 +194,9 @@ pip install -e python[all_npu] ``` ### Method 2: Using Docker Image + #### Obtain Image + You can download the SGLang image or build an image based on Dockerfile to obtain the Ascend NPU image. @@ -226,6 +235,7 @@ docker pull quay.io/ascend/sglang:main-cann9.0.0-910b 2. Build an image based on Dockerfile + ```bash Command # Clone the SGLang repository git clone https://github.com/sgl-project/sglang.git @@ -241,6 +251,7 @@ docker build --build-arg TARGETARCH= -t -f npu.Dockerfile ``` #### Create Docker + __Notice:__ `--privileged` and `--network=host` are required by RDMA, which is typically needed by Ascend NPU clusters. @@ -325,8 +336,11 @@ cat /proc/sys/vm/swappiness # shows 10 ``` ## Running SGLang Service + ### Running Service For Large Language Models + #### PD Mixed Scene + ```bash Command # Enabling CPU Affinity export SGLANG_SET_CPU_AFFINITY=1 @@ -338,6 +352,7 @@ python3 -m sglang.launch_server \ ``` #### PD Disaggregation Scene + 1. Launch Prefill Server @@ -439,6 +454,7 @@ python3 -m sglang.launch_server \ 3. Launch Router + ```bash Command python3 -m sglang_router.launch_router \ --pd-disaggregation \ @@ -450,7 +466,9 @@ python3 -m sglang_router.launch_router \ ``` ### Running Service For Multimodal Language Models + #### PD Mixed Scene + ```bash Command python3 -m sglang.launch_server \ --model-path Qwen/Qwen3-VL-30B-A3B-Instruct \ @@ -519,6 +537,7 @@ curl http://127.0.0.1:8000/v1/chat/completions \ ``` Some models return responses accompanied with thinking process content. To disable this output, configure parameters as follows: + ```bash Command curl http://127.0.0.1:8000/v1/chat/completions \ -H "Content-Type: application/json" \ diff --git a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_accuracy_evaluation.mdx b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_accuracy_evaluation.mdx index 2004ae422..ff4bd3e65 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_accuracy_evaluation.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_accuracy_evaluation.mdx @@ -100,6 +100,7 @@ export HF_ENDPOINT=https://hf-mirror.com # Start text model server sglang serve --model-path /home/weights/Qwen2.5-7B-Instruct --attention-backend ascend --host 0.0.0.0 --port 30000 & ``` + For more details of SGLang server, refer to the [Ascend NPU Quick Start](/docs/hardware-platforms/ascend-npus/ascend_npu_quick_start) #### Execute Accuracy Evaluation @@ -118,7 +119,7 @@ evalscope eval \ Upon completion, results similar to the following will be displayed: -``` +```text +---------------------+-----------+----------+----------+-------+---------+---------+ | Model | Dataset | Metric | Subset | Num | Score | Cat.0 | +=====================+===========+==========+==========+=======+=========+=========+ @@ -225,8 +226,10 @@ For more details, refer to the [EvalScope documentation](https://evalscope.readt 3. Ensure URL ends with `/v1`, e.g., `http://localhost:30000/v1` ### EvalScope SSL certificate verification failed + When using EvalScope commands without specifying a dataset or model path, it will attempt to download automatically, which may encounter an SSL certificate verification error: -``` + +```text File "/usr/local/python3.11.14/lib/python3.11/site-packages/requests/sessions.py", line 605, in get return self.request("GET", url, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -280,8 +283,10 @@ unset HTTPS_PROXY ``` ### Download Dataset Error + For this error -``` + +```text root@localhost:/home/# wget https://www.modelscope.cn/datasets/evalscope/MMStar/resolve/master/MMStar.tsv --2026-05-12 12:08:01-- https://www.modelscope.cn/datasets/evalscope/MMStar/resolve/master/MMStar.tsv Connecting to 141.5.152.215:6688... connected. @@ -289,8 +294,10 @@ ERROR: cannot verify www.modelscope.cn's certificate, issued by ‘CN=Huawei Web Self-signed certificate encountered. To connect to www.modelscope.cn insecurely, use `--no-check-certificate`. ``` -You can add `--no-check-certificate` -``` + +You can add `--no-check-certificate' + +```bash wget https://www.modelscope.cn/datasets/evalscope/MMStar/resolve/master/MMStar.tsv --no-check-certificate ``` diff --git a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_best_practice.mdx b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_best_practice.mdx new file mode 100644 index 000000000..cbf0ce447 --- /dev/null +++ b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_best_practice.mdx @@ -0,0 +1,6929 @@ +--- +title: "Best Practice on Ascend NPU" +metatags: + description: "Documentation for Best Practice on Ascend NPU" +--- +This section describes the best practice data of mainstream LLM models such as DeepSeek and Qwen on the Ascend NPU. If +you encounter issues or have any questions, please [open an issue](https://github.com/sgl-project/sglang/issues). + +## DeepSeek Series Models + +### Low Latency + +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
ModelHardwareCardsDeploy ModeDatasetTPOTQuantizationConfiguration
Deepseek-R1Atlas 800I A332PD Disaggregation6K+1.6K20msW8A8 INT8Optimal Configuration
Deepseek-R1Atlas 800I A332PD Disaggregation3.9K+1K19msW8A8 INT8Optimal Configuration
Deepseek-R1Atlas 800I A332PD Disaggregation3.5K+1.5K19msW8A8 INT8Optimal Configuration
Deepseek-R1Atlas 800I A332PD Disaggregation3.5K+1K19msW8A8 INT8Optimal Configuration
DeepSeek-V3.2Atlas 800I A332PD Disaggregation128K+1K26msW8A8 INT8Optimal Configuration
+ +### High Throughput + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
ModelHardwareCardsDeploy ModeDatasetTPOTQuantizationConfiguration
Deepseek-R1Atlas 800I A332PD Disaggregation3.5K+1.5K50msW8A8 INT8Optimal Configuration
Deepseek-R1Atlas 800I A324PD Disaggregation2K+2K50msW8A8 INT8Optimal Configuration
Deepseek-R1Atlas 800I A38PD Mixed2K+2K50msW4A8 INT8Optimal Configuration
Deepseek-R1Atlas 800I A316PD Disaggregation2K+2K50msW4A8 INT8Optimal Configuration
Deepseek-R1Atlas 800I A38PD Mixed3.5K+1.5K50msW4A8 INT8Optimal Configuration
Deepseek-R1Atlas 800I A316PD Disaggregation3.5K+1.5K50msW4A8 INT8Optimal Configuration
+ +## Qwen Series Models + +### Low Latency + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
ModelHardwareCardsDeploy ModeDatasetTPOTQuantizationConfiguration
Qwen3-235B-A22BAtlas 800I A38PD Mixed11K+1K10msBF16Optimal Configuration
Qwen3-32BAtlas 800I A34PD Mixed6K+1.5K18msBF16Optimal Configuration
Qwen3-32BAtlas 800I A34PD Mixed4K+1.5K11msBF16Optimal Configuration
Qwen3-32BAtlas 800I A38PD Mixed18K+4K6msBF16Optimal Configuration
Qwen3-32BAtlas 800I A28PD Mixed6K+1.5K18msW8A8 INT8Optimal Configuration
Qwen3-32BAtlas 800I A28PD Mixed4K+1.5K11msBF16Optimal Configuration
Qwen3-32BAtlas 800I A32PD Mixed1K+0.3K12msW8A8 INT8Optimal Configuration
Qwen3-32BAtlas 800I A32PD Mixed6K+1.5K17msW8A8 INT8Optimal Configuration
Qwen3-8BAtlas 800I A31PD Mixed1K+0.3K7msW8A8 INT8Optimal Configuration
Qwen3-8BAtlas 800I A31PD Mixed6K+1.5K12msW8A8 INT8Optimal Configuration
Qwen3-8BAtlas 800I A31PD Mixed3.5K+1.5K5msW8A8 INT8Optimal Configuration
Qwen3-30B-A3BAtlas 800I A31PD Mixed6K+1.5K10msW8A8 INT8Optimal Configuration
Qwen3-30B-A3BAtlas 800I A31PD Mixed1K+0.3K7msW8A8 INT8Optimal Configuration
Qwen3-Next-A3B-InstructAtlas 800I A32PD Mixed1K+0.3K14.21msW8A8 INT8Optimal Configuration
Qwen3-Next-A3B-InstructAtlas 800I A32PD Mixed6K+1.5K15.62msW8A8 INT8Optimal Configuration
Qwen3-Next-A3B-InstructAtlas 800I A31PD Mixed3.5K+1.5K20msW8A8 INT8Optimal Configuration
Qwen3-14BAtlas 800I A31PD Mixed3.5K+1.5K9msW8A8 INT8Optimal Configuration
Qwen3.5-27BAtlas 800I A32PD Mixed3.5K+1.5K20msW8A8 INT8Optimal Configuration
Qwen3.5-27BAtlas 800I A31PD Mixed16K+1K20msW8A8 INT8Optimal Configuration
Qwen3.5-27BAtlas 800I A31PD Mixed64K+1K20msW8A8 INT8Optimal Configuration
Qwen3.5-397B-A17BAtlas 800I A38PD Mixed3.5K+1.5K22msW4A8Optimal Configuration
+ +### High Throughput + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
ModelHardwareCardsDeploy ModeDatasetTPOTQuantizationConfiguration
Qwen3-235B-A22BAtlas 800I A324PD Disaggregation3.5K+1.5K50msW8A8 INT8Optimal Configuration
Qwen3-235B-A22BAtlas 800I A38PD Mixed3.5K+1.5K50msW8A8 INT8Optimal Configuration
Qwen3-235B-A22BAtlas 800I A38PD Mixed2K+2K50msW8A8 INT8Optimal Configuration
Qwen3-235B-A22BAtlas 800I A316PD Mixed2K+2K50msW8A8 INT8Optimal Configuration
Qwen3-32BAtlas 800I A32PD Mixed3.5K+1.5K50msW8A8 INT8Optimal Configuration
Qwen3-32BAtlas 800I A32PD Mixed2K+2K50msW8A8 INT8Optimal Configuration
Qwen3-30B-A3BAtlas 800I A31PD Mixed3.5K+1.5K50msW8A8 INT8Optimal Configuration
Qwen3-Coder-480B-A35B-InstructAtlas 800I A324PD Disaggregation3.5K+1.5K50msW8A8 INT8Optimal Configuration
Qwen3-Coder-480B-A35B-InstructAtlas 800I A316PD Mixed3.5K+1.5K50msW8A8 INT8Optimal Configuration
Qwen3-Coder-480B-A35B-InstructAtlas 800I A38PD Mixed3.5K+1.5K50msW8A8 INT8Optimal Configuration
Qwen3-Next-80B-A3B-InstructAtlas 800I A32PD Mixed3.5K+1.5K50msW8A8 INT8Optimal Configuration
Qwen3-32BAtlas 800I A28PD Mixed3.5K+1.5K50msW8A8 INT8Optimal Configuration
Qwen3-32BAtlas 800I A28PD Mixed2K+2K50msW8A8 INT8Optimal Configuration
Qwen3-14BAtlas 800I A31PD Mixed3.5K+1.5K50msW8A8 INT8Optimal Configuration
Qwen3-8BAtlas 800I A31PD Mixed3.5K+1.5K50msW8A8 INT8Optimal Configuration
Qwen3.5-27BAtlas 800I A31PD Mixed3.5K+1.5K50msW8A8 INT8Optimal Configuration
Qwen3.5-27BAtlas 800I A32PD Mixed16K+1K50msW8A8 INT8Optimal Configuration
Qwen3.5-27BAtlas 800I A32PD Mixed64K+1K50msW8A8 INT8Optimal Configuration
Qwen3.5-397B-A17BAtlas 800I A38PD Mixed3.5K+1.5K50msW4A8Optimal Configuration
+ +## MiniMax Series Models + +### Low Latency + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
ModelHardwareCardsDeploy ModeDatasetTPOTQuantizationConfiguration
MiniMax-M2.5Atlas 800I A38PD Mixed3.5K+1.5K20msW8A8 INT8Optimal Configuration
MiniMax-M2.5Atlas 800I A38PD Mixed128K+1K20msW8A8 INT8Optimal Configuration
+ +### High Throughput + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
ModelHardwareCardsDeploy ModeDatasetTPOTQuantizationConfiguration
MiniMax-M2.5Atlas 800I A38PD Mixed3.5K+1.5K50msW8A8 INT8Optimal Configuration
MiniMax-M2.5Atlas 800I A38PD Mixed64K+1K50msW8A8 INT8Optimal Configuration
MiniMax-M2.5Atlas 800I A38PD Mixed128K+1K50msW8A8 INT8Optimal Configuration
MiniMax-M2.5Atlas 800I A34PD Mixed64K+1K50msW8A8 INT8Optimal Configuration
MiniMax-M2.5Atlas 800I A316PD Disaggregation64K+1K50msW8A8 INT8Optimal Configuration
MiniMax-M2.5Atlas 800I A316PD Disaggregation128K+1K50msW8A8 INT8Optimal Configuration
+ +## MiMo-V2-Flash Series Models + +### Low Latency + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
ModelHardwareCardsDeploy ModeDatasetTTFTQuantizationConfiguration
MiMo-V2-FlashAtlas 800I A324PD Disaggregation16K+1974msW8A8 INT8Optimal Configuration
MiMo-V2-FlashAtlas 800I A324PD Disaggregation32K+12,147msW8A8 INT8Optimal Configuration
+ +### High Throughput + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
ModelHardwareCardsDeploy ModeDatasetTPOTQuantizationConfiguration
MiMo-V2-FlashAtlas 800I A324PD Disaggregation16K+1K16msW8A8 INT8Optimal Configuration
MiMo-V2-FlashAtlas 800I A324PD Disaggregation32K+1K12msW8A8 INT8Optimal Configuration
+ +## Kimi Series Models + +### Low Latency + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
ModelHardwareCardsDeploy ModeDatasetTPOTQuantizationConfiguration
Kimi-K2.5-w4a8Atlas 800I A38PD Mixed3.5K+1.5K20msW4A8 INT8Optimal Configuration
+ +### High Throughput + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
ModelHardwareCardsDeploy ModeDatasetTPOTQuantizationConfiguration
Kimi-K2.5-w4a8Atlas 800I A38PD Mixed3.5K+1.5K50msW4A8 INT8Optimal Configuration
+ +## GLM Series Models + +### High Throughput + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
ModelHardwareCardsDeploy ModeDatasetTPOTQuantizationConfiguration
GLM-5.1Atlas 800I A316PD Mixed3.5K+1.5K41msW4A8Optimal Configuration
GLM-5.1Atlas 800I A332PD Disaggregation16K+1K23msW4A8Optimal Configuration
GLM-5.1Atlas 800I A348PD Disaggregation64K+1K+90% cache hit45msW4A8Optimal Configuration
GLM-5.1Atlas 800I A348PD Disaggregation128K+1K+90% cache hit32msW4A8Optimal Configuration
+ +## Optimal Configuration + +### DeepSeek-R1 3_5K-1_5K 50ms on A3 32 Cards Disaggregation Mode + +Model: Deepseek R1 + +Hardware: Atlas 800I A3 32Card + +DeployMode: PD Disaggregation + +Dataset: random + +Input Output Length: 3.5K+1.5K + +TPOT: 50ms + +#### Model Deployment + +```bash Command +echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor +sysctl -w vm.swappiness=0 +sysctl -w kernel.numa_balancing=0 +sysctl -w kernel.sched_migration_cost_ns=50000 +export SGLANG_SET_CPU_AFFINITY=1 +unset ASCEND_LAUNCH_BLOCKING +source /usr/local/Ascend/ascend-toolkit/set_env.sh +source /usr/local/Ascend/nnal/atb/set_env.sh +export PATH=/usr/local/Ascend/8.5.0/compiler/bishengir/bin:$PATH + +export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True +export STREAMS_PER_DEVICE=32 +export HCCL_OP_EXPANSION_MODE=AIV +export SGLANG_NPU_USE_MLAPO=1 +export SGLANG_USE_FIA_NZ=1 +export SGLANG_NPU_USE_MULTI_STREAM=1 + +export ASCEND_MF_STORE_URL="tcp://your prefill ip1:24669" + +P_IP=('your prefill ip1' 'your prefill ip2') + +D_IP=('your decode ip1' 'your decode ip2') + +MODEL_PATH=xxx + +LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` +LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` +echo "${LOCAL_HOST1}" +echo "${LOCAL_HOST2}" +# prefill +for i in "${!P_IP[@]}"; +do + if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]]; + then + echo "${P_IP[$i]}" + export SGLANG_USE_AG_AFTER_QLORA=1 + export HCCL_BUFFSIZE=800 + export TASK_QUEUE_ENABLE=2 + export SGLANG_NPU_FUSED_MOE_MODE=2 + export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=131072 + + export HCCL_SOCKET_IFNAME=lo + export GLOO_SOCKET_IFNAME=lo + python -m sglang.launch_server --model-path ${MODEL_PATH} --disaggregation-mode prefill --host ${P_IP[$i]} \ + --port 8000 --disaggregation-bootstrap-port $((8998+$i)) --trust-remote-code --nnodes 1 --node-rank 0 \ + --tp-size 16 --mem-fraction-static 0.778 --attention-backend ascend --device npu --quantization modelslim \ + --disaggregation-transfer-backend ascend --max-running-requests 16 --disable-radix-cache \ + --chunked-prefill-size -1 --max-prefill-tokens 60000 --moe-a2a-backend ascend_fuseep --deepep-mode normal \ + --speculative-algorithm NEXTN --speculative-num-steps 1 --speculative-eagle-topk 1 --speculative-num-draft-tokens 2 \ + --dp-size 4 --enable-dp-attention --disable-shared-experts-fusion --dtype bfloat16 --enable-attn-tp-input-scattered + NODE_RANK=$i + break + fi +done + +# decode +for i in "${!D_IP[@]}"; +do + if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]]; + then + echo "${D_IP[$i]}" + export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 + export HCCL_BUFFSIZE=600 + export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=64 + export TASK_QUEUE_ENABLE=1 + export SGLANG_NPU_FUSED_MOE_MODE=1 + export SGLANG_LM_HEAD_TP=8 + export HCCL_SOCKET_IFNAME=xxx + export GLOO_SOCKET_IFNAME=xxx + python -m sglang.launch_server --model-path ${MODEL_PATH} --disaggregation-mode decode --host ${D_IP[$i]} \ + --port 8001 --trust-remote-code --dist-init-addr ${D_IP[0]}:5000 --nnodes 2 --node-rank $i --tp-size 32 --dp-size 32 \ + --mem-fraction-static 0.82 --max-running-requests 1024 --attention-backend ascend --device npu --quantization modelslim \ + --moe-a2a-backend ascend_fuseep --enable-dp-attention --deepep-mode low_latency --moe-dense-tp 1 \ + --cuda-graph-bs 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 --disaggregation-transfer-backend ascend --watchdog-timeout 9000 --context-length 8192 \ + --speculative-algorithm NEXTN --speculative-num-steps 1 --speculative-eagle-topk 1 --speculative-num-draft-tokens 2 \ + --tokenizer-worker-num 4 --disable-shared-experts-fusion --dtype bfloat16 \ + --load-balance-method round_robin + NODE_RANK=$i + break + fi +done + +``` + +```shell Command +export SGLANG_DP_ROUND_ROBIN=1 +python -m sglang_router.launch_router \ + --pd-disaggregation \ + --policy cache_aware \ + --prefill http://P_IP:8000 8998 \ + --prefill http://P_IP:8000 8999 \ + --decode http://D_IP:8001 \ + --host 127.0.0.1 \ + --port 6688 \ + --mini-lb +``` + +#### Benchmark + +We tested it based on the `RANDOM` dataset. + +```shell Command +python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 6688 --max-concurrency 1024 --random-input-len 3584 --random-output-len 1536 --num-prompts 7168 --random-range-ratio 1 --request-rate 40 +``` + +### DeepSeek-R1 2K-2K 50ms on A3 24 Cards Disaggregation Mode + +Model: Deepseek R1 + +Hardware: Atlas 800I A3 24Card + +DeployMode: PD Disaggregation + +Dataset: random + +Input Output Length: 2K+2K + +TPOT: 50ms + +#### Model Deployment + +```bash Command +echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor +sysctl -w vm.swappiness=0 +sysctl -w kernel.numa_balancing=0 +sysctl -w kernel.sched_migration_cost_ns=50000 + +unset ASCEND_LAUNCH_BLOCKING +source /usr/local/Ascend/ascend-toolkit/set_env.sh +source /usr/local/Ascend/nnal/atb/set_env.sh +export PATH=/usr/local/Ascend/8.5.0/compiler/bishengir/bin:$PATH + +export SGLANG_SET_CPU_AFFINITY=1 +export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True +export STREAMS_PER_DEVICE=32 +export SGLANG_NPU_USE_MLAPO=1 +export SGLANG_USE_FIA_NZ=1 + +export ASCEND_MF_STORE_URL="tcp://your prefill ip1:24669" + +P_IP=('your prefill ip1') +D_IP=('your decode ip1' 'your decode ip2') + +MODEL_PATH=xxx + +LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` +LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` +echo "${LOCAL_HOST1}" +echo "${LOCAL_HOST2}" +# prefill +for i in "${!P_IP[@]}"; +do + if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]]; + then + echo "${P_IP[$i]}" + export HCCL_BUFFSIZE=1600 + export TASK_QUEUE_ENABLE=2 + export SGLANG_USE_AG_AFTER_QLORA=1 + export HCCL_SOCKET_IFNAME=lo + export GLOO_SOCKET_IFNAME=lo + + python -m sglang.launch_server --model-path ${MODEL_PATH} --disaggregation-mode prefill --host ${P_IP[$i]} \ + --port 8000 --disaggregation-bootstrap-port $((8998+$i)) --trust-remote-code --nnodes 1 --node-rank 0 \ + --tp-size 16 --mem-fraction-static 0.8 --attention-backend ascend --device npu --quantization modelslim \ + --disaggregation-transfer-backend ascend --max-running-requests 20 --context-length 8192 --disable-radix-cache \ + --chunked-prefill-size -1 --max-prefill-tokens 28680 --moe-a2a-backend deepep --deepep-mode normal \ + --speculative-algorithm NEXTN --speculative-num-steps 1 --speculative-eagle-topk 1 --speculative-num-draft-tokens 2 \ + --dp-size 4 --enable-dp-attention --disable-shared-experts-fusion --dtype bfloat16 --enable-attn-tp-input-scattered + NODE_RANK=$i + break + fi +done + +# decode +for i in "${!D_IP[@]}"; +do + if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]]; + then + echo "${D_IP[$i]}" + export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 + export HCCL_BUFFSIZE=800 + export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=102 + export TASK_QUEUE_ENABLE=1 + export SGLANG_SCHEDULER_SKIP_ALL_GATHER=1 + export SGLANG_NPU_FUSED_MOE_MODE=1 + export HCCL_SOCKET_IFNAME=xxx + export GLOO_SOCKET_IFNAME=xxx + + python -m sglang.launch_server --model-path ${MODEL_PATH} --disaggregation-mode decode --host ${D_IP[$i]} \ + --port 8001 --trust-remote-code --dist-init-addr ${D_IP[0]}:5000 --nnodes 2 --node-rank $i --tp-size 32 --dp-size 32 \ + --mem-fraction-static 0.81 --max-running-requests 1088 --attention-backend ascend --device npu --quantization modelslim \ + --moe-a2a-backend ascend_fuseep --enable-dp-attention --deepep-mode low_latency --enable-dp-lm-head --moe-dense-tp 1 \ + --cuda-graph-bs 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 --disaggregation-transfer-backend ascend --watchdog-timeout 9000 --context-length 8192 \ + --speculative-algorithm NEXTN --speculative-num-steps 2 --speculative-eagle-topk 1 --speculative-num-draft-tokens 3 \ + --tokenizer-worker-num 4 --disable-shared-experts-fusion --dtype bfloat16 \ + --load-balance-method round_robin + NODE_RANK=$i + break + fi +done + +``` + +```bash Command +python -m sglang_router.launch_router \ + --pd-disaggregation \ + --policy cache_aware \ + --prefill http://P_IP:8000 8998 \ + --prefill http://P_IP:8000 8999 \ + --decode http://D_IP:8001 \ + --host 127.0.0.1 \ + --port 6688 \ + --mini-lb +``` + +#### Benchmark + +We tested it based on the `RANDOM` dataset. + +```bash Command +python -m sglang.bench_serving --dataset-name random --backend sglang \ +--host 127.0.0.1 \ +--port 6688 \ +--max-concurrency 1088 \ +--random-input-len 2048 \ +--random-output-len 2048 \ +--num-prompts 12800 \ +--random-range-ratio 1 \ +--request-rate 24 +``` + +### DeepSeek-R1 6K-1_6K 20ms on A3 32 Cards Disaggregation Mode + +Model: Deepseek R1 + +Hardware: Atlas 800I A3 32Card + +DeployMode: PD Disaggregation + +Dataset: random + +Input Output Length: 6K+1.6K + +TPOT: 20ms + +#### Model Deployment + +```bash Command +echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor +sysctl -w vm.swappiness=0 +sysctl -w kernel.numa_balancing=0 +sysctl -w kernel.sched_migration_cost_ns=50000 +export SGLANG_SET_CPU_AFFINITY=1 +unset ASCEND_LAUNCH_BLOCKING +source /usr/local/Ascend/ascend-toolkit/set_env.sh +source /usr/local/Ascend/nnal/atb/set_env.sh +export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True +export STREAMS_PER_DEVICE=32 +export ASCEND_MF_STORE_URL="tcp://your prefill ip1:24669" + +P_IP=('your prefill ip1' 'your prefill ip2') + +D_IP=('your decode ip1' 'your decode ip2') + +MODEL_PATH=xxx + +export SGLANG_NPU_USE_MLAPO=1 +export SGLANG_USE_FIA_NZ=1 + +LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` +LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` +echo "${LOCAL_HOST1}" +echo "${LOCAL_HOST2}" +# prefill +for i in "${!P_IP[@]}"; +do + if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]]; + then + echo "${P_IP[$i]}" + export HCCL_BUFFSIZE=1536 + export TASK_QUEUE_ENABLE=2 + + export HCCL_SOCKET_IFNAME=lo + export GLOO_SOCKET_IFNAME=lo + python -m sglang.launch_server --model-path ${MODEL_PATH} --disaggregation-mode prefill --host ${P_IP[$i]} \ + --port 8000 --disaggregation-bootstrap-port $((8998+$i)) --trust-remote-code --nnodes 1 --node-rank 0 \ + --tp-size 16 --mem-fraction-static 0.81 --attention-backend ascend --device npu --quantization modelslim \ + --disaggregation-transfer-backend ascend --max-running-requests 4 --disable-radix-cache \ + --chunked-prefill-size -1 --max-prefill-tokens 28680 --moe-a2a-backend deepep --deepep-mode normal \ + --speculative-algorithm NEXTN --speculative-num-steps 1 --speculative-eagle-topk 1 --speculative-num-draft-tokens 2 \ + --dp-size 2 --enable-dp-attention --disable-shared-experts-fusion --dtype bfloat16 --enable-attn-tp-input-scattered + NODE_RANK=$i + break + fi +done + +# decode +for i in "${!D_IP[@]}"; +do + if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]]; + then + echo "${D_IP[$i]}" + export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 + export HCCL_BUFFSIZE=650 + export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=16 + export TASK_QUEUE_ENABLE=1 + export SGLANG_SCHEDULER_SKIP_ALL_GATHER=1 + export HCCL_SOCKET_IFNAME=xxx + export GLOO_SOCKET_IFNAME=xxx + python -m sglang.launch_server --model-path ${MODEL_PATH} --disaggregation-mode decode --host ${D_IP[$i]} \ + --port 8001 --trust-remote-code --dist-init-addr DIP1:5000 --nnodes 2 --node-rank $i --tp-size 32 --dp-size 8 \ + --mem-fraction-static 0.75 --max-running-requests 32 --attention-backend ascend --device npu --quantization modelslim \ + --moe-a2a-backend deepep --enable-dp-attention --deepep-mode low_latency --enable-dp-lm-head --moe-dense-tp 1 \ + --cuda-graph-bs 2 4 6 --disaggregation-transfer-backend ascend --watchdog-timeout 9000 \ + --speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 \ + --tokenizer-worker-num 4 --disable-shared-experts-fusion --dtype bfloat16 \ + --load-balance-method round_robin + NODE_RANK=$i + break + fi +done + +``` + +```shell Command +export SGLANG_DP_ROUND_ROBIN=1 +python -m sglang_router.launch_router \ + --pd-disaggregation \ + --policy cache_aware \ + --prefill http://P_IP:8000 8998 \ + --prefill http://P_IP:8000 8999 \ + --decode http://D_IP:8001 \ + --host 127.0.0.1 \ + --port 6688 \ + --mini-lb +``` + +#### Benchmark + +We tested it based on the `RANDOM` dataset. + +```bash Command +python -m sglang.bench_serving --dataset-name random --backend sglang \ + --host 127.0.0.1 \ + --port 6688 \ + --max-concurrency 32 \ + --random-input-len 6000 \ + --random-output-len 1600 \ + --num-prompts 32 \ + --random-range-ratio 1 \ + --request-rate 16 +``` + +### DeepSeek-R1 3_9K-1K 19ms on A3 32 Cards Disaggregation Mode + +Model: Deepseek R1 + +Hardware: Atlas 800I A3 32Card + +DeployMode: PD Disaggregation + +Dataset: random + +Input Output Length: 3.9K+1K + +TPOT: 19ms + +#### Model Deployment + +```bash Command +echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor +sysctl -w vm.swappiness=0 +sysctl -w kernel.numa_balancing=0 +sysctl -w kernel.sched_migration_cost_ns=50000 + +unset ASCEND_LAUNCH_BLOCKING +source /usr/local/Ascend/ascend-toolkit/set_env.sh +source /usr/local/Ascend/nnal/atb/set_env.sh +export PATH=/usr/local/Ascend/8.5.0/compiler/bishengir/bin:$PATH + +export SGLANG_SET_CPU_AFFINITY=1 +export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True +export STREAMS_PER_DEVICE=32 +export SGLANG_NPU_USE_MLAPO=1 +export SGLANG_USE_FIA_NZ=1 +export ASCEND_MF_STORE_URL="tcp://your prefill ip1:24669" + +P_IP=('your prefill ip1' 'your prefill ip2') +D_IP=('your decode ip1' 'your decode ip2') + +MODEL_PATH=xxx + +LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` +LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` +echo "${LOCAL_HOST1}" +echo "${LOCAL_HOST2}" + +# prefill +for i in "${!P_IP[@]}"; +do + if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]]; + then + echo "${P_IP[$i]}" + export HCCL_BUFFSIZE=1536 + export TASK_QUEUE_ENABLE=2 + export HCCL_SOCKET_IFNAME=lo + export GLOO_SOCKET_IFNAME=lo + + python -m sglang.launch_server --model-path ${MODEL_PATH} --disaggregation-mode prefill --host ${P_IP[$i]} \ + --port 8000 --disaggregation-bootstrap-port $((8998+$i)) --trust-remote-code --nnodes 1 --node-rank 0 \ + --tp-size 16 --mem-fraction-static 0.81 --attention-backend ascend --device npu --quantization modelslim \ + --disaggregation-transfer-backend ascend --max-running-requests 4 --context-length 8192 --disable-radix-cache \ + --chunked-prefill-size -1 --max-prefill-tokens 28680 --moe-a2a-backend deepep --deepep-mode normal \ + --speculative-algorithm NEXTN --speculative-num-steps 1 --speculative-eagle-topk 1 --speculative-num-draft-tokens 2 \ + --dp-size 2 --enable-dp-attention --disable-shared-experts-fusion --dtype bfloat16 --enable-attn-tp-input-scattered + NODE_RANK=$i + break + fi +done + +# decode +for i in "${!D_IP[@]}"; +do + if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]]; + then + echo "${D_IP[$i]}" + export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 + export HCCL_BUFFSIZE=650 + export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=12 + export TASK_QUEUE_ENABLE=1 + export SGLANG_SCHEDULER_SKIP_ALL_GATHER=1 + export HCCL_SOCKET_IFNAME=xxx + export GLOO_SOCKET_IFNAME=xxx + python -m sglang.launch_server --model-path ${MODEL_PATH} --disaggregation-mode decode --host ${D_IP[$i]} \ + --port 8001 --trust-remote-code --dist-init-addr DIP1:5000 --nnodes 2 --node-rank $i --tp-size 32 --dp-size 16 \ + --mem-fraction-static 0.75 --max-running-requests 32 --attention-backend ascend --device npu --quantization modelslim \ + --moe-a2a-backend deepep --enable-dp-attention --deepep-mode low_latency --enable-dp-lm-head --moe-dense-tp 1 \ + --cuda-graph-bs 2 4 6 --disaggregation-transfer-backend ascend --watchdog-timeout 9000 --context-length 8192 \ + --speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 \ + --tokenizer-worker-num 4 --disable-shared-experts-fusion --dtype bfloat16 \ + --load-balance-method round_robin + NODE_RANK=$i + break + fi +done +``` + +```bash Command +export SGLANG_DP_ROUND_ROBIN=1 +python -m sglang_router.launch_router \ + --pd-disaggregation \ + --policy cache_aware \ + --prefill http://P_IP:8000 8998 \ + --prefill http://P_IP:8000 8999 \ + --decode http://D_IP:8001 \ + --host 127.0.0.1 \ + --port 6688 \ + --mini-lb +``` + +#### Benchmark + +We tested it based on the `RANDOM` dataset. + +```bash Command +python -m sglang.bench_serving --dataset-name random --backend sglang \ + --host 127.0.0.1 \ + --port 6688 \ + --max-concurrency 32 \ + --random-input-len 3900 \ + --random-output-len 1024 \ + --num-prompts 32 \ + --random-range-ratio 1 \ + --request-rate 16 +``` + +### DeepSeek-R1 3_5K-1_5K 19ms on A3 32 Cards Disaggregation Mode + +Model: Deepseek R1 + +Hardware: Atlas 800I A3 32Card + +DeployMode: PD Disaggregation + +Dataset: random + +Input Output Length: 3.5K+1.5K + +TPOT: 19ms + +#### Model Deployment + +Please Turn to [DeepSeek-R1 3_9K-1K 19ms on A3 32 Cards Disaggregation Mode](#deepseek-r1-3_9k-1k-19ms-on-a3-32-cards-disaggregation-mode) + +#### Benchmark + +We tested it based on the `RANDOM` dataset. + +```bash Command +python -m sglang.bench_serving --dataset-name random --backend sglang \ + --host 127.0.0.1 \ + --port 6688 \ + --max-concurrency 32 \ + --random-input-len 3500 \ + --random-output-len 1500 \ + --num-prompts 32 \ + --random-range-ratio 1 \ + --request-rate 16 +``` + +### DeepSeek-R1 3_5K-1K 19ms on A3 32 Cards Disaggregation Mode + +Model: Deepseek R1 + +Hardware: Atlas 800I A3 32Card + +DeployMode: PD Disaggregation + +Dataset: random + +Input Output Length: 3.5K+1K + +TPOT: 19ms + +#### Model Deployment + +Please Turn to [DeepSeek-R1 3_9K-1K 19ms on A3 32 Cards Disaggregation Mode](#deepseek-r1-3_9k-1k-19ms-on-a3-32-cards-disaggregation-mode) + +#### Benchmark + +We tested it based on the `RANDOM` dataset. + +```bash Command +python -m sglang.bench_serving --dataset-name random --backend sglang \ + --host 127.0.0.1 \ + --port 6688 \ + --max-concurrency 32 \ + --random-input-len 3500 \ + --random-output-len 1024 \ + --num-prompts 32 \ + --random-range-ratio 1 \ + --request-rate 16 +``` + +### DeepSeek-R1 2K-2K 50ms on A3 8 Cards Mixed Mode + +Model: Deepseek R1 + +Hardware: Atlas 800I A3 8Card + +DeployMode: PD Mixed + +Dataset: random + +Input Output Length: 2K+2K + +TPOT: 50ms + +#### Model Deployment + +```bash Command +echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor +sysctl -w vm.swappiness=0 +sysctl -w kernel.numa_balancing=0 +sysctl -w kernel.sched_migration_cost_ns=50000 + +export SGLANG_SET_CPU_AFFINITY=1 +unset https_proxy +unset http_proxy +unset HTTPS_PROXY +unset HTTP_PROXY +unset ASCEND_LAUNCH_BLOCKING +source /usr/local/Ascend/ascend-toolkit/set_env.sh +source /usr/local/Ascend/nnal/atb/set_env.sh +export PATH=/usr/local/Ascend/8.5.0/compiler/bishengir/bin:$PATH + +export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True +export STREAMS_PER_DEVICE=32 +export SGLANG_SCHEDULER_DECREASE_PREFILL_IDLE=1 +export SGLANG_PREFILL_DELAYER_MAX_DELAY_PASSES=200 + +export HCCL_SOCKET_IFNAME=lo +export GLOO_SOCKET_IFNAME=lo + +export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=88 +export HCCL_BUFFSIZE=1600 +export DEEPEP_NORMAL_LONG_SEQ_ROUND=10 +export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=512 + +MODEL_PATH=xxx + +export SGLANG_NPU_USE_MLAPO=1 +export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 +export SGLANG_USE_FIA_NZ=1 + +python3 -m sglang.launch_server --model-path ${MODEL_PATH} \ +--tp 16 \ +--trust-remote-code \ +--attention-backend ascend \ +--device npu \ +--quantization modelslim \ +--watchdog-timeout 9000 \ +--host 127.0.0.1 --port 6699 \ +--cuda-graph-bs 4 8 20 21 22 \ +--mem-fraction-static 0.78 \ +--max-running-requests 352 \ +--disable-radix-cache --chunked-prefill-size -1 --max-prefill-tokens 1500 \ +--moe-a2a-backend deepep --deepep-mode auto \ +--enable-dp-attention --dp-size 16 --enable-dp-lm-head \ +--speculative-algorithm NEXTN --speculative-num-steps 2 --speculative-eagle-topk 1 --speculative-num-draft-tokens 3 \ +--dtype bfloat16 +``` + +#### Benchmark + +We tested it based on the `RANDOM` dataset. + +```bash Command +python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 6699 --max-concurrency 352 --random-input-len 2048 --random-output-len 2048 --num-prompts 1408 --random-range-ratio 1 +``` + +### DeepSeek-R1 2K-2K 50ms on A3 16 Cards Disaggregation Mode + +Model: Deepseek R1 + +Hardware: Atlas 800I A3 16Card + +DeployMode: PD Disaggregation + +Dataset: random + +Input Output Length: 2K+2K + +TPOT: 50ms + +#### Model Deployment + +```bash Command +echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor +sysctl -w vm.swappiness=0 +sysctl -w kernel.numa_balancing=0 +sysctl -w kernel.sched_migration_cost_ns=50000 + +unset https_proxy +unset http_proxy +unset HTTPS_PROXY +unset HTTP_PROXY +unset ASCEND_LAUNCH_BLOCKING +source /usr/local/Ascend/ascend-toolkit/set_env.sh +source /usr/local/Ascend/nnal/atb/set_env.sh +export PATH=/usr/local/Ascend/8.5.0/compiler/bishengir/bin:$PATH + +export SGLANG_SET_CPU_AFFINITY=1 +export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True +export STREAMS_PER_DEVICE=32 + +export ASCEND_MF_STORE_URL="tcp://your prefill ip1:24667" + +P_IP=('your prefill ip1') + +D_IP=('your decode ip1') + +MODEL_PATH=xxx + +export SGLANG_NPU_USE_MLAPO=1 +export SGLANG_USE_FIA_NZ=1 +export ENABLE_MOE_NZ=1 + +LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` +LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` +echo "${LOCAL_HOST1}" +echo "${LOCAL_HOST2}" + +# prefill +for i in "${!P_IP[@]}"; +do + if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]]; + then + echo "${P_IP[$i]}" + export HCCL_BUFFSIZE=2600 + export TASK_QUEUE_ENABLE=2 + + export HCCL_SOCKET_IFNAME=lo + export GLOO_SOCKET_IFNAME=lo + python -m sglang.launch_server --model-path ${MODEL_PATH} --disaggregation-mode prefill --host ${P_IP[$i]} \ + --port 8000 --disaggregation-bootstrap-port $((8998+$i)) --trust-remote-code --nnodes 1 --node-rank 0 \ + --tp-size 16 --mem-fraction-static 0.7 --attention-backend ascend --device npu --quantization modelslim \ + --disaggregation-transfer-backend ascend --max-running-requests 32 --context-length 8192 --disable-radix-cache \ + --chunked-prefill-size -1 --max-prefill-tokens 10240 --moe-a2a-backend deepep --deepep-mode normal \ + --speculative-algorithm NEXTN --speculative-num-steps 1 --speculative-eagle-topk 1 --speculative-num-draft-tokens 2 \ + --dp-size 8 --enable-dp-attention --disable-shared-experts-fusion --dtype bfloat16 + NODE_RANK=$i + break + fi +done + +# decode +for i in "${!D_IP[@]}"; +do + if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]]; + then + echo "${D_IP[$i]}" + export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 + export HCCL_BUFFSIZE=900 + export SGLANG_DP_ROUND_ROBIN=1 + export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=112 + export TASK_QUEUE_ENABLE=1 + export HCCL_SOCKET_IFNAME=xxx + export GLOO_SOCKET_IFNAME=xxx + python -m sglang.launch_server --model-path ${MODEL_PATH} --disaggregation-mode decode --host ${D_IP[$i]} \ + --port 8001 --trust-remote-code --nnodes 1 --node-rank 0 --tp-size 16 --dp-size 16 \ + --mem-fraction-static 0.8 --max-running-requests 448 --attention-backend ascend --device npu --quantization modelslim \ + --moe-a2a-backend deepep --enable-dp-attention --deepep-mode low_latency --enable-dp-lm-head \ + --cuda-graph-bs 2 4 6 8 10 12 14 16 18 20 22 24 26 28 --disaggregation-transfer-backend ascend --watchdog-timeout 9000 --context-length 8192 \ + --speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 \ + --disable-shared-experts-fusion --dtype bfloat16 --tokenizer-worker-num 4 \ + --load-balance-method round_robin + NODE_RANK=$i + break + fi +done + +``` + +```bash Command +python -m sglang_router.launch_router \ + --pd-disaggregation \ + --policy cache_aware \ + --prefill http://P_IP:8000 8998 \ + --decode http://D_IP:8001 \ + --host 127.0.0.1 \ + --port 6688 \ + --mini-lb +``` + +#### Benchmark + +We tested it based on the `RANDOM` dataset. + +```bash Command +python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 6688 --max-concurrency 448 --random-input-len 2048 --random-output-len 2048 --num-prompts 1792 --random-range-ratio 1 --request-rate 32 +``` + +### DeepSeek-R1 3_5K-1_5K 50ms on A3 8 Cards Mixed Mode + +Model: Deepseek R1 + +Hardware: Atlas 800I A3 8Card + +DeployMode: PD Mixed + +Dataset: random + +Input Output Length: 3.5K+1.5K + +TPOT: 50ms + +#### Model Deployment + +```bash Command +echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor +sysctl -w vm.swappiness=0 +sysctl -w kernel.numa_balancing=0 +sysctl -w kernel.sched_migration_cost_ns=50000 + +unset https_proxy +unset http_proxy +unset HTTPS_PROXY +unset HTTP_PROXY +unset ASCEND_LAUNCH_BLOCKING +source /usr/local/Ascend/ascend-toolkit/set_env.sh +source /usr/local/Ascend/nnal/atb/set_env.sh +export PATH=/usr/local/Ascend/8.5.0/compiler/bishengir/bin:$PATH + +export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True + +export STREAMS_PER_DEVICE=32 +export HCCL_SOCKET_IFNAME=lo +export GLOO_SOCKET_IFNAME=lo +export SGLANG_SCHEDULER_DECREASE_PREFILL_IDLE=1 +export SGLANG_PREFILL_DELAYER_MAX_DELAY_PASSES=200 +export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=56 +export HCCL_BUFFSIZE=1200 +export DEEPEP_NORMAL_LONG_SEQ_ROUND=10 +export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=512 +export SGLANG_NPU_USE_MLAPO=1 +export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 +export SGLANG_USE_FIA_NZ=1 + +MODEL_PATH=xxx + +python3 -m sglang.launch_server --model-path ${MODEL_PATH} \ +--tp 16 \ +--trust-remote-code \ +--attention-backend ascend \ +--device npu \ +--quantization modelslim \ +--watchdog-timeout 9000 \ +--host 127.0.0.1 --port 6699 \ +--cuda-graph-bs 4 8 12 14 \ +--mem-fraction-static 0.77 \ +--max-running-requests 224 \ +--context-length 8188 --disable-radix-cache --chunked-prefill-size -1 --max-prefill-tokens 3000 \ +--moe-a2a-backend deepep --deepep-mode auto \ +--enable-dp-attention --dp-size 16 --enable-dp-lm-head \ +--speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 \ +--dtype bfloat16 +``` + +#### Benchmark + +We tested it based on the `RANDOM` dataset. + +```bash Command +python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 6699 --max-concurrency 224 --random-input-len 3500 --random-output-len 1500 --num-prompts 896 --random-range-ratio 1 +``` + +### DeepSeek-R1 3_5K-1_5K 50ms on A3 16 Cards Disaggregation Mode + +Model: Deepseek R1 + +Hardware: Atlas 800I A3 16Card + +DeployMode: PD Disaggregation + +Dataset: random + +Input Output Length: 3.5K+1.5K + +TPOT: 50ms + +#### Model Deployment + +```bash Command +echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor +sysctl -w vm.swappiness=0 +sysctl -w kernel.numa_balancing=0 +sysctl -w kernel.sched_migration_cost_ns=50000 + +unset https_proxy +unset http_proxy +unset HTTPS_PROXY +unset HTTP_PROXY +unset ASCEND_LAUNCH_BLOCKING +source /usr/local/Ascend/ascend-toolkit/set_env.sh +source /usr/local/Ascend/nnal/atb/set_env.sh +export PATH=/usr/local/Ascend/8.5.0/compiler/bishengir/bin:$PATH + +export SGLANG_SET_CPU_AFFINITY=1 +export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True +export STREAMS_PER_DEVICE=32 + +export ASCEND_MF_STORE_URL="tcp://your prefill ip1:24667" + +P_IP=('your prefill ip1') + +D_IP=('your decode ip1') + +MODEL_PATH=xxx + +export SGLANG_NPU_USE_MLAPO=1 +export SGLANG_USE_FIA_NZ=1 +export ENABLE_MOE_NZ=1 + +LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` +LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` +echo "${LOCAL_HOST1}" +echo "${LOCAL_HOST2}" + +# prefill +for i in "${!P_IP[@]}"; +do + if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]]; + then + echo "${P_IP[$i]}" + export HCCL_BUFFSIZE=3500 + export TASK_QUEUE_ENABLE=2 + + export HCCL_SOCKET_IFNAME=lo + export GLOO_SOCKET_IFNAME=lo + python -m sglang.launch_server --model-path ${MODEL_PATH} --disaggregation-mode prefill --host ${P_IP[$i]} \ + --port 8000 --disaggregation-bootstrap-port $((8998+$i)) --trust-remote-code --nnodes 1 --node-rank 0 \ + --tp-size 16 --mem-fraction-static 0.62 --attention-backend ascend --device npu --quantization modelslim \ + --disaggregation-transfer-backend ascend --max-running-requests 32 --context-length 8192 --disable-radix-cache \ + --chunked-prefill-size -1 --max-prefill-tokens 20480 --moe-a2a-backend deepep --deepep-mode normal \ + --speculative-algorithm NEXTN --speculative-num-steps 1 --speculative-eagle-topk 1 --speculative-num-draft-tokens 2 \ + --dp-size 8 --enable-dp-attention --disable-shared-experts-fusion --dtype bfloat16 + NODE_RANK=$i + break + fi +done + +# decode +for i in "${!D_IP[@]}"; +do + if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]]; + then + echo "${D_IP[$i]}" + export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 + export HCCL_BUFFSIZE=800 + export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=78 + export TASK_QUEUE_ENABLE=1 + export HCCL_SOCKET_IFNAME=xxx + export GLOO_SOCKET_IFNAME=xxx + python -m sglang.launch_server --model-path ${MODEL_PATH} --disaggregation-mode decode --host ${D_IP[$i]} \ + --port 8001 --trust-remote-code --nnodes 1 --node-rank 0 --tp-size 16 --dp-size 16 \ + --mem-fraction-static 0.805 --max-running-requests 416 --attention-backend ascend --device npu --quantization modelslim \ + --moe-a2a-backend deepep --enable-dp-attention --deepep-mode low_latency --enable-dp-lm-head \ + --cuda-graph-bs 2 4 6 8 10 12 14 16 18 20 22 24 26 --disaggregation-transfer-backend ascend --watchdog-timeout 9000 --context-length 8192 \ + --speculative-algorithm NEXTN --speculative-num-steps 2 --speculative-eagle-topk 1 --speculative-num-draft-tokens 3 \ + --disable-shared-experts-fusion --dtype bfloat16 --tokenizer-worker-num 4 \ + --load-balance-method round_robin + NODE_RANK=$i + break + fi +done + +``` + +```bash Command +python -m sglang_router.launch_router \ + --pd-disaggregation \ + --policy cache_aware \ + --prefill http://P_IP:8000 8998 \ + --decode http://D_IP:8001 \ + --host 127.0.0.1 \ + --port 6688 \ + --mini-lb +``` + +#### Benchmark + +We tested it based on the `RANDOM` dataset. + +```bash Command +python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 6688 --max-concurrency 416 --random-input-len 3500 --random-output-len 1500 --num-prompts 1664 --random-range-ratio 1 +``` + +### DeepSeek-V3.2 128K-1K 26ms on A3 32 Cards Disaggregation Mode + +Model: DeepSeek-V3.2-W8A8 + +Hardware: Atlas 800I A3 32Card + +DeployMode: PD Disaggregation + +Dataset: random + +Input Output Length: 128K+1K + +TPOT: 26ms + +#### Model Deployment + +```bash Command +echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor +sysctl -w vm.swappiness=0 +sysctl -w kernel.numa_balancing=0 +sysctl -w kernel.sched_migration_cost_ns=50000 + +unset https_proxy +unset http_proxy +unset HTTPS_PROXY +unset HTTP_PROXY +unset ASCEND_LAUNCH_BLOCKING + +source /usr/local/Ascend/ascend-toolkit/set_env.sh +source /usr/local/Ascend/nnal/atb/set_env.sh +export LD_LIBRARY_PATH=/usr/local/Ascend/ascend-toolkit/latest/opp/vendors/customize/op_api/lib/:${LD_LIBRARY_PATH} +export PATH=/usr/local/Ascend/8.5.0/compiler/bishengir/bin:$PATH + +export SGLANG_SET_CPU_AFFINITY=1 +export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True +export STREAMS_PER_DEVICE=32 +export ASCEND_MF_STORE_URL="tcp://your prefill ip1:24670" + +P_IP=('your prefill ip1' 'your prefill ip2') +D_IP=('your decode ip1' 'your decode ip2') +MODEL_PATH=xxx + +LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` +LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` +echo "${LOCAL_HOST1}" +echo "${LOCAL_HOST2}" + +# prefill +for i in "${!P_IP[@]}"; +do + if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]]; + then + echo "${P_IP[$i]}" + export HCCL_BUFFSIZE=1200 + export TASK_QUEUE_ENABLE=2 + export HCCL_SOCKET_IFNAME=xxx + export GLOO_SOCKET_IFNAME=xxx + + python3 -m sglang.launch_server --model-path ${MODEL_PATH} \ + --tp 32 \ + --trust-remote-code \ + --attention-backend ascend \ + --device npu \ + --watchdog-timeout 9000 \ + --host ${P_IP[$i]} --port 8000 \ + --mem-fraction-static 0.73 \ + --disable-radix-cache --chunked-prefill-size -1 --max-prefill-tokens 68000 \ + --max-running-requests 1 \ + --moe-a2a-backend deepep --deepep-mode normal \ + --quantization modelslim \ + --disaggregation-transfer-backend ascend \ + --disaggregation-mode prefill \ + --disable-cuda-graph \ + --nnodes 2 --node-rank $i \ + --disaggregation-bootstrap-port 8995 \ + --moe-dense-tp-size 1 \ + --enable-dsa-prefill-context-parallel \ + --dsa-prefill-cp-mode in-seq-split \ + --attn-cp-size 32 \ + --speculative-algorithm NEXTN --speculative-num-steps 1 --speculative-eagle-topk 1 --speculative-num-draft-tokens 2 \ + --dist-init-addr ${P_IP[0]}:10000 + break + fi +done + + +# decode +for i in "${!D_IP[@]}"; +do + if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]]; + then + echo "${D_IP[$i]}" + export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 + + export TASK_QUEUE_ENABLE=0 + export SGLANG_SCHEDULER_SKIP_ALL_GATHER=1 + + export HCCL_SOCKET_IFNAME=xxx + export GLOO_SOCKET_IFNAME=xxx + + DP=8 + export HCCL_BUFFSIZE=400 + export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=8 + + python3 -m sglang.launch_server --model-path ${MODEL_PATH} \ + --tp 32 \ + --dp ${DP} \ + --ep 32 \ + --moe-dense-tp-size 1 \ + --enable-dp-attention \ + --enable-dp-lm-head \ + --trust-remote-code \ + --attention-backend ascend \ + --device npu \ + --watchdog-timeout 9000 \ + --host ${D_IP[$i]} --port 8001 \ + --mem-fraction-static 0.79 \ + --disable-radix-cache \ + --chunked-prefill-size -1 --max-prefill-tokens 68000 \ + --max-running-requests 32 \ + --cuda-graph-max-bs 4 \ + --moe-a2a-backend deepep \ + --deepep-mode low_latency \ + --quantization modelslim \ + --speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 \ + --disaggregation-transfer-backend ascend \ + --disaggregation-mode decode \ + --nnodes 2 --node-rank $i \ + --dist-init-addr ${D_IP[0]}:10000 + break + fi +done +``` + + +```bash Command +python -m sglang_router.launch_router \ + --pd-disaggregation \ + --policy cache_aware \ + --prefill http://P_IP1:8000 8995 \ + --decode http://D_IP1:8001 \ + --host 127.0.0.1 \ + --port 6688 \ + --mini-lb +``` + +#### Benchmark + +We tested it based on the `RANDOM` dataset. + +```bash Command +python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 6688 --max-concurrency 8 --random-input-len 131076 --random-output-len 1024 --num-prompts 8 --random-range-ratio 1 +``` + +### Qwen3-235B-A22B 3_5K-1_5K 50ms on A3 24 Cards Disaggregation Mode + +Model: Qwen3-235B-A22B-W8A8 + +Hardware: Atlas 800I A3 24Card + +DeployMode: PD Disaggregation + +Dataset: random + +Input Output Length: 3.5K+1.5K + +TPOT: 50ms + +#### Model Deployment + +```bash Command +echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor +sysctl -w vm.swappiness=0 +sysctl -w kernel.numa_balancing=0 +sysctl -w kernel.sched_migration_cost_ns=50000 + +unset https_proxy +unset http_proxy +unset HTTPS_PROXY +unset HTTP_PROXY +unset ASCEND_LAUNCH_BLOCKING + +source /usr/local/Ascend/ascend-toolkit/latest/opp/vendors/customize/bin/set_env.bash + +export SGLANG_SET_CPU_AFFINITY=1 +export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True +export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600 +export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 +export SGLANG_DP_ROUND_ROBIN=1 +export SGLANG_NPU_FUSED_MOE_MODE=2 + +MODEL_PATH=xxx +export ASCEND_MF_STORE_URL="tcp://your prefill ip1:24667" +P_IP=('your prefill ip1') +D_IP=('your decode ip1' 'your decode ip2') + +LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` +LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` + +echo "${LOCAL_HOST1}" +echo "${LOCAL_HOST2}" + + +for i in "${!P_IP[@]}"; +do + if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]]; + then + echo "${P_IP[$i]}" + source /usr/local/Ascend/ascend-toolkit/set_env.sh + source /usr/local/Ascend/nnal/atb/set_env.sh + export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=188416 + export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=1024 + export DEEPEP_NORMAL_LONG_SEQ_ROUND=16 + export HCCL_BUFFSIZE=4300 + export TASK_QUEUE_ENABLE=2 + export HCCL_SOCKET_IFNAME=lo + export GLOO_SOCKET_IFNAME=lo + export STREAMS_PER_DEVICE=32 + + # Prefill + python -m sglang.launch_server --model-path ${MODEL_PATH} --disaggregation-mode prefill \ + --host ${P_IP[$i]} --port 8000 --disaggregation-bootstrap-port 8995 --trust-remote-code \ + --nnodes 1 --node-rank $i --tp-size 16 --dp-size 16 --mem-fraction-static 0.6 \ + --disable-radix-cache \ + --attention-backend ascend --device npu --quantization modelslim --disaggregation-transfer-backend ascend \ + --speculative-algorithm EAGLE3 --speculative-draft-model-path xxx \ + --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 \ + --speculative-draft-model-quantization unquant \ + --max-running-requests 128 --chunked-prefill-size 94208 --max-prefill-tokens 262144 \ + --enable-dp-attention \ + --moe-a2a-backend ascend_fuseep --dtype bfloat16 + NODE_RANK=$i + break + fi +done + + +for i in "${!D_IP[@]}"; +do + if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]]; + then + echo "${D_IP[$i]}" + source /usr/local/Ascend/ascend-toolkit/set_env.sh + source /usr/local/Ascend/nnal/atb/set_env.sh + export DP_ROUND_ROBIN=1 + export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=65536 + export HCCL_BUFFSIZE=800 + export HCCL_SOCKET_IFNAME=data0.3001 + export GLOO_SOCKET_IFNAME=data0.3001 + export STREAMS_PER_DEVICE=32 + + python -m sglang.launch_server --model-path ${MODEL_PATH} --disaggregation-mode decode \ + --host ${D_IP[$i]} --port 8001 --trust-remote-code \ + --nnodes 2 --node-rank $i --tp-size 32 --dp-size 32 --mem-fraction-static 0.83 --max-running-requests 768 \ + --attention-backend ascend --device npu --quantization modelslim --enable-dp-attention \ + --moe-a2a-backend ascend_fuseep --cuda-graph-bs 6 8 12 15 18 20 22 24 \ + --speculative-algorithm EAGLE3 --speculative-draft-model-path xxx \ + --speculative-draft-model-quantization unquant \ + --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 \ + --dist-init-addr xxx:5000 \ + --disaggregation-transfer-backend ascend --watchdog-timeout 9000 --context-length 8192 \ + --enable-dp-lm-head --dtype bfloat16 --tokenizer-worker-num 4 \ + --load-balance-method round_robin + NODE_RANK=$i + break + fi +done + +``` + +```shell Command +export SGLANG_DP_ROUND_ROBIN=1 +python -m sglang_router.launch_router \ + --pd-disaggregation \ + --policy cache_aware \ + --prefill http://PIP:8000 8995 \ + --decode http://DIP:8001 \ + --host 127.0.0.1 \ + --port 6688 \ + --mini-lb +``` + +#### Benchmark + +We tested it based on the `RANDOM` dataset. + +```shell Command +python -m sglang.bench_serving --dataset-name random --backend sglang-oai --host 127.0.0.1 --port 7239 --max-concurrency 860 --random-input-len 3500 --random-output-len 1500 --num-prompts 3440 --random-range-ratio 1 +``` + +### Qwen3-235B-A22B 3_5K-1_5K 50ms on A3 8 Cards Mixed Mode + +Model: Qwen3-235B-A22B-W8A8 + +Hardware: Atlas 800I A3 8Card + +DeployMode: PD Mixed + +Dataset: random + +Input Output Length: 3.5K+1.5K + +TPOT: 50ms + +#### Model Deployment + +```bash Command +echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor +sysctl -w vm.swappiness=0 +sysctl -w kernel.numa_balancing=0 +sysctl -w kernel.sched_migration_cost_ns=50000 + +export SGLANG_SET_CPU_AFFINITY=1 +unset https_proxy +unset http_proxy +unset HTTPS_PROXY +unset HTTP_PROXY +unset ASCEND_LAUNCH_BLOCKING +source /usr/local/Ascend/ascend-toolkit/set_env.sh +source /usr/local/Ascend/nnal/atb/set_env.sh +source /usr/local/Ascend/ascend-toolkit/latest/opp/vendors/customize/bin/set_env.bash +export PATH=/usr/local/Ascend/8.5.0/compiler/bishengir/bin:$PATH + +export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True + +MODEL_PATH=xxx + +export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600 + +LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` +LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` + +echo "${LOCAL_HOST1}" +echo "${LOCAL_HOST2}" + +export HCCL_BUFFSIZE=570 +export HCCL_SOCKET_IFNAME=lo +export GLOO_SOCKET_IFNAME=lo +export HCCL_OP_EXPANSION_MODE="AIV" +export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 +export SGLANG_SCHEDULER_DECREASE_PREFILL_IDLE=1 +export SGLANG_PREFILL_DELAYER_MAX_DELAY_PASSES=100 + +export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=188416 +export SGLANG_NPU_FUSED_MOE_MODE=2 + +python -m sglang.launch_server --model-path $MODEL_PATH \ + --host 127.0.0.1 --port 7439 --trust-remote-code --nnodes 1 --node-rank 0 \ + --attention-backend ascend --device npu --quantization modelslim \ + --max-running-requests 432 --context-length 8192 --dtype bfloat16 \ + --chunked-prefill-size 94208 --max-prefill-tokens 458880 --sampling-backend ascend \ + --speculative-algorithm EAGLE3 --speculative-draft-model-path xxx \ + --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 \ + --disable-radix-cache --moe-a2a-backend ascend_fuseep --speculative-draft-model-quantization unquant \ + --tp 16 --dp-size 16 --enable-dp-attention --enable-dp-lm-head --mem-fraction-static 0.8 --cuda-graph-bs 1 2 4 8 16 20 24 26 27 +``` + +#### Benchmark + +We tested it based on the `RANDOM` dataset. + +```shell Command +python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7439 --max-concurrency 272 --random-input-len 3500 --random-output-len 1500 --num-prompts 1088 --random-range-ratio 1 +``` + +### Qwen3-235B-A22B 2K-2K 50ms on A3 8 Cards Mixed Mode + +Model: Qwen3-235B-A22B-W8A8 + +Hardware: Atlas 800I A3 8Card + +DeployMode: PD Mixed + +Dataset: random + +Input Output Length: 2K+2K + +TPOT: 50ms + +#### Model Deployment + +```bash Command +echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor +sysctl -w vm.swappiness=0 +sysctl -w kernel.numa_balancing=0 +sysctl -w kernel.sched_migration_cost_ns=50000 + +unset https_proxy +unset http_proxy +unset HTTPS_PROXY +unset HTTP_PROXY +unset ASCEND_LAUNCH_BLOCKING +source /usr/local/Ascend/ascend-toolkit/set_env.sh +source /usr/local/Ascend/nnal/atb/set_env.sh +source /usr/local/Ascend/ascend-toolkit/latest/opp/vendors/customize/bin/set_env.bash +export PATH=/usr/local/Ascend/8.5.0/compiler/bishengir/bin:$PATH + +export SGLANG_SET_CPU_AFFINITY=1 +export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True +export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600 + +MODEL_PATH=xxx + +LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` +LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` + +echo "${LOCAL_HOST1}" +echo "${LOCAL_HOST2}" + +export HCCL_BUFFSIZE=450 +export HCCL_SOCKET_IFNAME=xxx +export GLOO_SOCKET_IFNAME=xxx +export HCCL_OP_EXPANSION_MODE="AIV" +export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 +export SGLANG_SCHEDULER_DECREASE_PREFILL_IDLE=1 +export SGLANG_PREFILL_DELAYER_MAX_DELAY_PASSES=100 +export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=147456 +export SGLANG_NPU_FUSED_MOE_MODE=2 + +python -m sglang.launch_server --model-path $MODEL_PATH \ + --host 127.0.0.1 --port 7439 --trust-remote-code --nnodes 1 --node-rank 0 \ + --attention-backend ascend --device npu --quantization modelslim \ + --max-running-requests 624 --context-length 8192 --dtype bfloat16 \ + --chunked-prefill-size 73728 --max-prefill-tokens 458880 --speculative-draft-model-quantization unquant \ + --speculative-algorithm EAGLE3 --speculative-draft-model-path xxx \ + --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 \ + --disable-radix-cache --moe-a2a-backend ascend_fuseep \ + --tp 16 --dp-size 16 --enable-dp-attention --enable-dp-lm-head --mem-fraction-static 0.83 --cuda-graph-bs 4 8 16 24 28 29 30 32 34 36 37 38 39 +``` + +#### Benchmark + +We tested it based on the `RANDOM` dataset. + +```shell Command +python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7439 --max-concurrency 480 --random-input-len 2048 --random-output-len 2048 --num-prompts 480 --random-range-ratio 1 +``` + +### Qwen3-235B-A22B 2K-2K 50ms on A3 16 Cards Mixed Mode + +Model: Qwen3-235B-A22B-W8A8 + +Hardware: Atlas 800I A3 16Card + +DeployMode: PD Mixed + +Dataset: random + +Input Output Length: 2K+2K + +TPOT: 50ms + +#### Model Deployment + +```bash Command +echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor +sysctl -w vm.swappiness=0 +sysctl -w kernel.numa_balancing=0 +sysctl -w kernel.sched_migration_cost_ns=50000 + +unset https_proxy +unset http_proxy +unset HTTPS_PROXY +unset HTTP_PROXY +unset ASCEND_LAUNCH_BLOCKING +source /usr/local/Ascend/ascend-toolkit/set_env.sh +source /usr/local/Ascend/nnal/atb/set_env.sh +source /usr/local/Ascend/ascend-toolkit/latest/opp/vendors/customize/bin/set_env.bash + +export SGLANG_SET_CPU_AFFINITY=1 +export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True +export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600 + +MODEL_PATH=xxx + +LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` +LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` + +echo "${LOCAL_HOST1}" +echo "${LOCAL_HOST2}" + +export HCCL_BUFFSIZE=1600 +export HCCL_SOCKET_IFNAME=xxx +export GLOO_SOCKET_IFNAME=xxx +export HCCL_OP_EXPANSION_MODE="AIV" + +MIX_IP=('IP1' 'IP2') + +for i in "${!MIX_IP[@]}"; +do + if [[ "$LOCAL_HOST1" == "${MIX_IP[$i]}" || "$LOCAL_HOST2" == "${MIX_IP[$i]}" ]]; + then + echo "${MIX_IP[$i]}" + export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 + + python -m sglang.launch_server --model-path ${MODEL_PATH} \ + --host 127.0.0.1 --port 7439 --trust-remote-code \ + --nnodes 2 --node-rank $i --tp-size 32 --dp-size 32 --mem-fraction-static 0.8 --max-running-requests 768 \ + --attention-backend ascend --device npu --quantization modelslim --enable-dp-attention \ + --moe-a2a-backend deepep --deepep-mode auto --cuda-graph-bs 6 8 10 12 18 24 \ + --dist-init-addr ${MIX_IP[0]}:5000 --chunked-prefill-size 131072 --max-prefill-tokens 458880 \ + --speculative-algorithm EAGLE3 --speculative-draft-model-path xxx --speculative-draft-model-quantization= unquant \ + --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 \ + --context-length 8192 --disable-radix-cache \ + --enable-dp-lm-head --dtype bfloat16 + NODE_RANK=$i + break + fi +done + +``` + +#### Benchmark + +We tested it based on the `RANDOM` dataset. + +```shell Command +python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7439 --max-concurrency 768 --random-input-len 2000 --random-output-len 2000 --num-prompts 768 --random-range-ratio 1 +``` + +### Qwen3-235B-A22B 11K-1K 10ms on A3 8 Cards Mixed Mode + +Model: Qwen3-235B-A22B-W8A8 + +Hardware: Atlas 800I A3 8Card + +DeployMode: PD Mixed + +Dataset: random + +Input Output Length: 11K+1K + +TPOT: 10ms + +#### Model Deployment + +```shell Command +echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor +sysctl -w vm.swappiness=0 +sysctl -w kernel.numa_balancing=0 +sysctl -w kernel.sched_migration_cost_ns=50000 + +export SGLANG_SET_CPU_AFFINITY=1 + +unset https_proxy +unset http_proxy +unset HTTPS_PROXY +unset HTTP_PROXY +unset ASCEND_LAUNCH_BLOCKING +source /usr/local/Ascend/ascend-toolkit/set_env.sh +source /usr/local/Ascend/nnal/atb/set_env.sh +source /usr/local/Ascend/ascend-toolkit/latest/opp/vendors/customize/bin/set_env.bash +export PATH=/usr/local/Ascend/8.5.0/compiler/bishengir/bin:$PATH + +export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True + +MODEL_PATH=xxx + +export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600 + +LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` +LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` + +echo "${LOCAL_HOST1}" +echo "${LOCAL_HOST2}" + +export HCCL_BUFFSIZE=1600 +export HCCL_SOCKET_IFNAME=xxx +export GLOO_SOCKET_IFNAME=xxx +export HCCL_OP_EXPANSION_MODE="AIV" +export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 + +python -m sglang.launch_server --model-path $MODEL_PATH \ + --host 127.0.0.1 --port 7439 --trust-remote-code --nnodes 1 --node-rank 0 \ + --attention-backend ascend --device npu --quantization modelslim \ + --max-running-requests 1 --dtype bfloat16 \ + --chunked-prefill-size -1 --max-prefill-tokens 16384 --speculative-draft-model-quantization unquant \ + --speculative-algorithm EAGLE3 --speculative-draft-model-path xxx \ + --speculative-num-steps 4 --speculative-eagle-topk 1 --speculative-num-draft-tokens 5 \ + --disable-radix-cache --enable-dp-lm-head \ + --tp 16 --mem-fraction-static 0.78 --cuda-graph-bs 1 + +``` + +#### Benchmark + +We tested it based on the `RANDOM` dataset. + +```shell Command +python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7439 --max-concurrency 1 --random-input-len 11000 --random-output-len 1000 --num-prompts 1 --random-range-ratio 1 +``` + +### Qwen3-32B 6K-1_5K 18ms on A3 4 Cards Mixed Mode + +Model: Qwen3-32B + +Hardware: Atlas 800I A3 4Card + +DeployMode: PD Mixed + +Dataset: random + +Input Output Length: 6K+1.5K + +TPOT: 18ms + +#### Model Deployment + +```shell Command +echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor +sysctl -w vm.swappiness=0 +sysctl -w kernel.numa_balancing=0 +sysctl -w kernel.sched_migration_cost_ns=50000 + +export SGLANG_SET_CPU_AFFINITY=1 + +unset https_proxy +unset http_proxy +unset HTTPS_PROXY +unset HTTP_PROXY +unset ASCEND_LAUNCH_BLOCKING +source /usr/local/Ascend/ascend-toolkit/set_env.sh +source /usr/local/Ascend/nnal/atb/set_env.sh +source /usr/local/Ascend/ascend-toolkit/latest/opp/vendors/customize/bin/set_env.bash +export PATH=/usr/local/Ascend/8.5.0/compiler/bishengir/bin:$PATH + +MODEL_PATH=xxx + +export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600 + +LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` +LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` + +echo "${LOCAL_HOST1}" +echo "${LOCAL_HOST2}" + +export HCCL_BUFFSIZE=400 +export HCCL_SOCKET_IFNAME=xxx +export GLOO_SOCKET_IFNAME=xxx +export HCCL_OP_EXPANSION_MODE="AIV" +export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 + +python -m sglang.launch_server --model-path $MODEL_PATH \ + --host 127.0.0.1 --port 7439 --trust-remote-code --nnodes 1 --node-rank 0 \ + --attention-backend ascend --device npu \ + --max-running-requests 32 \ + --disable-radix-cache \ + --chunked-prefill-size 24576 --max-prefill-tokens 65536 \ + --speculative-algorithm EAGLE3 --speculative-draft-model-path xxx \ + --speculative-num-steps 4 --speculative-eagle-topk 1 --speculative-num-draft-tokens 5 \ + --tp-size 8 --mem-fraction-static 0.72 --cuda-graph-bs 8 16 24 32 --dtype bfloat16 + +``` + +#### Benchmark + +We tested it based on the `RANDOM` dataset. + +```shell Command +python3 -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7439 --max-concurrency 32 --random-output-len 1500 --random-input-len 6000 --num-prompts 32 --random-range-ratio 1 +``` + +### Qwen3-32B 4K-1_5K 11ms on A3 4 Cards Mixed Mode + +Model: Qwen3-32B + +Hardware: Atlas 800I A3 4Card + +DeployMode: PD Mixed + +Dataset: random + +Input Output Length: 4K+1.5K + +TPOT: 11ms + +#### Model Deployment + +```shell Command +echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor +sysctl -w vm.swappiness=0 +sysctl -w kernel.numa_balancing=0 +sysctl -w kernel.sched_migration_cost_ns=50000 + +export SGLANG_SET_CPU_AFFINITY=1 +unset https_proxy +unset http_proxy +unset HTTPS_PROXY +unset HTTP_PROXY +unset ASCEND_LAUNCH_BLOCKING +source /usr/local/Ascend/ascend-toolkit/set_env.sh +source /usr/local/Ascend/nnal/atb/set_env.sh +source /usr/local/Ascend/ascend-toolkit/latest/opp/vendors/customize/bin/set_env.bash +export PATH=/usr/local/Ascend/8.5.0/compiler/bishengir/bin:$PATH + +MODEL_PATH=xxx + +export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600 + +LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` +LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` + +echo "${LOCAL_HOST1}" +echo "${LOCAL_HOST2}" + +export HCCL_BUFFSIZE=400 +export HCCL_SOCKET_IFNAME=lo +export GLOO_SOCKET_IFNAME=lo +export HCCL_OP_EXPANSION_MODE="AIV" +export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 + +python -m sglang.launch_server --model-path $MODEL_PATH \ + --host 127.0.0.1 --port 7339 --trust-remote-code --nnodes 1 --node-rank 0 \ + --attention-backend ascend --device npu \ + --max-running-requests 1 \ + --disable-radix-cache \ + --speculative-algorithm EAGLE3 --speculative-draft-model-path xxx \ + --speculative-num-steps 4 --speculative-eagle-topk 1 --speculative-num-draft-tokens 5 \ + --chunked-prefill-size 24576 --max-prefill-tokens 65536 \ + --tp-size 8 --mem-fraction-static 0.72 --cuda-graph-bs 1 --dtype bfloat16 + +``` + +#### Benchmark + +We tested it based on the `RANDOM` dataset. + +```shell Command +python3 -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7239 --random-range-ratio 1 --max-concurrency 1 --random-output-len 1500 --random-input-len 4096 --num-prompts 4 +``` + +### Qwen3-32B 18K-4K 6ms on A3 8 Cards Mixed Mode + +Model: Qwen3-32B + +Hardware: Atlas 800I A3 8Card + +DeployMode: PD Mixed + +Dataset: random + +Input Output Length: 18K+4K + +TPOT: 6ms + +#### Model Deployment + +```shell Command +echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor +sysctl -w vm.swappiness=0 +sysctl -w kernel.numa_balancing=0 +sysctl -w kernel.sched_migration_cost_ns=50000 + +export SGLANG_SET_CPU_AFFINITY=1 +unset https_proxy +unset http_proxy +unset HTTPS_PROXY +unset HTTP_PROXY +unset ASCEND_LAUNCH_BLOCKING +source /usr/local/Ascend/ascend-toolkit/set_env.sh +source /usr/local/Ascend/nnal/atb/set_env.sh +source /usr/local/Ascend/ascend-toolkit/latest/opp/vendors/customize/bin/set_env.bash +export PATH=/usr/local/Ascend/8.5.0/compiler/bishengir/bin:$PATH + +MODEL_PATH=xxx + +export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600 + +LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` +LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` + +echo "${LOCAL_HOST1}" +echo "${LOCAL_HOST2}" + +export HCCL_BUFFSIZE=400 +export HCCL_SOCKET_IFNAME=lo +export GLOO_SOCKET_IFNAME=lo +export HCCL_OP_EXPANSION_MODE="AIV" +export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 + +python -m sglang.launch_server --model-path $MODEL_PATH \ + --host 127.0.0.1 --port 7339 --trust-remote-code --nnodes 1 --node-rank 0 \ + --attention-backend ascend --device npu \ + --max-running-requests 1 \ + --disable-radix-cache --speculative-draft-model-quantization unquant \ + --speculative-algorithm EAGLE3 --speculative-draft-model-path xxx \ + --speculative-num-steps 4 --speculative-eagle-topk 1 --speculative-num-draft-tokens 5 \ + --chunked-prefill-size -1 --max-prefill-tokens 65536 \ + --tp-size 16 --mem-fraction-static 0.72 --cuda-graph-bs 1 --dtype bfloat16 +``` + +#### Benchmark + +We tested it based on the `RANDOM` dataset. + +```shell Command +python3 -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7339 --random-range-ratio 1 --max-concurrency 1 --random-output-len 18000 --random-input-len 4000 --num-prompts 1 +``` + +### Qwen3-32B 3_5K-1_5K 50ms on A3 2 Cards Mixed Mode + +Model: Qwen3-32B + +Hardware: Atlas 800I A3 2Card + +DeployMode: PD Mixed + +Dataset: random + +Input Output Length: 3.5K+1.5K + +TPOT: 50ms + +#### Model Deployment + +```shell Command +echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor +sysctl -w vm.swappiness=0 +sysctl -w kernel.numa_balancing=0 +sysctl -w kernel.sched_migration_cost_ns=50000 + +export SGLANG_SET_CPU_AFFINITY=1 +unset https_proxy +unset http_proxy +unset HTTPS_PROXY +unset HTTP_PROXY +unset ASCEND_LAUNCH_BLOCKING +source /usr/local/Ascend/ascend-toolkit/set_env.sh +source /usr/local/Ascend/nnal/atb/set_env.sh +source /usr/local/Ascend/ascend-toolkit/latest/opp/vendors/customize/bin/set_env.bash + + +MODEL_PATH=xxx + +export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600 + +LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` +LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` + +echo "${LOCAL_HOST1}" +echo "${LOCAL_HOST2}" + +export HCCL_BUFFSIZE=400 +export HCCL_SOCKET_IFNAME=lo +export GLOO_SOCKET_IFNAME=lo +export HCCL_OP_EXPANSION_MODE="AIV" +export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 + +python -m sglang.launch_server --model-path $MODEL_PATH \ + --host 127.0.0.1 --port 7239 --trust-remote-code --nnodes 1 --node-rank 0 \ + --attention-backend ascend --device npu --quantization modelslim \ + --max-running-requests 78 \ + --disable-radix-cache --speculative-draft-model-quantization unquant \ + --chunked-prefill-size -1 --max-prefill-tokens 49152 \ + --speculative-algorithm EAGLE3 --speculative-draft-model-path xxx \ + --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 \ + --tp-size 4 --mem-fraction-static 0.72 --cuda-graph-bs 16 32 64 68 72 78 --dtype bfloat16 +``` + +#### Benchmark + +We tested it based on the `RANDOM` dataset. + +```shell Command +python3 -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7239 --max-concurrency 78 --random-output-len 1500 --random-input-len 3500 --num-prompts 312 --random-range-ratio 1 +``` + +### Qwen3-32B 2K-2K 50ms on A3 2 Cards Mixed Mode + +Model: Qwen3-32B + +Hardware: Atlas 800I A3 2Card + +DeployMode: PD Mixed + +Dataset: random + +Input Output Length: 2K+2K + +TPOT: 50ms + +#### Model Deployment + +```shell Command +echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor +sysctl -w vm.swappiness=0 +sysctl -w kernel.numa_balancing=0 +sysctl -w kernel.sched_migration_cost_ns=50000 + +export SGLANG_SET_CPU_AFFINITY=1 +unset https_proxy +unset http_proxy +unset HTTPS_PROXY +unset HTTP_PROXY +unset ASCEND_LAUNCH_BLOCKING +source /usr/local/Ascend/ascend-toolkit/set_env.sh +source /usr/local/Ascend/nnal/atb/set_env.sh +source /usr/local/Ascend/ascend-toolkit/latest/opp/vendors/customize/bin/set_env.bash +export PATH=/usr/local/Ascend/8.5.0/compiler/bishengir/bin:$PATH + +MODEL_PATH=xxx + +export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600 + +LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` +LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` + +echo "${LOCAL_HOST1}" +echo "${LOCAL_HOST2}" + +export HCCL_BUFFSIZE=400 +export HCCL_SOCKET_IFNAME=lo +export GLOO_SOCKET_IFNAME=lo +export HCCL_OP_EXPANSION_MODE="AIV" +export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 + +python -m sglang.launch_server --model-path $MODEL_PATH \ + --host 127.0.0.1 --port 7239 --trust-remote-code --nnodes 1 --node-rank 0 \ + --attention-backend ascend --device npu --quantization modelslim \ + --max-running-requests 120 \ + --disable-radix-cache --speculative-draft-model-quantization unquant \ + --speculative-algorithm EAGLE3 --speculative-draft-model-path xxx \ + --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 \ + --chunked-prefill-size -1 --max-prefill-tokens 49152 \ + --tp-size 4 --mem-fraction-static 0.7 --cuda-graph-bs 54 60 66 72 78 84 90 108 114 120 --dtype bfloat16 + +``` + +#### Benchmark + +We tested it based on the `RANDOM` dataset. + +```shell Command +python3 -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7239 --max-concurrency 120 --random-output-len 2000 --random-input-len 2000 --num-prompts 480 --random-range-ratio 1 +``` + +### Qwen3-30B-A3B 3_5K-1_5K 50ms on A3 1 Card Mixed Mode + +Model: Qwen3-30B-A3B-Instruct-2507 + +Hardware: Atlas 800I A3 1Card + +DeployMode: PD Mixed + +Dataset: random + +Input Output Length: 3.5K+1.5K + +TPOT: 50ms + +#### Model Deployment + +```bash Command +echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor +sysctl -w vm.swappiness=0 +sysctl -w kernel.numa_balancing=0 +sysctl -w kernel.sched_migration_cost_ns=50000 + +unset https_proxy +unset http_proxy +unset HTTPS_PROXY +unset HTTP_PROXY +unset ASCEND_LAUNCH_BLOCKING +source /usr/local/Ascend/ascend-toolkit/set_env.sh +source /usr/local/Ascend/nnal/atb/set_env.sh +source /usr/local/Ascend/ascend-toolkit/latest/opp/vendors/customize/bin/set_env.bash +export PATH=/usr/local/Ascend/8.5.0/compiler/bishengir/bin:$PATH + +MODEL_PATH=xxx + +export SGLANG_SET_CPU_AFFINITY=1 +export ASCEND_LAUNCH_BLOCKING=0 +export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600 +export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True + +LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` +LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` + +echo "${LOCAL_HOST1}" +echo "${LOCAL_HOST2}" + +export HCCL_BUFFSIZE=400 +export HCCL_SOCKET_IFNAME=lo +export GLOO_SOCKET_IFNAME=lo +export HCCL_OP_EXPANSION_MODE="AIV" +export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 +export SGLANG_SCHEDULER_DECREASE_PREFILL_IDLE=1 +export SGLANG_PREFILL_DELAYER_MAX_DELAY_PASSES=200 + +python -m sglang.launch_server --model-path $MODEL_PATH \ + --host 127.0.0.1 --port 7239 --trust-remote-code --nnodes 1 --node-rank 0 \ + --attention-backend ascend --device npu --quantization modelslim \ + --max-running-requests 162 \ + --disable-radix-cache \ + --speculative-draft-model-quantization unquant \ + --speculative-algorithm EAGLE3 --speculative-draft-model-path xxx \ + --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 \ + --chunked-prefill-size -1 --max-prefill-tokens 35000 \ + --tp-size 2 --mem-fraction-static 0.87 --cuda-graph-bs 1 5 15 40 70 100 120 130 140 146 150 154 156 158 160 162 \ + --dtype bfloat16 +``` + +#### Benchmark + +We tested it based on the `RANDOM` dataset. + +```shell Command +python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7239 --max-concurrency 156 --random-input-len 3500 --random-output-len 1500 --num-prompts 624 --random-range-ratio 1 +``` + +### Qwen3-Coder-480B-A35B-Instruct 3_5K-1_5K 50ms on A3 24 Cards Disaggregation Mode + +Model: Qwen3-Coder-480B-A35B-Instruct + +Hardware: Atlas 800I A3 24Card + +DeployMode: PD Disaggregation + +Dataset: random + +Input Output Length: 3.5K+1.5K + +TPOT: 50ms + +#### Model Deployment + +```bash Command +echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor +sysctl -w vm.swappiness=0 +sysctl -w kernel.numa_balancing=0 +sysctl -w kernel.sched_migration_cost_ns=50000 + +unset https_proxy +unset http_proxy +unset HTTPS_PROXY +unset HTTP_PROXY +unset ASCEND_LAUNCH_BLOCKING + +source /usr/local/Ascend/ascend-toolkit/latest/opp/vendors/customize/bin/set_env.bash + +export SGLANG_SET_CPU_AFFINITY=1 +export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True +export STREAMS_PER_DEVICE=32 +export SGLANG_NPU_FUSED_MOE_MODE=2 + +MODEL_PATH=xxx +export ASCEND_MF_STORE_URL="tcp://PIP:24667" +P_IP=('PIP') +D_IP=('DIP1' 'DIP2') +export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600 + +LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` +LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` + +echo "${LOCAL_HOST1}" +echo "${LOCAL_HOST2}" + + +for i in "${!P_IP[@]}"; +do + if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]]; + then + echo "${P_IP[$i]}" + source /usr/local/Ascend/ascend-toolkit/set_env.sh + source /usr/local/Ascend/nnal/atb/set_env.sh + export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=327680 + export HCCL_BUFFSIZE=1550 + export TASK_QUEUE_ENABLE=2 + export HCCL_SOCKET_IFNAME=lo + export GLOO_SOCKET_IFNAME=lo + + python -m sglang.launch_server --model-path ${MODEL_PATH} --disaggregation-mode prefill \ + --host ${P_IP[$i]} --port 8000 --disaggregation-bootstrap-port 8995 --trust-remote-code \ + --nnodes 1 --node-rank $i --tp-size 16 --dp-size 2 --mem-fraction-static 0.7 \ + --disable-radix-cache \ + --attention-backend ascend --device npu --quantization modelslim --disaggregation-transfer-backend ascend \ + --max-running-requests 16 --chunked-prefill-size 20480 --max-prefill-tokens 20480 \ + --enable-dp-attention \ + --moe-a2a-backend ascend_fuseep --dtype bfloat16 \ + --disable-overlap-schedule + NODE_RANK=$i + break + fi +done + +for i in "${!D_IP[@]}"; +do + if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]]; + then + echo "${D_IP[$i]}" + source /usr/local/Ascend/ascend-toolkit/set_env.sh + source /usr/local/Ascend/nnal/atb/set_env.sh + export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=65536 + export HCCL_BUFFSIZE=600 + export SGLANG_NPU_FUSED_MOE_MODE=2 + export HCCL_SOCKET_IFNAME=xxx + export GLOO_SOCKET_IFNAME=xxx + + python -m sglang.launch_server --model-path ${MODEL_PATH} --disaggregation-mode decode \ + --host ${D_IP[$i]} --port 8001 --trust-remote-code \ + --nnodes 2 --node-rank $i --tp-size 32 --dp-size 4 --mem-fraction-static 0.75 --max-running-requests 544 \ + --attention-backend ascend --device npu --quantization modelslim --enable-dp-attention \ + --moe-a2a-backend ascend_fuseep --cuda-graph-bs 16 32 56 72 80 88 96 104 112 120 128 136 \ + --dist-init-addr DIP1:5000 \ + --disaggregation-transfer-backend ascend --watchdog-timeout 9000 --context-length 8192 \ + --enable-dp-lm-head --dtype bfloat16 --tokenizer-worker-num 4 --load-balance-method round_robin + NODE_RANK=$i + break + fi +done + +``` + +```bash Command +python -m sglang_router.launch_router \ + --pd-disaggregation \ + --policy cache_aware \ + --prefill http://PIP:8000 8995 \ + --decode http://DIP:8001 \ + --host 127.0.0.1 \ + --port 6688 \ + --mini-lb +``` + +#### Benchmark + +We tested it based on the `RANDOM` dataset. + +```shell Command +python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7239 --max-concurrency 410 --random-input-len 3500 --random-output-len 1500 --num-prompts 1640 --random-range-ratio 1 --request-rate 8 +``` + +### Qwen3-Coder-480B-A35B-Instruct 3_5K-1_5K 50ms on A3 16 Cards Mixed Mode + +Model: Qwen3-Coder-480B-A35B-Instruct + +Hardware: Atlas 800I A3 16Card + +DeployMode: PD Mixed + +Dataset: random + +Input Output Length: 3.5K+1.5K + +TPOT: 50ms + +#### Model Deployment + +```bash Command +echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor +sysctl -w vm.swappiness=0 +sysctl -w kernel.numa_balancing=0 +sysctl -w kernel.sched_migration_cost_ns=50000 + +unset https_proxy +unset http_proxy +unset HTTPS_PROXY +unset HTTP_PROXY +unset ASCEND_LAUNCH_BLOCKING +source /usr/local/Ascend/ascend-toolkit/set_env.sh +source /usr/local/Ascend/nnal/atb/set_env.sh +source /usr/local/Ascend/ascend-toolkit/latest/opp/vendors/customize/bin/set_env.bash + +export SGLANG_SET_CPU_AFFINITY=1 +export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True +export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=72 +export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600 + +MODEL_PATH=xxx + +LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` +LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` + +echo "${LOCAL_HOST1}" +echo "${LOCAL_HOST2}" + +export HCCL_BUFFSIZE=1800 +export HCCL_SOCKET_IFNAME=xxx +export GLOO_SOCKET_IFNAME=xxx +export HCCL_OP_EXPANSION_MODE="AIV" + +MIX_IP=('IP1' 'IP2') + +for i in "${!MIX_IP[@]}"; +do + if [[ "$LOCAL_HOST1" == "${MIX_IP[$i]}" || "$LOCAL_HOST2" == "${MIX_IP[$i]}" ]]; + then + echo "${MIX_IP[$i]}" + + python -m sglang.launch_server --model-path $MODEL_PATH \ + --host 127.0.0.1 --port 7439 --trust-remote-code --nnodes 2 --node-rank $i \ + --dist-init-addr 141.61.133.128:5000 \ + --attention-backend ascend --device npu --quantization modelslim \ + --max-running-requests 288 --context-length 8192 --dtype bfloat16 \ + --chunked-prefill-size 114688 --max-prefill-tokens 458880 \ + --disable-radix-cache --moe-a2a-backend deepep --deepep-mode auto \ + --tp 32 --dp-size 4 --enable-dp-attention --enable-dp-lm-head --mem-fraction-static 0.7 --cuda-graph-bs 56 64 72 + NODE_RANK=$i + break + fi +done +``` + +#### Benchmark + +We tested it based on the `RANDOM` dataset. + +```shell Command +python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7439 --max-concurrency 288 --random-input-len 3500 --random-output-len 1500 --num-prompts 1152 --random-range-ratio 1 --request-rate 20 +``` + +### Qwen3-Coder-480B-A35B-Instruct 3_5K-1_5K 50ms on A3 8 Cards Mixed Mode + +Model: Qwen3-Coder-480B-A35B-Instruct + +Hardware: Atlas 800I A3 8Card + +DeployMode: PD Mixed + +Dataset: random + +Input Output Length: 3.5K+1.5K + +TPOT: 50ms + +#### Model Deployment + +```bash Command +echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor +sysctl -w vm.swappiness=0 +sysctl -w kernel.numa_balancing=0 +sysctl -w kernel.sched_migration_cost_ns=50000 + +unset https_proxy +unset http_proxy +unset HTTPS_PROXY +unset HTTP_PROXY +unset ASCEND_LAUNCH_BLOCKING +source /usr/local/Ascend/ascend-toolkit/set_env.sh +source /usr/local/Ascend/nnal/atb/set_env.sh +source /usr/local/Ascend/ascend-toolkit/latest/opp/vendors/customize/bin/set_env.bash +export PATH=/usr/local/Ascend/8.5.0/compiler/bishengir/bin:$PATH + +export SGLANG_SET_CPU_AFFINITY=1 +export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True +export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600 + +MODEL_PATH=xxx + +LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` +LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` + +echo "${LOCAL_HOST1}" +echo "${LOCAL_HOST2}" + +export HCCL_BUFFSIZE=2100 +export HCCL_SOCKET_IFNAME=lo +export GLOO_SOCKET_IFNAME=lo +export HCCL_OP_EXPANSION_MODE="AIV" + +python -m sglang.launch_server --model-path $MODEL_PATH \ +--host 127.0.0.1 --port 7439 --trust-remote-code --nnodes 1 --node-rank 0 \ +--attention-backend ascend --device npu --quantization modelslim \ +--max-running-requests 80 --context-length 8192 --dtype bfloat16 \ +--chunked-prefill-size 28672 --max-prefill-tokens 458880 \ +--disable-radix-cache --moe-a2a-backend deepep --deepep-mode auto --enable-dp-attention --enable-dp-lm-head \ +--tp 16 --dp-size 4 --mem-fraction-static 0.7 --cuda-graph-bs 16 20 24 +``` + +#### Benchmark + +We tested it based on the `RANDOM` dataset. + +```shell Command +python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7439 --max-concurrency 80 --random-input-len 3500 --random-output-len 1500 --num-prompts 320 --random-range-ratio 1 +``` + +### Qwen3-Next-80B-A3B-Instruct 3_5K-1_5K 50ms on A3 2 Cards Mixed Mode + +Model: Qwen3-Next-80B-A3B-Instruct + +Hardware: Atlas 800I A3 2Card + +DeployMode: PD Mixed + +Dataset: random + +Input Output Length: 3.5K+1.5K + +TPOT: 50ms + +#### Model Deployment + +```shell Command +export cann_path=/usr/local/Ascend/ascend-toolkit/latest +source /usr/local/Ascend/driver/bin/setenv.bash +source ${cann_path}/../set_env.sh +source ${cann_path}/../../nnal/atb/set_env.sh +source ${cann_path}/opp/vendors/customize/bin/set_env.bash +export ASCEND_HOME_PATH=${cann_path} +source /usr/local/Ascend/8.5.0/bisheng_toolkit/set_env.sh + +echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor +sysctl -w vm.swappiness=0 +sysctl -w kernel.numa_balancing=0 +sysctl -w kernel.sched_migration_cost_ns=50000 + +export SGLANG_SET_CPU_AFFINITY=1 +export LD_LIBRARY_PATH=/usr/local/Ascend/cann-9.0.0/opp/vendors/custom_transformer/op_api/lib:${LD_LIBRARY_PATH} + +export STREAMS_PER_DEVICE=32 +export HCCL_SOCKET_IFNAME=lo +export GLOO_SOCKET_IFNAME=lo + +export HCCL_OP_EXPANSION_MODE=AIV +export HCCL_ALGO="level0:NA;level1:ring" + +export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 +export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=330 +export ASCEND_USE_FIA=1 +export SGLANG_NPU_USE_MULTI_STREAM=0 +export SGLANG_WARMUP_TIMEOUT=3600 +export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 +export FORCE_DRAFT_MODEL_NON_QUANT=1 + +ZBAL_HCCL_OP="allreduce,_allgather_base,allgather,broadcast,scatter,reduce_scatter,_reduce_scatter_base,alltoall_base" +export HCCL_BUFFSIZE=64 +export SGLANG_ZBAL_LOCAL_MEM_SIZE=59648 +export SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 +export SGLANG_ZBAL_BOOTSTRAP_URL="tcp://127.0.0.1:24669" + +export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True +export ZBAL_NPU_ALLOC_CONF=use_vmm_for_static_memory:True +export ZBAL_ENABLE_GRAPH=1 +MODEL_PATH=/home/weights/Qwen3-Next-80B-A3B-Instruct-W8A8 + +python3 -m sglang.launch_server --model-path ${MODEL_PATH} \ + --page-size 128 \ + --tp-size 4 \ + --trust-remote-code \ + --attention-backend ascend \ + --device npu \ + --watchdog-timeout 9000 \ + --host 127.0.0.1 --port 6699 \ + --mem-fraction-static 0.75 \ + --disable-radix-cache --max-prefill-tokens 14080 --context-length 26384 \ + --chunked-prefill-size -1 --max-running-requests 300 \ + --mamba-ssm-dtype bfloat16 \ + --quantization modelslim \ + --speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 --speculative-draft-model-quantization unquant \ + --speculative-draft-model-path /home/weights/Qwen3-Next-80B-A3B-Instruct \ + --dp-size 2 --enable-dp-attention --enable-dp-lm-head \ + --moe-a2a-backend deepep --deepep-mode auto \ + --cuda-graph-bs 1 2 3 4 5 6 7 8 10 12 14 16 18 20 22 24 26 28 30 32 40 44 48 52 56 60 64 72 80 88 96 104 112 120 128 136 144 150 +``` + +#### Benchmark + +We tested it based on the `RANDOM` dataset. + +```shell Command +python3 -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 6699 --max-concurrency 300 --random-output-len 1536 --random-input-len 3584 --num-prompts 300 --random-range-ratio 1 +``` + +### Qwen3-32B 6K-1_5K 18ms on A2 8 Cards Mixed Mode + +Model: Qwen3-32B + +Hardware: Atlas 800I A2 8Card + +DeployMode: PD Mixed + +Dataset: random + +Input Output Length: 6K+1.5K + +TPOT: 18ms + +#### Model Deployment + +```shell Command +echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor +sysctl -w vm.swappiness=0 +sysctl -w kernel.numa_balancing=0 +sysctl -w kernel.sched_migration_cost_ns=50000 + +export SGLANG_SET_CPU_AFFINITY=1 +unset https_proxy +unset http_proxy +unset HTTPS_PROXY +unset HTTP_PROXY +unset ASCEND_LAUNCH_BLOCKING +source /usr/local/Ascend/ascend-toolkit/set_env.sh +source /usr/local/Ascend/nnal/atb/set_env.sh +source /usr/local/Ascend/ascend-toolkit/latest/opp/vendors/customize/bin/set_env.bash +export PATH=/usr/local/Ascend/8.5.0/compiler/bishengir/bin:$PATH + +MODEL_PATH=xxx + +export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600 + +LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` +LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` + +echo "${LOCAL_HOST1}" +echo "${LOCAL_HOST2}" + +export HCCL_BUFFSIZE=400 +export HCCL_SOCKET_IFNAME=lo +export GLOO_SOCKET_IFNAME=lo +export HCCL_OP_EXPANSION_MODE="AIV" +export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 + +python -m sglang.launch_server --model-path $MODEL_PATH \ + --host 127.0.0.1 --port 7439 --trust-remote-code --nnodes 1 --node-rank 0 \ + --attention-backend ascend --device npu --quantization modelslim \ + --max-running-requests 32 \ + --disable-radix-cache \ + --chunked-prefill-size 24576 --max-prefill-tokens 65536 \ + --speculative-algorithm EAGLE3 --speculative-draft-model-path xxx \ + --speculative-num-steps 4 --speculative-eagle-topk 1 --speculative-num-draft-tokens 5 \ + --tp-size 8 --mem-fraction-static 0.72 --cuda-graph-bs 8 16 24 32 --dtype bfloat16 +``` + +#### Benchmark + +We tested it based on the `RANDOM` dataset. + +```shell Command +python3 -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7439 --max-concurrency 32 --random-output-len 1500 --random-input-len 6000 --num-prompts 32 --random-range-ratio 1 +``` + +### Qwen3-32B 4K-1_5K 11ms on A2 8 Cards Mixed Mode + +Model: Qwen3-32B + +Hardware: Atlas 800I A2 8Card + +DeployMode: PD Mixed + +Dataset: random + +Input Output Length: 4K+1.5K + +TPOT: 11ms + +#### Model Deployment + +```shell Command +echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor +sysctl -w vm.swappiness=0 +sysctl -w kernel.numa_balancing=0 +sysctl -w kernel.sched_migration_cost_ns=50000 + +export SGLANG_SET_CPU_AFFINITY=1 + +unset https_proxy +unset http_proxy +unset HTTPS_PROXY +unset HTTP_PROXY +unset ASCEND_LAUNCH_BLOCKING +source /usr/local/Ascend/ascend-toolkit/set_env.sh +source /usr/local/Ascend/nnal/atb/set_env.sh +source /usr/local/Ascend/ascend-toolkit/latest/opp/vendors/customize/bin/set_env.bash +export PATH=/usr/local/Ascend/8.5.0/compiler/bishengir/bin:$PATH + +export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True + +MODEL_PATH=xxx + +export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600 + +LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` +LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` + +echo "${LOCAL_HOST1}" +echo "${LOCAL_HOST2}" + +export HCCL_BUFFSIZE=400 +export HCCL_SOCKET_IFNAME=lo +export GLOO_SOCKET_IFNAME=lo +export HCCL_OP_EXPANSION_MODE="AIV" +export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 + +python -m sglang.launch_server --model-path $MODEL_PATH \ + --host 127.0.0.1 --port 7339 --trust-remote-code --nnodes 1 --node-rank 0 \ + --attention-backend ascend --device npu \ + --max-running-requests 32 \ + --disable-radix-cache \ + --speculative-draft-model-quantization unquant \ + --speculative-algorithm EAGLE3 --speculative-draft-model-path xxx \ + --speculative-num-steps 4 --speculative-eagle-topk 1 --speculative-num-draft-tokens 5 \ + --chunked-prefill-size -1 --max-prefill-tokens 65536 \ + --tp-size 8 --mem-fraction-static 0.72 --cuda-graph-bs 1 4 6 12 18 24 30 32 --dtype bfloat16 +``` + +#### Benchmark + +We tested it based on the `RANDOM` dataset. + +```shell Command +python3 -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7339 --random-range-ratio 1 --max-concurrency 1 --random-output-len 1500 --random-input-len 4096 --num-prompts 4 +``` + +### Qwen3-32B 1K-0_3K 12ms on A3 2 Cards Mixed Mode + +Model: Qwen3-32B + +Hardware: Atlas 800I A3 2Card + +DeployMode: PD Mixed + +Dataset: random + +Input Output Length: 1K+0.3K + +TPOT: 12ms + +#### Model Deployment + +```bash Command +echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor +sysctl -w vm.swappiness=0 +sysctl -w kernel.numa_balancing=0 +sysctl -w kernel.sched_migration_cost_ns=50000 + +export SGLANG_SET_CPU_AFFINITY=1 + +unset https_proxy +unset http_proxy +unset HTTPS_PROXY +unset HTTP_PROXY +unset ASCEND_LAUNCH_BLOCKING +source /usr/local/Ascend/ascend-toolkit/set_env.sh +source /usr/local/Ascend/nnal/atb/set_env.sh +source /usr/local/Ascend/ascend-toolkit/latest/opp/vendors/customize/bin/set_env.bash +export PATH=/usr/local/Ascend/8.5.0/compiler/bishengir/bin:$PATH + +export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True + +MODEL_PATH=xxx + +export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600 + +LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` +LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` + +echo "${LOCAL_HOST1}" +echo "${LOCAL_HOST2}" + +export HCCL_SOCKET_IFNAME=lo +export GLOO_SOCKET_IFNAME=lo +export HCCL_OP_EXPANSION_MODE="AIV" +export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 + +python -m sglang.launch_server --model-path $MODEL_PATH \ + --host 127.0.0.1 --port 7339 --trust-remote-code --nnodes 1 --node-rank 0 \ + --attention-backend ascend --device npu --quantization modelslim \ + --max-running-requests 16 \ + --disable-radix-cache \ + --speculative-draft-model-quantization unquant \ + --speculative-algorithm EAGLE3 --speculative-draft-model-path xxx \ + --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 \ + --chunked-prefill-size -1 --max-prefill-tokens 16384 \ + --tp-size 4 --mem-fraction-static 0.843 --cuda-graph-bs 1 4 8 16 --dtype bfloat16 +``` + +#### Benchmark + +We tested it based on the `RANDOM` dataset. + +```bash Command +python3 -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7339 --random-range-ratio 1 --max-concurrency 16 --random-output-len 300 --random-input-len 1024 --num-prompts 16 +``` + +### Qwen3-32B 6K-1_5K 17ms on A3 2 Cards Mixed Mode + +Model: Qwen3-32B + +Hardware: Atlas 800I A3 2Card + +DeployMode: PD Mixed + +Dataset: random + +Input Output Length: 6K+1.5K + +TPOT: 17ms + +#### Model Deployment + +```bash Command +echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor +sysctl -w vm.swappiness=0 +sysctl -w kernel.numa_balancing=0 +sysctl -w kernel.sched_migration_cost_ns=50000 + +export SGLANG_SET_CPU_AFFINITY=1 + +unset https_proxy +unset http_proxy +unset HTTPS_PROXY +unset HTTP_PROXY +unset ASCEND_LAUNCH_BLOCKING +source /usr/local/Ascend/ascend-toolkit/set_env.sh +source /usr/local/Ascend/nnal/atb/set_env.sh +source /usr/local/Ascend/ascend-toolkit/latest/opp/vendors/customize/bin/set_env.bash +export PATH=/usr/local/Ascend/8.5.0/compiler/bishengir/bin:$PATH + +export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True + +MODEL_PATH=xxx + +export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600 + +LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` +LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` + +echo "${LOCAL_HOST1}" +echo "${LOCAL_HOST2}" + +export HCCL_SOCKET_IFNAME=lo +export GLOO_SOCKET_IFNAME=lo +export HCCL_OP_EXPANSION_MODE="AIV" +export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 + +python -m sglang.launch_server --model-path $MODEL_PATH \ + --host 127.0.0.1 --port 7339 --trust-remote-code --nnodes 1 --node-rank 0 \ + --attention-backend ascend --device npu --quantization modelslim \ + --max-running-requests 16 \ + --disable-radix-cache \ + --speculative-draft-model-quantization unquant \ + --speculative-algorithm EAGLE3 --speculative-draft-model-path xxx \ + --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 \ + --chunked-prefill-size -1 --max-prefill-tokens 16384 \ + --tp-size 4 --mem-fraction-static 0.843 --cuda-graph-bs 1 4 10 15 16 --dtype bfloat16 +``` + +#### Benchmark + +We tested it based on the `RANDOM` dataset. + +```bash Command +python3 -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7339 --random-range-ratio 1 --max-concurrency 16 --random-output-len 1500 --random-input-len 6144 --num-prompts 16 +``` + +### Qwen3-8B 1K-0_3K 7ms on A3 1 Cards Mixed Mode + +Model: Qwen3-8B + +Hardware: Atlas 800I A3 1Card + +DeployMode: PD Mixed + +Dataset: random + +Input Output Length: 1K+0.3K + +TPOT: 7ms + +#### Model Deployment + +```bash Command +echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor +sysctl -w vm.swappiness=0 +sysctl -w kernel.numa_balancing=0 +sysctl -w kernel.sched_migration_cost_ns=50000 + +export SGLANG_SET_CPU_AFFINITY=1 + +unset https_proxy +unset http_proxy +unset HTTPS_PROXY +unset HTTP_PROXY +unset ASCEND_LAUNCH_BLOCKING +source /usr/local/Ascend/ascend-toolkit/set_env.sh +source /usr/local/Ascend/nnal/atb/set_env.sh +source /usr/local/Ascend/ascend-toolkit/latest/opp/vendors/customize/bin/set_env.bash +export PATH=/usr/local/Ascend/8.5.0/compiler/bishengir/bin:$PATH + +export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True + +MODEL_PATH=xxx + +export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600 + +LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` +LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` + +echo "${LOCAL_HOST1}" +echo "${LOCAL_HOST2}" + +export HCCL_SOCKET_IFNAME=lo +export GLOO_SOCKET_IFNAME=lo +export HCCL_OP_EXPANSION_MODE="AIV" +export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 + +python -m sglang.launch_server --model-path $MODEL_PATH \ + --host 127.0.0.1 --port 7339 --trust-remote-code --nnodes 1 --node-rank 0 \ + --attention-backend ascend --device npu --quantization modelslim \ + --max-running-requests 16 \ + --disable-radix-cache \ + --speculative-draft-model-quantization unquant \ + --speculative-algorithm EAGLE3 --speculative-draft-model-path xxx \ + --speculative-num-steps 4 --speculative-eagle-topk 1 --speculative-num-draft-tokens 5 \ + --chunked-prefill-size -1 --max-prefill-tokens 16384 \ + --tp-size 2 --mem-fraction-static 0.894 --cuda-graph-bs 1 2 4 6 9 10 15 16 --dtype bfloat16 +``` + +#### Benchmark + +We tested it based on the `RANDOM` dataset. + +```bash Command +python3 -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7339 --random-range-ratio 1 --max-concurrency 16 --random-output-len 300 --random-input-len 1024 --num-prompts 16 +``` + +### Qwen3-8B 6K-1_5K 12ms on A3 1 Cards Mixed Mode + +Model: Qwen3-8B + +Hardware: Atlas 800I A3 1Card + +DeployMode: PD Mixed + +Dataset: random + +Input Output Length: 6K+1.5K + +TPOT: 12ms + +#### Model Deployment + +```bash Command +echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor +sysctl -w vm.swappiness=0 +sysctl -w kernel.numa_balancing=0 +sysctl -w kernel.sched_migration_cost_ns=50000 + +export SGLANG_SET_CPU_AFFINITY=1 + +unset https_proxy +unset http_proxy +unset HTTPS_PROXY +unset HTTP_PROXY +unset ASCEND_LAUNCH_BLOCKING +source /usr/local/Ascend/ascend-toolkit/set_env.sh +source /usr/local/Ascend/nnal/atb/set_env.sh +source /usr/local/Ascend/ascend-toolkit/latest/opp/vendors/customize/bin/set_env.bash +export PATH=/usr/local/Ascend/8.5.0/compiler/bishengir/bin:$PATH + +export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True + +MODEL_PATH=xxx + +export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600 + +LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` +LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` + +echo "${LOCAL_HOST1}" +echo "${LOCAL_HOST2}" + +export HCCL_SOCKET_IFNAME=lo +export GLOO_SOCKET_IFNAME=lo +export HCCL_OP_EXPANSION_MODE="AIV" +export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 + +python -m sglang.launch_server --model-path $MODEL_PATH \ + --host 127.0.0.1 --port 7339 --trust-remote-code --nnodes 1 --node-rank 0 \ + --attention-backend ascend --device npu --quantization modelslim \ + --max-running-requests 16 \ + --disable-radix-cache \ + --speculative-draft-model-quantization unquant \ + --speculative-algorithm EAGLE3 --speculative-draft-model-path xxx \ + --speculative-num-steps 4 --speculative-eagle-topk 1 --speculative-num-draft-tokens 5 \ + --chunked-prefill-size -1 --max-prefill-tokens 16384 \ + --tp-size 2 --mem-fraction-static 0.894 --cuda-graph-bs 1 5 15 16 --dtype bfloat16 +``` + +#### Benchmark + +We tested it based on the `RANDOM` dataset. + +```bash Command +python3 -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7339 --random-range-ratio 1 --max-concurrency 16 --random-output-len 1500 --random-input-len 6144 --num-prompts 16 +``` + +### Qwen3-32B 3_5K-1_5K 50ms on A2 8 Cards Mixed Mode + +Model: Qwen3-32B + +Hardware: Atlas 800I A2 8Card + +DeployMode: PD Mixed + +Dataset: random + +Input Output Length: 3.5K+1.5K + +TPOT: 50ms + +#### Model Deployment + +```shell Command +echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor +sysctl -w vm.swappiness=0 +sysctl -w kernel.numa_balancing=0 +sysctl -w kernel.sched_migration_cost_ns=50000 + +export SGLANG_SET_CPU_AFFINITY=1 + +unset https_proxy +unset http_proxy +unset HTTPS_PROXY +unset HTTP_PROXY +unset ASCEND_LAUNCH_BLOCKING +source /usr/local/Ascend/ascend-toolkit/set_env.sh +source /usr/local/Ascend/nnal/atb/set_env.sh +source /usr/local/Ascend/ascend-toolkit/latest/opp/vendors/customize/bin/set_env.bash +export PATH=/usr/local/Ascend/8.5.0/compiler/bishengir/bin:$PATH + +MODEL_PATH=xxx + +export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600 + +LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` +LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` + +echo "${LOCAL_HOST1}" +echo "${LOCAL_HOST2}" + +export HCCL_BUFFSIZE=400 +export HCCL_SOCKET_IFNAME=lo +export GLOO_SOCKET_IFNAME=lo +export HCCL_OP_EXPANSION_MODE="AIV" +export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 + +python -m sglang.launch_server --model-path $MODEL_PATH \ + --host 127.0.0.1 --port 7239 --trust-remote-code --nnodes 1 --node-rank 0 \ + --attention-backend ascend --device npu \ + --max-running-requests 78 \ + --disable-radix-cache --speculative-draft-model-quantization unquant \ + --chunked-prefill-size -1 --max-prefill-tokens 65536 \ + --speculative-algorithm EAGLE3 --speculative-draft-model-path xxx \ + --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 \ + --tp-size 4 --mem-fraction-static 0.72 --cuda-graph-bs 1 4 8 16 32 64 68 72 78 --dtype bfloat16 --base-gpu-id 4 +``` + +#### Benchmark + +We tested it based on the `RANDOM` dataset. + +```shell Command +python3 -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7239 --max-concurrency 78 --random-output-len 1500 --random-input-len 3500 --num-prompts 312 --random-range-ratio 1 +``` + +### Qwen3-32B 2K-2K 50ms on A2 8 Cards Mixed Mode + +Model: Qwen3-32B + +Hardware: Atlas 800I A2 8Card + +DeployMode: PD Mixed + +Dataset: random + +Input Output Length: 2K+2K + +TPOT: 50ms + +#### Model Deployment + +```shell Command +echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor +sysctl -w vm.swappiness=0 +sysctl -w kernel.numa_balancing=0 +sysctl -w kernel.sched_migration_cost_ns=50000 + +export SGLANG_SET_CPU_AFFINITY=1 +unset https_proxy +unset http_proxy +unset HTTPS_PROXY +unset HTTP_PROXY +unset ASCEND_LAUNCH_BLOCKING +source /usr/local/Ascend/ascend-toolkit/set_env.sh +source /usr/local/Ascend/nnal/atb/set_env.sh +source /usr/local/Ascend/ascend-toolkit/latest/opp/vendors/customize/bin/set_env.bash +export PATH=/usr/local/Ascend/8.5.0/compiler/bishengir/bin:$PATH + +MODEL_PATH=xxx + +export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600 + +LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` +LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` + +echo "${LOCAL_HOST1}" +echo "${LOCAL_HOST2}" + +export HCCL_BUFFSIZE=400 +export HCCL_SOCKET_IFNAME=lo +export GLOO_SOCKET_IFNAME=lo +export HCCL_OP_EXPANSION_MODE="AIV" +export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 + +python -m sglang.launch_server --model-path $MODEL_PATH \ + --host 127.0.0.1 --port 7239 --trust-remote-code --nnodes 1 --node-rank 0 \ + --attention-backend ascend --device npu \ + --max-running-requests 120 \ + --disable-radix-cache \ + --speculative-algorithm EAGLE3 --speculative-draft-model-path xxx \ + --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 --speculative-draft-model-quantization unquant \ + --chunked-prefill-size -1 --max-prefill-tokens 49152 --base-gpu-id 4 \ + --tp-size 4 --mem-fraction-static 0.7 --cuda-graph-bs 54 60 66 72 78 84 90 108 114 120 --dtype bfloat16 +``` + +#### Benchmark + +We tested it based on the `RANDOM` dataset. + +```shell Command +python3 -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7239 --max-concurrency 120 --random-output-len 2000 --random-input-len 2000 --num-prompts 120 --random-range-ratio 1 +``` + +### Qwen3-30B-A3B 6K-1_5K 10ms on A3 1 Cards Mixed Mode + +Model: Qwen3-30B-A3B + +Hardware: Atlas 800I A3 1Card + +DeployMode: PD Mixed + +Dataset: random + +Input Output Length: 6K+1.5K + +TPOT: 10ms + +#### Model Deployment + +```bash Command +export SGLANG_SET_CPU_AFFINITY=1 + +unset https_proxy +unset http_proxy +unset HTTPS_PROXY +unset HTTP_PROXY +unset ASCEND_LAUNCH_BLOCKING + +export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True + +MODEL_PATH=xxx + +export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600 + +LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` +LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` + +echo "${LOCAL_HOST1}" +echo "${LOCAL_HOST2}" + +export HCCL_BUFFSIZE=400 +export HCCL_SOCKET_IFNAME=lo +export GLOO_SOCKET_IFNAME=lo +export HCCL_OP_EXPANSION_MODE="AIV" +export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 + +python -m sglang.launch_server --model-path $MODEL_PATH \ + --host 127.0.0.1 --port 7339 --trust-remote-code --nnodes 1 --node-rank 0 \ + --attention-backend ascend --device npu \ + --max-running-requests 16 \ + --disable-radix-cache \ + --speculative-draft-model-quantization unquant \ + --speculative-algorithm EAGLE3 --speculative-draft-model-path xxx \ + --speculative-num-steps 4 --speculative-eagle-topk 1 --speculative-num-draft-tokens 5 \ + --chunked-prefill-size -1 --max-prefill-tokens 35000 \ + --tp-size 2 --mem-fraction-static 0.6 --cuda-graph-bs 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 --dtype bfloat16 +``` + +#### Benchmark + +We tested it based on the `RANDOM` dataset. + +```bash Command +python3 -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7339 --random-range-ratio 1 --max-concurrency 16 --random-output-len 1500 --random-input-len 6144 --num-prompts 16 +``` + +### Qwen3-30B-A3B 1K-0_3K 7ms on A3 1 Cards Mixed Mode + +Model: Qwen3-30B-A3B + +Hardware: Atlas 800I A3 1Card + +DeployMode: PD Mixed + +Dataset: random + +Input Output Length: 1K+0.3K + +TPOT: 7ms + +#### Model Deployment + +```bash Command +export SGLANG_SET_CPU_AFFINITY=1 + +unset https_proxy +unset http_proxy +unset HTTPS_PROXY +unset HTTP_PROXY +unset ASCEND_LAUNCH_BLOCKING + +export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True + +MODEL_PATH=xxx + +export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600 + +LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` +LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` + +echo "${LOCAL_HOST1}" +echo "${LOCAL_HOST2}" + +export HCCL_BUFFSIZE=400 +export HCCL_SOCKET_IFNAME=lo +export GLOO_SOCKET_IFNAME=lo +export HCCL_OP_EXPANSION_MODE="AIV" +export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 + +python -m sglang.launch_server --model-path $MODEL_PATH \ + --host 127.0.0.1 --port 7339 --trust-remote-code --nnodes 1 --node-rank 0 \ + --attention-backend ascend --device npu \ + --max-running-requests 8 \ + --disable-radix-cache \ + --speculative-draft-model-quantization unquant \ + --speculative-algorithm EAGLE3 --speculative-draft-model-path xxx \ + --speculative-num-steps 4 --speculative-eagle-topk 1 --speculative-num-draft-tokens 5 \ + --chunked-prefill-size -1 --max-prefill-tokens 35000 \ + --tp-size 2 --mem-fraction-static 0.7 --cuda-graph-bs 1 2 3 4 5 6 7 8 --dtype bfloat16 +``` + +#### Benchmark + +We tested it based on the `RANDOM` dataset. + +```bash Command +python3 -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7339 --random-range-ratio 1 --max-concurrency 8 --random-output-len 300 --random-input-len 1024 --num-prompts 8 +``` + +### Qwen3-Next 1K-0_3K 14_21ms on A3 2 Cards Mixed Mode + +Model: Qwen3-Next-80B-A3B-Instruct + +Hardware: Atlas 800I A3 2Card + +DeployMode: PD Mixed + +Dataset: random + +Input Output Length: 1K+0.3K + +TPOT: 14.21ms + +#### Model Deployment + +```bash Command +echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor +sysctl -w vm.swappiness=0 +sysctl -w kernel.numa_balancing=0 +sysctl -w kernel.sched_migration_cost_ns=50000 + +export SGLANG_SET_CPU_AFFINITY=1 + +unset https_proxy +unset http_proxy +unset HTTPS_PROXY +unset HTTP_PROXY +unset ASCEND_LAUNCH_BLOCKING +source /usr/local/Ascend/ascend-toolkit/set_env.sh +source /usr/local/Ascend/nnal/atb/set_env.sh +source /usr/local/Ascend/ascend-toolkit/latest/opp/vendors/customize/bin/set_env.bash +export PATH=/usr/local/Ascend/8.5.0/compiler/bishengir/bin:$PATH + +export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True +export STREAMS_PER_DEVICE=32 +export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=330 +export DEEPEP_NORMAL_LONG_SEQ_ROUND=5 +export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=3000 +export DEEPEP_NORMAL_COMBINE_ENABLE_LONG_SEQ=1 + +export ASCEND_USE_FIA=1 +export SGLANG_NPU_USE_MULTI_STREAM=1 + +export SGLANG_WARMUP_TIMEOUT=3600 +export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 +export FORCE_DRAFT_MODEL_NON_QUANT=1 + +MODEL_PATH=xxx + +export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600 + +LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` +LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` + +echo "${LOCAL_HOST1}" +echo "${LOCAL_HOST2}" + +export HCCL_BUFFSIZE=2000 +export HCCL_SOCKET_IFNAME=lo +export GLOO_SOCKET_IFNAME=lo +export HCCL_OP_EXPANSION_MODE="AIV" +export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 + +python3 -m sglang.launch_server --model-path ${MODEL_PATH} \ + --page-size 128 \ + --tp-size 4 \ + --trust-remote-code \ + --attention-backend ascend \ + --device npu \ + --watchdog-timeout 9000 \ + --host 127.0.0.1 --port 6699 \ + --mem-fraction-static 0.75 \ + --disable-radix-cache --max-prefill-tokens 14080 --context-length 26384 \ + --speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 --speculative-draft-model-quantization unquant \ + --chunked-prefill-size -1 --max-running-requests 312 \ + --cuda-graph-bs 2 4 16 32 48 64 80 96 128 140 156 \ + --mamba-ssm-dtype bfloat16 \ + --base-gpu-id 0 \ + --speculative-draft-model-path /home/weights/Qwen3-Next-80B-A3B-Instruct \ + --moe-a2a-backend deepep --deepep-mode auto \ +``` + +#### Benchmark + +We tested it based on the `RANDOM` dataset. + +```bash Command +python3 -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 6699 --random-range-ratio 1 --max-concurrency 16 --random-output-len 300 --random-input-len 1024 --num-prompts 16 +``` + +### Qwen3-Next 6K-1_5K 15_62ms on A3 2 Cards Mixed Mode + +Model: Qwen3-Next-80B-A3B-Instruct + +Hardware: Atlas 800I A3 2Card + +DeployMode: PD Mixed + +Dataset: random + +Input Output Length: 6K+1.5K + +TPOT: 15.62ms + +#### Model Deployment + +```bash Command +echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor +sysctl -w vm.swappiness=0 +sysctl -w kernel.numa_balancing=0 +sysctl -w kernel.sched_migration_cost_ns=50000 + +export SGLANG_SET_CPU_AFFINITY=1 + +unset https_proxy +unset http_proxy +unset HTTPS_PROXY +unset HTTP_PROXY +unset ASCEND_LAUNCH_BLOCKING +source /usr/local/Ascend/ascend-toolkit/set_env.sh +source /usr/local/Ascend/nnal/atb/set_env.sh +source /usr/local/Ascend/ascend-toolkit/latest/opp/vendors/customize/bin/set_env.bash +export PATH=/usr/local/Ascend/8.5.0/compiler/bishengir/bin:$PATH + +export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True +export STREAMS_PER_DEVICE=32 +export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=330 +export DEEPEP_NORMAL_LONG_SEQ_ROUND=5 +export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=3000 +export DEEPEP_NORMAL_COMBINE_ENABLE_LONG_SEQ=1 + +export ASCEND_USE_FIA=1 +export SGLANG_NPU_USE_MULTI_STREAM=1 + +export SGLANG_WARMUP_TIMEOUT=3600 +export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 +export FORCE_DRAFT_MODEL_NON_QUANT=1 + +MODEL_PATH=xxx + +export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600 + +LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` +LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` + +echo "${LOCAL_HOST1}" +echo "${LOCAL_HOST2}" + +export HCCL_BUFFSIZE=2000 +export HCCL_SOCKET_IFNAME=lo +export GLOO_SOCKET_IFNAME=lo +export HCCL_OP_EXPANSION_MODE="AIV" +export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 + +python3 -m sglang.launch_server --model-path ${MODEL_PATH} \ + --page-size 128 \ + --tp-size 4 \ + --trust-remote-code \ + --attention-backend ascend \ + --device npu \ + --watchdog-timeout 9000 \ + --host 127.0.0.1 --port 6699 \ + --mem-fraction-static 0.75 \ + --disable-radix-cache --max-prefill-tokens 14080 --context-length 26384 \ + --speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 --speculative-draft-model-quantization unquant \ + --chunked-prefill-size -1 --max-running-requests 312 \ + --cuda-graph-bs 2 4 16 32 48 64 80 96 128 140 156 \ + --mamba-ssm-dtype bfloat16 \ + --base-gpu-id 0 \ + --speculative-draft-model-path /home/weights/Qwen3-Next-80B-A3B-Instruct \ + --moe-a2a-backend deepep --deepep-mode auto \ +``` + +#### Benchmark + +We tested it based on the `RANDOM` dataset. + +```bash Command +python3 -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 6699 --random-range-ratio 1 --max-concurrency 16 --random-output-len 1500 --random-input-len 6144 --num-prompts 16 +``` + +### Qwen3-14B 3_5K-1_5K 9ms on A3 1 Cards Mixed Mode + +Model: Qwen3-14B + +Hardware: Atlas 800I A3 1Card + +DeployMode: PD Mixed + +Dataset: random + +Input Output Length: 3.5K+1.5K + +TPOT: 9ms + +#### Model Deployment + +```bash Command +echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor +sysctl -w vm.swappiness=0 +sysctl -w kernel.numa_balancing=0 +sysctl -w kernel.sched_migration_cost_ns=50000 + +unset https_proxy +unset http_proxy +unset HTTPS_PROXY +unset HTTP_PROXY +unset ASCEND_LAUNCH_BLOCKING +source /usr/local/Ascend/ascend-toolkit/set_env.sh +source /usr/local/Ascend/nnal/atb/set_env.sh +source /usr/local/Ascend/ascend-toolkit/latest/opp/vendors/customize/bin/set_env.bash +export PATH=/usr/local/Ascend/8.5.0/compiler/bishengir/bin:$PATH + +MODEL_PATH=xxx + +export SGLANG_SET_CPU_AFFINITY=1 +export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600 +export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True +export HCCL_OP_EXPANSION_MODE="AIV" +export STREAMS_PER_DEVICE=32 +export HCCL_SOCKET_IFNAME=lo +export GLOO_SOCKET_IFNAME=lo +export ASCEND_USE_FIA=0 +export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 + +LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` +LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` + +echo "${LOCAL_HOST1}" +echo "${LOCAL_HOST2}" + +python -m sglang.launch_server --model-path $MODEL_PATH \ + --host 127.0.0.1 --port 7239 --trust-remote-code --nnodes 1 --node-rank 0 \ + --attention-backend ascend --device npu \ + --disable-radix-cache --mem-fraction-static 0.8 \ + --tp-size 1 --dp-size 1 \ + --sampling-backend ascend --max-running-requests 8 \ + --served-model-name Qwen3-14B \ + --chunked-prefill-size -1 \ + --cuda-graph-bs 8 \ + --dtype bfloat16 \ + --speculative-draft-model-quantization unquant \ + --speculative-algorithm EAGLE3 --speculative-draft-model-path xxx \ + --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 \ + --schedule-conservativeness 0.01 +``` + +#### Benchmark + +We tested it based on the `RANDOM` dataset. + +```bash Command +python3 -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7239 --max-concurrency 1 --random-output-len 1500 --random-input-len 3500 --num-prompts 8 --random-range-ratio 1 +``` + +### Qwen3-14B 3_5K-1_5K 50ms on A3 1 Cards Mixed Mode + +Model: Qwen3-14B + +Hardware: Atlas 800I A3 1Card + +DeployMode: PD Mixed + +Dataset: random + +Input Output Length: 3.5K+1.5K + +TPOT: 50ms + +#### Model Deployment + +```bash Command +echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor +sysctl -w vm.swappiness=0 +sysctl -w kernel.numa_balancing=0 +sysctl -w kernel.sched_migration_cost_ns=50000 + +unset https_proxy +unset http_proxy +unset HTTPS_PROXY +unset HTTP_PROXY +unset ASCEND_LAUNCH_BLOCKING +source /usr/local/Ascend/ascend-toolkit/set_env.sh +source /usr/local/Ascend/nnal/atb/set_env.sh +source /usr/local/Ascend/ascend-toolkit/latest/opp/vendors/customize/bin/set_env.bash +export PATH=/usr/local/Ascend/8.5.0/compiler/bishengir/bin:$PATH + +MODEL_PATH=xxx + +export SGLANG_SET_CPU_AFFINITY=1 +export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600 +export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True +export HCCL_OP_EXPANSION_MODE="AIV" +export STREAMS_PER_DEVICE=32 +export HCCL_SOCKET_IFNAME=lo +export GLOO_SOCKET_IFNAME=lo +export ASCEND_USE_FIA=0 +export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 +export SGLANG_SCHEDULER_DECREASE_PREFILL_IDLE=1 +export SGLANG_PREFILL_DELAYER_MAX_DELAY_PASSES=200 + +LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` +LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` + +echo "${LOCAL_HOST1}" +echo "${LOCAL_HOST2}" + +python -m sglang.launch_server --model-path $MODEL_PATH \ + --host 127.0.0.1 --port 7239 --trust-remote-code --nnodes 1 --node-rank 0 \ + --attention-backend ascend --device npu \ + --disable-radix-cache --mem-fraction-static 0.89 \ + --tp-size 1 --dp-size 2 \ + --sampling-backend ascend --max-running-requests 144 \ + --max-prefill-tokens 12288 \ + --served-model-name Qwen3-14B \ + --chunked-prefill-size -1 \ + --cuda-graph-bs 8 16 32 44 48 50 52 \ + --dtype bfloat16 \ + --speculative-draft-model-quantization unquant \ + --speculative-algorithm EAGLE3 --speculative-draft-model-path xxx \ + --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 \ + --schedule-conservativeness 0.01 +``` + +#### Benchmark + +We tested it based on the `RANDOM` dataset. + +```bash Command +python3 -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7239 --max-concurrency 144 --random-output-len 1500 --random-input-len 3500 --num-prompts 576 --random-range-ratio 1 +``` + +### Qwen3-8B 3_5K-1_5K 50ms on A3 1 Cards Mixed Mode + +Model: Qwen3-8B + +Hardware: Atlas 800I A3 1Card + +DeployMode: PD Mixed + +Dataset: random + +Input Output Length: 3.5K+1.5K + +TPOT: 50ms + +#### Model Deployment + +```bash Command +echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor +sysctl -w vm.swappiness=0 +sysctl -w kernel.numa_balancing=0 +sysctl -w kernel.sched_migration_cost_ns=50000 + +unset https_proxy +unset http_proxy +unset HTTPS_PROXY +unset HTTP_PROXY +unset ASCEND_LAUNCH_BLOCKING +source /usr/local/Ascend/ascend-toolkit/set_env.sh +source /usr/local/Ascend/nnal/atb/set_env.sh +source /usr/local/Ascend/ascend-toolkit/latest/opp/vendors/customize/bin/set_env.bash +export PATH=/usr/local/Ascend/8.5.0/compiler/bishengir/bin:$PATH + +MODEL_PATH=xxx + +export SGLANG_SET_CPU_AFFINITY=1 +export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600 +export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True +export HCCL_OP_EXPANSION_MODE="AIV" +export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 +export SGLANG_SCHEDULER_DECREASE_PREFILL_IDLE=1 +export SGLANG_PREFILL_DELAYER_MAX_DELAY_PASSES=50 +export HCCL_SOCKET_IFNAME=lo +export GLOO_SOCKET_IFNAME=lo + +LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` +LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` + +echo "${LOCAL_HOST1}" +echo "${LOCAL_HOST2}" + +python -m sglang.launch_server --model-path $MODEL_PATH \ + --host 127.0.0.1 --port 7239 --trust-remote-code --nnodes 1 --node-rank 0 \ + --attention-backend ascend --device npu \ + --disable-radix-cache --mem-fraction-static 0.9 \ + --tp-size 1 \ + --max-running-requests 70 \ + --max-prefill-tokens 16384 \ + --served-model-name Qwen3-8B \ + --chunked-prefill-size 16384 \ + --cuda-graph-bs 8 12 24 36 48 51 55 60 63 64 66 68 70 \ + --dtype bfloat16 \ + --speculative-draft-model-quantization unquant \ + --speculative-algorithm EAGLE3 --speculative-draft-model-path xxx \ + --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 +``` + +#### Benchmark + +We tested it based on the `RANDOM` dataset. + +```bash Command +python3 -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7239 --max-concurrency 64 --random-output-len 1500 --random-input-len 3500 --num-prompts 256 --random-range-ratio 1 +``` + +### Qwen3-8B 3_5K-1_5K 5ms on A3 1 Cards Mixed Mode + +Model: Qwen3-8B + +Hardware: Atlas 800I A3 1Card + +DeployMode: PD Mixed + +Dataset: random + +Input Output Length: 3.5K+1.5K + +TPOT: 5ms + +#### Model Deployment + +```bash Command +echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor +sysctl -w vm.swappiness=0 +sysctl -w kernel.numa_balancing=0 +sysctl -w kernel.sched_migration_cost_ns=50000 + +unset https_proxy +unset http_proxy +unset HTTPS_PROXY +unset HTTP_PROXY +unset ASCEND_LAUNCH_BLOCKING +source /usr/local/Ascend/ascend-toolkit/set_env.sh +source /usr/local/Ascend/nnal/atb/set_env.sh +source /usr/local/Ascend/ascend-toolkit/latest/opp/vendors/customize/bin/set_env.bash +export PATH=/usr/local/Ascend/8.5.0/compiler/bishengir/bin:$PATH + +MODEL_PATH=xxx + +export SGLANG_SET_CPU_AFFINITY=1 +export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600 +export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True +export HCCL_OP_EXPANSION_MODE="AIV" +export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 +export HCCL_SOCKET_IFNAME=lo +export GLOO_SOCKET_IFNAME=lo + +LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` +LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` + +echo "${LOCAL_HOST1}" +echo "${LOCAL_HOST2}" + +python -m sglang.launch_server --model-path $MODEL_PATH \ + --host 127.0.0.1 --port 7239 --trust-remote-code --nnodes 1 --node-rank 0 \ + --attention-backend ascend --device npu \ + --disable-radix-cache --mem-fraction-static 0.894 \ + --tp-size 2 \ + --max-running-requests 1 \ + --max-prefill-tokens 16384 \ + --served-model-name Qwen3-8B \ + --chunked-prefill-size -1 \ + --cuda-graph-bs 1 \ + --dtype bfloat16 \ + --speculative-draft-model-quantization unquant \ + --speculative-algorithm EAGLE3 --speculative-draft-model-path xxx \ + --speculative-num-steps 4 --speculative-eagle-topk 1 --speculative-num-draft-tokens 5 +``` + +#### Benchmark + +We tested it based on the `RANDOM` dataset. + +```bash Command +python3 -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 7239 --max-concurrency 1 --random-output-len 1500 --random-input-len 3500 --num-prompts 4 --random-range-ratio 1 +``` + +### Qwen3-Next 3_5K-1_5K 20ms on A3 1 Cards Mixed Mode + +Model: Qwen3-Next-80B-A3B-Instruct + +Hardware: Atlas 800I A3 1Card + +DeployMode: PD Mixed + +Dataset: random + +Input Output Length: 3.5K+1.5K + +TPOT: 20ms + +#### Model Deployment + +```bash Command +echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor +sysctl -w vm.swappiness=0 +sysctl -w kernel.numa_balancing=0 +sysctl -w kernel.sched_migration_cost_ns=50000 + +unset https_proxy +unset http_proxy +unset HTTPS_PROXY +unset HTTP_PROXY +unset ASCEND_LAUNCH_BLOCKING +source /usr/local/Ascend/ascend-toolkit/set_env.sh +source /usr/local/Ascend/nnal/atb/set_env.sh +source /usr/local/Ascend/ascend-toolkit/latest/opp/vendors/customize/bin/set_env.bash +export PATH=/usr/local/Ascend/8.5.0/compiler/bishengir/bin:$PATH + +export SGLANG_SET_CPU_AFFINITY=1 +export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True +export STREAMS_PER_DEVICE=32 +export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=400 +export DEEPEP_NORMAL_LONG_SEQ_ROUND=10 +export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=2048 +export HCCL_OP_EXPANSION_MODE="AIV" +export TASK_QUEUE_ENABLE=1 +export ASCEND_USE_FIA=1 +export SGLANG_NPU_USE_MULTI_STREAM=0 +export SGLANG_WARMUP_TIMEOUT=3600 +export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 +export FORCE_DRAFT_MODEL_NON_QUANT=1 +export HCCL_BUFFSIZE=2000 +export ZBCCL_LOCAL_MEM_SIZE=60416 +export SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 + +export ZBCCL_BOOTSTRAP_URL=tcp://127.0.0.1:24669 +export ZBCCL_NPU_ALLOC_CONF=use_vmm_for_static_memory:True +export ZBCCL_ENABLE_GRAPH=1 + +export HCCL_SOCKET_IFNAME=lo +export GLOO_SOCKET_IFNAME=lo + +MODEL_PATH=xxx + +LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` +LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` + +echo "${LOCAL_HOST1}" +echo "${LOCAL_HOST2}" + +python3 -m sglang.launch_server --model-path ${MODEL_PATH} \ + --page-size 128 \ + --tp-size 2 \ + --trust-remote-code \ + --attention-backend ascend \ + --device npu \ + --watchdog-timeout 9000 \ + --host 127.0.0.1 --port 6699 \ + --mem-fraction-static 0.85 \ + --disable-radix-cache --max-prefill-tokens 28672 --context-length 26384 --max-total-tokens 122304 \ + --speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 --speculative-draft-model-quantization unquant \ + --chunked-prefill-size -1 --max-running-requests 2 \ + --cuda-graph-bs 2 \ + --mamba-ssm-dtype bfloat16 \ + --speculative-draft-model-path /path/to/Qwen3-Next-80B-A3B-Instruct +``` + +#### Benchmark + +We tested it based on the `RANDOM` dataset. + +```bash Command +python3 -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 6699 --random-range-ratio 1 --max-concurrency 1 --random-output-len 1500 --random-input-len 3500 --num-prompts 1 +``` + +### Qwen3.5-27B 3_5K-1_5K 20ms on A3 2 Cards Mixed Mode + +Model: Eco-Tech/Qwen3.5-27B-w8a8-mtp + +Hardware: Atlas 800I A3 2Card + +DeployMode: PD Mixed + +Dataset: random + +Input Output Length: 3.5K+1.5K + +TPOT: 20ms + +#### Model Deployment + +```bash Command +# high performance cpu +echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor +sysctl -w vm.swappiness=0 +sysctl -w kernel.numa_balancing=0 +sysctl -w kernel.sched_migration_cost_ns=50000 +# bind cpu +export SGLANG_SET_CPU_AFFINITY=1 + +unset https_proxy +unset http_proxy +unset HTTPS_PROXY +unset HTTP_PROXY + +# on-demand set device +export ASCEND_RT_VISIBLE_DEVICES=0,1,2,3 + +export ASCEND_LAUNCH_BLOCKING=1 +export STREAMS_PER_DEVICE=32 +export HCCL_BUFFSIZE=3000 +export HCCL_OP_EXPANSION_MODE="AIV" +export HCCL_SOCKET_IFNAME=lo +export GLOO_SOCKET_IFNAME=lo +export SGLANG_NPU_PROFILING=0 +export SGLANG_DISAGGEGATION_WAITING_TIMEOUT=3600 +export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=0 + +MODEL_PATH=xxx + +python -m sglang.launch_server --model-path ${MODEL_PATH} \ + --attention-backend ascend \ + --host 127.0.0.1 --port 6699 \ + --device npu \ + --tp-size 4\ + --trust-remote-code \ + --watchdog-timeout 9000 \ + --chunked-prefill-size -1 \ + --max-prefill-tokens 186000 \ + --enable-prefill-delayer \ + --prefill-delayer-max-delay-passes 200 \ + --disable-radix-cache \ + --mem-fraction-static 0.94 \ + --max-total-tokens 700000 \ + --max-running-requests 38 \ + --max-mamba-cache-size 200 \ + --quantization modelslim \ + --dtype bfloat16 \ + --mamba-ssm-dtype bfloat16 \ + --enable-multimodal \ + --mm-attention-backend ascend_attn \ + --cuda-graph-bs 1 2 4 8 12 18 24 32 34 36 38 \ + --speculative-algorithm NEXTN \ + --speculative-num-steps 3 \ + --speculative-eagle-topk 1 \ + --speculative-num-draft-tokens 4 +``` + +#### Benchmark + +We tested it based on the `RANDOM` dataset. + +```bash Command +python3 -m sglang.bench_serving --backend sglang --host 127.0.0.1 --port 6699 --dataset-name random --max-concurrency 38 --num-prompts 152 --random-range-ratio 1 --random-output-len 1500 --random-input-len 3500 +``` + +### Qwen3.5-27B 16K-1K 20ms on A3 1 Cards Mixed Mode + +Model: Eco-Tech/Qwen3.5-27B-w8a8-mtp + +Hardware: Atlas 800I A3 1Card + +DeployMode: PD Mixed + +Dataset: random + +Input Output Length: 16K+1K + +TPOT: 20ms + +#### Model Deployment + +```bash Command +# high performance cpu +echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor +sysctl -w vm.swappiness=0 +sysctl -w kernel.numa_balancing=0 +sysctl -w kernel.sched_migration_cost_ns=50000 +# bind cpu +export SGLANG_SET_CPU_AFFINITY=1 + +unset https_proxy +unset http_proxy +unset HTTPS_PROXY +unset HTTP_PROXY +# cann +source /usr/local/Ascend/ascend-toolkit/set_env.sh +source /usr/local/Ascend/nnal/atb/set_env.sh + +export STREAMS_PER_DEVICE=32 +export HCCL_OP_EXPANSION_MODE=AIV +export HCCL_SOCKET_IFNAME=lo +export GLOO_SOCKET_IFNAME=lo +export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=0 +export SGLANG_SCHEDULER_DECREASE_PREFILL_IDLE=1 +export SGLANG_PREFILL_DELAYER_MAX_DELAY_PASSES=100 + +# on-demand set device +export ASCEND_RT_VISIBLE_DEVICES=8,9 + +MODEL_PATH=xxx + +sglang serve --model-path ${MODEL_PATH} \ + --attention-backend ascend \ + --device npu \ + --tp-size 2 --nnodes 1 --node-rank 0 \ + --chunked-prefill-size -1 --max-prefill-tokens 65000 \ + --disable-radix-cache \ + --trust-remote-code \ + --host 127.0.0.1 --max-running-requests 32 --max-mamba-cache-size 32 \ + --mem-fraction-static 0.85 \ + --port 8001 \ + --cuda-graph-bs 2 3 4 5 6 \ + --enable-multimodal \ + --quantization modelslim \ + --mm-attention-backend ascend_attn \ + --dtype bfloat16 --mamba-ssm-dtype bfloat16 --max-total-tokens 310000 \ + --speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 +``` + +#### Benchmark + +We tested it based on the `RANDOM` dataset. + +```bash Command +python3 -m sglang.bench_serving --backend sglang --host 127.0.0.1 --port 8001 --dataset-name random --max-concurrency 32 --num-prompts 128 --random-range-ratio 1 --random-output-len 1000 --random-input-len 16000 +``` + +### Qwen3.5-27B 64K-1K 20ms on A3 1 Cards Mixed Mode + +Model: Eco-Tech/Qwen3.5-27B-w8a8-mtp + +Hardware: Atlas 800I A3 1Card + +DeployMode: PD Mixed + +Dataset: random + +Input Output Length: 64K+1K + +TPOT: 20ms + +#### Model Deployment + +```bash Command +# high performance cpu +echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor +sysctl -w vm.swappiness=0 +sysctl -w kernel.numa_balancing=0 +sysctl -w kernel.sched_migration_cost_ns=50000 +# bind cpu +export SGLANG_SET_CPU_AFFINITY=1 + +unset https_proxy +unset http_proxy +unset HTTPS_PROXY +unset HTTP_PROXY + +# cann +source /usr/local/Ascend/ascend-toolkit/set_env.sh +source /usr/local/Ascend/nnal/atb/set_env.sh + +export STREAMS_PER_DEVICE=32 +export HCCL_OP_EXPANSION_MODE=AIV +export HCCL_SOCKET_IFNAME=lo +export GLOO_SOCKET_IFNAME=lo +export SGLANG_NPU_PROFILING=1 +export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=0 +# on-demand set device +export ASCEND_RT_VISIBLE_DEVICES=4,5 + +MODEL_PATH=xxx + +python -m sglang.launch_server --model-path ${MODEL_PATH} \ + --attention-backend ascend \ + --device npu \ + --tp-size 2 --nnodes 1 --node-rank 0 \ + --chunked-prefill-size -1 --max-prefill-tokens 130000 \ + --disable-radix-cache \ + --trust-remote-code \ + --host 127.0.0.1 --max-running-requests 32 --max-mamba-cache-size 18 \ + --mem-fraction-static 0.5 \ + --port 8004 \ + --cuda-graph-bs 2 3 4 \ + --enable-multimodal \ + --quantization modelslim \ + --mm-attention-backend ascend_attn \ + --dtype bfloat16 --mamba-ssm-dtype bfloat16 --max-total-tokens 280000 \ + --speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 +``` + +#### Benchmark + +We tested it based on the `RANDOM` dataset. + +```bash Command +python3 -m sglang.bench_serving --backend sglang --host 127.0.0.1 --port 8004 --dataset-name random --max-concurrency 9 --num-prompts 36 --random-range-ratio 1 --random-output-len 1000 --random-input-len 64000 +``` + +### Qwen3.5-27B 3_5K-1_5K 50ms on A3 1 Cards Mixed Mode + +Model: Eco-Tech/Qwen3.5-27B-w8a8-mtp + +Hardware: Atlas 800I A3 1Card + +DeployMode: PD Mixed + +Dataset: random + +Input Output Length: 3.5K+1.5K + +TPOT: 50ms + +#### Model Deployment + +```bash Command +# high performance cpu +echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor +sysctl -w vm.swappiness=0 +sysctl -w kernel.numa_balancing=0 +sysctl -w kernel.sched_migration_cost_ns=50000 +# bind cpu +export SGLANG_SET_CPU_AFFINITY=1 + +unset https_proxy +unset http_proxy +unset HTTPS_PROXY +unset HTTP_PROXY + +# cann +source /usr/local/Ascend/ascend-toolkit/set_env.sh +source /usr/local/Ascend/nnal/atb/set_env.sh + +export STREAMS_PER_DEVICE=32 +export HCCL_OP_EXPANSION_MODE=AIV +export HCCL_SOCKET_IFNAME=lo +export GLOO_SOCKET_IFNAME=lo +export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=0 +export SGLANG_SCHEDULER_DECREASE_PREFILL_IDLE=1 +export SGLANG_PREFILL_DELAYER_MAX_DELAY_PASSES=100 + +MODEL_PATH=xxx + +python -m sglang.launch_server --model-path ${MODEL_PATH} \ + --attention-backend ascend \ + --device npu \ + --tp-size 2 --nnodes 1 --node-rank 0 \ + --chunked-prefill-size -1 --max-prefill-tokens 60000 \ + --disable-radix-cache \ + --trust-remote-code \ + --host 127.0.0.1 --max-running-requests 48 --max-mamba-cache-size 60 \ + --mem-fraction-static 0.7 \ + --port 8000 \ + --cuda-graph-bs 2 8 16 32 48 \ + --enable-multimodal \ + --quantization modelslim \ + --mm-attention-backend ascend_attn \ + --dtype bfloat16 --mamba-ssm-dtype bfloat16 \ + --speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 +``` + +#### Benchmark + +We tested it based on the `RANDOM` dataset. + +```bash Command +python3 -m sglang.bench_serving --backend sglang --host 127.0.0.1 --port 8000 --dataset-name random --max-concurrency 48 --num-prompts 192 --random-range-ratio 1 --random-output-len 1500 --random-input-len 3500 +``` + +### Qwen3.5-27B 16K-1K 50ms on A3 2 Cards Mixed Mode + +Model: Eco-Tech/Qwen3.5-27B-w8a8-mtp + +Hardware: Atlas 800I A3 2Card + +DeployMode: PD Mixed + +Dataset: random + +Input Output Length: 16K+1K + +TPOT: 50ms + +#### Model Deployment + +```bash Command +# high performance cpu +echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor +sysctl -w vm.swappiness=0 +sysctl -w kernel.numa_balancing=0 +sysctl -w kernel.sched_migration_cost_ns=50000 +# bind cpu +export SGLANG_SET_CPU_AFFINITY=1 + +unset https_proxy +unset http_proxy +unset HTTPS_PROXY +unset HTTP_PROXY + +# cann +source /usr/local/Ascend/ascend-toolkit/set_env.sh +source /usr/local/Ascend/nnal/atb/set_env.sh + +export STREAMS_PER_DEVICE=32 +export HCCL_OP_EXPANSION_MODE=AIV +export HCCL_SOCKET_IFNAME=lo +export GLOO_SOCKET_IFNAME=lo +export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 +export SGLANG_SCHEDULER_DECREASE_PREFILL_IDLE=1 +export SGLANG_PREFILL_DELAYER_MAX_DELAY_PASSES=30 +# on-demand set device +export ASCEND_RT_VISIBLE_DEVICES=0,1,2,3 + +MODEL_PATH=xxx + +python3 -m sglang.launch_server --model-path ${MODEL_PATH} \ + --attention-backend ascend \ + --device npu \ + --tp-size 4 --nnodes 1 --node-rank 0 \ + --chunked-prefill-size -1 --max-prefill-tokens 50000 \ + --disable-radix-cache \ + --trust-remote-code \ + --host 127.0.0.1 --max-running-requests 28 --max-mamba-cache-size 50 \ + --mem-fraction-static 0.7 \ + --port 8001 \ + --cuda-graph-bs 2 8 12 16 20 24 28\ + --enable-multimodal \ + --quantization modelslim \ + --mm-attention-backend ascend_attn \ + --dtype bfloat16 --mamba-ssm-dtype bfloat16 \ + --speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 +``` + +#### Benchmark + +We tested it based on the `RANDOM` dataset. + +```bash Command +python3 -m sglang.bench_serving --backend sglang --host 127.0.0.1 --port 8001 --dataset-name random --max-concurrency 28 --num-prompts 152 --random-range-ratio 1 --random-output-len 1000 --random-input-len 16000 +``` + +### Qwen3.5-27B 64K-1K 50ms on A3 2 Cards Mixed Mode + +Model: Eco-Tech/Qwen3.5-27B-w8a8-mtp + +Hardware: Atlas 800I A3 2Card + +DeployMode: PD Mixed + +Dataset: random + +Input Output Length: 64K+1K + +TPOT: 50ms + +#### Model Deployment + +```bash Command +# high performance cpu +echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor +sysctl -w vm.swappiness=0 +sysctl -w kernel.numa_balancing=0 +sysctl -w kernel.sched_migration_cost_ns=50000 +# bind cpu +export SGLANG_SET_CPU_AFFINITY=1 + +unset https_proxy +unset http_proxy +unset HTTPS_PROXY +unset HTTP_PROXY + +# cann +source /usr/local/Ascend/ascend-toolkit/set_env.sh +source /usr/local/Ascend/nnal/atb/set_env.sh + +export STREAMS_PER_DEVICE=32 +export HCCL_OP_EXPANSION_MODE=AIV +export HCCL_SOCKET_IFNAME=lo +export GLOO_SOCKET_IFNAME=lo +export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=0 +export SGLANG_SCHEDULER_DECREASE_PREFILL_IDLE=1 +export SGLANG_PREFILL_DELAYER_MAX_DELAY_PASSES=100 +# on-demand set device +export ASCEND_RT_VISIBLE_DEVICES=4,5,6,7 + +MODEL_PATH=xxx + +python3 -m sglang.launch_server --model-path ${MODEL_PATH} \ + --attention-backend ascend \ + --device npu \ + --tp-size 4 --nnodes 1 --node-rank 0 \ + --chunked-prefill-size -1 --max-prefill-tokens 200000 \ + --disable-radix-cache \ + --trust-remote-code \ + --host 127.0.0.1 --max-running-requests 32 --max-mamba-cache-size 22 \ + --mem-fraction-static 0.5 \ + --port 9000 \ + --cuda-graph-bs 2 4 8 11 12 13 \ + --enable-multimodal \ + --quantization modelslim \ + --mm-attention-backend ascend_attn \ + --dtype bfloat16 --mamba-ssm-dtype bfloat16 --max-total-tokens 850000 \ + --speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 +``` + +#### Benchmark + +We tested it based on the `RANDOM` dataset. + +```bash Command +python3 -m sglang.bench_serving --backend sglang --host 127.0.0.1 --port 9000 --dataset-name random --max-concurrency 9 --num-prompts 36 --random-range-ratio 1 --random-output-len 1000 --random-input-len 64000 +``` + +### Qwen3.5-397B-A17B 3_5K-1_5K 22ms on A3 8 Cards Mixed Mode + +Model: Qwen3.5-397B-A17B + +Hardware: Atlas 800I A3 8Card + +DeployMode: PD Mixed + +Dataset: random + +Input Output Length: 3.5K+1.5K + +TPOT: 22ms + +#### Model Deployment + +```bash Command +# high performance cpu +echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor +sysctl -w vm.swappiness=0 +sysctl -w kernel.numa_balancing=0 +sysctl -w kernel.sched_migration_cost_ns=50000 +# bind cpu +export SGLANG_SET_CPU_AFFINITY=1 + +unset https_proxy +unset http_proxy +unset HTTPS_PROXY +unset HTTP_PROXY +unset ASCEND_LAUNCH_BLOCKING +source /usr/local/Ascend/ascend-toolkit/set_env.sh +source /usr/local/Ascend/nnal/atb/set_env.sh +source /usr/local/Ascend/ascend-toolkit/latest/opp/vendors/customize/bin/set_env.bash +export PATH=/usr/local/Ascend/8.5.0/compiler/bishengir/bin:$PATH + +export ASCEND_USE_FIA=1 +export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=128 +export HCCL_BUFFSIZE=3000 +export DEEPEP_NORMAL_LONG_SEQ_ROUND=32 +export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=3584 +export STREAMS_PER_DEVICE=32 +export HCCL_OP_EXPANSION_MODE=AIV +export HCCL_SOCKET_IFNAME=lo +export GLOO_SOCKET_IFNAME=lo +export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 +export SGLANG_NPU_USE_MULTI_STREAM=1 + +export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True +export SGLANG_ZBAL_LOCAL_MEM_SIZE=58624 +export SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 +export SGLANG_ZBAL_BOOTSTRAP_URL="tcp://127.0.0.1:24669" +export ZBAL_NPU_ALLOC_CONF=use_vmm_for_static_memory:True +export ZBAL_ENABLE_GRAPH=1 + +MODEL_PATH=xxx + +python3 -m sglang.launch_server \ +--model-path $MODEL_PATH \ +--attention-backend ascend \ +--device npu \ +--tp-size 16 \ +--chunked-prefill-size -1 --max-prefill-tokens 35000 \ +--disable-radix-cache \ +--trust-remote-code \ +--host 127.0.0.1 --max-running-requests 160 \ +--mem-fraction-static 0.8 \ +--port 6699 \ +--cuda-graph-bs 2 4 6 8 10 12 14 16 18 20 \ +--quantization modelslim \ +--enable-multimodal --moe-a2a-backend deepep --deepep-mode auto \ +--mm-attention-backend ascend_attn \ +--dtype bfloat16 --mamba-ssm-dtype bfloat16 --max-total-tokens 128000 \ +--speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 \ +--speculative-draft-model-quantization unquant \ +--dp-size 8 --enable-dp-attention --enable-dp-lm-head \ +--enable-prefill-delayer --prefill-delayer-max-delay-passes 100 +``` + +#### Benchmark + +We tested it based on the `RANDOM` dataset. + +```bash Command +python3 -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 6699 --random-range-ratio 1 --max-concurrency 120 --random-output-len 1500 --random-input-len 3500 --num-prompts 480 +``` + +### Qwen3.5-397B-A17B 3_5K-1_5K 50ms on A3 8 Cards Mixed Mode + +Model: Qwen3.5-397B-A17B + +Hardware: Atlas 800I A3 8Card + +DeployMode: PD Mixed + +Dataset: random + +Input Output Length: 3.5K+1.5K + +TPOT: 50ms + +#### Model Deployment + +```bash Command +# high performance cpu +echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor +sysctl -w vm.swappiness=0 +sysctl -w kernel.numa_balancing=0 +sysctl -w kernel.sched_migration_cost_ns=50000 +# bind cpu +export SGLANG_SET_CPU_AFFINITY=1 + +unset https_proxy +unset http_proxy +unset HTTPS_PROXY +unset HTTP_PROXY +unset ASCEND_LAUNCH_BLOCKING +source /usr/local/Ascend/ascend-toolkit/set_env.sh +source /usr/local/Ascend/nnal/atb/set_env.sh +source /usr/local/Ascend/ascend-toolkit/latest/opp/vendors/customize/bin/set_env.bash +export PATH=/usr/local/Ascend/8.5.0/compiler/bishengir/bin:$PATH + +export ASCEND_USE_FIA=1 +export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=128 +export HCCL_BUFFSIZE=3000 +export DEEPEP_NORMAL_LONG_SEQ_ROUND=32 +export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=3584 +export STREAMS_PER_DEVICE=32 +export HCCL_OP_EXPANSION_MODE=AIV +export HCCL_SOCKET_IFNAME=lo +export GLOO_SOCKET_IFNAME=lo +export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 +export SGLANG_NPU_USE_MULTI_STREAM=1 + +export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True +export SGLANG_ZBAL_LOCAL_MEM_SIZE=59648 +export SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 +export SGLANG_ZBAL_BOOTSTRAP_URL="tcp://127.0.0.1:24669" +export ZBAL_NPU_ALLOC_CONF=use_vmm_for_static_memory:True +export ZBAL_ENABLE_GRAPH=1 + +MODEL_PATH=xxx + +python3 -m sglang.launch_server \ +--model-path $MODEL_PATH \ +--attention-backend ascend \ +--device npu \ +--tp-size 16 \ +--chunked-prefill-size -1 --max-prefill-tokens 17500 \ +--disable-radix-cache \ +--trust-remote-code \ +--host 127.0.0.1 --max-running-requests 432 \ +--mem-fraction-static 0.75 \ +--port 6699 \ +--cuda-graph-bs 2 4 6 8 12 16 20 24 28 32 36 40 44 48 52 56 \ +--quantization modelslim \ +--enable-multimodal --moe-a2a-backend deepep --deepep-mode auto \ +--mm-attention-backend ascend_attn \ +--dtype bfloat16 --mamba-ssm-dtype bfloat16 --max-total-tokens 280000 \ +--dp-size 8 --enable-dp-attention --enable-dp-lm-head \ +--speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 \ +--speculative-draft-model-quantization unquant \ +--enable-prefill-delayer --prefill-delayer-max-delay-passes 200 +``` + +#### Benchmark + +We tested it based on the `RANDOM` dataset. + +```bash Command +python3 -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 6699 --random-range-ratio 1 --max-concurrency 352 --random-output-len 1500 --random-input-len 3500 --num-prompts 1408 +``` + +### MiniMax-M2.5 3_5K-1_5K Low Latency on A3 8 Cards Mixed Mode + +Model: MiniMax-M2.5 + +Hardware: Atlas 800I A3 8Card + +DeployMode: PD Mixed + +Dataset: random + +Input Output Length: 3.5K+1.5K + +#### Model Deployment + +```bash Command +echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor +sysctl -w vm.swappiness=0 +sysctl -w kernel.numa_balancing=0 +sysctl -w kernel.sched_migration_cost_ns=50000 + +unset https_proxy +unset http_proxy +unset HTTPS_PROXY +unset HTTP_PROXY +unset ASCEND_LAUNCH_BLOCKING + +source /usr/local/Ascend/ascend-toolkit/set_env.sh +source /usr/local/Ascend/nnal/atb/set_env.sh +export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True +export STREAMS_PER_DEVICE=32 +export HCCL_SOCKET_IFNAME=lo +export GLOO_SOCKET_IFNAME=lo + +export HCCL_OP_EXPANSION_MODE=AIV +export TASK_QUEUE_ENABLE=1 + +export HCCL_BUFFSIZE=1500 +export ASCEND_USE_FIA=1 +export SGLANG_SET_CPU_AFFINITY=1 +export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 +export SGLANG_NPU_USE_MULTI_STREAM=1 +export SGLANG_NPU_FUSED_MOE_MODE=2 +export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=224000 + +MODEL_PATH=/path/to/MiniMax-M2.5-w8a8-QuaRot +EAGLE_MODEL_PATH=/path/to/MiniMax-M2.5-eagle-model +export PYTHONPATH=${EAGLE_MODEL_PATH}:$PYTHONPATH +export SGLANG_EXTERNAL_MODEL_PACKAGE=custom_eagle3 + +python -m sglang.launch_server \ + --model-path $MODEL_PATH \ + --host 127.0.0.1 \ + --port 32001 \ + --tp-size 16 \ + --dp-size 16 \ + --enable-dp-attention \ + --mem-fraction-static 0.75 \ + --max-running-requests 128 \ + --disable-radix-cache \ + --chunked-prefill-size -1 --max-prefill-token 8192 \ + --cuda-graph-bs 2 4 6 8 \ + --moe-a2a-backend ascend_fuseep --deepep-mode auto --quantization modelslim \ + --speculative-algorithm EAGLE3 \ + --speculative-draft-model-path $EAGLE_MODEL_PATH \ + --speculative-num-steps 3 \ + --speculative-eagle-topk 1 \ + --speculative-num-draft-tokens 4 \ + --speculative-draft-model-quantization unquant \ + --dtype bfloat16 \ + --tokenizer-worker-num 2 \ + --prefill-delayer-max-delay-passes 500 \ + --enable-prefill-delayer +``` + +#### Benchmark + +We tested it based on the `RANDOM` dataset. + +```shell Command +python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 32001 --random-input-len 3500 --random-output-len 1500 --num-prompts 320 --random-range-ratio 1 --max-concurrency 80 +``` +### MiniMax-M2.5 128K-1K Low Latency on A3 8 Cards Mixed Mode + +Model: MiniMax-M2.5 + +Hardware: Atlas 800I A3 8Card + +DeployMode: PD Mixed + +Dataset: random + +Input Output Length: 128K+1K + +#### Model Deployment + +```bash Command +echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor +sysctl -w vm.swappiness=0 +sysctl -w kernel.numa_balancing=0 +sysctl -w kernel.sched_migration_cost_ns=50000 + +unset https_proxy +unset http_proxy +unset HTTPS_PROXY +unset HTTP_PROXY +unset ASCEND_LAUNCH_BLOCKING + +source /usr/local/Ascend/ascend-toolkit/set_env.sh +source /usr/local/Ascend/nnal/atb/set_env.sh +export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True +export STREAMS_PER_DEVICE=32 +export HCCL_SOCKET_IFNAME=lo +export GLOO_SOCKET_IFNAME=lo + +export TASK_QUEUE_ENABLE=1 + +export ASCEND_USE_FIA=1 +export HCCL_BUFFSIZE=1600 +export SGLANG_SET_CPU_AFFINITY=1 +export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 +export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=640 +export DEEPEP_NORMAL_LONG_SEQ_ROUND=64 +export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=2048 +export DEEPEP_NORMAL_COMBINE_ENABLE_LONG_SEQ=1 +export SGLANG_NPU_FUSED_MOE_MODE=2 +export SGLANG_NPU_DEEPEP_USE_FUSED_MOE_DECODE=1 +export SGLANG_NPU_FUSEEP_DECODE_ONLY=1 + +MODEL_PATH=/path/to/MiniMax-M2.5-w8a8-QuaRot +EAGLE_MODEL_PATH=/path/to/MiniMax-M2.5-eagle-model +export PYTHONPATH=${EAGLE_MODEL_PATH}:$PYTHONPATH +export SGLANG_EXTERNAL_MODEL_PACKAGE=custom_eagle3 + +python -m sglang.launch_server \ + --model-path $MODEL_PATH \ + --host 127.0.0.1 \ + --port 32000 \ + --tp-size 16 \ + --dp-size 2 \ + --enable-dp-attention \ + --prefill-delayer-max-delay-passes 100 \ + --enable-prefill-delayer \ + --mem-fraction-static 0.65 \ + --max-running-requests 8 \ + --chunked-prefill-size -1 --max-prefill-token 130000 \ + --cuda-graph-bs 1 2 4 \ + --moe-a2a-backend ascend_fuseep --deepep-mode auto --quantization modelslim \ + --speculative-algorithm EAGLE3 \ + --speculative-draft-model-path $EAGLE_MODEL_PATH \ + --speculative-num-steps 3 \ + --speculative-eagle-topk 1 \ + --speculative-num-draft-tokens 4 \ + --speculative-draft-model-quantization unquant \ + --dtype bfloat16 \ + --trust-remote-code \ + --tokenizer-worker-num 8 +``` + +#### Benchmark + +We tested it based on the `RANDOM` dataset. + +```shell Command +python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 32000 --random-input-len 131072 --random-output-len 1024 --num-prompts 8 --random-range-ratio 1 --max-concurrency 2 +``` +### MiniMax-M2.5 3_5K-1_5K High Throughput on A3 8 Cards Mixed Mode + +Model: MiniMax-M2.5 + +Hardware: Atlas 800I A3 8Card + +DeployMode: PD Mixed + +Dataset: random + +Input Output Length: 3.5K+1.5K + +#### Model Deployment + +```bash Command +echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor +sysctl -w vm.swappiness=0 +sysctl -w kernel.numa_balancing=0 +sysctl -w kernel.sched_migration_cost_ns=50000 + +unset https_proxy +unset http_proxy +unset HTTPS_PROXY +unset HTTP_PROXY +unset ASCEND_LAUNCH_BLOCKING + +source /usr/local/Ascend/ascend-toolkit/set_env.sh +source /usr/local/Ascend/nnal/atb/set_env.sh +export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True +export STREAMS_PER_DEVICE=32 +export HCCL_SOCKET_IFNAME=lo +export GLOO_SOCKET_IFNAME=lo + +export HCCL_OP_EXPANSION_MODE=AIV +export TASK_QUEUE_ENABLE=1 + +export HCCL_BUFFSIZE=800 +export ASCEND_USE_FIA=1 +export SGLANG_SET_CPU_AFFINITY=1 +export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 +export SGLANG_NPU_FUSED_MOE_MODE=2 +export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=204800 + +MODEL_PATH=/path/to/MiniMax-M2.5-w8a8-QuaRot +EAGLE_MODEL_PATH=/path/to/MiniMax-M2.5-eagle-model +export PYTHONPATH=${EAGLE_MODEL_PATH}:$PYTHONPATH +export SGLANG_EXTERNAL_MODEL_PACKAGE=custom_eagle3 + +python -m sglang.launch_server \ + --model-path $MODEL_PATH \ + --host 127.0.0.1 \ + --port 32001 \ + --tp-size 16 \ + --enable-dp-attention \ + --dp-size 16 \ + --mem-fraction-static 0.75 \ + --max-running-requests 480 \ + --disable-radix-cache \ + --prefill-delayer-max-delay-passes 500 \ + --enable-prefill-delayer \ + --chunked-prefill-size -1 --max-prefill-token 8192 \ + --cuda-graph-bs 8 16 24 32 48 64 80 \ + --moe-a2a-backend ascend_fuseep --deepep-mode auto --quantization modelslim \ + --speculative-algorithm EAGLE3 \ + --speculative-draft-model-path $EAGLE_MODEL_PATH \ + --speculative-num-steps 3 \ + --speculative-eagle-topk 1 \ + --speculative-num-draft-tokens 4 \ + --speculative-draft-model-quantization unquant \ + --dtype bfloat16 +``` + +#### Benchmark + +We tested it based on the `RANDOM` dataset. + +```shell Command +python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 32001 --random-input-len 3500 --random-output-len 1500 --num-prompts 1280 --random-range-ratio 1 --max-concurrency 320 +``` + +### MiniMax-M2.5 64K-1K High Throughput on A3 8 Cards Mixed Mode + +Model: MiniMax-M2.5 + +Hardware: Atlas 800I A3 8Card + +DeployMode: PD Mixed + +Dataset: random + +Input Output Length: 64K+1K + +#### Model Deployment + +```bash Command +echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor +sysctl -w vm.swappiness=0 +sysctl -w kernel.numa_balancing=0 +sysctl -w kernel.sched_migration_cost_ns=50000 + +unset https_proxy +unset http_proxy +unset HTTPS_PROXY +unset HTTP_PROXY +unset ASCEND_LAUNCH_BLOCKING + +source /usr/local/Ascend/ascend-toolkit/set_env.sh +source /usr/local/Ascend/nnal/atb/set_env.sh +export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True +export STREAMS_PER_DEVICE=32 +export HCCL_SOCKET_IFNAME=lo +export GLOO_SOCKET_IFNAME=lo + +export TASK_QUEUE_ENABLE=1 + +export ASCEND_USE_FIA=1 +export HCCL_BUFFSIZE=1600 +export SGLANG_SET_CPU_AFFINITY=1 +export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 +export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=640 +export DEEPEP_NORMAL_LONG_SEQ_ROUND=64 +export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=2048 +export DEEPEP_NORMAL_COMBINE_ENABLE_LONG_SEQ=1 +export SGLANG_NPU_FUSED_MOE_MODE=2 +export SGLANG_NPU_DEEPEP_USE_FUSED_MOE_DECODE=1 +export SGLANG_NPU_FUSEEP_DECODE_ONLY=1 + +MODEL_PATH=/path/to/MiniMax-M2.5-w8a8-QuaRot +EAGLE_MODEL_PATH=/path/to/MiniMax-M2.5-eagle-model +export PYTHONPATH=${EAGLE_MODEL_PATH}:$PYTHONPATH +export SGLANG_EXTERNAL_MODEL_PACKAGE=custom_eagle3 + +python -m sglang.launch_server \ + --model-path $MODEL_PATH \ + --host 127.0.0.1 \ + --port 32000 \ + --tp-size 16 \ + --dp-size 2 \ + --enable-dp-attention \ + --prefill-delayer-max-delay-passes 100 \ + --enable-prefill-delayer \ + --mem-fraction-static 0.65 \ + --max-running-requests 72 \ + --chunked-prefill-size -1 --max-prefill-token 180000 \ + --cuda-graph-bs 8 16 24 32 40 \ + --moe-a2a-backend ascend_fuseep --deepep-mode auto --quantization modelslim \ + --speculative-algorithm EAGLE3 \ + --speculative-draft-model-path $EAGLE_MODEL_PATH \ + --speculative-num-steps 3 \ + --speculative-eagle-topk 1 \ + --speculative-num-draft-tokens 4 \ + --speculative-draft-model-quantization unquant \ + --dtype bfloat16 \ + --trust-remote-code \ + --tokenizer-worker-num 8 +``` + +#### Benchmark + +We tested it based on the `RANDOM` dataset. + +```shell Command +python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 32000 --random-input-len 65536 --random-output-len 1024 --num-prompts 144 --random-range-ratio 1 --max-concurrency 36 +``` +### MiniMax-M2.5 128K-1K High Throughput on A3 8 Cards Mixed Mode + +Model: MiniMax-M2.5 + +Hardware: Atlas 800I A3 8Card + +DeployMode: PD Mixed + +Dataset: random + +Input Output Length: 128K+1K + +#### Model Deployment + +```bash Command +echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor +sysctl -w vm.swappiness=0 +sysctl -w kernel.numa_balancing=0 +sysctl -w kernel.sched_migration_cost_ns=50000 + +unset https_proxy +unset http_proxy +unset HTTPS_PROXY +unset HTTP_PROXY +unset ASCEND_LAUNCH_BLOCKING + +source /usr/local/Ascend/ascend-toolkit/set_env.sh +source /usr/local/Ascend/nnal/atb/set_env.sh +export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True +export STREAMS_PER_DEVICE=32 +export HCCL_SOCKET_IFNAME=lo +export GLOO_SOCKET_IFNAME=lo +export TASK_QUEUE_ENABLE=1 + +export ASCEND_USE_FIA=1 +export HCCL_BUFFSIZE=1600 +export SGLANG_SET_CPU_AFFINITY=1 +export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 +export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=640 +export DEEPEP_NORMAL_LONG_SEQ_ROUND=64 +export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=2048 +export DEEPEP_NORMAL_COMBINE_ENABLE_LONG_SEQ=1 +export SGLANG_NPU_FUSED_MOE_MODE=2 +export SGLANG_NPU_DEEPEP_USE_FUSED_MOE_DECODE=1 +export SGLANG_NPU_FUSEEP_DECODE_ONLY=1 + +MODEL_PATH=/path/to/MiniMax-M2.5-w8a8-QuaRot +EAGLE_MODEL_PATH=/path/to/MiniMax-M2.5-eagle-model +export PYTHONPATH=${EAGLE_MODEL_PATH}:$PYTHONPATH +export SGLANG_EXTERNAL_MODEL_PACKAGE=custom_eagle3 + +python -m sglang.launch_server \ + --model-path $MODEL_PATH \ + --host 127.0.0.1 \ + --port 32000 \ + --tp-size 16 \ + --dp-size 2 \ + --enable-dp-attention \ + --prefill-delayer-max-delay-passes 100 \ + --enable-prefill-delayer \ + --mem-fraction-static 0.65 \ + --max-running-requests 36 \ + --chunked-prefill-size -1 --max-prefill-token 130000 \ + --cuda-graph-bs 8 16 24 \ + --moe-a2a-backend ascend_fuseep --deepep-mode auto --quantization modelslim \ + --speculative-algorithm EAGLE3 \ + --speculative-draft-model-path $EAGLE_MODEL_PATH \ + --speculative-num-steps 3 \ + --speculative-eagle-topk 1 \ + --speculative-num-draft-tokens 4 \ + --speculative-draft-model-quantization unquant \ + --dtype bfloat16 \ + --trust-remote-code \ + --tokenizer-worker-num 8 +``` + +#### Benchmark + +We tested it based on the `RANDOM` dataset. + +```shell Command +python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 32000 --random-input-len 131072 --random-output-len 1024 --num-prompts 128 --random-range-ratio 1 --max-concurrency 32 +``` +### MiniMax-M2.5 64K-1K High Throughput on A3 4 Cards Mixed Mode + +Model: MiniMax-M2.5 + +Hardware: Atlas 800I A3 4Card + +DeployMode: PD Mixed + +Dataset: random + +Input Output Length: 64K+1K + +#### Model Deployment + +```bash Command +echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor +sysctl -w vm.swappiness=0 +sysctl -w kernel.numa_balancing=0 +sysctl -w kernel.sched_migration_cost_ns=50000 + +unset https_proxy +unset http_proxy +unset HTTPS_PROXY +unset HTTP_PROXY +unset ASCEND_LAUNCH_BLOCKING + +source /usr/local/Ascend/ascend-toolkit/set_env.sh +source /usr/local/Ascend/nnal/atb/set_env.sh +export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True +export STREAMS_PER_DEVICE=32 +export HCCL_SOCKET_IFNAME=lo +export GLOO_SOCKET_IFNAME=lo +export TASK_QUEUE_ENABLE=1 + +export ASCEND_USE_FIA=0 +export HCCL_BUFFSIZE=1600 +export SGLANG_SET_CPU_AFFINITY=1 +export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 +export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=640 +export DEEPEP_NORMAL_LONG_SEQ_ROUND=64 +export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=2048 +export DEEPEP_NORMAL_COMBINE_ENABLE_LONG_SEQ=1 +export SGLANG_NPU_FUSED_MOE_MODE=2 +export SGLANG_NPU_DEEPEP_USE_FUSED_MOE_DECODE=1 +export SGLANG_NPU_FUSEEP_DECODE_ONLY=1 + +MODEL_PATH=/path/to/MiniMax-M2.5-w8a8-QuaRot +EAGLE_MODEL_PATH=/path/to/MiniMax-M2.5-eagle-model +export PYTHONPATH=${EAGLE_MODEL_PATH}:$PYTHONPATH +export SGLANG_EXTERNAL_MODEL_PACKAGE=custom_eagle3 + +python -m sglang.launch_server \ + --model-path $MODEL_PATH \ + --host 127.0.0.1 \ + --port 32000 \ + --tp-size 8 \ + --enable-dp-attention \ + --prefill-delayer-max-delay-passes 500 \ + --enable-prefill-delayer \ + --mem-fraction-static 0.65 \ + --max-running-requests 36 \ + --chunked-prefill-size -1 --max-prefill-token 150000 \ + --cuda-graph-bs 8 16 24 32 40 \ + --moe-a2a-backend ascend_fuseep --deepep-mode auto --quantization modelslim \ + --speculative-algorithm EAGLE3 \ + --speculative-draft-model-path $EAGLE_MODEL_PATH \ + --speculative-num-steps 3 \ + --speculative-eagle-topk 1 \ + --speculative-num-draft-tokens 4 \ + --speculative-draft-model-quantization unquant \ + --dtype bfloat16 \ + --trust-remote-code \ + --tokenizer-worker-num 8 +``` + +#### Benchmark + +We tested it based on the `RANDOM` dataset. + +```shell Command +python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 32000 --random-input-len 65536 --random-output-len 1024 --num-prompts 144 --random-range-ratio 1 --max-concurrency 36 +``` +### MiniMax-M2.5 64K-1K High Throughput on A3 16 Cards Disaggregation Mode + +Model: MiniMax-M2.5 + +Hardware: Atlas 800I A3 16Card + +DeployMode: PD Disaggregation + +Dataset: random + +Input Output Length: 64K+1K + +#### Model Deployment + +```bash Command +echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor +sysctl -w vm.swappiness=0 +sysctl -w kernel.numa_balancing=0 +sysctl -w kernel.sched_migration_cost_ns=50000 + +export SGLANG_SET_CPU_AFFINITY=1 +unset https_proxy +unset http_proxy +unset HTTPS_PROXY +unset HTTP_PROXY +unset ASCEND_LAUNCH_BLOCKING +source /usr/local/Ascend/ascend-toolkit/set_env.sh +source /usr/local/Ascend/nnal/atb/set_env.sh +export PATH=/usr/local/Ascend/8.5.0/compiler/bishengir/bin:$PATH + +export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True +export STREAMS_PER_DEVICE=32 + +export ASCEND_MF_STORE_URL="tcp://your_prefill_ip:24667" + +P_IP=('your_prefill_ip') +D_IP=('your_decode_ip') +D_MASTER="${D_IP[0]}:8001" +MODEL_PATH=/path/to/MiniMax-M2.5-w8a8-QuaRot + +EAGLE_MODEL_PATH=/path/to/MiniMax-M2.5-eagle-model +export PYTHONPATH=${EAGLE_MODEL_PATH}:$PYTHONPATH +export SGLANG_EXTERNAL_MODEL_PACKAGE=custom_eagle3 + +LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` +LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` + +# prefill +for i in "${!P_IP[@]}"; +do + if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]]; + then + echo "${P_IP[$i]}" + export HCCL_SOCKET_IFNAME=your_nic + export GLOO_SOCKET_IFNAME=your_nic + export ASCEND_USE_FIA=1 + export HCCL_BUFFSIZE=2500 + export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 + export TASK_QUEUE_ENABLE=2 + export DEEPEP_NORMAL_LONG_SEQ_ROUND=64 + export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=2048 + export DEEPEP_NORMAL_COMBINE_ENABLE_LONG_SEQ=1 + python -m sglang.launch_server --model-path ${MODEL_PATH} --disaggregation-mode prefill --host ${P_IP[$i]} \ + --port 32000 --disaggregation-bootstrap-port $((8998+$i)) --trust-remote-code --nnodes 1 --node-rank 0 \ + --tp-size 16 --mem-fraction-static 0.43 --attention-backend ascend --device npu --quantization modelslim \ + --disaggregation-transfer-backend ascend --max-running-requests 128 \ + --chunked-prefill-size -1 --max-prefill-tokens 58000 --moe-a2a-backend deepep --deepep-mode normal \ + --tokenizer-worker-num 16 \ + --dp-size 2 --enable-dp-attention --dtype bfloat16 --load-balance-method round_robin \ + --speculative-algorithm EAGLE3 \ + --speculative-draft-model-path $EAGLE_MODEL_PATH \ + --speculative-num-steps 3 \ + --speculative-eagle-topk 1 \ + --speculative-num-draft-tokens 4 \ + --speculative-draft-model-quantization unquant --skip-server-warmup + NODE_RANK=$i + break + fi +done + +# decode +for i in "${!D_IP[@]}"; +do + if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]]; + then + echo "${D_IP[$i]}" + export HCCL_BUFFSIZE=1600 + export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=640 + export HCCL_SOCKET_IFNAME=your_nic + export GLOO_SOCKET_IFNAME=your_nic + export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 + export SGLANG_NPU_FUSED_MOE_MODE=2 + export SGLANG_DISAGGREGATION_NUM_PRE_ALLOCATE_REQS=96 + + python -m sglang.launch_server --model-path ${MODEL_PATH} --disaggregation-mode decode --host ${D_IP[$i]} \ + --cuda-graph-bs 8 16 24 32 40 \ + --port 33000 --trust-remote-code \ + --tp-size 16 --mem-fraction-static 0.76 --attention-backend ascend --device npu --quantization modelslim \ + --nnodes 1 --node-rank $i --dist-init-addr $D_MASTER \ + --disaggregation-transfer-backend ascend --max-running-requests 80 \ + --chunked-prefill-size -1 --moe-a2a-backend ascend_fuseep --deepep-mode low_latency \ + --tokenizer-worker-num 16 \ + --dp-size 2 --enable-dp-attention --dtype bfloat16 \ + --load-balance-method round_robin \ + --speculative-algorithm EAGLE3 \ + --speculative-draft-model-path $EAGLE_MODEL_PATH \ + --speculative-num-steps 3 \ + --speculative-eagle-topk 1 \ + --speculative-num-draft-tokens 4 \ + --speculative-draft-model-quantization unquant + + NODE_RANK=$i + break + fi +done +``` + +```shell Command +python -m sglang_router.launch_router \ + --pd-disaggregation \ + --policy round_robin \ + --prefill http://your_prefill_ip:32000 8998 \ + --decode http://your_decode_ip:33000 \ + --host 127.0.0.1 \ + --mini-lb \ + --port 6688 +``` + +#### Benchmark + +We tested it based on the `RANDOM` dataset. + +```shell Command +python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 6688 --random-input-len 65536 --random-output-len 1024 --num-prompts 640 --random-range-ratio 1 --max-concurrency 160 +``` +### MiniMax-M2.5 128K-1K High Throughput on A3 16 Cards Disaggregation Mode + +Model: MiniMax-M2.5 + +Hardware: Atlas 800I A3 16Card + +DeployMode: PD Disaggregation + +Dataset: random + +Input Output Length: 128K+1K + +#### Model Deployment + +```bash Command +echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor +sysctl -w vm.swappiness=0 +sysctl -w kernel.numa_balancing=0 +sysctl -w kernel.sched_migration_cost_ns=50000 + +export SGLANG_SET_CPU_AFFINITY=1 +unset https_proxy +unset http_proxy +unset HTTPS_PROXY +unset HTTP_PROXY +unset ASCEND_LAUNCH_BLOCKING +source /usr/local/Ascend/ascend-toolkit/set_env.sh +source /usr/local/Ascend/nnal/atb/set_env.sh +export PATH=/usr/local/Ascend/8.5.0/compiler/bishengir/bin:$PATH + +export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True +export STREAMS_PER_DEVICE=32 + +export ASCEND_MF_STORE_URL="tcp://your_prefill_ip:24667" + +P_IP=('your_prefill_ip') +D_IP=('your_decode_ip') +D_MASTER="${D_IP[0]}:8001" +MODEL_PATH=/path/to/MiniMax-M2.5-w8a8-QuaRot + +EAGLE_MODEL_PATH=/path/to/MiniMax-M2.5-eagle-model +export PYTHONPATH=${EAGLE_MODEL_PATH}:$PYTHONPATH +export SGLANG_EXTERNAL_MODEL_PACKAGE=custom_eagle3 + +LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` +LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` + +# prefill +for i in "${!P_IP[@]}"; +do + if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]]; + then + echo "${P_IP[$i]}" + export HCCL_SOCKET_IFNAME=your_nic + export GLOO_SOCKET_IFNAME=your_nic + export ASCEND_USE_FIA=1 + export HCCL_BUFFSIZE=2500 + export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 + export TASK_QUEUE_ENABLE=2 + export DEEPEP_NORMAL_LONG_SEQ_ROUND=64 + export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=2048 + export DEEPEP_NORMAL_COMBINE_ENABLE_LONG_SEQ=1 + python -m sglang.launch_server --model-path ${MODEL_PATH} --disaggregation-mode prefill --host ${P_IP[$i]} \ + --port 32000 --disaggregation-bootstrap-port $((8998+$i)) --trust-remote-code --nnodes 1 --node-rank 0 \ + --tp-size 16 --mem-fraction-static 0.43 --attention-backend ascend --device npu --quantization modelslim \ + --disaggregation-transfer-backend ascend --max-running-requests 128 \ + --chunked-prefill-size -1 --max-prefill-tokens 130000 --moe-a2a-backend deepep --deepep-mode normal \ + --tokenizer-worker-num 16 \ + --dp-size 2 --enable-dp-attention --dtype bfloat16 --load-balance-method round_robin \ + --speculative-algorithm EAGLE3 \ + --speculative-draft-model-path $EAGLE_MODEL_PATH \ + --speculative-num-steps 2 \ + --speculative-eagle-topk 1 \ + --speculative-num-draft-tokens 3 \ + --speculative-draft-model-quantization unquant --skip-server-warmup + NODE_RANK=$i + break + fi +done + +# decode +for i in "${!D_IP[@]}"; +do + if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]]; + then + echo "${D_IP[$i]}" + export HCCL_BUFFSIZE=1600 + export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=640 + export HCCL_SOCKET_IFNAME=your_nic + export GLOO_SOCKET_IFNAME=your_nic + export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 + export SGLANG_NPU_FUSED_MOE_MODE=2 + export SGLANG_DISAGGREGATION_NUM_PRE_ALLOCATE_REQS=96 + + python -m sglang.launch_server --model-path ${MODEL_PATH} --disaggregation-mode decode --host ${D_IP[$i]} \ + --cuda-graph-bs 2 4 8 \ + --port 33000 --trust-remote-code \ + --tp-size 16 --mem-fraction-static 0.76 --attention-backend ascend --device npu --quantization modelslim \ + --nnodes 1 --node-rank $i --dist-init-addr $D_MASTER \ + --disaggregation-transfer-backend ascend --max-running-requests 80 \ + --chunked-prefill-size -1 --moe-a2a-backend ascend_fuseep --deepep-mode low_latency \ + --tokenizer-worker-num 8 \ + --dp-size 2 --enable-dp-attention --dtype bfloat16 \ + --load-balance-method round_robin \ + --speculative-algorithm EAGLE3 \ + --speculative-draft-model-path $EAGLE_MODEL_PATH \ + --speculative-num-steps 2 \ + --speculative-eagle-topk 1 \ + --speculative-num-draft-tokens 3 \ + --speculative-draft-model-quantization unquant + + NODE_RANK=$i + break + fi +done +``` + +```shell Command +python -m sglang_router.launch_router \ + --pd-disaggregation \ + --policy round_robin \ + --prefill http://your_prefill_ip:32000 8998 \ + --decode http://your_decode_ip:33000 \ + --host 127.0.0.1 \ + --mini-lb \ + --port 6688 +``` + +#### Benchmark + +We tested it based on the `RANDOM` dataset. + +```shell Command +python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 6688 --random-input-len 131072 --random-output-len 1024 --num-prompts 192 --random-range-ratio 1 --max-concurrency 48 +``` + + +### Kimi K2.5 w4a8 3_5K-1_5K 20ms on A3 8 Cards Mixed Mode + +Model: Kimi-K2.5-w4a8 + +Hardware: Atlas 800I A3 8Card + +DeployMode: PD Mixed + +Dataset: random + +Input Output Length: 3.5K+1.5K + +TPOT: 20ms + +#### Model Deployment + +```bash Command +echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor +sysctl -w vm.swappiness=0 +sysctl -w kernel.numa_balancing=0 +sysctl -w kernel.sched_migration_cost_ns=50000 + +unset https_proxy +unset http_proxy +unset HTTPS_PROXY +unset HTTP_PROXY +unset ASCEND_LAUNCH_BLOCKING +source /usr/local/Ascend/ascend-toolkit/set_env.sh +source /usr/local/Ascend/nnal/atb/set_env.sh +source /usr/local/Ascend/ascend-toolkit/latest/opp/vendors/customize/bin/set_env.bash + +export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600 +export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True +export HCCL_SOCKET_IFNAME=lo +export GLOO_SOCKET_IFNAME=lo +export STREAMS_PER_DEVICE=32 +export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 +export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=48 +export HCCL_BUFFSIZE=1200 +export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 +export SGLANG_NPU_USE_MLAPO=1 +export SGLANG_NPU_USE_MULTI_STREAM=1 +export SGLANG_SCHEDULER_DECREASE_PREFILL_IDLE=1 +export SGLANG_PREFILL_DELAYER_MAX_DELAY_PASSES=200 + +MODEL_PATH=xxx +DRAFT_PATH=xxx + +python3 -m sglang.launch_server \ + --model-path $MODEL_PATH --quantization modelslim --dtype bfloat16 \ + --model-loader-extra-config '{"enable_multithread_load": true}' \ + --host 0.0.0.0 --port 6699 \ + --trust-remote-code --device npu --attention-backend ascend \ + --tp-size 16 --base-gpu-id 0 --mem-fraction-static 0.78 --max-running-requests 64 \ + --chunked-prefill-size 32768 --context-length 8192 --max-prefill-tokens 16384 \ + --enable-multimodal --mm-attention-backend ascend_attn --sampling-backend ascend \ + --enable-dp-attention --dp-size 16 \ + --moe-a2a-backend deepep --deepep-mode auto \ + --cuda-graph-bs 1 2 3 4 --disable-radix-cache \ + --speculative-algorithm EAGLE3 \ + --speculative-draft-model-path $DRAFT_PATH \ + --speculative-num-steps 4 --speculative-eagle-topk 1 \ + --speculative-num-draft-tokens 5 \ + --speculative-draft-model-quantization unquant +``` + +#### Benchmark + +We tested it based on the `RANDOM` dataset. + +```bash Command +python3 -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 6699 --random-range-ratio 1 --max-concurrency 64 --random-output-len 1500 --random-input-len 3500 --num-prompts 64 +``` + + +### Kimi K2.5 w4a8 3_5K-1_5K 50ms on A3 8 Cards Mixed Mode + +Model: Kimi-K2.5-w4a8 + +Hardware: Atlas 800I A3 8Card + +DeployMode: PD Mixed + +Dataset: random + +Input Output Length: 3.5K+1.5K + +TPOT: 50ms + +#### Model Deployment + +```bash Command +echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor +sysctl -w vm.swappiness=0 +sysctl -w kernel.numa_balancing=0 +sysctl -w kernel.sched_migration_cost_ns=50000 + +unset https_proxy +unset http_proxy +unset HTTPS_PROXY +unset HTTP_PROXY +unset ASCEND_LAUNCH_BLOCKING +source /usr/local/Ascend/ascend-toolkit/set_env.sh +source /usr/local/Ascend/nnal/atb/set_env.sh +source /usr/local/Ascend/ascend-toolkit/latest/opp/vendors/customize/bin/set_env.bash + +export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600 +export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True +export HCCL_SOCKET_IFNAME=lo +export GLOO_SOCKET_IFNAME=lo +export STREAMS_PER_DEVICE=32 +export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 +export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=96 +export HCCL_BUFFSIZE=1200 +export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 +export SGLANG_SCHEDULER_DECREASE_PREFILL_IDLE=1 +export SGLANG_PREFILL_DELAYER_MAX_DELAY_PASSES=200 + +MODEL_PATH=xxx +DRAFT_PATH=xxx + +python3 -m sglang.launch_server \ + --model-path $MODEL_PATH --quantization modelslim --dtype bfloat16 \ + --model-loader-extra-config '{"enable_multithread_load": true}' \ + --host 0.0.0.0 --port 6699 \ + --trust-remote-code --device npu --attention-backend ascend \ + --tp-size 16 --base-gpu-id 0 --mem-fraction-static 0.7 --max-running-requests 120 \ + --chunked-prefill-size 32768 --context-length 8192 --max-prefill-tokens 16384 \ + --enable-multimodal --mm-attention-backend ascend_attn --sampling-backend ascend \ + --enable-dp-attention --dp-size 16 \ + --moe-a2a-backend deepep --deepep-mode auto \ + --cuda-graph-bs 1 2 4 8 12 16 24 32 48 64 96 120 --disable-radix-cache \ + --speculative-algorithm EAGLE3 \ + --speculative-draft-model-path $DRAFT_PATH \ + --speculative-num-steps 4 --speculative-eagle-topk 1 \ + --speculative-num-draft-tokens 5 \ + --speculative-draft-model-quantization unquant +``` + +#### Benchmark + +We tested it based on the `RANDOM` dataset. + +```bash Command +python3 -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 6699 --random-range-ratio 1 --max-concurrency 120 --random-output-len 1500 --random-input-len 3500 --num-prompts 120 +``` + +### GLM-5.1 3_5K-1_5K 41ms on A3 16 Cards Mixed Mode + +Model: [GLM-5.1](https://www.modelscope.cn/models/Eco-Tech/GLM-5.1-w4a8) + +The model is quantized, with MTP layers excluded from quantization. + +Hardware: Atlas 800I A3 16Card + +DeployMode: PD Mixed + +Dataset: random + +Input Output Length: 3.5K+1.5K + +TPOT: 41ms + +#### Model Deployment + +```bash Command +echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor +sysctl -w vm.swappiness=0 +sysctl -w kernel.numa_balancing=0 +sysctl -w kernel.sched_migration_cost_ns=50000 +export SGLANG_SET_CPU_AFFINITY=1 + +unset https_proxy +unset http_proxy +unset HTTPS_PROXY +unset HTTP_PROXY +unset ASCEND_LAUNCH_BLOCKING + +source /usr/local/Ascend/ascend-toolkit/set_env.sh +source /usr/local/Ascend/nnal/atb/set_env.sh + +export PYTHONPATH=/path/to/sglang/python:$PYTHONPATH + +export STREAMS_PER_DEVICE=32 + +export HCCL_SOCKET_IFNAME=your_nic +export GLOO_SOCKET_IFNAME=your_nic + +MODEL_PATH=/path/to/GLM-5.1-w4a8 + +P_IP=('your ip1' 'your ip2') +P_MASTER="${P_IP[0]}:4567" +export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600 + +export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 + +LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` +LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` + +echo "${LOCAL_HOST1}" +echo "${LOCAL_HOST2}" + +export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=32 +export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 + +for i in "${!P_IP[@]}"; +do + if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]]; + then + echo "${P_IP[$i]}" + export HCCL_BUFFSIZE=2500 + python -m sglang.launch_server \ + --model-path $MODEL_PATH \ + --attention-backend ascend \ + --device npu \ + --dist-init-addr ${P_IP[0]}:5000 \ + --tp-size 32 --nnodes 2 --node-rank $i \ + --dp-size 16 --enable-dp-attention \ + --chunked-prefill-size 131072 --max-prefill-tokens 280000 \ + --trust-remote-code \ + --host 127.0.0.1 \ + --mem-fraction-static 0.65 \ + --port 8001 \ + --served-model-name glm-5 \ + --cuda-graph-max-bs 8 \ + --max-running-requests 128 \ + --quantization modelslim \ + --speculative-draft-model-quantization unquant \ + --moe-a2a-backend deepep --deepep-mode auto \ + --load-balance-method round_robin \ + --speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 + NODE_RANK=$i + break + fi +done +``` + + +**Quantization Configuration:** + +- `--quantization modelslim` is only applicable for quantized models. +- `--speculative-draft-model-quantization unquant` should be configured based on model specs, turned on for non-quantized MTP layers. + + +#### Benchmark + +We tested it based on the `RANDOM` dataset. + +```bash Command +python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 8001 --random-range-ratio 1 --random-output-len 1500 --random-input-len 3500 --num-prompts 320 +``` + +### GLM-5.1 16K-1K 23ms on A3 32 Cards Disaggregation Mode + +Model: [GLM-5.1](https://www.modelscope.cn/models/Eco-Tech/GLM-5.1-w4a8) + +Hardware: Atlas 800I A3 32Card + +DeployMode: PD Disaggregation + +Dataset: random + +Input Output Length: 16K+1K + +TPOT: 23ms + +#### Model Deployment + +```bash Command +echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor +sysctl -w vm.swappiness=0 +sysctl -w kernel.numa_balancing=0 +sysctl -w kernel.sched_migration_cost_ns=50000 +export SGLANG_SET_CPU_AFFINITY=1 + +unset https_proxy +unset http_proxy +unset HTTPS_PROXY +unset HTTP_PROXY +unset ASCEND_LAUNCH_BLOCKING +source /usr/local/Ascend/ascend-toolkit/set_env.sh +source /usr/local/Ascend/nnal/atb/set_env.sh +export LD_LIBRARY_PATH=/usr/local/Ascend/ascend-toolkit/latest/opp/vendors/customize/op_api/lib/:${LD_LIBRARY_PATH} +export PATH=/usr/local/Ascend/8.5.0/compiler/bishengir/bin:$PATH + +export PYTHONPATH=/path/to/sglang/python:$PYTHONPATH + +export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True +export STREAMS_PER_DEVICE=32 + +export ASCEND_MF_STORE_URL="tcp://${P_IP[0]}:24707" +export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600 + +P_IP=('your prefill ip1' 'your prefill ip2') +D_IP=('your decode ip1' 'your decode ip2') + +MODEL_PATH=/path/to/GLM-5.1-w4a8 + +LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` +LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` +echo "${LOCAL_HOST1}" +echo "${LOCAL_HOST2}" + +# prefill +for i in "${!P_IP[@]}"; +do + if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]]; + then + echo "${P_IP[$i]}" + export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 + export TASK_QUEUE_ENABLE=2 + export ENABLE_PROFILING=0 + export HCCL_SOCKET_IFNAME=your_nic + export GLOO_SOCKET_IFNAME=your_nic + + export HCCL_BUFFSIZE=8 + unset PYTORCH_NPU_ALLOC_CONF + export SGLANG_ZBAL_LOCAL_MEM_SIZE=61184 + export SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 + export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True + export ZBAL_NPU_ALLOC_CONF=use_vmm_for_static_memory:True + export SGLANG_ZBAL_BOOTSTRAP_URL="tcp://${P_IP[0]}:24672" + + python -m sglang.launch_server --model-path ${MODEL_PATH} --disaggregation-mode prefill --host ${P_IP[$i]} \ + --port 8000 --disaggregation-bootstrap-port 8998 --dist-init-addr ${P_IP[0]}:5000 --trust-remote-code --nnodes 2 --node-rank $i \ + --tp-size 32 --mem-fraction-static 0.75 --attention-backend ascend --device npu --quantization modelslim \ + --disaggregation-transfer-backend ascend --max-running-requests 64 \ + --served-model-name glm-5 --chunked-prefill-size 524288 --max-prefill-tokens 180000 --moe-a2a-backend deepep --deepep-mode normal \ + --disable-shared-experts-fusion --disable-cuda-graph --dtype bfloat16 \ + --dp-size 4 --enable-dp-attention \ + --load-balance-method round_robin \ + --enable-nsa-prefill-context-parallel \ + --nsa-prefill-cp-mode in-seq-split \ + --attn-cp-size 8 \ + --enable-dp-lm-head --moe-dense-tp 1 \ + --speculative-draft-model-quantization unquant \ + --speculative-algorithm NEXTN --speculative-num-steps 1 --speculative-eagle-topk 1 --speculative-num-draft-tokens 2 + NODE_RANK=$i + break + fi +done + +# decode +for i in "${!D_IP[@]}"; +do + if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]]; + then + echo "${D_IP[$i]}" + export SGLANG_SPEC_ENABLE_OVERLAP_REFLOW=1 + export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 + export HCCL_BUFFSIZE=650 + export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=64 + export TASK_QUEUE_ENABLE=0 + export HCCL_SOCKET_IFNAME=your_nic + export GLOO_SOCKET_IFNAME=your_nic + + export SGLANG_NPU_USE_MULTI_STREAM=1 + + python -m sglang.launch_server --model-path ${MODEL_PATH} --disaggregation-mode decode --host ${D_IP[$i]} \ + --port 8003 --trust-remote-code --dist-init-addr ${D_IP[0]}:5000 --nnodes 2 --node-rank $i --tp-size 32 --dp-size 32 --ep-size 32 \ + --mem-fraction-static 0.87 --max-running-requests 128 --attention-backend ascend --device npu --quantization modelslim \ + --served-model-name glm-5 --moe-a2a-backend deepep --enable-dp-attention --deepep-mode low_latency \ + --cuda-graph-bs 1 2 3 --disaggregation-transfer-backend ascend --watchdog-timeout 9000 --context-length 180000 \ + --tokenizer-worker-num 4 --disable-shared-experts-fusion --dtype bfloat16 --load-balance-method round_robin \ + --speculative-draft-model-quantization unquant \ + --speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 + NODE_RANK=$i + break + fi +done +``` + +```shell Command +python -m sglang_router.launch_router \ + --pd-disaggregation \ + --policy round_robin \ + --prefill http://your_prefill_ip1:8000 8998 \ + --decode http://your_decode_ip1:8003 \ + --host 127.0.0.1 \ + --port 6688 +``` + +#### Benchmark + +We tested it based on the `RANDOM` dataset. + +```bash Command +python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 8003 --random-range-ratio 1 --random-output-len 1000 --random-input-len 16000 --num-prompts 192 +``` + +### GLM-5.1 64K-1K-90%_cache_hit 45ms on A3 48 Cards Disaggregation Mode + +Model: [GLM-5.1](https://www.modelscope.cn/models/Eco-Tech/GLM-5.1-w4a8) + +Hardware: Atlas 800I A3 48Card + +DeployMode: PD Disaggregation + +Dataset: random (90% cache hit) + +Input Output Length: 64K+1K + +TPOT: 45ms + +#### Model Deployment + +```bash Command +echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor +sysctl -w vm.swappiness=0 +sysctl -w kernel.numa_balancing=0 +sysctl -w kernel.sched_migration_cost_ns=50000 +export SGLANG_SET_CPU_AFFINITY=1 + +unset https_proxy +unset http_proxy +unset HTTPS_PROXY +unset HTTP_PROXY +unset ASCEND_LAUNCH_BLOCKING +source /usr/local/Ascend/ascend-toolkit/set_env.sh +source /usr/local/Ascend/nnal/atb/set_env.sh +export LD_LIBRARY_PATH=/usr/local/Ascend/ascend-toolkit/latest/opp/vendors/customize/op_api/lib/:${LD_LIBRARY_PATH} +export PATH=/usr/local/Ascend/8.5.0/compiler/bishengir/bin:$PATH + +export PYTHONPATH=/path/to/sglang/python:$PYTHONPATH + +export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True +export STREAMS_PER_DEVICE=32 + +export ASCEND_MF_STORE_URL="tcp://${P_IP[0]}:24709" +export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=1200 +export SGLANG_DISAGGREGATION_WAITING_TIMEOUT=1200 + +P_IP=('your prefill ip1' 'your prefill ip2' 'your prefill ip3' 'your prefill ip4') +D_IP=('your decode ip1' 'your decode ip2') + +MODEL_PATH=/path/to/GLM-5.1-w4a8 + +LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` +LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` +echo "${LOCAL_HOST1}" +echo "${LOCAL_HOST2}" + +# prefill +for i in "${!P_IP[@]}"; +do + if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]]; + then + echo "${P_IP[$i]}" + export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 + export TASK_QUEUE_ENABLE=2 + export ENABLE_PROFILING=0 + export HCCL_SOCKET_IFNAME=your_nic + export GLOO_SOCKET_IFNAME=your_nic + + export ZBAL_HCCL_OP="send,recv" + export HCCL_BUFFSIZE=128 + unset PYTORCH_NPU_ALLOC_CONF + export SGLANG_ZBAL_LOCAL_MEM_SIZE=61184 + export SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 + export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True + export ZBAL_NPU_ALLOC_CONF=use_vmm_for_static_memory:True + export SGLANG_ZBAL_BOOTSTRAP_URL="tcp://${P_IP[$i]}:24691" + + python -m sglang.launch_server --model-path ${MODEL_PATH} --disaggregation-mode prefill --host ${P_IP[$i]} \ + --port 8000 --disaggregation-bootstrap-port $((8998 + i)) --trust-remote-code --nnodes 1 --node-rank 0 \ + --tp-size 4 --mem-fraction-static 0.72 --attention-backend ascend --device npu --quantization modelslim \ + --disaggregation-transfer-backend ascend --max-running-requests 16 \ + --served-model-name glm-5 --chunked-prefill-size 16384 --max-prefill-tokens 180000 --moe-a2a-backend deepep --deepep-mode normal \ + --disable-shared-experts-fusion --disable-cuda-graph --dtype bfloat16 \ + --speculative-draft-model-quantization unquant \ + --enable-nsa-prefill-context-parallel \ + --nsa-prefill-cp-mode in-seq-split \ + --attn-cp-size 4 \ + --enable-dp-lm-head --moe-dense-tp 1 \ + --speculative-algorithm NEXTN --speculative-num-steps 1 --speculative-eagle-topk 1 --speculative-num-draft-tokens 2 \ + --pp-size 4 + NODE_RANK=$i + break + fi +done + +# decode +for i in "${!D_IP[@]}"; +do + if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]]; + then + echo "${D_IP[$i]}" + export SGLANG_SPEC_ENABLE_OVERLAP_REFLOW=1 + export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 + export HCCL_BUFFSIZE=300 + export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=40 + export TASK_QUEUE_ENABLE=0 + export HCCL_SOCKET_IFNAME=your_nic + export GLOO_SOCKET_IFNAME=your_nic + + export SGLANG_NPU_USE_MULTI_STREAM=1 + export SGLANG_LM_HEAD_TP=4 + + python -m sglang.launch_server --model-path ${MODEL_PATH} --disaggregation-mode decode --host ${D_IP[$i]} \ + --port 8003 --trust-remote-code --dist-init-addr ${D_IP[0]}:5000 --nnodes 2 --node-rank $i --tp-size 32 --dp-size 32 --enable-dp-attention --ep-size 32 \ + --mem-fraction-static 0.85 --max-running-requests 320 --attention-backend ascend --device npu --quantization modelslim \ + --served-model-name glm-5 --moe-a2a-backend deepep --deepep-mode low_latency \ + --cuda-graph-bs 1 2 3 4 5 6 7 8 9 10 --disaggregation-transfer-backend ascend --watchdog-timeout 9000 --context-length 180000 \ + --tokenizer-worker-num 4 --disable-shared-experts-fusion --dtype bfloat16 --load-balance-method round_robin \ + --speculative-draft-model-quantization unquant \ + --speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 \ + --disaggregation-enable-decode-radix-cache + NODE_RANK=$i + break + fi +done +``` + +```shell Command +python -m sglang_router.launch_router \ + --pd-disaggregation \ + --policy round_robin \ + --prefill http://your_prefill_ip1:8000 8998 \ + --prefill http://your_prefill_ip2:8000 8999 \ + --prefill http://your_prefill_ip3:8000 9000 \ + --prefill http://your_prefill_ip4:8000 9001 \ + --decode http://your_decode_ip1:8003 \ + --host 127.0.0.1 \ + --port 6688 +``` + +#### Benchmark + +We tested it based on the `RANDOM` dataset (90% cache hit), this dataset is generated through [this tool](https://github.com/rayn-zzz/aisbench_auto_tools_prefix/tree/main). + +```bash Command +python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 8003 --random-range-ratio 1 --random-output-len 1000 --random-input-len 64000 --num-prompts 192 +``` + +### GLM-5.1 128K-1K-90%_cache_hit 32ms on A3 48 Cards Disaggregation Mode + +Model: [GLM-5.1](https://www.modelscope.cn/models/Eco-Tech/GLM-5.1-w4a8) + +Hardware: Atlas 800I A3 48Card + +DeployMode: PD Disaggregation + +Dataset: random (90% cache hit) + +Input Output Length: 128K+1K + +TPOT: 32ms + +#### Model Deployment + +```bash Command +echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor +sysctl -w vm.swappiness=0 +sysctl -w kernel.numa_balancing=0 +sysctl -w kernel.sched_migration_cost_ns=50000 +export SGLANG_SET_CPU_AFFINITY=1 + +unset https_proxy +unset http_proxy +unset HTTPS_PROXY +unset HTTP_PROXY +unset ASCEND_LAUNCH_BLOCKING +source /usr/local/Ascend/ascend-toolkit/set_env.sh +source /usr/local/Ascend/nnal/atb/set_env.sh +export LD_LIBRARY_PATH=/usr/local/Ascend/ascend-toolkit/latest/opp/vendors/customize/op_api/lib/:${LD_LIBRARY_PATH} +export PATH=/usr/local/Ascend/8.5.0/compiler/bishengir/bin:$PATH + +export PYTHONPATH=/path/to/sglang/python:$PYTHONPATH + +export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True +export STREAMS_PER_DEVICE=32 + +export ASCEND_MF_STORE_URL="tcp://${P_IP[0]}:24709" +export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=1200 +export SGLANG_DISAGGREGATION_WAITING_TIMEOUT=1200 + +P_IP=('your prefill ip1' 'your prefill ip2') +P1_IP=('your prefill ip3' 'your prefill ip4') +D_IP=('your decode ip1' 'your decode ip2') + +MODEL_PATH=/path/to/GLM-5.1-w4a8 + +LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'` +LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'` +echo "${LOCAL_HOST1}" +echo "${LOCAL_HOST2}" + +# prefill group 1 +for i in "${!P_IP[@]}"; +do + if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]]; + then + echo "${P_IP[$i]}" + export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 + export TASK_QUEUE_ENABLE=2 + export ENABLE_PROFILING=0 + export HCCL_SOCKET_IFNAME=your_nic + export GLOO_SOCKET_IFNAME=your_nic + + export ZBAL_HCCL_OP="send,recv" + export HCCL_BUFFSIZE=128 + unset PYTORCH_NPU_ALLOC_CONF + export SGLANG_ZBAL_LOCAL_MEM_SIZE=61184 + export SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 + export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True + export ZBAL_NPU_ALLOC_CONF=use_vmm_for_static_memory:True + export SGLANG_ZBAL_BOOTSTRAP_URL="tcp://${P_IP[0]}:24691" + + python -m sglang.launch_server --model-path ${MODEL_PATH} --disaggregation-mode prefill --host ${P_IP[$i]} \ + --port 8000 --disaggregation-bootstrap-port 8998 --trust-remote-code --nnodes 2 --node-rank $i --dist-init-addr ${P_IP[0]}:5000 \ + --tp-size 4 --mem-fraction-static 0.72 --attention-backend ascend --device npu --quantization modelslim \ + --disaggregation-transfer-backend ascend --max-running-requests 32 \ + --served-model-name glm-5 --chunked-prefill-size 16384 --max-prefill-tokens 180000 --moe-a2a-backend deepep --deepep-mode normal \ + --disable-shared-experts-fusion --disable-cuda-graph --dtype bfloat16 \ + --speculative-draft-model-quantization unquant \ + --enable-nsa-prefill-context-parallel \ + --nsa-prefill-cp-mode in-seq-split \ + --attn-cp-size 4 \ + --enable-dp-lm-head --moe-dense-tp 1 \ + --speculative-algorithm NEXTN --speculative-num-steps 1 --speculative-eagle-topk 1 --speculative-num-draft-tokens 2 \ + --pp-size 8 + NODE_RANK=$i + break + fi +done + +# prefill group 2 +for i in "${!P1_IP[@]}"; +do + if [[ "$LOCAL_HOST1" == "${P1_IP[$i]}" || "$LOCAL_HOST2" == "${P1_IP[$i]}" ]]; + then + echo "${P1_IP[$i]}" + export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 + export TASK_QUEUE_ENABLE=2 + export ENABLE_PROFILING=0 + export HCCL_SOCKET_IFNAME=your_nic + export GLOO_SOCKET_IFNAME=your_nic + + export ZBAL_HCCL_OP="send,recv" + export HCCL_BUFFSIZE=128 + unset PYTORCH_NPU_ALLOC_CONF + export SGLANG_ZBAL_LOCAL_MEM_SIZE=61184 + export SGLANG_ENABLE_TP_MEMORY_INBALANCE_CHECK=0 + export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True + export ZBAL_NPU_ALLOC_CONF=use_vmm_for_static_memory:True + export SGLANG_ZBAL_BOOTSTRAP_URL="tcp://${P1_IP[0]}:24691" + + python -m sglang.launch_server --model-path ${MODEL_PATH} --disaggregation-mode prefill --host ${P1_IP[$i]} \ + --port 8000 --disaggregation-bootstrap-port 8999 --trust-remote-code --nnodes 2 --node-rank $i --dist-init-addr ${P1_IP[0]}:5000 \ + --tp-size 4 --mem-fraction-static 0.72 --attention-backend ascend --device npu --quantization modelslim \ + --disaggregation-transfer-backend ascend --max-running-requests 32 \ + --served-model-name glm-5 --chunked-prefill-size 16384 --max-prefill-tokens 180000 --moe-a2a-backend deepep --deepep-mode normal \ + --disable-shared-experts-fusion --disable-cuda-graph --dtype bfloat16 \ + --speculative-draft-model-quantization unquant \ + --enable-nsa-prefill-context-parallel \ + --nsa-prefill-cp-mode in-seq-split \ + --attn-cp-size 4 \ + --enable-dp-lm-head --moe-dense-tp 1 \ + --speculative-algorithm NEXTN --speculative-num-steps 1 --speculative-eagle-topk 1 --speculative-num-draft-tokens 2 \ + --pp-size 8 + NODE_RANK=$i + break + fi +done + +# decode +for i in "${!D_IP[@]}"; +do + if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]]; + then + echo "${D_IP[$i]}" + export SGLANG_SPEC_ENABLE_OVERLAP_REFLOW=1 + export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 + export HCCL_BUFFSIZE=200 + export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=24 + export TASK_QUEUE_ENABLE=0 + export HCCL_SOCKET_IFNAME=your_nic + export GLOO_SOCKET_IFNAME=your_nic + + export SGLANG_NPU_USE_MULTI_STREAM=1 + + python -m sglang.launch_server --model-path ${MODEL_PATH} --disaggregation-mode decode --host ${D_IP[$i]} \ + --port 8003 --trust-remote-code --dist-init-addr ${D_IP[0]}:5000 --nnodes 2 --node-rank $i --tp-size 32 --dp-size 32 --enable-dp-attention --ep-size 32 \ + --mem-fraction-static 0.865 --max-running-requests 96 --attention-backend ascend --device npu --quantization modelslim \ + --served-model-name glm-5 --moe-a2a-backend deepep --deepep-mode low_latency \ + --cuda-graph-bs 1 2 3 4 5 6 --disaggregation-transfer-backend ascend --watchdog-timeout 9000 \ + --tokenizer-worker-num 32 --disable-shared-experts-fusion --dtype bfloat16 --load-balance-method round_robin \ + --speculative-draft-model-quantization unquant \ + --speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 \ + --disaggregation-decode-enable-radix-cache + NODE_RANK=$i + break + fi +done +``` + +```shell Command +python -m sglang_router.launch_router \ + --pd-disaggregation \ + --policy round_robin \ + --prefill http://your_prefill_ip1:8000 8998 \ + --prefill http://your_prefill_ip3:8000 8999 \ + --decode http://your_decode_ip1:8003 \ + --host 127.0.0.1 \ + --port 6688 +``` + +#### Benchmark + +We tested it based on the `RANDOM` dataset (90% cache hit), this dataset is generated through [this tool](https://github.com/rayn-zzz/aisbench_auto_tools_prefix/tree/main). + +```bash Command +python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 8003 --random-range-ratio 1 --random-output-len 1000 --random-input-len 131072 --num-prompts 192 +``` + +### MiMo-V2-Flash 16K+1 974ms on A3 24 Cards Disaggregation Mode + +**Model:** MiMo-V2-Flash + +**Hardware:** Atlas 800I A3 24 Cards (8 cards for prefill node, 16 cards for decode node) + +**Deploy Mode:** PD Disaggregation + +**Dataset:** random + +**Input/Output Length:** 16K+1 + +**TTFT:** 974ms (P99: 2,068ms) + +**Quantization:** W8A8 INT8 + +#### Model Deployment + +**Prefill Node:** + +```bash +# High performance CPU configuration +echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor +sysctl -w vm.swappiness=0 +sysctl -w kernel.numa_balancing=0 +sysctl -w kernel.sched_migration_cost_ns=50000 + +# Bind CPU +export SGLANG_SET_CPU_AFFINITY=1 + +# Proxy settings (unset if needed) +unset https_proxy +unset http_proxy +unset HTTPS_PROXY +unset HTTP_PROXY +unset ASCEND_LAUNCH_BLOCKING + +# Performance environment variables +export STREAMS_PER_DEVICE=32 +export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=32 +export HCCL_BUFFSIZE=1024 +export HCCL_OP_EXPANSION_MODE=AIV +export HCCL_SOCKET_IFNAME=lo +export GLOO_SOCKET_IFNAME=lo +export SGLANG_NPU_PROFILING=0 +export SGLANG_NPU_PROFILING_STAGE="prefill" +export DEEPEP_NORMAL_LONG_SEQ_ROUND=32 +export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=3584 +export ASCEND_MF_STORE_URL="tcp://:24669" +export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=3600 +export SGLANG_DISAGGREGATION_WAITING_TIMEOUT=3600 +export SGLANG_ENABLE_SPEC_V2=1 +export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=0 +export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 +export SGLANG_DEEPEP_BF16_DISPATCH=0 +export SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK=1 +export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True +export HCCL_CONNECT_TIMEOUT=1800 +export ASCEND_USE_FIA=1 + +# Launch prefill server +python3 -m sglang.launch_server \ + --model-path /path/to/MiMo-V2-Flash-w8a8-all-0512 \ + --attention-backend ascend \ + --device npu \ + --tp-size 8 --nnodes 1 --node-rank 0 \ + --chunked-prefill-size 8192 \ + --trust-remote-code --port 10000 \ + --host --max-running-requests 64 \ + --mem-fraction-static 0.8 \ + --swa-full-tokens-ratio 0.3 \ + --disaggregation-mode prefill --disaggregation-transfer-backend ascend \ + --disaggregation-bootstrap-port 8996 \ + --disable-radix-cache \ + --disable-cuda-graph \ + --disable-piecewise-cuda-graph \ + --dp-size 2 +``` + +**Decode Node:** + +```bash +# High performance CPU configuration +echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor +sysctl -w vm.swappiness=0 +sysctl -w kernel.numa_balancing=0 +sysctl -w kernel.sched_migration_cost_ns=50000 + +# Bind CPU +export SGLANG_SET_CPU_AFFINITY=1 + +# Proxy settings (unset if needed) +unset https_proxy +unset http_proxy +unset HTTPS_PROXY +unset HTTP_PROXY +unset ASCEND_LAUNCH_BLOCKING + +# Performance environment variables +export STREAMS_PER_DEVICE=32 +export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=128 +export HCCL_BUFFSIZE=800 +export HCCL_OP_EXPANSION_MODE=AIV +export HCCL_SOCKET_IFNAME=lo +export GLOO_SOCKET_IFNAME=lo +export SGLANG_NPU_PROFILING=0 +export SGLANG_NPU_PROFILING_STAGE="prefill" +export DEEPEP_NORMAL_LONG_SEQ_ROUND=32 +export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=3584 +export ASCEND_MF_STORE_URL="tcp://:24669" +export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=3600 +export SGLANG_DISAGGREGATION_WAITING_TIMEOUT=3600 +export SGLANG_ENABLE_SPEC_V2=1 +export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=0 +export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 +export SGLANG_DEEPEP_BF16_DISPATCH=0 +export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True +export HCCL_CONNECT_TIMEOUT=1800 +export SGLANG_PROFILE_WITH_STACK=True +export ASCEND_USE_FIA=1 + +# Launch decode server +python3 -m sglang.launch_server \ + --model-path /path/to/MiMo-V2-Flash-w8a8-all-0512/ \ + --attention-backend ascend \ + --device npu \ + --tp-size 16 --nnodes 1 --node-rank 0 \ + --trust-remote-code --port 10001 \ + --host --max-running-requests 64 \ + --mem-fraction-static 0.8 \ + --swa-full-tokens-ratio 0.3 \ + --cuda-graph-bs 1 2 4 8 12 16 20 24 28 32 \ + --disaggregation-mode decode --disaggregation-transfer-backend ascend \ + --disaggregation-bootstrap-port 8996 \ + --speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 \ + --enable-multi-layer-eagle \ + --disable-radix-cache \ + --dp-size 2 --enable-dp-attention --enable-dp-lm-head \ + --moe-a2a-backend deepep --deepep-mode low_latency +``` + +**Router:** + +```bash +python -m sglang_router.launch_router \ + --pd-disaggregation \ + --prefill http://:10000 \ + --decode http://:10001 \ + --host \ + --port 9903 \ + --health-check-interval-secs 3600 \ + --mini-lb +``` + +#### Benchmark + +We tested it based on the `RANDOM` dataset with throughput focus (0.4 request rate). + +```bash +python3 -m sglang.bench_serving --backend sglang --host --port 9903 --dataset-path /path/to/ShareGPT_V3_unfiltered_cleaned_split.json --dataset-name random --tokenize-prompt --random-input-len 16000 --random-output-len 1 --request-rate 0.4 --random-range-ratio 1 --num-prompts 128 --max-concurrency 64 +``` + +### MiMo-V2-Flash 32K+1 2,147ms on A3 24 Cards Disaggregation Mode + +**Model:** MiMo-V2-Flash + +**Hardware:** Atlas 800I A3 24 Cards (8 cards for prefill node, 16 cards for decode node) + +**Deploy Mode:** PD Disaggregation + +**Dataset:** random + +**Input/Output Length:** 32K+1 + +**TTFT:** 2,147ms (P99: 4,546ms) + +**Quantization:** W8A8 INT8 + +#### Model Deployment + +Same deployment configuration as [MiMo-V2-Flash 16K+1 974ms](#mimo-v2-flash-16k-1-974ms-on-a3-24-cards-disaggregation-mode). + +### MiMo-V2-Flash 16K+1K 16ms on A3 24 Cards Disaggregation Mode + +**Model:** MiMo-V2-Flash + +**Hardware:** Atlas 800I A3 24 Cards (8 cards for prefill node, 16 cards for decode node) + +**Deploy Mode:** PD Disaggregation + +**Dataset:** random + +**Input/Output Length:** 16K+1K + +**TTFT:** 11,326ms (P99: 28,160ms) + +**TPOT:** 16ms (P99: 23ms) + +**Quantization:** W8A8 INT8 + +#### Model Deployment + +Same deployment configuration as [MiMo-V2-Flash 16K+1 974ms](#mimo-v2-flash-16k-1-974ms-on-a3-24-cards-disaggregation-mode). + +#### Benchmark + +We tested it based on the `RANDOM` dataset with low latency focus (infinite request rate). + +```bash +python3 -m sglang.bench_serving --backend sglang --host --port 9903 --dataset-path /path/to/ShareGPT_V3_unfiltered_cleaned_split.json --dataset-name random --tokenize-prompt --random-input-len 16000 --random-output-len 1000 --request-rate inf --random-range-ratio 1 --num-prompts 128 --max-concurrency 64 +``` + +### MiMo-V2-Flash 32K+1K 12ms on A3 24 Cards Disaggregation Mode + +**Model:** MiMo-V2-Flash + +**Hardware:** Atlas 800I A3 24 Cards (8 cards for prefill node, 16 cards for decode node) + +**Deploy Mode:** PD Disaggregation + +**Dataset:** random + +**Input/Output Length:** 32K+1K + +**TTFT:** 35,144ms (P99: 55,004ms) + +**TPOT:** 12ms (P99: 17ms) + +**Quantization:** W8A8 INT8 + +#### Model Deployment + +Same deployment configuration as [MiMo-V2-Flash 16K+1 974ms](#mimo-v2-flash-16k-1-974ms-on-a3-24-cards-disaggregation-mode). diff --git a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_faq.mdx b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_faq.mdx index dd321d94b..6047d17cf 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_faq.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_faq.mdx @@ -43,16 +43,16 @@ Thread 0x0000fff873f6f120 (most recent call first): (SGLangEngine pid=3872176) [Error]: An internal error occurs in the runtime module on the host. (SGLangEngine pid=3872176) Rectify the fault based on the error information in the ascend log. (SGLangEngine pid=3872176) [PID: 3874122] 2026-04-14-12:14:41.897.548 AclNN_Runtime_Error(EZ9903): aclrtLaunchKerneWithHostArgs failed, return: 507000 -(SGLangEngine pid=3872176) Solution: In this scenario, collect the plog when the fault occurs and locate the fault based on the plog. -(SGLangEngine pid=3872176) TraceBack (most recent call last): -(SGLangEngine pid=3872176) Check kernel task failed, stream_id=2028, task_id=48, retCode=0x7080005.[FUNC:LaunchKernel][FILE:context.cc][LINE:1585] -(SGLangEngine pid=3872176) rtsLaunchKernelWithHostArgs execution failed, reason=kernel type error[FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:61] -(SGLangEngine pid=3872176) rtsLaunchKernelWithHostArgs failed, runtime result = 507000.[FUNC:ReportCallError][FILE:Log_inner.cpp][LINE:148] -(SGLangEngine pid=3872176) aclrtLaunchKerneWWithHostArgs failed, return: 507000 -(SGLangEngine pid=3872176) Launch kernel failed. -(SGLangEngine pid=3872176) #### KernelLaunch failed: /home/850b160/cann-8.5.8/opp/built-in/op_impl/ai_core/tbe//kernel/ascend910_93/ops_legacy/fill/Fill_41dadce325bOf810d03359af2a38990b_high_performance.o -(SGLangEngine pid=3872176) Kernel Run failed. opType: 18, Fill -(SGLangEngine pid=3872176) launch failed for Fill, errno:361001. +(SGLangEngine pid=3872176) Solution: In this scenario, collect the plog when the fault occurs and locate the fault based on the plog. +(SGLangEngine pid=3872176) TraceBack (most recent call last): +(SGLangEngine pid=3872176) Check kernel task failed, stream_id=2028, task_id=48, retCode=0x7080005.[FUNC:LaunchKernel][FILE:context.cc][LINE:1585] +(SGLangEngine pid=3872176) rtsLaunchKernelWithHostArgs execution failed, reason=kernel type error[FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:61] +(SGLangEngine pid=3872176) rtsLaunchKernelWithHostArgs failed, runtime result = 507000.[FUNC:ReportCallError][FILE:Log_inner.cpp][LINE:148] +(SGLangEngine pid=3872176) aclrtLaunchKerneWWithHostArgs failed, return: 507000 +(SGLangEngine pid=3872176) Launch kernel failed. +(SGLangEngine pid=3872176) #### KernelLaunch failed: /home/850b160/cann-8.5.8/opp/built-in/op_impl/ai_core/tbe//kernel/ascend910_93/ops_legacy/fill/Fill_41dadce325bOf810d03359af2a38990b_high_performance.o +(SGLangEngine pid=3872176) Kernel Run failed. opType: 18, Fill +(SGLangEngine pid=3872176) launch failed for Fill, errno:361001. (SGLangEngine pid=3872176) (SGLangEngine pid=3872176) Exception raised from operator() at build/CMakeFiles/torch_npu.dir/compiler depend.ts:26 (most recent call first): (SGLangEngine pid=3872176) frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string, std::allocator >)+ 0xb0 (0xffff806848c0 in /root/anaconda3/envs/slime_re/Lib/python3.11/site-packages/torch/lib/libc10.so) @@ -76,7 +76,7 @@ Too many captured graphs cause a conflict in the graph mode update stream. Each ### Error message -``` +```text [root@os-node-created-6z9tp pd_7p1d_tp2_20260414_030537]# grep -nR -E "aivec error|ACL" *,log prefill 1.log:5739:EZ9999[PID: 164243] 2026-04-14-06:40:35.033.745 (EZ9999): The error from device(chipId:0, dieId:1), serial number is 1, there is an exception of aivec error, core id is 23, error code = 0, dump info: pc start: 0x12420156a000, current: 0x12420156ad74, vec error info: 0xdb1751f50e, mte error info: 0x98f6388707, ifu error info: 0x212c93fc00000, ccu error info: 0x998e981c00000000, cube error info: 0, biu error info: 0, aic error mask: 0x6500020bd00028c, para base: 0x12c100dccc00.[FUNC:PrintCoreInfo][FILE:device error core proc.cc][LINE:347] prefill_1.log:5747:EZ9999[PID: 164242] 2026-04-14-06:40:35.034.456 (EZ9999): The error from device(chipId:0, dieId:0), serial number is 1, there is an exception of aivec error, core id is 45, error code = O, dump info: pc start: 0x12400156a000, current: 0x12400156ad74, vec error info: 0x7304583407, mte error info: 0x27e2b1c765, ifu error info: 0x212c93fc00000, ccu error info: 0x6e0836f300000000, cube error info: 0, biu error info: 0O, aic error mask: 0x6500020bd00028c, para base: 0x12c100dccc0O.[FUNC:PrintCoreInfo][FILE:device_error_core_proc.cc][LINE:347] @@ -206,7 +206,8 @@ If the NPUs are not occupied, you can use `--tp` to deploy across multiple devic ## 5. How to update sgl-kernel-npu ### Solution -```shell + +```bash git clone https://github.com/sgl-project/sgl-kernel-npu.git source /usr/local/Ascend/ascend-toolkit/set_env.sh @@ -281,12 +282,15 @@ The machine cannot directly access the HuggingFace server due to network restric --dataset-path /path/to/ShareGPT_V3_unfiltered_cleaned_split.json \ ... ``` + ## 7. Unexpected type fp8 ### Cause + FP8 model is not supported ### Solution + Replace model weights, e.g., switch Qwen/Qwen3.5-27B-FP8 to Eco-Tech/Qwen3.5-27B-w8a8-mtp. ## 8. Docker image versions: stable release vs. daily build diff --git a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_operator_development.mdx b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_operator_development.mdx index 2488d64eb..799b7ac7c 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_operator_development.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_operator_development.mdx @@ -148,6 +148,7 @@ extern "C" __global__ __aicore__ void helloworld( ``` **Key points:** + - Class methods must be marked with `__aicore__`, indicating they run on AICore. - Use `AscendC::TPipe` + `AscendC::TQue` to build a pipeline that overlaps data movement and computation. @@ -201,6 +202,7 @@ HOST_API at::Tensor helloworld(const at::Tensor &x, const at::Tensor &y) ``` **Key points:** + - The namespace must be `sglang::npu_kernel`. - Function signatures follow the pattern `at::Tensor (const at::Tensor &input, ...)`. @@ -249,6 +251,7 @@ TORCH_LIBRARY_IMPL(npu, PrivateUse1, m) ``` **Schema conventions:** + - The namespace is fixed to `npu`. In SGLang, operators are called via `torch.ops.npu.()`. - Output tensor parameters use the `Tensor(a!)` mutating annotation. @@ -258,6 +261,7 @@ TORCH_LIBRARY_IMPL(npu, PrivateUse1, m) [PyTorch Schema Reference](https://github.com/pytorch/pytorch/blob/main/aten/src/ATen/native/README.md#func). **Implementation binding rules:** + - The device name is fixed to `PrivateUse1` (PyTorch NPU backend identifier). - Use the `TORCH_FN` macro to bind to the implementation function. - For complex operators with optional parameters, use lambda expressions to @@ -436,6 +440,7 @@ python tests/python/sgl_kernel_npu/test_helloworld.py ``` **Testing checklist:** + - Cover typical input shapes (power-of-2 sizes and non-standard sizes). - Cover different data types (bf16 / fp16, etc.). - For operators with in-place behavior, verify correctness of output tensors. @@ -493,11 +498,13 @@ submitting a PR. steps above. Ensure all tests pass. 4. **Run pre-commit**: Ensure code formatting compliance. 5. **Commit and push**: + ```bash git add . git commit -m "feat: add operator" git push origin feature/add-my-op ``` + 6. **Create a PR**: Open a Pull Request on GitHub from your branch to `sgl-project/sgl-kernel-npu:main`. 7. **Wait for CI and review**: CI checks include linting, compilation, and diff --git a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_operator_performance_optimizing.mdx b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_operator_performance_optimizing.mdx index eab466f2d..8051b3891 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_operator_performance_optimizing.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_operator_performance_optimizing.mdx @@ -17,6 +17,7 @@ The msProf tool is used to collect and analyze key performance indicators of ope ```bash msprof op python3 test_xxxxx.py ``` + The following is a case of using msprof for data collection. | Attribute | Value | @@ -65,7 +66,9 @@ The following is a case of using msprof for data collection. | cube_utilization(%) | 0 | Below is the field-by-field breakdown of the operator performance record, aligned with the official specification. + ### 1. Basic Identification Fields + | Field | Value | Definition (per official docs) | |-------|-------|--------------------------------| | Name | DequantSwigluQuant_int32_high_performance_100000000 | **Op Name**: Name of the fused operator (dequantization + SwiGLU activation + quantization), with an int32 high-performance implementation suffix. | @@ -74,6 +77,7 @@ Below is the field-by-field breakdown of the operator performance record, aligne | Accelerator Core | AI_VECTOR_CORE | **Task Type**: The operator runs on the AI Vector Core; other common types include AI_CORE (matrix computation core) and AI_CPU. | ### 2. Timing & Scheduling Fields + | Field | Value | Definition (per official docs) | |-------|-------|--------------------------------| | Start Time(us) | 1774489226717521.715 | **Task Start Time**: Absolute start timestamp of the operator task on the device side, in microseconds. | @@ -81,6 +85,7 @@ Below is the field-by-field breakdown of the operator performance record, aligne | Wait Time(us) | 0 | **Task Wait Time**: Time interval between the end of the previous task and the start of the current task. A value of 0 means no idle wait between task dispatches. | ### 3. Core Configuration & Precision Fields + | Field | Value | Definition (per official docs) | |-------|-------|--------------------------------| | Block Dim | 36 | **Block Num**: Number of parallel thread blocks for the operator task, corresponding to Block Dim in the SIMT programming model. One AI Vector Core executes only one thread block at a time, so this value reflects the scale of occupied parallel compute resources. | @@ -88,6 +93,7 @@ Below is the field-by-field breakdown of the operator performance record, aligne | HF32 Eligible | NO | **HF32 Eligible**: Indicates whether the HF32 high-precision floating-point format is enabled; `NO` means it is not used. This field is reported only at the `--task-time=l1` collection level. | ### 4. Input & Output Information + | Field | Value | Definition (per official docs) | |-------|-------|--------------------------------| | Input Shapes | 163840,1024;128,1024;16384;;;;128 | **Input Shapes**: Dimensions of each input tensor, separated by semicolons; empty values represent scalar inputs. Breakdown: 7 inputs with shapes `[163840,1024]`, `[128,1024]`, `[163840]`, 3 scalars, and `[128]`. | @@ -99,6 +105,7 @@ Below is the field-by-field breakdown of the operator performance record, aligne | Context ID | N/A | **Context ID**: Identifier for sub-tasks at Sub Task granularity; N/A means no sub-task splitting for this operator. | ### 5. AI Core Performance Metrics (aic_* series) + | Field | Value | Definition (per official docs) | |-------|-------|--------------------------------| | aicore_time(us) | 0 | Theoretical execution time on AI Core, in microseconds. | @@ -111,6 +118,7 @@ Below is the field-by-field breakdown of the operator performance record, aligne | aic_icache_miss_rate | 0 | Instruction cache miss rate of AI Core. | ### 6. AI Vector Core Performance Metrics (aiv_* series) + | Field | Value | Definition & Interpretation | |-------|-------|------------------------------| | aiv_time(us) | 59.128 | **aiv_time**: Theoretical execution time on Vector Core under ideal conditions (all blocks scheduled simultaneously with equal duration). In practice, this value is slightly smaller than real execution time due to staggered block startup. | @@ -122,11 +130,11 @@ Below is the field-by-field breakdown of the operator performance record, aligne | aiv_icache_miss_rate | 0.005 | Vector Core instruction cache miss rate of 0.5%, extremely low and indicates efficient instruction fetch. | ### 7. Utilization Metrics + | Field | Value | Definition (per official docs) | |-------|-------|--------------------------------| | cube_utilization(%) | 0 | **cube_utilization**: Utilization rate of the matrix multiplication unit. The value is 0 because the operator is purely vector-based. | - ## Optimization ### Specification diff --git a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_optimization.mdx b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_optimization.mdx index ff4fef1dc..d283f9d4e 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_optimization.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_optimization.mdx @@ -390,6 +390,7 @@ The following environment variables are used in other best practice configuratio The table shows feature-by-feature compatibility of SGLang capabilities on Ascend NPU. Features that have performance impact on Ascend are included. Each cell at the intersection of two features indicates whether they can be used together. The symbols used are defined as follows: + - 🟢 = Full compatibility - 🟠 = Partial compatibility - ❌ = No compatibility @@ -791,7 +792,6 @@ Enables Expert Parallelism Load Balancing to dynamically redistribute experts ac Casts model weight tensors to the Ascend NPU ACL FRACTAL_NZ format (format 29) for improved memory access efficiency on Da Vinci AI cores. Enabled by default globally; disable with `SGLANG_NPU_DISABLE_ACL_FORMAT_WEIGHT=1` when the format conflicts with other features (e.g., EPLB). Applied to linear weights, MoE weights, quantized weights, MLA QKV projections, and KV Cache (via `PA_NZ` / `nzcache` modes for FIA). Related env var: `SGLANG_USE_FIA_NZ=1` (requires `SGLANG_NPU_USE_MLAPO=1`). - ## See Also - [Ascend NPU Best Practice](/docs/hardware-platforms/ascend-npus/best_practice) — Complete deployment configurations and benchmark results for all supported models diff --git a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_performance_testing.mdx b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_performance_testing.mdx index de51acf2f..0ff7a76ac 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_performance_testing.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_performance_testing.mdx @@ -436,6 +436,7 @@ The output format is **hardcoded in `bench_serving.py`**. All formatting decisio ##### Time per Output Token (TPOT) – Excluding First Token + Formula: (E2E Latency - TTFT) / (Number of output tokens - 1) @@ -460,7 +461,6 @@ Formula: (E2E Latency - TTFT) / (Number of output tokens - 1)
- ## 3. Online Service: Multimodal Model Test `Qwen/Qwen2.5-VL-7B-Instruct` for vision-language tasks. diff --git a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_profiling.mdx b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_profiling.mdx index e36747e36..d25470405 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_profiling.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_profiling.mdx @@ -259,6 +259,7 @@ python3 -m sglang.profiler \ --num-steps 3 \ --cpu --gpu & ``` + ```bash Command # Terminal 2: Wait for "Waiting for N steps" output from Terminal 1, then send requests. # The profiler starts recording once /start_profile is received by the server. @@ -407,15 +408,18 @@ them via: - **When profiling starts**: server log outputs `Profiling starts. Traces will be saved to: (with profile id: )` -``` + +```text [2026-05-19 13:23:15] Profiling starts. Traces will be saved to: /tmp/1779196995.6948605 (with profile id: 1779196995.6979997) [2026-05-19 13:23:15] [WARNING] [350443] profiler.py: Invalid parameter export_type: None, reset it to text. [2026-05-19 13:23:15] [WARNING] [350443] profiler.py: Invalid parameter export_type: None, reset it to text. [2026-05-19 13:23:15] INFO: 127.0.0.1:40714 - "POST /start_profile HTTP/1.1" 200 OK ``` + - **When profiling stops**: server log outputs `Profiling done. Traces are saved to: ` -``` + +```text [2026-05-19 13:23:17] Stop profiling... [2026-05-19 13:23:17] [WARNING] [350443] profiler.py: Incorrect schedule: Stop profiler while current state is RECORD which may result in incomplete parsed data. [rank0]:[W519 13:23:17.084812760 compiler_depend.ts:3136] Warning: The indexFromRank 0is not equal indexFromCurDevice 4 , which might be normal if the number of devices on your collective communication server is inconsistent.Otherwise, you need to check if the current device is correct when calling the interface.If it's incorrect, it might have introduced an error. (function operator()) @@ -424,8 +428,10 @@ them via: [2026-05-19 13:23:32] [INFO] [352725] profiler.py: All profiling data parsed in a total time of 0:00:14.305669 [2026-05-19 13:23:32] Profiling done. Traces are saved to: /tmp/1779196995.6948605 ``` + - **CLI output**: `sglang.profiler` outputs `Dump profiling traces to ` -``` + +```text Dump profiling traces to /tmp/1779243331.3219 Waiting for 10 steps and the trace to be flushed.... (profile_by_stage=False) ``` diff --git a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_quantization.mdx b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_quantization.mdx index bbb7ced8f..18c6b8edf 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_quantization.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_quantization.mdx @@ -312,6 +312,7 @@ python3 -m sglang.launch_server \ ``` > **Implementation Notes:** +> > - GGUF weights are pre-dequantized to FP16/BF16 during model loading on CPU, then transferred to NPU for inference. This trades higher memory usage for faster runtime performance (no per-forward-pass dequantization overhead). > - MoE layers use `npu_grouped_matmul` and `npu_moe_init_routing` / `npu_moe_finalize_routing` for high-performance expert computation. > - TP (tensor parallelism) sharding is supported for both dense and MoE GGUF models. diff --git a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_support_new_models.mdx b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_support_new_models.mdx index f87657a83..b45f8b841 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_support_new_models.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_support_new_models.mdx @@ -64,7 +64,7 @@ Please note all your testing and benchmarking results in PR description. ### Benchmark -- **(Required) MMMU**: follow MMMU benchmark [README.md](https://github.com/sgl-project/sglang/blob/main/benchmark/mmmu/README.md) to get SGLang vs. HF Transformer accuracy comparison. The accuracy score from SGLang run should not be much lower than that from HF Transformer run. Similarly, follow https://docs.sglang.io/developer_guide/benchmark_and_profiling.html to get performance comparison: TTFT and throughput must meet or exceed baselines (e.g., HF Transformer). +- **(Required) MMMU**: follow MMMU benchmark [README.md](https://github.com/sgl-project/sglang/blob/main/benchmark/mmmu/README.md) to get SGLang vs. HF Transformer accuracy comparison. The accuracy score from SGLang run should not be much lower than that from HF Transformer run. Similarly, follow [https://docs.sglang.io/developer_guide/benchmark_and_profiling.html](https://docs.sglang.io/developer_guide/benchmark_and_profiling.html) to get performance comparison: TTFT and throughput must meet or exceed baselines (e.g., HF Transformer). - **(Optional) Other evals**: If you ran other evals, please note the results in PR description. @@ -332,7 +332,7 @@ Using the same Llama wrapper from the previous section, here is how to package a 1. Create your project -``` +```text sglang_custom_project/ |----setup.py |----custom_llm/ @@ -404,8 +404,8 @@ Update the `config.json` under your HuggingFace model checkpoint directory so th ```json Config { - "architectures": ["LlamaWrapper"], - ... + "architectures": ["LlamaWrapper"], + ... } ``` @@ -435,7 +435,7 @@ For example, let's build a custom model based on Qwen2-VL-Instruct that takes th Create the project: -``` +```text sglang_custom_project_vl/ |----setup.py |----custom_vlm/ diff --git a/docs_new/docs/hardware-platforms/ascend-npus/diffusion/disaggregation.mdx b/docs_new/docs/hardware-platforms/ascend-npus/diffusion/disaggregation.mdx index d3adf0bb2..c39522a24 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/diffusion/disaggregation.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/diffusion/disaggregation.mdx @@ -24,6 +24,7 @@ pip install . ``` Before run servers ensure that *.so files of mooncake in LD_LIBRARY_PATH + ```bash export LD_LIBRARY_PATH=/usr/local/python3.11.14/lib/python3.11/site-packages/mooncake:$LD_LIBRARY_PATH ``` diff --git a/docs_new/docs/hardware-platforms/ascend-npus/mindspore_backend.mdx b/docs_new/docs/hardware-platforms/ascend-npus/mindspore_backend.mdx index 32adcbb7d..a143f02da 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/mindspore_backend.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/mindspore_backend.mdx @@ -29,7 +29,6 @@ pip install -e . ``` - ## Run Model Current SGLang-MindSpore supports Qwen3 and DeepSeek V3/R1 models. This doc uses Qwen3-8B as an example. @@ -164,6 +163,7 @@ export PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python # to avoid protobuf binary ## Support + For MindSpore-specific issues: - Refer to the [MindSpore documentation](https://www.mindspore.cn/)