diff --git a/docs_new/docs/hardware-platforms/ascend-npus/ascend_contribution_guide.mdx b/docs_new/docs/hardware-platforms/ascend-npus/ascend_contribution_guide.mdx index 995189617..69c4092a3 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/ascend_contribution_guide.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/ascend_contribution_guide.mdx @@ -83,7 +83,7 @@ For detailed instructions, refer to [test/README.md](https://github.com/sgl-proj If you need to use model which is not in `python/sglang/test/ascend/test_ascend_utils.py` list. Follow these steps: 1. Register account and upload your model to [modelscope](https://modelscope.cn/models). -2. Make sure your model is pre-cached on the CI server and is on the way "/data/ascend-ci-share-pkking-sglang/modelscope/hub/models/{your_model_repo}/{your_model}". +2. Make sure your model is pre-cached on the CI server and is at the path "/data/ascend-ci-share-pkking-sglang/modelscope/hub/models/{your_model_repo}/{your_model}". If this is not the case, use following command on CI server: ```bash @@ -184,7 +184,7 @@ Users listed in [CI_PERMISSIONS.json](https://github.com/sgl-project/sglang/blob - In a file, put core data structures at the top of the file. Put utility functions at the bottom of the file. - Keep tests run fast. - If a single test file run longer than 500 seconds, split it into multiple smaller files (e.g., `test_eagle_infer_a.py`, `test_eagle_infer_b.py`). - - If a single job in a github workflow runs longer than 30 mins, split it into smaller jobs/steps. + - If a single job in a github workflow runs longer than 30 min, split it into smaller jobs/steps. - Reuse server launches in your unit tests to make tests run faster. - Never use `pickle.loads()`, `pickle.load()`, or `recv_pyobj()` to deserialize untrusted or network-received data. Python’s [pickle module is not secure](https://docs.python.org/3/library/pickle.html) — it can execute arbitrary code during deserialization. Use safe serialization formats such as [msgpack](https://github.com/jcrist/msgspec) or JSON instead. - When supporting new hardware or features, follow these guidelines: diff --git a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu.mdx b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu.mdx index b41884b57..b71de7601 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu.mdx @@ -61,7 +61,7 @@ You can install SGLang using any of the methods below. Please go through `System ### Obtain CANN Image -Ensure sufficient disk space before pulling images. Each Docker image requires at least **30 GB** of free space. +Ensure sufficient disk space before pulling images. Each Docker image requires at least **30GB** of free space. You can obtain the dependency of a specified version of CANN through an image. @@ -200,7 +200,7 @@ pip install -e python[all_npu] You can download the SGLang image or build an image based on Dockerfile to obtain the Ascend NPU image. -Ensure sufficient disk space before pulling images. Each Docker image requires at least **30 GB** of free space. If you need to download model weights, check the model size at [ModelScope](https://www.modelscope.cn/models) to reserve enough space. +Ensure sufficient disk space before pulling images. Each Docker image requires at least **30GB** of free space. If you need to download model weights, check the model size at [ModelScope](https://www.modelscope.cn/models) to reserve enough space. 1. Download SGLang image @@ -348,7 +348,7 @@ python3 -m sglang.launch_server \ --model-path meta-llama/Llama-3.1-8B-Instruct \ --attention-backend ascend \ --host 127.0.0.1 \ - --port 8000 + --port 30000 ``` #### PD Disaggregation Scene @@ -376,7 +376,7 @@ python3 -m sglang.launch_server \ --base-gpu-id 0 \ --tp-size 1 \ --host 127.0.0.1 \ - --port 8000 + --port 30001 ``` @@ -401,7 +401,7 @@ python3 -m sglang.launch_server \ --base-gpu-id 0 \ --tp-size 1 \ --host 127.0.0.1 \ - --port 8000 + --port 30001 ``` @@ -426,7 +426,7 @@ python3 -m sglang.launch_server \ --base-gpu-id 1 \ --tp-size 1 \ --host 127.0.0.1 \ - --port 8001 + --port 30002 ``` @@ -447,7 +447,7 @@ python3 -m sglang.launch_server \ --base-gpu-id 1 \ --tp-size 1 \ --host 127.0.0.1 \ - --port 8001 + --port 30002 ``` @@ -459,12 +459,17 @@ python3 -m sglang.launch_server \ python3 -m sglang_router.launch_router \ --pd-disaggregation \ --policy cache_aware \ - --prefill http://127.0.0.1:8000 8995 \ - --decode http://127.0.0.1:8001 \ + --prefill http://127.0.0.1:30001 8995 \ + --decode http://127.0.0.1:30002 \ --host 127.0.0.1 \ - --port 6688 + --port 30000 ``` + +The `8995` in command script is the disaggregation bootstrap port. It must match the +`--disaggregation-bootstrap-port` value set on the prefill server in step 1. + + ### Running Service For Multimodal Language Models #### PD Mixed Scene @@ -473,7 +478,7 @@ python3 -m sglang_router.launch_router \ python3 -m sglang.launch_server \ --model-path Qwen/Qwen3-VL-30B-A3B-Instruct \ --host 127.0.0.1 \ - --port 8000 \ + --port 30000 \ --tp 4 \ --device npu \ --attention-backend ascend \ @@ -494,20 +499,18 @@ The port you use depends on your deployment mode: | Scenario | Where to send requests | |---|---| -| Non-PD (single server) | The server's `--port` (e.g., `8000` in the examples above) | +| Non-PD (single server) | The server's `--port` (e.g., `30000` in the examples above) | | Non-PD (multi-node) | The primary node's (`--node-rank 0`) `--port`; do **not** send requests to worker nodes | -| PD disaggregation | The router's `--port` (e.g., `6688` in the examples above); do **not** send requests directly to prefill or decode servers | +| PD disaggregation | The router's `--port` (e.g., `30000` in the examples above); do **not** send requests directly to prefill or decode servers | -SGLang serves on port `30000` by default if `--port` is not specified. The examples in this guide use explicit ports for clarity. - -If you are using PD disaggregation, replace `8000` with your router's port (e.g., `6688`) in the following examples. +SGLang defaults to port `30000` when `--port` is not specified. The examples in this guide use explicit ports for clarity. ### Health Check ```bash Command -curl http://127.0.0.1:8000/health +curl http://127.0.0.1:30000/health ``` A successful response returns HTTP 200 with an empty body. @@ -515,7 +518,7 @@ A successful response returns HTTP 200 with an empty body. ### Generate (Native Endpoint) ```bash Command -curl http://127.0.0.1:8000/generate \ +curl http://127.0.0.1:30000/generate \ -H "Content-Type: application/json" \ -d '{ "text": "What is the capital of France?", @@ -528,7 +531,7 @@ The expected output should contain "Paris". ### Chat Completions (OpenAI-Compatible) ```bash Command -curl http://127.0.0.1:8000/v1/chat/completions \ +curl http://127.0.0.1:30000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "meta-llama/Llama-3.1-8B-Instruct", @@ -539,7 +542,7 @@ curl http://127.0.0.1:8000/v1/chat/completions \ Some models return responses accompanied with thinking process content. To disable this output, configure parameters as follows: ```bash Command -curl http://127.0.0.1:8000/v1/chat/completions \ +curl http://127.0.0.1:30000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "Eco-Tech/Qwen3.5-27B-w8a8-mtp", @@ -557,7 +560,7 @@ The image URL in the example below references an external resource (`raw.githubu ```bash Command -curl http://127.0.0.1:8000/v1/chat/completions \ +curl http://127.0.0.1:30000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "Qwen3-VL-30B-A3B-Instruct", diff --git a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_accuracy_evaluation.mdx b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_accuracy_evaluation.mdx index 435b17dce..efad7af0e 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_accuracy_evaluation.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_accuracy_evaluation.mdx @@ -17,7 +17,7 @@ This document describes how to perform accuracy evaluation for SGLang models run ## Environment Setup -Ensure sufficient disk space before proceeding. The Docker image requires at least **30 GB** of free space. If you need to download model weights, check the model size at [ModelScope](https://www.modelscope.cn/models) to reserve enough space. +Ensure sufficient disk space before proceeding. The Docker image requires at least **30GB** of free space. If you need to download model weights, check the model size at [ModelScope](https://www.modelscope.cn/models) to reserve enough space. First, launch the SGLang environment using the provided container image: @@ -295,7 +295,7 @@ ERROR: cannot verify www.modelscope.cn's certificate, issued by ‘ To connect to www.modelscope.cn insecurely, use `--no-check-certificate`. ``` -You can add `--no-check-certificate' +You can add `--no-check-certificate` ```bash wget https://www.modelscope.cn/datasets/evalscope/MMStar/resolve/master/MMStar.tsv --no-check-certificate diff --git a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_operator_development.mdx b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_operator_development.mdx index 166d75a0d..2c7fc9088 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_operator_development.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_operator_development.mdx @@ -23,8 +23,9 @@ on Ascend hardware. ## Directory Structure -The identifiers `sgl_kenel_npu_ops.h` and `KernalHelloworld` in this guide match the spelling -used in the upstream [sgl-kernel-npu](https://github.com/sgl-project/sgl-kernel-npu) repository and are kept verbatim for consistency. +The identifiers `sgl_kenel_npu_ops.h`, `KernalHelloworld`, and `retrive_*` (e.g. `retrive_index`, +`retrive_next_token`, `retrive_next_sibling`) in this guide match the spelling used in the +upstream [sgl-kernel-npu](https://github.com/sgl-project/sgl-kernel-npu) repository and are kept verbatim for consistency. ```text diff --git a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_operator_performance_optimizing.mdx b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_operator_performance_optimizing.mdx index 8051b3891..91ea7f747 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_operator_performance_optimizing.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_operator_performance_optimizing.mdx @@ -144,7 +144,7 @@ Below is the field-by-field breakdown of the operator performance record, aligne - AI Core: the core that actually performs matrix/vector computation - Vector Unit: responsible for SIMD computation (similar to CUDA Core) - Scalar Unit: responsible for control/loop -- L0/L1/L2 cache: The smaller the size, the faster the speed. L0 is only 64 KB, L1 is 256 KB, and L2 is shared. +- L0/L1/L2 cache: The smaller the size, the faster the speed. L0 is only 64KB, L1 is 256KB, and L2 is shared. #### 2. Ascend memory hierarchy (from fastest to slowest) @@ -168,6 +168,6 @@ Must be vectorized. 1. Ascend 910 series usually has only 40 or 48 vector cores. If the number of grids exceeds 40 or 48 vector cores, the grids will be delivered in a queue, resulting in a long waiting time. Therefore, the number of cores for high-performance implementation does not exceed the number of vector cores. 2. Try to use up all the UB as much as possible. Move a large block size at a time to ensure that the bound is in the MTE. No Redundant Copy. 3. If the offset is a negative number, the current triton-ascend considers it as a discrete memory access scenario. As a result, the performance severely deteriorates, and the data is read from the entire DMA block instead of being read in scalar mode. -4. The UB of the Ascend hardware requires that the size of the tail axis of the tensor can be exactly divided by 32 bytes. If the length of the tail axis is insufficient, the length of the tail axis is automatically supplemented. For example, the performance deteriorates exponentially due to automatic supplementation for the Tensor whose shape is (2048, 3). In this situation, you can perform the transposition operation to change the alignment axis to a lower dimension. In addition, the transposition operation is affected by the automatic supplement rule. Therefore, special skills are also required to avoid supplementation. +4. The UB of the Ascend hardware requires that the size of the tail axis of the tensor can be exactly divided by 32bytes. If the length of the tail axis is insufficient, the length of the tail axis is automatically supplemented. For example, the performance deteriorates exponentially due to automatic supplementation for the Tensor whose shape is (2048, 3). In this situation, you can perform the transposition operation to change the alignment axis to a lower dimension. In addition, the transposition operation is affected by the automatic supplement rule. Therefore, special skills are also required to avoid supplementation. 5. Use Double Buffer, parallelizes computation and data transfer. While computing one block of data, another block of data is being transferred to L1. 6. If hostbound behavior is severe, core binding can be used to address it. diff --git a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_optimization.mdx b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_optimization.mdx index 552c6cd80..17e7b4ae9 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_optimization.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_optimization.mdx @@ -137,7 +137,7 @@ The following system-level tuning steps reduce OS interference and improve CPU s `TASK_QUEUE_ENABLE` - Controls the ASCEND Runtime task queue optimization level: `0` = disabled, `1` = default optimization, `2` = aggressive optimization with greater task fusion and overlap. Higher levels improve throughput but may interfere with CUDA Graph-launched tasks. Start with `1` for general use. Use `2` for throughput-critical prefill workloads; use `0` for decode where CUDA Graph compatibility is needed. + Controls the ASCEND Runtime task queue optimization level: `0` = disabled, `1` = default optimization, `2` = aggressive optimization with greater task fusion and overlap. Higher levels improve throughput but may interfere with NPU Graph-launched tasks. Start with `1` for general use. Use `2` for throughput-critical prefill workloads; use `0` for decode where NPU Graph compatibility is needed. Prefill: `2`, Decode: `0` @@ -742,7 +742,7 @@ Each device holds a subset of experts and routes tokens via all-to-all communica ### Context Parallelism (`--attn-cp-size`) -Splits long input sequences across devices so that the KV cache and attention computation for a single request are distributed. This enables serving very long context lengths (e.g., 128K tokens) that exceed single-device memory. On Ascend, `--attn-cp-size` must equal `--tp-size`. +Splits long input sequences across devices so that the KV cache and attention computation for a single request are distributed. This enables serving very long context lengths (e.g., 128k tokens) that exceed single-device memory. On Ascend, `--attn-cp-size` must equal `--tp-size`. ### PD Disaggregation (`--disaggregation-mode`) diff --git a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_quick_start.mdx b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_quick_start.mdx index ff868ea29..7908cf5d6 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_quick_start.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_quick_start.mdx @@ -20,7 +20,7 @@ see [SGLang installation with NPUs support](/docs/hardware-platforms/ascend-npus ## Setup environment using container -Ensure sufficient disk space before proceeding. The Docker image requires at least **30 GB** of free space. If you need to download model weights, check the model size at [ModelScope](https://www.modelscope.cn/models) to reserve enough space. +Ensure sufficient disk space before proceeding. The Docker image requires at least **30GB** of free space. If you need to download model weights, check the model size at [ModelScope](https://www.modelscope.cn/models) to reserve enough space. diff --git a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_support_features.mdx b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_support_features.mdx index ed60c0821..4c5b88f6b 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_support_features.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_support_features.mdx @@ -77,7 +77,7 @@ click [Server Arguments](../../advanced_features/server_arguments). A2, A3 - `--model-loader-`
`extra-config` + `--model-loader-extra-config` `{}` Type: str A2, A3 @@ -185,7 +185,7 @@ click [Server Arguments](../../advanced_features/server_arguments). `--grpc-mode` `False` - `False` + bool flag (set to enable) Planned @@ -300,13 +300,13 @@ click [Server Arguments](../../advanced_features/server_arguments). Special For GPU - `--modelopt-checkpoint-`
`restore-path` + `--modelopt-checkpoint-restore-path` `None` Type: str Special For GPU - `--modelopt-checkpoint-`
`save-path` + `--modelopt-checkpoint-save-path` `None` Type: str Special For GPU @@ -400,7 +400,7 @@ click [Server Arguments](../../advanced_features/server_arguments). A2, A3 - `--enable-priority-`
`scheduling` + `--enable-priority-scheduling` `False` bool flag
(set to enable) A2, A3 @@ -418,13 +418,13 @@ click [Server Arguments](../../advanced_features/server_arguments). A2, A3 - `--schedule-low-priority-`
`values-first` + `--schedule-low-priority-values-first` `False` bool flag
(set to enable) A2, A3 - `--priority-scheduling-`
`preemption-threshold` + `--priority-scheduling-preemption-threshold` `10` Type: int A2, A3 @@ -502,7 +502,7 @@ click [Server Arguments](../../advanced_features/server_arguments). A2, A3 - `--abort-on-priority-`
`when-disabled` + `--abort-on-priority-when-disabled` `False` bool flag
(set to enable) A2, A3 @@ -614,13 +614,13 @@ click [Server Arguments](../../advanced_features/server_arguments). A2, A3 - `--constrained-json-`
`whitespace-pattern` + `--constrained-json-whitespace-pattern` `None` Type: str A2, A3 - `--constrained-json-`
`disable-any-whitespace` + `--constrained-json-disable-any-whitespace` `False` bool flag (set to enable) A2, A3 @@ -763,19 +763,19 @@ click [Server Arguments](../../advanced_features/server_arguments). A2, A3 - `--enable-metrics-for-`
`all-schedulers` + `--enable-metrics-for-all-schedulers` `False` bool flag
(set to enable) A2, A3 - `--tokenizer-metrics-`
`custom-labels-header` + `--tokenizer-metrics-custom-labels-header` `x-custom-labels` Type: str A2, A3 - `--tokenizer-metrics-`
`allowed-custom-labels` + `--tokenizer-metrics-allowed-custom-labels` `None` List[str] A2, A3 @@ -787,25 +787,25 @@ click [Server Arguments](../../advanced_features/server_arguments). A2, A3 - `--bucket-time-to-`
`first-token` + `--bucket-time-to-first-token` `None` List[float] A2, A3 - `--bucket-inter-token-`
`latency` + `--bucket-inter-token-latency` `None` List[float] A2, A3 - `--bucket-e2e-request-`
`latency` + `--bucket-e2e-request-latency` `None` List[float] A2, A3 - `--collect-tokens-`
`histogram` + `--collect-tokens-histogram` `False` bool flag
(set to enable) A2, A3 @@ -835,7 +835,7 @@ click [Server Arguments](../../advanced_features/server_arguments). A2, A3 - `--enable-request-time-`
`stats-logging` + `--enable-request-time-stats-logging` `False` bool flag
(set to enable) A2, A3 @@ -917,13 +917,13 @@ click [Server Arguments](../../advanced_features/server_arguments). - `--export-metrics-to-`
`file` + `--export-metrics-to-file` `False` bool flag
(set to enable) A2, A3 - `--export-metrics-to-`
`file-dir` + `--export-metrics-to-file-dir` `None` Type: str A2, A3 @@ -1143,13 +1143,13 @@ click [Server Arguments](../../advanced_features/server_arguments). - `--json-model-override-`
`args` + `--json-model-override-args` `{}` Type: str A2, A3 - `--preferred-sampling-`
`params` + `--preferred-sampling-params` `None` Type: str A2, A3 @@ -1345,7 +1345,7 @@ click [Server Arguments](../../advanced_features/server_arguments). Special for GPU - `--disable-flashinfer-`
`autotune` + `--disable-flashinfer-autotune` `False` bool flag
(set to enable) Special for GPU @@ -1386,7 +1386,7 @@ click [Server Arguments](../../advanced_features/server_arguments). A2, A3 - `--speculative-draft-model-`
`revision` + `--speculative-draft-model-revision` `None` Type: str,
`branch name`,
`tag name`,
`commit id` A2, A3 @@ -1422,13 +1422,13 @@ click [Server Arguments](../../advanced_features/server_arguments). A2, A3 - `--speculative-accept-`
`threshold-single` + `--speculative-accept-threshold-single` `1.0` Type: float Special for GPU - `--speculative-accept-`
`threshold-acc` + `--speculative-accept-threshold-acc` `1.0` Type: float Special for GPU @@ -1440,19 +1440,19 @@ click [Server Arguments](../../advanced_features/server_arguments). A2, A3 - `--speculative-attention-`
`mode` + `--speculative-attention-mode` `prefill` `prefill`,
`decode` A2, A3 - `--speculative-moe-runner-`
`backend` + `--speculative-moe-runner-backend` `None` `auto` A2, A3 - `--speculative-moe-a2a-`
`backend` + `--speculative-moe-a2a-backend` `None` `ascend_fuseep` (the only supported value on Ascend NPU) A2, A3 @@ -1499,43 +1499,43 @@ click [Server Arguments](../../advanced_features/server_arguments). - `--speculative-ngram-`
`min-match-window-size` + `--speculative-ngram-min-match-window-size` `1` Type: int Experimental - `--speculative-ngram-`
`max-match-window-size` + `--speculative-ngram-max-match-window-size` `12` Type: int Experimental - `--speculative-ngram-`
`min-bfs-breadth` + `--speculative-ngram-min-bfs-breadth` `1` Type: int Experimental - `--speculative-ngram-`
`max-bfs-breadth` + `--speculative-ngram-max-bfs-breadth` `10` Type: int Experimental - `--speculative-ngram-`
`match-type` + `--speculative-ngram-match-type` `BFS` `BFS`,
`PROB` Experimental. `BFS` uses recency-based expansion; `PROB` uses frequency-based expansion. - `--speculative-ngram-`
`max-trie-depth` + `--speculative-ngram-max-trie-depth` `18` Type: int Experimental - `--speculative-ngram-`
`capacity` + `--speculative-ngram-capacity` `10000000` Type: int Experimental @@ -1618,13 +1618,13 @@ click [Server Arguments](../../advanced_features/server_arguments). Special for GPU - `--flashinfer-mxfp4-`
`moe-precision` + `--flashinfer-mxfp4-moe-precision` `default` `default`,
`bf16` Special for GPU - `--enable-flashinfer-`
`allreduce-fusion` + `--enable-flashinfer-allreduce-fusion` `False` bool flag
(set to enable) Special for GPU @@ -1685,31 +1685,31 @@ If the value is int8, you must also set the environment variable:DEEP_NORMAL_MOD A2, A3 - `--eplb-rebalance-layers-`
`per-chunk` + `--eplb-rebalance-layers-per-chunk` `None` Type: int A2, A3 - `--eplb-min-rebalancing-`
`utilization-threshold` + `--eplb-min-rebalancing-utilization-threshold` `1.0` Type: float A2, A3 - `--expert-distribution-`
`recorder-mode` + `--expert-distribution-recorder-mode` `None` `stat`,
`stat_approx`,
`per_pass`,
`per_token` A2, A3 - `--expert-distribution-`
`recorder-buffer-size` + `--expert-distribution-recorder-buffer-size` `None` Type: int A2, A3 - `--enable-expert-distribution-`
`metrics` + `--enable-expert-distribution-metrics` `False` bool flag (set to enable) A2, A3 @@ -1814,7 +1814,7 @@ If the value is int8, you must also set the environment variable:DEEP_NORMAL_MOD - `--enable-hierarchical-`
`cache` + `--enable-hierarchical-cache` `False` bool flag
(set to enable).
Currently, mamba cache is not supported. A2, A3 @@ -1850,19 +1850,19 @@ If the value is int8, you must also set the environment variable:DEEP_NORMAL_MOD A2, A3 - `--hicache-storage-`
`backend` + `--hicache-storage-backend` `None` `file` A2, A3 - `--hicache-storage-`
`prefetch-policy` + `--hicache-storage-prefetch-policy` `timeout` `best_effort`,
`wait_complete`,
`timeout` Special for GPU - `--hicache-storage-`
`backend-extra-config` + `--hicache-storage-backend-extra-config` `None` Type: str Special for GPU @@ -2072,12 +2072,6 @@ If the value is int8, you must also set the environment variable:DEEP_NORMAL_MOD bool flag
(set to enable) A2, A3 - - `--cuda-graph-max-bs-decode` - `None` - Type: int - A2, A3 - `--cuda-graph-bs` `None` @@ -2091,13 +2085,13 @@ If the value is int8, you must also set the environment variable:DEEP_NORMAL_MOD A2, A3 - `--disable-cuda-graph-`
`padding` + `--disable-cuda-graph-padding` `False` bool flag
(set to enable) A2, A3 - `--enable-profile-`
`cuda-graph` + `--enable-profile-cuda-graph` `False` bool flag
(set to enable) A2, A3 @@ -2121,25 +2115,25 @@ If the value is int8, you must also set the environment variable:DEEP_NORMAL_MOD Special for GPU - `--disable-flashinfer-`
`cutlass-moe-fp4-allgather` + `--disable-flashinfer-cutlass-moe-fp4-allgather` `False` bool flag
(set to enable) Special for GPU - `--enable-tokenizer-`
`batch-encode` + `--enable-tokenizer-batch-encode` `False` bool flag
(set to enable) A2, A3 - `--disable-tokenizer-`
`batch-decode` + `--disable-tokenizer-batch-decode` `False` bool flag
(set to enable) A2, A3 - `--disable-custom-`
`all-reduce` + `--disable-custom-all-reduce` `False` bool flag
(set to enable) Special for GPU @@ -2157,19 +2151,19 @@ If the value is int8, you must also set the environment variable:DEEP_NORMAL_MOD A2, A3 - `--enable-torch-`
`symm-mem` + `--enable-torch-symm-mem` `False` bool flag
(set to enable) Special for GPU - `--disable-overlap`
`-schedule` + `--disable-overlap-schedule` `False` bool flag
(set to enable) A2, A3 - `--enable-mixed-`
`chunk` + `--enable-mixed-chunk` `False` bool flag
(set to enable) A2, A3 @@ -2193,49 +2187,49 @@ If the value is int8, you must also set the environment variable:DEEP_NORMAL_MOD A2, A3 - `--enable-two-`
`batch-overlap` + `--enable-two-batch-overlap` `False` bool flag
(set to enable) Planned - `--enable-single-`
`batch-overlap` + `--enable-single-batch-overlap` `False` bool flag
(set to enable) Special for GPU - `--tbo-token-`
`distribution-threshold` + `--tbo-token-distribution-threshold` `0.48` Type: float Planned - `--enable-torch-`
`compile` + `--enable-torch-compile` `False` bool flag
(set to enable) A2, A3 - `--enable-torch-`
`compile-debug-mode` + `--enable-torch-compile-debug-mode` `False` bool flag
(set to enable) A2, A3 - `--enforce-piecewise-`
`cuda-graph` + `--enforce-piecewise-cuda-graph` `False` bool flag
(set to enable);
Currently, Llama-3.1-8B-Instruct and Qwen2.5-7B-Instruct models are supported. A2, A3 - `--piecewise-cuda-`
`graph-tokens` + `--piecewise-cuda-graph-tokens` `None` Type: JSON
list A2, A3 - `--piecewise-cuda-`
`graph-compiler` + `--piecewise-cuda-graph-compiler` `eager` `eager` A2, A3 @@ -2247,7 +2241,7 @@ If the value is int8, you must also set the environment variable:DEEP_NORMAL_MOD A2, A3 - `--piecewise-cuda-`
`graph-max-tokens` + `--piecewise-cuda-graph-max-tokens` `None` Type: int A2, A3 @@ -2265,25 +2259,25 @@ If the value is int8, you must also set the environment variable:DEEP_NORMAL_MOD Special for GPU - `--triton-attention-`
`reduce-in-fp32` + `--triton-attention-reduce-in-fp32` `False` bool flag
(set to enable) Special for GPU - `--triton-attention-`
`num-kv-splits` + `--triton-attention-num-kv-splits` `8` Type: int Special for GPU - `--triton-attention-`
`split-tile-size` + `--triton-attention-split-tile-size` `None` Type: int Special for GPU - `--delete-ckpt-`
`after-loading` + `--delete-ckpt-after-loading` `False` bool flag
(set to enable) A2, A3 @@ -2295,13 +2289,13 @@ If the value is int8, you must also set the environment variable:DEEP_NORMAL_MOD A2, A3 - `--enable-weights-`
`cpu-backup` + `--enable-weights-cpu-backup` `False` bool flag
(set to enable) A2, A3 - `--enable-draft-weights-`
`cpu-backup` + `--enable-draft-weights-cpu-backup` `False` bool flag
(set to enable) A2, A3 @@ -2313,19 +2307,19 @@ If the value is int8, you must also set the environment variable:DEEP_NORMAL_MOD A2, A3 - `--enable-custom-`
`logit-processor` + `--enable-custom-logit-processor` `False` bool flag
(set to enable) A2, A3 - `--flashinfer-mla-`
`disable-ragged` + `--flashinfer-mla-disable-ragged` `False` bool flag
(set to enable) Special for GPU - `--disable-shared-`
`experts-fusion` + `--disable-shared-experts-fusion` `True` bool flag
(set to enable) A2, A3 @@ -2337,37 +2331,37 @@ If the value is int8, you must also set the environment variable:DEEP_NORMAL_MOD A2, A3 - `--disable-chunked-`
`prefix-cache` + `--disable-chunked-prefix-cache` `True` bool flag
(set to enable) A2, A3 - `--disable-fast-`
`image-processor` + `--disable-fast-image-processor` `False` bool flag
(set to enable) A2, A3 - `--keep-mm-feature-`
`on-device` + `--keep-mm-feature-on-device` `False` bool flag
(set to enable) A2, A3 - `--enable-return-`
`hidden-states` + `--enable-return-hidden-states` `False` bool flag
(set to enable) A2, A3 - `--enable-return-`
`routed-experts` + `--enable-return-routed-experts` `False` bool flag
(set to enable) A2, A3 - `--scheduler-recv-`
`interval` + `--scheduler-recv-interval` `1` Type: int A2, A3 @@ -2379,7 +2373,7 @@ If the value is int8, you must also set the environment variable:DEEP_NORMAL_MOD A2, A3 - `--enable-deterministic-`
`inference` + `--enable-deterministic-inference` `False` bool flag
(set to enable) A2, A3 @@ -2397,13 +2391,13 @@ If the value is int8, you must also set the environment variable:DEEP_NORMAL_MOD A2, A3 - `--enable-layerwise-`
`nvtx-marker` + `--enable-layerwise-nvtx-marker` `False` bool flag
(set to enable) Special for GPU - `--enable-attn-tp-`
`input-scattered` + `--enable-attn-tp-input-scattered` `False` bool flag
(set to enable) Experimental @@ -2421,7 +2415,7 @@ If the value is int8, you must also set the environment variable:DEEP_NORMAL_MOD A2, A3 - `--enable-fused-qk-`
`norm-rope` + `--enable-fused-qk-norm-rope` `False` bool flag
(set to enable) Special for GPU @@ -2461,19 +2455,19 @@ If the value is int8, you must also set the environment variable:DEEP_NORMAL_MOD - `--enable-dynamic-`
`batch-tokenizer` + `--enable-dynamic-batch-tokenizer` `False` bool flag
(set to enable) A2, A3 - `--dynamic-batch-`
`tokenizer-batch-size` + `--dynamic-batch-tokenizer-batch-size` `32` Type: int A2, A3 - `--dynamic-batch-`
`tokenizer-batch-timeout` + `--dynamic-batch-tokenizer-batch-timeout` `0.002` Type: float A2, A3 @@ -2501,19 +2495,19 @@ If the value is int8, you must also set the environment variable:DEEP_NORMAL_MOD - `--debug-tensor-dump-`
`output-folder` + `--debug-tensor-dump-output-folder` `None` Type: str A2, A3 - `--debug-tensor-dump-`
`layers` + `--debug-tensor-dump-layers` `None` List[int] A2, A3 - `--debug-tensor-dump-`
`input-file` + `--debug-tensor-dump-input-file` `None` Type: str A2, A3 @@ -2566,7 +2560,7 @@ If the value is int8, you must also set the environment variable:DEEP_NORMAL_MOD Special for GPU - `--disaggregation-decode-`
`enable-offload-kvcache` + `--disaggregation-decode-enable-offload-kvcache` `False` bool flag
(set to enable) A2, A3 @@ -2578,7 +2572,7 @@ If the value is int8, you must also set the environment variable:DEEP_NORMAL_MOD A2, A3 - `--disaggregation-decode-`
`polling-interval` + `--disaggregation-decode-polling-interval` `1` Type: int A2, A3 @@ -2683,7 +2677,7 @@ If the value is int8, you must also set the environment variable:DEEP_NORMAL_MOD A2, A3 - `--weight-loader-disable-`
`mmap` + `--weight-loader-disable-mmap` `False` bool flag
(set to enable) A2, A3 @@ -2701,25 +2695,25 @@ If the value is int8, you must also set the environment variable:DEEP_NORMAL_MOD A2, A3 - `--remote-instance-weight-`
`loader-seed-instance-ip` + `--remote-instance-weight-loader-seed-instance-ip` `None` Type: str Special for GPU - `--remote-instance-weight-`
`loader-seed-instance-service-port` + `--remote-instance-weight-loader-seed-instance-service-port` `None` Type: int Special for GPU - `--remote-instance-weight-`
`loader-send-weights-group-ports` + `--remote-instance-weight-loader-send-weights-group-ports` `None` Type: JSON
list Special for GPU - `--remote-instance-weight-`
`loader-backend` + `--remote-instance-weight-loader-backend` `nccl` `transfer_engine`,
`nccl` Special for GPU @@ -2731,7 +2725,7 @@ If the value is int8, you must also set the environment variable:DEEP_NORMAL_MOD A2, A3 - `--remote-instance-weight-`
`loader-start-seed-via-transfer-engine` + `--remote-instance-weight-loader-start-seed-via-transfer-engine` `False` bool flag
(set to enable) Special for GPU @@ -2800,7 +2794,7 @@ If the value is int8, you must also set the environment variable:DEEP_NORMAL_MOD - `--enable-broadcast-mm-`
`inputs-process` + `--enable-broadcast-mm-inputs-process` `False` bool flag
(set to enable) A2, A3 @@ -2977,7 +2971,7 @@ NPU, like Ktransformer, checkpoint-engine etc. - `--checkpoint-engine-`
`wait-weights-`
`before-ready` + `--checkpoint-engine-wait-weights-before-ready` `False` bool flag (set to enable) @@ -3007,14 +3001,14 @@ NPU, like Ktransformer, checkpoint-engine etc. Type: int - `--kt-max-deferred-`
`experts-per-token` + `--kt-max-deferred-experts-per-token` `None` Type: int -The following parameters have some functional deficiencies on community +The following parameters have known functional deficiencies in the upstream community codebase diff --git a/docs_new/docs/hardware-platforms/ascend-npus/best_practice/deepseek_r1.mdx b/docs_new/docs/hardware-platforms/ascend-npus/best_practice/deepseek_r1.mdx index df807da78..e44f72674 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/best_practice/deepseek_r1.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/best_practice/deepseek_r1.mdx @@ -14,18 +14,18 @@ On A3 each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration | | --- | --- | --- | --- | --- | --- | --- | --- | -| DeepSeek-R1 | Atlas 800I A3 | 32 | PD Disaggregation | 3.5K+1.5K | 16ms | W8A8 INT8 | [Optimal Configuration](#deepseek-r1-w8a8-2p1d-32p-in3k5-out1k5-16ms) | -| DeepSeek-R1 | Atlas 800I A3 | 32 | PD Disaggregation | 3.5K+1K | 19.0ms | W8A8 INT8 | [Optimal Configuration](#deepseek-r1-w8a8-2p1d-32p-in3k5-out1k-19-0ms) | -| DeepSeek-R1 | Atlas 800I A3 | 32 | PD Disaggregation | 3.9K+1K | 19.0ms | W8A8 INT8 | [Optimal Configuration](#deepseek-r1-w8a8-2p1d-32p-in3k9-out1k-19-0ms) | -| DeepSeek-R1 | Atlas 800I A3 | 32 | PD Disaggregation | 6K+1.6K | 20.5ms | W8A8 INT8 | [Optimal Configuration](#deepseek-r1-w8a8-2p1d-32p-in6k-out1k6-20-5ms) | +| DeepSeek-R1 | Atlas 800I A3 | 32 | PD Disaggregation | 3.5k+1.5k | 16ms | W8A8 INT8 | [Optimal Configuration](#deepseek-r1-w8a8-2p1d-32p-in3k5-out1k5-16ms) | +| DeepSeek-R1 | Atlas 800I A3 | 32 | PD Disaggregation | 3.5k+1k | 19.0ms | W8A8 INT8 | [Optimal Configuration](#deepseek-r1-w8a8-2p1d-32p-in3k5-out1k-19-0ms) | +| DeepSeek-R1 | Atlas 800I A3 | 32 | PD Disaggregation | 3.9k+1k | 19.0ms | W8A8 INT8 | [Optimal Configuration](#deepseek-r1-w8a8-2p1d-32p-in3k9-out1k-19-0ms) | +| DeepSeek-R1 | Atlas 800I A3 | 32 | PD Disaggregation | 6k+1.6k | 20.5ms | W8A8 INT8 | [Optimal Configuration](#deepseek-r1-w8a8-2p1d-32p-in6k-out1k6-20-5ms) | ### High Throughput | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration | | --- | --- | --- | --- | --- | --- | --- | --- | -| DeepSeek-R1 | Atlas 800I A3 | 16 | PD Disaggregation | 3.5K+1.5K | 50ms | W4A8 INT8 | [Optimal Configuration](#deepseek-r1-w4a8-1p1d-16p-in3k5-out1k5-50ms) | -| DeepSeek-R1 | Atlas 800I A3 | 8 | PD Mixed | 3.5K+1.5K | 50ms | W4A8 INT8 | [Optimal Configuration](#deepseek-r1-w4a8-8p-in3k5-out1k5-50ms) | -| DeepSeek-R1 | Atlas 800I A3 | 32 | PD Disaggregation | 3.5K+1.5K | 50ms | W8A8 INT8 | [Optimal Configuration](#deepseek-r1-w8a8-2p1d-32p-in3k5-out1k5-50ms) | +| DeepSeek-R1 | Atlas 800I A3 | 16 | PD Disaggregation | 3.5k+1.5k | 50ms | W4A8 INT8 | [Optimal Configuration](#deepseek-r1-w4a8-1p1d-16p-in3k5-out1k5-50ms) | +| DeepSeek-R1 | Atlas 800I A3 | 8 | PD Mixed | 3.5k+1.5k | 50ms | W4A8 INT8 | [Optimal Configuration](#deepseek-r1-w4a8-8p-in3k5-out1k5-50ms) | +| DeepSeek-R1 | Atlas 800I A3 | 32 | PD Disaggregation | 3.5k+1.5k | 50ms | W8A8 INT8 | [Optimal Configuration](#deepseek-r1-w8a8-2p1d-32p-in3k5-out1k5-50ms) | ## Optimal Configuration @@ -41,7 +41,7 @@ On A3 each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend **Quantization**: W4A8 INT8 -**Dataset**: 3.5K+1.5K +**Dataset**: 3.5k+1.5k **TPOT**: 50ms @@ -241,7 +241,7 @@ python -m sglang.bench_serving \ **Quantization**: W4A8 INT8 -**Dataset**: 3.5K+1.5K +**Dataset**: 3.5k+1.5k **TPOT**: 50ms @@ -347,7 +347,7 @@ python -m sglang.bench_serving \ **Quantization**: W8A8 INT8 -**Dataset**: 3.5K+1.5K +**Dataset**: 3.5k+1.5k **TPOT**: 16ms @@ -552,7 +552,7 @@ python -m sglang.bench_serving \ **Quantization**: W8A8 INT8 -**Dataset**: 3.5K+1.5K +**Dataset**: 3.5k+1.5k **TPOT**: 50ms @@ -761,7 +761,7 @@ python -m sglang.bench_serving \ **Quantization**: W8A8 INT8 -**Dataset**: 3.5K+1K +**Dataset**: 3.5k+1k **TPOT**: 19.0ms @@ -966,7 +966,7 @@ python -m sglang.bench_serving \ **Quantization**: W8A8 INT8 -**Dataset**: 3.9K+1K +**Dataset**: 3.9k+1k **TPOT**: 19.0ms @@ -1171,7 +1171,7 @@ python -m sglang.bench_serving \ **Quantization**: W8A8 INT8 -**Dataset**: 6K+1.6K +**Dataset**: 6k+1.6k **TPOT**: 20.5ms diff --git a/docs_new/docs/hardware-platforms/ascend-npus/best_practice/deepseek_v3_2.mdx b/docs_new/docs/hardware-platforms/ascend-npus/best_practice/deepseek_v3_2.mdx index a9a027990..0de937d22 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/best_practice/deepseek_v3_2.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/best_practice/deepseek_v3_2.mdx @@ -14,14 +14,14 @@ On A3 each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration | | --- | --- | --- | --- | --- | --- | --- | --- | -| DeepSeek-V3.2 | Atlas 800I A3 | 32 | PD Disaggregation | 128K+1K | 26ms | W8A8 INT8 | [Optimal Configuration](#deepseek-v3-2-w8a8-1p1d-32p-in128k-out1k-26ms) | -| DeepSeek-V3.2 | Atlas 800I A3 | 32 | PD Disaggregation | 128K+1K | 26ms | W8A8 INT8 | [Optimal Configuration](#deepseek-v3-2-w8a8-1p1d-32p-in128k-out1k-bs8) | +| DeepSeek-V3.2 | Atlas 800I A3 | 32 | PD Disaggregation | 128k+1k | 26ms | W8A8 INT8 | [Optimal Configuration](#deepseek-v3-2-w8a8-1p1d-32p-in128k-out1k-26ms) | +| DeepSeek-V3.2 | Atlas 800I A3 | 32 | PD Disaggregation | 128k+1k | 26ms | W8A8 INT8 | [Optimal Configuration](#deepseek-v3-2-w8a8-1p1d-32p-in128k-out1k-bs8) | ### High Throughput | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration | | --- | --- | --- | --- | --- | --- | --- | --- | -| DeepSeek-V3.2 | Atlas 800I A3 | 32 | PD Disaggregation | 128K+1K | 107ms | W8A8 INT8 | [Optimal Configuration](#deepseek-v3-2-w8a8-1p1d-32p-in128k-out1k-bs16) | +| DeepSeek-V3.2 | Atlas 800I A3 | 32 | PD Disaggregation | 128k+1k | 107ms | W8A8 INT8 | [Optimal Configuration](#deepseek-v3-2-w8a8-1p1d-32p-in128k-out1k-bs16) | ## Optimal Configuration @@ -39,7 +39,7 @@ On A3 each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend **Quantization**: W8A8 INT8 -**Dataset**: 128K+1K +**Dataset**: 128k+1k **TPOT**: 26ms @@ -238,7 +238,7 @@ python -m sglang.bench_serving \ **Quantization**: W8A8 INT8 -**Dataset**: 128K+1K +**Dataset**: 128k+1k **TPOT**: 107ms @@ -437,7 +437,7 @@ python -m sglang.bench_serving \ **Quantization**: W8A8 INT8 -**Dataset**: 128K+1K +**Dataset**: 128k+1k **TPOT**: 26ms diff --git a/docs_new/docs/hardware-platforms/ascend-npus/best_practice/glm5_1.mdx b/docs_new/docs/hardware-platforms/ascend-npus/best_practice/glm5_1.mdx index 90f69ac6e..6a10dcb50 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/best_practice/glm5_1.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/best_practice/glm5_1.mdx @@ -14,20 +14,20 @@ On A3 each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | TTFT | Quantization | Configuration | | --- | --- | --- | --- | --- | --- | --- | --- | --- | -| GLM-5.1 | Atlas 800I A3 | 32 | PD Disaggregation | 65K+1.5K (90% prefix cache hit rate) | 25ms | - | W4A8 INT8 | [Optimal Configuration](#glm-5-1-w4a8-1p1d-32p-in65k-out1k5-prefix90-25ms) | +| GLM-5.1 | Atlas 800I A3 | 32 | PD Disaggregation | 65k+1.5k (90% prefix cache hit rate) | 25ms | - | W4A8 INT8 | [Optimal Configuration](#glm-5-1-w4a8-1p1d-32p-in65k-out1k5-prefix90-25ms) | ### High Throughput | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | TTFT | Quantization | Configuration | | --- | --- | --- | --- | --- | --- | --- | --- | --- | -| GLM-5.1 | Atlas 800I A3 | 16 | PD Mixed | 3.5K+1.5K | 50ms | - | W4A8 INT8 | [Optimal Configuration](#glm-5-1-w4a8-16p-in3k5-out1k5-50ms) | -| GLM-5.1 | Atlas 800I A3 | 32 | PD Disaggregation | 128K+1K | 56.4ms | 13.1s | W4A8 INT8 | [Optimal Configuration](#glm-5-1-w4a8-1p1d-32p-in128k-out1k-56-4ms) | -| GLM-5.1 | Atlas 800I A3 | 32 | PD Disaggregation | 16K+1K | 50ms | - | W4A8 INT8 | [Optimal Configuration](#glm-5-1-w4a8-1p1d-32p-in16k-out1k-50ms) | -| GLM-5.1 | Atlas 800I A3 | 32 | PD Disaggregation | 64K+1K | 55.2ms | 7.58s | W4A8 INT8 | [Optimal Configuration](#glm-5-1-w4a8-1p1d-32p-in64k-out1k-55-2ms) | -| GLM-5.1 | Atlas 800I A3 | 32 | PD Disaggregation | 64K+1K | 50ms | - | W4A8 INT8 | [Optimal Configuration](#glm-5-1-w4a8-1p1d-32p-in64k-out1k-50ms) | -| GLM-5.1 | Atlas 800I A3 | 48 | PD Disaggregation | 65K+1.5K (100% prefix cache hit rate) | 33ms | - | W4A8 INT8 | [Optimal Configuration](#glm-5-1-w4a8-1p1d-48p-in65k-out1k5-prefix100-33ms) | -| GLM-5.1 | Atlas 800I A3 | 48 | PD Disaggregation | 128K+1K (90% prefix cache hit rate) | 50ms | - | W4A8 INT8 | [Optimal Configuration](#glm-5-1-w4a8-2p1d-48p-in128k-out1k-prefix90-50ms) | -| GLM-5.1 | Atlas 800I A3 | 48 | PD Disaggregation | 64K+1K (90% prefix cache hit rate) | 50ms | - | W4A8 INT8 | [Optimal Configuration](#glm-5-1-w4a8-4p1d-48p-in64k-out1k-prefix90-50ms) | +| GLM-5.1 | Atlas 800I A3 | 16 | PD Mixed | 3.5k+1.5k | 50ms | - | W4A8 INT8 | [Optimal Configuration](#glm-5-1-w4a8-16p-in3k5-out1k5-50ms) | +| GLM-5.1 | Atlas 800I A3 | 32 | PD Disaggregation | 128k+1k | 56.4ms | 13.1s | W4A8 INT8 | [Optimal Configuration](#glm-5-1-w4a8-1p1d-32p-in128k-out1k-56-4ms) | +| GLM-5.1 | Atlas 800I A3 | 32 | PD Disaggregation | 16k+1k | 50ms | - | W4A8 INT8 | [Optimal Configuration](#glm-5-1-w4a8-1p1d-32p-in16k-out1k-50ms) | +| GLM-5.1 | Atlas 800I A3 | 32 | PD Disaggregation | 64k+1k | 55.2ms | 7.58s | W4A8 INT8 | [Optimal Configuration](#glm-5-1-w4a8-1p1d-32p-in64k-out1k-55-2ms) | +| GLM-5.1 | Atlas 800I A3 | 32 | PD Disaggregation | 64k+1k | 50ms | - | W4A8 INT8 | [Optimal Configuration](#glm-5-1-w4a8-1p1d-32p-in64k-out1k-50ms) | +| GLM-5.1 | Atlas 800I A3 | 48 | PD Disaggregation | 65k+1.5k (100% prefix cache hit rate) | 33ms | - | W4A8 INT8 | [Optimal Configuration](#glm-5-1-w4a8-1p1d-48p-in65k-out1k5-prefix100-33ms) | +| GLM-5.1 | Atlas 800I A3 | 48 | PD Disaggregation | 128k+1k (90% prefix cache hit rate) | 50ms | - | W4A8 INT8 | [Optimal Configuration](#glm-5-1-w4a8-2p1d-48p-in128k-out1k-prefix90-50ms) | +| GLM-5.1 | Atlas 800I A3 | 48 | PD Disaggregation | 64k+1k (90% prefix cache hit rate) | 50ms | - | W4A8 INT8 | [Optimal Configuration](#glm-5-1-w4a8-4p1d-48p-in64k-out1k-prefix90-50ms) | ## Optimal Configuration @@ -45,7 +45,7 @@ On A3 each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend **Quantization**: W4A8 INT8 -**Dataset**: 3.5K+1.5K +**Dataset**: 3.5k+1.5k **TPOT**: 50ms @@ -163,7 +163,7 @@ python -m sglang.bench_serving \ **Quantization**: W4A8 INT8 -**Dataset**: 128K+1K +**Dataset**: 128k+1k **TPOT**: 56.4ms @@ -368,7 +368,7 @@ python -m sglang.bench_serving \ **Quantization**: W4A8 INT8 -**Dataset**: 16K+1K +**Dataset**: 16k+1k **TPOT**: 50ms @@ -573,7 +573,7 @@ python -m sglang.bench_serving \ **Quantization**: W4A8 INT8 -**Dataset**: 64K+1K +**Dataset**: 64k+1k **TPOT**: 55.2ms @@ -778,7 +778,7 @@ python -m sglang.bench_serving \ **Quantization**: W4A8 INT8 -**Dataset**: 64K+1K +**Dataset**: 64k+1k **TPOT**: 50ms @@ -981,7 +981,7 @@ python -m sglang.bench_serving \ **Quantization**: W4A8 INT8 -**Dataset**: 65K+1.5K (90% prefix cache hit rate) +**Dataset**: 65k+1.5k (90% prefix cache hit rate) **TPOT**: 25ms @@ -1161,7 +1161,7 @@ python -m sglang_router.launch_router \ We tested it based on the `generated-shared-prefix` dataset with 90% cache hit (`repeat_rate = 0.9`): `--gsp-system-prompt-len 59904` = `int(66560 * 0.9)` is the shared prefix portion. -`--gsp-question-len 6655` = `int(66560 * (1 - 0.9))` is the unique per-request suffix. +`--gsp-question-len 6656` = `int(66560 * (1 - 0.9))` is the unique per-request suffix. `--gsp-num-groups 1` keeps all requests in one prefix group for maximum cache reuse. ```bash Command @@ -1173,7 +1173,7 @@ python -m sglang.bench_serving \ --gsp-num-groups 1 \ --gsp-prompts-per-group 480 \ --gsp-system-prompt-len 59904 \ - --gsp-question-len 6655 \ + --gsp-question-len 6656 \ --gsp-output-len 1536 \ --max-concurrency 100 \ --num-prompts 480 \ @@ -1192,7 +1192,7 @@ python -m sglang.bench_serving \ **Quantization**: W4A8 INT8 -**Dataset**: 65K+1.5K (100% prefix cache hit rate) +**Dataset**: 65k+1.5k (100% prefix cache hit rate) **TPOT**: 33ms @@ -1390,7 +1390,7 @@ python -m sglang.bench_serving \ **Quantization**: W4A8 INT8 -**Dataset**: 128K+1K (90% prefix cache hit rate) +**Dataset**: 128k+1k (90% prefix cache hit rate) **TPOT**: 50ms @@ -1604,7 +1604,7 @@ python -m sglang.bench_serving \ **Quantization**: W4A8 INT8 -**Dataset**: 64K+1K (90% prefix cache hit rate) +**Dataset**: 64k+1k (90% prefix cache hit rate) **TPOT**: 50ms diff --git a/docs_new/docs/hardware-platforms/ascend-npus/best_practice/kimi_k2_6.mdx b/docs_new/docs/hardware-platforms/ascend-npus/best_practice/kimi_k2_6.mdx index c40052d72..0fd91b4f3 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/best_practice/kimi_k2_6.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/best_practice/kimi_k2_6.mdx @@ -14,20 +14,20 @@ On A3 each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | TTFT | Quantization | Configuration | | --- | --- | --- | --- | --- | --- | --- | --- | --- | -| Kimi-K2.6 | Atlas 800I A3 | 8 | PD Mixed | 3.5K+1.5K | 20ms | - | W4A8 INT8 | [Optimal Configuration](#kimi-k2-6-w4a8-8p-in3k5-out1k5-20ms) | +| Kimi-K2.6 | Atlas 800I A3 | 8 | PD Mixed | 3.5k+1.5k | 20ms | - | W4A8 INT8 | [Optimal Configuration](#kimi-k2-6-w4a8-8p-in3k5-out1k5-20ms) | ### High Throughput | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | TTFT | Quantization | Configuration | | --- | --- | --- | --- | --- | --- | --- | --- | --- | -| Kimi-K2.6 | Atlas 800I A3 | 16 | PD Mixed | 64K+1K | 100ms | - | W4A8 INT8 | [Optimal Configuration](#kimi-k2-6-w4a8-16p-in64k-out1k-100ms) | -| Kimi-K2.6 | Atlas 800I A3 | 16 | PD Disaggregation | 128K+1K | 100ms | - | W4A8 INT8 | [Optimal Configuration](#kimi-k2-6-w4a8-1p1d-16p-in128k-out1k-100ms) | -| Kimi-K2.6 | Atlas 800I A3 | 16 | PD Disaggregation | 128K+1K (90% prefix cache hit rate) | 100ms | 5s | W4A8 INT8 | [Optimal Configuration](#kimi-k2-6-w4a8-1p1d-16p-in128k-out1k-prefix90-100ms) | -| Kimi-K2.6 | Atlas 800I A3 | 16 | PD Disaggregation | 64K+1.5K | 100ms | - | W4A8 INT8 | [Optimal Configuration](#kimi-k2-6-w4a8-1p1d-16p-in64k-out1k5-100ms) | -| Kimi-K2.6 | Atlas 800I A3 | 16 | PD Disaggregation | 64K+1.5K (90% prefix cache hit rate) | 100ms | 3s | W4A8 INT8 | [Optimal Configuration](#kimi-k2-6-w4a8-1p1d-16p-in64k-out1k5-prefix90-100ms) | +| Kimi-K2.6 | Atlas 800I A3 | 16 | PD Mixed | 64k+1k | 100ms | - | W4A8 INT8 | [Optimal Configuration](#kimi-k2-6-w4a8-16p-in64k-out1k-100ms) | +| Kimi-K2.6 | Atlas 800I A3 | 16 | PD Disaggregation | 128k+1k | 100ms | - | W4A8 INT8 | [Optimal Configuration](#kimi-k2-6-w4a8-1p1d-16p-in128k-out1k-100ms) | +| Kimi-K2.6 | Atlas 800I A3 | 16 | PD Disaggregation | 128k+1k (90% prefix cache hit rate) | 100ms | 5s | W4A8 INT8 | [Optimal Configuration](#kimi-k2-6-w4a8-1p1d-16p-in128k-out1k-prefix90-100ms) | +| Kimi-K2.6 | Atlas 800I A3 | 16 | PD Disaggregation | 64k+1.5k | 100ms | - | W4A8 INT8 | [Optimal Configuration](#kimi-k2-6-w4a8-1p1d-16p-in64k-out1k5-100ms) | +| Kimi-K2.6 | Atlas 800I A3 | 16 | PD Disaggregation | 64k+1.5k (90% prefix cache hit rate) | 100ms | 3s | W4A8 INT8 | [Optimal Configuration](#kimi-k2-6-w4a8-1p1d-16p-in64k-out1k5-prefix90-100ms) | | Kimi-K2.6 | Atlas 800I A3 | 8 | PD Mixed | 1024x1024 (30)+1024 | 50ms | - | W4A8 INT8 | [Optimal Configuration](#kimi-k2-6-w4a8-8p-in1024x1024-30-out1024-50ms) | | Kimi-K2.6 | Atlas 800I A3 | 8 | PD Mixed | 1080p_30+256 | 50ms | - | W4A8 INT8 | [Optimal Configuration](#kimi-k2-6-w4a8-8p-in1080p-30-out256-50ms) | -| Kimi-K2.6 | Atlas 800I A3 | 8 | PD Mixed | 3.5K+1.5K | 50ms | - | W4A8 INT8 | [Optimal Configuration](#kimi-k2-6-w4a8-8p-in3k5-out1k5-50ms) | +| Kimi-K2.6 | Atlas 800I A3 | 8 | PD Mixed | 3.5k+1.5k | 50ms | - | W4A8 INT8 | [Optimal Configuration](#kimi-k2-6-w4a8-8p-in3k5-out1k5-50ms) | ## Optimal Configuration @@ -45,7 +45,7 @@ On A3 each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend **Quantization**: W4A8 INT8 -**Dataset**: 64K+1K +**Dataset**: 64k+1k **TPOT**: 100ms @@ -170,7 +170,7 @@ python -m sglang.bench_serving \ **Quantization**: W4A8 INT8 -**Dataset**: 128K+1K +**Dataset**: 128k+1k **TPOT**: 100ms @@ -367,7 +367,7 @@ python -m sglang.bench_serving \ **Quantization**: W4A8 INT8 -**Dataset**: 128K+1K (90% prefix cache hit rate) +**Dataset**: 128k+1k (90% prefix cache hit rate) **TPOT**: 100ms @@ -572,7 +572,7 @@ python -m sglang.bench_serving \ **Quantization**: W4A8 INT8 -**Dataset**: 64K+1.5K +**Dataset**: 64k+1.5k **TPOT**: 100ms @@ -767,7 +767,7 @@ python -m sglang.bench_serving \ **Quantization**: W4A8 INT8 -**Dataset**: 64K+1.5K (90% prefix cache hit rate) +**Dataset**: 64k+1.5k (90% prefix cache hit rate) **TPOT**: 100ms @@ -1196,7 +1196,7 @@ python -m sglang.bench_serving \ **Quantization**: W4A8 INT8 -**Dataset**: 3.5K+1.5K +**Dataset**: 3.5k+1.5k **TPOT**: 20ms @@ -1309,7 +1309,7 @@ python -m sglang.bench_serving \ **Quantization**: W4A8 INT8 -**Dataset**: 3.5K+1.5K +**Dataset**: 3.5k+1.5k **TPOT**: 50ms diff --git a/docs_new/docs/hardware-platforms/ascend-npus/best_practice/mimo_v2_flash.mdx b/docs_new/docs/hardware-platforms/ascend-npus/best_practice/mimo_v2_flash.mdx index b4d4aa3c2..e2a80c1b5 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/best_practice/mimo_v2_flash.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/best_practice/mimo_v2_flash.mdx @@ -14,15 +14,15 @@ On A3 each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | TTFT | Quantization | Configuration | | --- | --- | --- | --- | --- | --- | --- | --- | --- | -| MiMo-V2-Flash | Atlas 800I A3 | 12 | PD Disaggregation | 16K+1K | 20ms | - | W8A8 INT8 | [Optimal Configuration](#mimo-v2-flash-1p1d-12p-in16k-out1k-tpot-20ms) | -| MiMo-V2-Flash | Atlas 800I A3 | 12 | PD Disaggregation | 32K+1K | 20ms | - | W8A8 INT8 | [Optimal Configuration](#mimo-v2-flash-1p1d-12p-in32k-out1k-tpot-20ms) | +| MiMo-V2-Flash | Atlas 800I A3 | 12 | PD Disaggregation | 16k+1k | 20ms | - | W8A8 INT8 | [Optimal Configuration](#mimo-v2-flash-1p1d-12p-in16k-out1k-tpot-20ms) | +| MiMo-V2-Flash | Atlas 800I A3 | 12 | PD Disaggregation | 32k+1k | 20ms | - | W8A8 INT8 | [Optimal Configuration](#mimo-v2-flash-1p1d-12p-in32k-out1k-tpot-20ms) | ### High Throughput | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | TTFT | Quantization | Configuration | | --- | --- | --- | --- | --- | --- | --- | --- | --- | -| MiMo-V2-Flash | Atlas 800I A3 | 12 | PD Disaggregation | 16K+1 | - | 5s | W8A8 INT8 | [Optimal Configuration](#mimo-v2-flash-1p1d-12p-in16k-out1-ttft-5s) | -| MiMo-V2-Flash | Atlas 800I A3 | 12 | PD Disaggregation | 32K+1 | - | 5s | W8A8 INT8 | [Optimal Configuration](#mimo-v2-flash-1p1d-12p-in32k-out1-ttft-5s) | +| MiMo-V2-Flash | Atlas 800I A3 | 12 | PD Disaggregation | 16k+1 | - | 5s | W8A8 INT8 | [Optimal Configuration](#mimo-v2-flash-1p1d-12p-in16k-out1-ttft-5s) | +| MiMo-V2-Flash | Atlas 800I A3 | 12 | PD Disaggregation | 32k+1 | - | 5s | W8A8 INT8 | [Optimal Configuration](#mimo-v2-flash-1p1d-12p-in32k-out1-ttft-5s) | ## Optimal Configuration @@ -40,7 +40,7 @@ On A3 each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend **Quantization**: W8A8 INT8 -**Dataset**: 16K+1 +**Dataset**: 16k+1 **TTFT**: 5s @@ -226,7 +226,7 @@ python -m sglang.bench_serving \ **Quantization**: W8A8 INT8 -**Dataset**: 16K+1K +**Dataset**: 16k+1k **TPOT**: 20ms @@ -412,7 +412,7 @@ python -m sglang.bench_serving \ **Quantization**: W8A8 INT8 -**Dataset**: 32K+1 +**Dataset**: 32k+1 **TTFT**: 5s @@ -598,7 +598,7 @@ python -m sglang.bench_serving \ **Quantization**: W8A8 INT8 -**Dataset**: 32K+1K +**Dataset**: 32k+1k **TPOT**: 20ms diff --git a/docs_new/docs/hardware-platforms/ascend-npus/best_practice/minimax_m2_5.mdx b/docs_new/docs/hardware-platforms/ascend-npus/best_practice/minimax_m2_5.mdx index 75c49a046..e7fe0333b 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/best_practice/minimax_m2_5.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/best_practice/minimax_m2_5.mdx @@ -14,16 +14,16 @@ On A3 each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration | | --- | --- | --- | --- | --- | --- | --- | --- | -| MiniMax-M2.5 | Atlas 800I A3 | 8 | PD Mixed | 128K+1K (90% prefix cache hit rate) | 24.44ms | W8A8 INT8 | [Optimal Configuration](#minimax-m2-5-w8a8-8p-in128k-out1k-prefix90-24-44ms) | -| MiniMax-M2.5 | Atlas 800I A3 | 8 | PD Mixed | 3.5K+1.5K | 20ms | W8A8 INT8 | [Optimal Configuration](#minimax-m2-5-w8a8-8p-in3k5-out1k5-20ms) | +| MiniMax-M2.5 | Atlas 800I A3 | 8 | PD Mixed | 128k+1k (90% prefix cache hit rate) | 24.44ms | W8A8 INT8 | [Optimal Configuration](#minimax-m2-5-w8a8-8p-in128k-out1k-prefix90-24-44ms) | +| MiniMax-M2.5 | Atlas 800I A3 | 8 | PD Mixed | 3.5k+1.5k | 20ms | W8A8 INT8 | [Optimal Configuration](#minimax-m2-5-w8a8-8p-in3k5-out1k5-20ms) | ### High Throughput | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration | | --- | --- | --- | --- | --- | --- | --- | --- | -| MiniMax-M2.5 | Atlas 800I A3 | 4 | PD Mixed | 32K+1K | 50ms | W8A8 INT8 | [Optimal Configuration](#minimax-m2-5-w8a8-4p-in32k-out1k-50ms) | -| MiniMax-M2.5 | Atlas 800I A3 | 4 | PD Mixed | 64K+1K (90% prefix cache hit rate) | 50ms | W8A8 INT8 | [Optimal Configuration](#minimax-m2-5-w8a8-4p-in64k-out1k-prefix90-50ms) | -| MiniMax-M2.5 | Atlas 800I A3 | 8 | PD Mixed | 3.5K+1.5K | 50ms | W8A8 INT8 | [Optimal Configuration](#minimax-m2-5-w8a8-8p-in3k5-out1k5-50ms) | +| MiniMax-M2.5 | Atlas 800I A3 | 4 | PD Mixed | 32k+1k | 50ms | W8A8 INT8 | [Optimal Configuration](#minimax-m2-5-w8a8-4p-in32k-out1k-50ms) | +| MiniMax-M2.5 | Atlas 800I A3 | 4 | PD Mixed | 64k+1k (90% prefix cache hit rate) | 50ms | W8A8 INT8 | [Optimal Configuration](#minimax-m2-5-w8a8-4p-in64k-out1k-prefix90-50ms) | +| MiniMax-M2.5 | Atlas 800I A3 | 8 | PD Mixed | 3.5k+1.5k | 50ms | W8A8 INT8 | [Optimal Configuration](#minimax-m2-5-w8a8-8p-in3k5-out1k5-50ms) | ## Optimal Configuration @@ -41,7 +41,7 @@ On A3 each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend **Quantization**: W8A8 INT8 -**Dataset**: 32K+1K +**Dataset**: 32k+1k **TPOT**: 50ms @@ -151,7 +151,7 @@ python -m sglang.bench_serving \ **Quantization**: W8A8 INT8 -**Dataset**: 64K+1K (90% prefix cache hit rate) +**Dataset**: 64k+1k (90% prefix cache hit rate) **TPOT**: 50ms @@ -260,7 +260,7 @@ python -m sglang.bench_serving \ **Quantization**: W8A8 INT8 -**Dataset**: 128K+1K (90% prefix cache hit rate) +**Dataset**: 128k+1k (90% prefix cache hit rate) **TPOT**: 24.44ms @@ -371,7 +371,7 @@ python -m sglang.bench_serving \ **Quantization**: W8A8 INT8 -**Dataset**: 3.5K+1.5K +**Dataset**: 3.5k+1.5k **TPOT**: 20ms @@ -476,7 +476,7 @@ python -m sglang.bench_serving \ **Quantization**: W8A8 INT8 -**Dataset**: 3.5K+1.5K +**Dataset**: 3.5k+1.5k **TPOT**: 50ms diff --git a/docs_new/docs/hardware-platforms/ascend-npus/best_practice/qwen3-8b.mdx b/docs_new/docs/hardware-platforms/ascend-npus/best_practice/qwen3-8b.mdx index b5e4d7f2e..4da722b70 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/best_practice/qwen3-8b.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/best_practice/qwen3-8b.mdx @@ -14,14 +14,14 @@ On A3 each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration | | --- | --- | --- | --- | --- | --- | --- | --- | -| Qwen3-8B | Atlas 800I A3 | 1 | PD Mixed | 3.5K+1.5K | 5ms | W8A8 INT8 | [Optimal Configuration](#qwen3-8b-w8a8-1p-in3k5-out1k5-5ms) | -| Qwen3-8B | Atlas 800I A3 | 1 | PD Mixed | 6K+1.5K | 11.79ms | W8A8 INT8 | [Optimal Configuration](#qwen3-8b-w8a8-1p-in6k-out1k5-bs16) | +| Qwen3-8B | Atlas 800I A3 | 1 | PD Mixed | 3.5k+1.5k | 5ms | W8A8 INT8 | [Optimal Configuration](#qwen3-8b-w8a8-1p-in3k5-out1k5-5ms) | +| Qwen3-8B | Atlas 800I A3 | 1 | PD Mixed | 6k+1.5k | 11.79ms | W8A8 INT8 | [Optimal Configuration](#qwen3-8b-w8a8-1p-in6k-out1k5-bs16) | ### High Throughput | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration | | --- | --- | --- | --- | --- | --- | --- | --- | -| Qwen3-8B | Atlas 800I A3 | 1 | PD Mixed | 3.5K+1.5K | 37ms | W8A8 INT8 | [Optimal Configuration](#qwen3-8b-w8a8-1p-in3k5-out1k5-37ms) | +| Qwen3-8B | Atlas 800I A3 | 1 | PD Mixed | 3.5k+1.5k | 37ms | W8A8 INT8 | [Optimal Configuration](#qwen3-8b-w8a8-1p-in3k5-out1k5-37ms) | ## Optimal Configuration @@ -39,7 +39,7 @@ On A3 each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend **Quantization**: W8A8 INT8 -**Dataset**: 3.5K+1.5K +**Dataset**: 3.5k+1.5k **TPOT**: 37ms @@ -137,7 +137,7 @@ python -m sglang.bench_serving \ **Quantization**: W8A8 INT8 -**Dataset**: 3.5K+1.5K +**Dataset**: 3.5k+1.5k **TPOT**: 5ms @@ -233,7 +233,7 @@ python -m sglang.bench_serving \ **Quantization**: W8A8 INT8 -**Dataset**: 6K+1.5K +**Dataset**: 6k+1.5k **TPOT**: 11.79ms diff --git a/docs_new/docs/hardware-platforms/ascend-npus/best_practice/qwen3_235b_a22b.mdx b/docs_new/docs/hardware-platforms/ascend-npus/best_practice/qwen3_235b_a22b.mdx index 505e0054d..f324e9c2e 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/best_practice/qwen3_235b_a22b.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/best_practice/qwen3_235b_a22b.mdx @@ -14,13 +14,13 @@ On A3 each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration | | --- | --- | --- | --- | --- | --- | --- | --- | -| Qwen3-235B-A22B | Atlas 800I A3 | 8 | PD Mixed | 11K+1.5K | 8ms | BF16 | [Optimal Configuration](#qwen3-235b-a22b-bf16-8p-in11k-out1k5-8ms) | +| Qwen3-235B-A22B | Atlas 800I A3 | 8 | PD Mixed | 11k+1.5k | 8ms | BF16 | [Optimal Configuration](#qwen3-235b-a22b-bf16-8p-in11k-out1k5-8ms) | ### High Throughput | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration | | --- | --- | --- | --- | --- | --- | --- | --- | -| Qwen3-235B-A22B | Atlas 800I A3 | 8 | PD Mixed | 3.5K+1.5K | 50.1ms | W8A8 INT8 | [Optimal Configuration](#qwen3-235b-a22b-w8a8-8p-in3k5-out1k5-50-1ms) | +| Qwen3-235B-A22B | Atlas 800I A3 | 8 | PD Mixed | 3.5k+1.5k | 50.1ms | W8A8 INT8 | [Optimal Configuration](#qwen3-235b-a22b-w8a8-8p-in3k5-out1k5-50-1ms) | ## Optimal Configuration @@ -38,7 +38,7 @@ On A3 each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend **Quantization**: BF16 -**Dataset**: 11K+1.5K +**Dataset**: 11k+1.5k **TPOT**: 8ms @@ -135,7 +135,7 @@ python -m sglang.bench_serving \ **Quantization**: W8A8 INT8 -**Dataset**: 3.5K+1.5K +**Dataset**: 3.5k+1.5k **TPOT**: 50.1ms diff --git a/docs_new/docs/hardware-platforms/ascend-npus/best_practice/qwen3_30b_a3b.mdx b/docs_new/docs/hardware-platforms/ascend-npus/best_practice/qwen3_30b_a3b.mdx index 4df7b29c6..74981b8ac 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/best_practice/qwen3_30b_a3b.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/best_practice/qwen3_30b_a3b.mdx @@ -14,15 +14,15 @@ On A3 each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration | | --- | --- | --- | --- | --- | --- | --- | --- | -| Qwen3-30B-A3B | Atlas 800I A3 | 1 | PD Mixed | 3.5K+1.5K | 10ms | W8A8 INT8 | [Optimal Configuration](#qwen3-30b-a3b-w8a8-1p-in3k5-out1k5-10ms) | -| Qwen3-30B-A3B | Atlas 800I A3 | 1 | PD Mixed | 6K+1.5K | 10.25ms | W8A8 INT8 | [Optimal Configuration](#qwen3-30b-a3b-w8a8-1p-in6k-out1k5-bs16) | +| Qwen3-30B-A3B | Atlas 800I A3 | 1 | PD Mixed | 3.5k+1.5k | 10ms | W8A8 INT8 | [Optimal Configuration](#qwen3-30b-a3b-w8a8-1p-in3k5-out1k5-10ms) | +| Qwen3-30B-A3B | Atlas 800I A3 | 1 | PD Mixed | 6k+1.5k | 10.25ms | W8A8 INT8 | [Optimal Configuration](#qwen3-30b-a3b-w8a8-1p-in6k-out1k5-bs16) | ### High Throughput | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration | | --- | --- | --- | --- | --- | --- | --- | --- | -| Qwen3-30B-A3B | Atlas 800I A3 | 1 | PD Mixed | 1K+100 | 10000ms | BF16 | [Optimal Configuration](#qwen3-30b-a3b-bf16-1p-in1k-out100) | -| Qwen3-30B-A3B | Atlas 800I A3 | 1 | PD Mixed | 3.5K+1.5K | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-30b-a3b-w8a8-1p-in3k5-out1k5-50ms) | +| Qwen3-30B-A3B | Atlas 800I A3 | 1 | PD Mixed | 1k+100 | 10000ms | BF16 | [Optimal Configuration](#qwen3-30b-a3b-bf16-1p-in1k-out100) | +| Qwen3-30B-A3B | Atlas 800I A3 | 1 | PD Mixed | 3.5k+1.5k | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-30b-a3b-w8a8-1p-in3k5-out1k5-50ms) | ## Optimal Configuration @@ -40,7 +40,7 @@ On A3 each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend **Quantization**: BF16 -**Dataset**: 1K+100 +**Dataset**: 1k+100 **TPOT**: 10000ms @@ -144,7 +144,7 @@ python -m sglang.bench_serving \ **Quantization**: W8A8 INT8 -**Dataset**: 3.5K+1.5K +**Dataset**: 3.5k+1.5k **TPOT**: 10ms @@ -244,7 +244,7 @@ python -m sglang.bench_serving \ **Quantization**: W8A8 INT8 -**Dataset**: 3.5K+1.5K +**Dataset**: 3.5k+1.5k **TPOT**: 50ms @@ -344,7 +344,7 @@ python -m sglang.bench_serving \ **Quantization**: W8A8 INT8 -**Dataset**: 6K+1.5K +**Dataset**: 6k+1.5k **TPOT**: 10.25ms diff --git a/docs_new/docs/hardware-platforms/ascend-npus/best_practice/qwen3_32b.mdx b/docs_new/docs/hardware-platforms/ascend-npus/best_practice/qwen3_32b.mdx index f750ab0ab..0fcc2ed96 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/best_practice/qwen3_32b.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/best_practice/qwen3_32b.mdx @@ -14,14 +14,14 @@ On A3 each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration | | --- | --- | --- | --- | --- | --- | --- | --- | -| Qwen3-32B | Atlas 800I A3 | 8 | PD Mixed | 18K+4K | 6ms | BF16 | [Optimal Configuration](#qwen3-32b-bf16-8p-in18k-out4k-6ms) | +| Qwen3-32B | Atlas 800I A3 | 8 | PD Mixed | 18k+4k | 6ms | BF16 | [Optimal Configuration](#qwen3-32b-bf16-8p-in18k-out4k-6ms) | ### High Throughput | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration | | --- | --- | --- | --- | --- | --- | --- | --- | -| Qwen3-32B | Atlas 800I A2 | 2 | PD Mixed | 3.5K+1.5K | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-32b-w8a8-2p-in3k5-out1k5-50ms-a2) | -| Qwen3-32B | Atlas 800I A3 | 2 | PD Mixed | 3.5K+1.5K | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-32b-w8a8-2p-in3k5-out1k5-50ms) | +| Qwen3-32B | Atlas 800I A2 | 2 | PD Mixed | 3.5k+1.5k | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-32b-w8a8-2p-in3k5-out1k5-50ms-a2) | +| Qwen3-32B | Atlas 800I A3 | 2 | PD Mixed | 3.5k+1.5k | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-32b-w8a8-2p-in3k5-out1k5-50ms) | ## Optimal Configuration @@ -39,7 +39,7 @@ On A3 each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend **Quantization**: BF16 -**Dataset**: 18K+4K +**Dataset**: 18k+4k **TPOT**: 6ms @@ -136,7 +136,7 @@ python -m sglang.bench_serving \ **Quantization**: W8A8 INT8 -**Dataset**: 3.5K+1.5K +**Dataset**: 3.5k+1.5k **TPOT**: 50ms @@ -235,7 +235,7 @@ python -m sglang.bench_serving \ **Quantization**: W8A8 INT8 -**Dataset**: 3.5K+1.5K +**Dataset**: 3.5k+1.5k **TPOT**: 50ms diff --git a/docs_new/docs/hardware-platforms/ascend-npus/best_practice/qwen3_5_397b.mdx b/docs_new/docs/hardware-platforms/ascend-npus/best_practice/qwen3_5_397b.mdx index f25a06764..45b53e24c 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/best_practice/qwen3_5_397b.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/best_practice/qwen3_5_397b.mdx @@ -14,21 +14,21 @@ On A3 each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration | | --- | --- | --- | --- | --- | --- | --- | --- | -| Qwen3.5-397B | Atlas 800I A3 | 8 | PD Mixed | 128K+1K | 20ms | W4A8 INT8 | [Optimal Configuration](#qwen3-5-397b-w4a8-8p-in128k-out1k-20ms) | -| Qwen3.5-397B | Atlas 800I A3 | 8 | PD Mixed | 16K+1K | 20ms | W4A8 INT8 | [Optimal Configuration](#qwen3-5-397b-w4a8-8p-in16k-out1k-20ms) | -| Qwen3.5-397B | Atlas 800I A3 | 8 | PD Mixed | 3.5K+1.5K | 20ms | W4A8 INT8 | [Optimal Configuration](#qwen3-5-397b-w4a8-8p-in3k5-out1k5-20ms) | -| Qwen3.5-397B | Atlas 800I A3 | 8 | PD Mixed | 64K+1K | 20ms | W4A8 INT8 | [Optimal Configuration](#qwen3-5-397b-w4a8-8p-in64k-out1k-20ms) | +| Qwen3.5-397B | Atlas 800I A3 | 8 | PD Mixed | 128k+1k | 20ms | W4A8 INT8 | [Optimal Configuration](#qwen3-5-397b-w4a8-8p-in128k-out1k-20ms) | +| Qwen3.5-397B | Atlas 800I A3 | 8 | PD Mixed | 16k+1k | 20ms | W4A8 INT8 | [Optimal Configuration](#qwen3-5-397b-w4a8-8p-in16k-out1k-20ms) | +| Qwen3.5-397B | Atlas 800I A3 | 8 | PD Mixed | 3.5k+1.5k | 20ms | W4A8 INT8 | [Optimal Configuration](#qwen3-5-397b-w4a8-8p-in3k5-out1k5-20ms) | +| Qwen3.5-397B | Atlas 800I A3 | 8 | PD Mixed | 64k+1k | 20ms | W4A8 INT8 | [Optimal Configuration](#qwen3-5-397b-w4a8-8p-in64k-out1k-20ms) | ### High Throughput | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration | | --- | --- | --- | --- | --- | --- | --- | --- | -| Qwen3.5-397B | Atlas 800I A3 | 8 | PD Mixed | 128K+1K | 50ms | W4A8 INT8 | [Optimal Configuration](#qwen3-5-397b-w4a8-8p-in128k-out1k-50ms) | -| Qwen3.5-397B | Atlas 800I A3 | 8 | PD Mixed | 128K+1K (90% prefix cache hit rate) | 50ms | W4A8 INT8 | [Optimal Configuration](#qwen3-5-397b-w4a8-8p-in128k-out1k-prefix90-50ms) | -| Qwen3.5-397B | Atlas 800I A3 | 8 | PD Mixed | 16K+1K | 50ms | W4A8 INT8 | [Optimal Configuration](#qwen3-5-397b-w4a8-8p-in16k-out1k-50ms) | -| Qwen3.5-397B | Atlas 800I A3 | 8 | PD Mixed | 3.5K+1.5K | 50ms | W4A8 INT8 | [Optimal Configuration](#qwen3-5-397b-w4a8-8p-in3k5-out1k5-50ms) | -| Qwen3.5-397B | Atlas 800I A3 | 8 | PD Mixed | 64K+1K | 50ms | W4A8 INT8 | [Optimal Configuration](#qwen3-5-397b-w4a8-8p-in64k-out1k-50ms) | -| Qwen3.5-397B | Atlas 800I A3 | 8 | PD Mixed | 64K+1K (90% prefix cache hit rate) | 50ms | W4A8 INT8 | [Optimal Configuration](#qwen3-5-397b-w4a8-8p-in64k-out1k-prefix90-50ms) | +| Qwen3.5-397B | Atlas 800I A3 | 8 | PD Mixed | 128k+1k | 50ms | W4A8 INT8 | [Optimal Configuration](#qwen3-5-397b-w4a8-8p-in128k-out1k-50ms) | +| Qwen3.5-397B | Atlas 800I A3 | 8 | PD Mixed | 128k+1k (90% prefix cache hit rate) | 50ms | W4A8 INT8 | [Optimal Configuration](#qwen3-5-397b-w4a8-8p-in128k-out1k-prefix90-50ms) | +| Qwen3.5-397B | Atlas 800I A3 | 8 | PD Mixed | 16k+1k | 50ms | W4A8 INT8 | [Optimal Configuration](#qwen3-5-397b-w4a8-8p-in16k-out1k-50ms) | +| Qwen3.5-397B | Atlas 800I A3 | 8 | PD Mixed | 3.5k+1.5k | 50ms | W4A8 INT8 | [Optimal Configuration](#qwen3-5-397b-w4a8-8p-in3k5-out1k5-50ms) | +| Qwen3.5-397B | Atlas 800I A3 | 8 | PD Mixed | 64k+1k | 50ms | W4A8 INT8 | [Optimal Configuration](#qwen3-5-397b-w4a8-8p-in64k-out1k-50ms) | +| Qwen3.5-397B | Atlas 800I A3 | 8 | PD Mixed | 64k+1k (90% prefix cache hit rate) | 50ms | W4A8 INT8 | [Optimal Configuration](#qwen3-5-397b-w4a8-8p-in64k-out1k-prefix90-50ms) | ## Optimal Configuration @@ -46,7 +46,7 @@ On A3 each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend **Quantization**: W4A8 INT8 -**Dataset**: 128K+1K +**Dataset**: 128k+1k **TPOT**: 20ms @@ -157,7 +157,7 @@ python -m sglang.bench_serving \ **Quantization**: W4A8 INT8 -**Dataset**: 128K+1K +**Dataset**: 128k+1k **TPOT**: 50ms @@ -268,7 +268,7 @@ python -m sglang.bench_serving \ **Quantization**: W4A8 INT8 -**Dataset**: 128K+1K (90% prefix cache hit rate) +**Dataset**: 128k+1k (90% prefix cache hit rate) **TPOT**: 50ms @@ -378,7 +378,7 @@ python -m sglang.bench_serving \ **Quantization**: W4A8 INT8 -**Dataset**: 16K+1K +**Dataset**: 16k+1k **TPOT**: 20ms @@ -493,7 +493,7 @@ python -m sglang.bench_serving \ **Quantization**: W4A8 INT8 -**Dataset**: 16K+1K +**Dataset**: 16k+1k **TPOT**: 50ms @@ -608,7 +608,7 @@ python -m sglang.bench_serving \ **Quantization**: W4A8 INT8 -**Dataset**: 3.5K+1.5K +**Dataset**: 3.5k+1.5k **TPOT**: 20ms @@ -722,7 +722,7 @@ python -m sglang.bench_serving \ **Quantization**: W4A8 INT8 -**Dataset**: 3.5K+1.5K +**Dataset**: 3.5k+1.5k **TPOT**: 50ms @@ -836,7 +836,7 @@ python -m sglang.bench_serving \ **Quantization**: W4A8 INT8 -**Dataset**: 64K+1K +**Dataset**: 64k+1k **TPOT**: 20ms @@ -951,7 +951,7 @@ python -m sglang.bench_serving \ **Quantization**: W4A8 INT8 -**Dataset**: 64K+1K +**Dataset**: 64k+1k **TPOT**: 50ms @@ -1066,7 +1066,7 @@ python -m sglang.bench_serving \ **Quantization**: W4A8 INT8 -**Dataset**: 64K+1K (90% prefix cache hit rate) +**Dataset**: 64k+1k (90% prefix cache hit rate) **TPOT**: 50ms diff --git a/docs_new/docs/hardware-platforms/ascend-npus/best_practice/qwen3_6_27b.mdx b/docs_new/docs/hardware-platforms/ascend-npus/best_practice/qwen3_6_27b.mdx index a56c82d1f..4a1709c0b 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/best_practice/qwen3_6_27b.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/best_practice/qwen3_6_27b.mdx @@ -16,11 +16,11 @@ On A3 each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend | --- | --- | --- | --- | --- | --- | --- | --- | | Qwen3.6-27B | Atlas 800I A3 | 1 | PD Mixed | 1024x1024 (30)+1024 | 50ms | BF16 | [Optimal Configuration](#qwen3-6-27b-1p-in1024x1024-30-out1024-50ms) | | Qwen3.6-27B | Atlas 800I A3 | 1 | PD Mixed | 1080p_30+256 | 50ms | BF16 | [Optimal Configuration](#qwen3-6-27b-1p-in1080p-30-out256-50ms) | -| Qwen3.6-27B | Atlas 800I A3 | 2 | PD Mixed | 64K+1K (90% prefix cache hit rate) | 50ms | BF16 | [Optimal Configuration](#qwen3-6-27b-2p-in64k-out1k-prefix90-50ms) | -| Qwen3.6-27B | Atlas 800I A3 | 1 | PD Mixed | 3.5K+1.5K | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-6-27b-w8a8-1p-in3k5-out1k5-50ms) | -| Qwen3.6-27B | Atlas 800I A3 | 1 | PD Mixed | 64K+1K | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-6-27b-w8a8-1p-in64k-out1k-50ms) | -| Qwen3.6-27B | Atlas 800I A3 | 2 | PD Mixed | 128K+1K | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-6-27b-w8a8-2p-in128k-out1k-50ms) | -| Qwen3.6-27B | Atlas 800I A3 | 2 | PD Mixed | 16K+1K | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-6-27b-w8a8-2p-in16k-out1k-50ms) | +| Qwen3.6-27B | Atlas 800I A3 | 2 | PD Mixed | 64k+1k (90% prefix cache hit rate) | 50ms | BF16 | [Optimal Configuration](#qwen3-6-27b-2p-in64k-out1k-prefix90-50ms) | +| Qwen3.6-27B | Atlas 800I A3 | 1 | PD Mixed | 3.5k+1.5k | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-6-27b-w8a8-1p-in3k5-out1k5-50ms) | +| Qwen3.6-27B | Atlas 800I A3 | 1 | PD Mixed | 64k+1k | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-6-27b-w8a8-1p-in64k-out1k-50ms) | +| Qwen3.6-27B | Atlas 800I A3 | 2 | PD Mixed | 128k+1k | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-6-27b-w8a8-2p-in128k-out1k-50ms) | +| Qwen3.6-27B | Atlas 800I A3 | 2 | PD Mixed | 16k+1k | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-6-27b-w8a8-2p-in16k-out1k-50ms) | ## Optimal Configuration @@ -238,7 +238,7 @@ python -m sglang.bench_serving \ **Quantization**: BF16 -**Dataset**: 64K+1K (90% prefix cache hit rate) +**Dataset**: 64k+1k (90% prefix cache hit rate) **TPOT**: 50ms @@ -341,7 +341,7 @@ python -m sglang.bench_serving \ **Quantization**: W8A8 INT8 -**Dataset**: 3.5K+1.5K +**Dataset**: 3.5k+1.5k **TPOT**: 50ms @@ -439,7 +439,7 @@ python -m sglang.bench_serving \ **Quantization**: W8A8 INT8 -**Dataset**: 64K+1K +**Dataset**: 64k+1k **TPOT**: 50ms @@ -531,7 +531,7 @@ python -m sglang.bench_serving \ **Quantization**: W8A8 INT8 -**Dataset**: 128K+1K +**Dataset**: 128k+1k **TPOT**: 50ms @@ -625,7 +625,7 @@ python -m sglang.bench_serving \ **Quantization**: W8A8 INT8 -**Dataset**: 16K+1K +**Dataset**: 16k+1k **TPOT**: 50ms diff --git a/docs_new/docs/hardware-platforms/ascend-npus/best_practice/qwen3_6_35b_a3b.mdx b/docs_new/docs/hardware-platforms/ascend-npus/best_practice/qwen3_6_35b_a3b.mdx index b8c8692ff..9fee6228e 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/best_practice/qwen3_6_35b_a3b.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/best_practice/qwen3_6_35b_a3b.mdx @@ -14,7 +14,7 @@ On A3 each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration | | --- | --- | --- | --- | --- | --- | --- | --- | -| Qwen3.6-35B-A3B | Atlas 800I A3 | 1 | PD Mixed | 254K+1K | 16.1ms | BF16 | [Optimal Configuration](#qwen3-6-35b-a3b-1p-in254k-out1k) | +| Qwen3.6-35B-A3B | Atlas 800I A3 | 1 | PD Mixed | 254k+1k | 16.1ms | BF16 | [Optimal Configuration](#qwen3-6-35b-a3b-1p-in254k-out1k) | ### High Throughput @@ -22,12 +22,12 @@ On A3 each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend | --- | --- | --- | --- | --- | --- | --- | --- | | Qwen3.6-35B-A3B | Atlas 800I A3 | 1 | PD Mixed | 1024x1024 (30)+1024 | 50ms | BF16 | [Optimal Configuration](#qwen3-6-35b-a3b-1p-in1024x1024-30-out1024-50ms) | | Qwen3.6-35B-A3B | Atlas 800I A3 | 1 | PD Mixed | 1080p_30+256 | 50ms | BF16 | [Optimal Configuration](#qwen3-6-35b-a3b-1p-in1080p-30-out256-50ms) | -| Qwen3.6-35B-A3B | Atlas 800I A3 | 1 | PD Mixed | 128K+1K | 50ms | BF16 | [Optimal Configuration](#qwen3-6-35b-a3b-1p-in128k-out1k-50ms) | -| Qwen3.6-35B-A3B | Atlas 800I A3 | 1 | PD Mixed | 128K+1K (90% prefix cache hit rate) | 50ms | BF16 | [Optimal Configuration](#qwen3-6-35b-a3b-1p-in128k-out1k-prefix90-50ms) | -| Qwen3.6-35B-A3B | Atlas 800I A3 | 1 | PD Mixed | 3.5K+1.5K | 50ms | BF16 | [Optimal Configuration](#qwen3-6-35b-a3b-1p-in3k5-out1k5-50ms) | -| Qwen3.6-35B-A3B | Atlas 800I A3 | 1 | PD Mixed | 64K+1K | 50ms | BF16 | [Optimal Configuration](#qwen3-6-35b-a3b-1p-in64k-out1k-50ms) | -| Qwen3.6-35B-A3B | Atlas 800I A3 | 1 | PD Mixed | 64K+1K (90% prefix cache hit rate) | 50ms | BF16 | [Optimal Configuration](#qwen3-6-35b-a3b-1p-in64k-out1k-prefix90-50ms) | -| Qwen3.6-35B-A3B | Atlas 800I A3 | 2 | PD Mixed | 984K+1K | 40.91ms | BF16 | [Optimal Configuration](#qwen3-6-35b-a3b-2p-in984k-out1k) | +| Qwen3.6-35B-A3B | Atlas 800I A3 | 1 | PD Mixed | 128k+1k | 50ms | BF16 | [Optimal Configuration](#qwen3-6-35b-a3b-1p-in128k-out1k-50ms) | +| Qwen3.6-35B-A3B | Atlas 800I A3 | 1 | PD Mixed | 128k+1k (90% prefix cache hit rate) | 50ms | BF16 | [Optimal Configuration](#qwen3-6-35b-a3b-1p-in128k-out1k-prefix90-50ms) | +| Qwen3.6-35B-A3B | Atlas 800I A3 | 1 | PD Mixed | 3.5k+1.5k | 50ms | BF16 | [Optimal Configuration](#qwen3-6-35b-a3b-1p-in3k5-out1k5-50ms) | +| Qwen3.6-35B-A3B | Atlas 800I A3 | 1 | PD Mixed | 64k+1k | 50ms | BF16 | [Optimal Configuration](#qwen3-6-35b-a3b-1p-in64k-out1k-50ms) | +| Qwen3.6-35B-A3B | Atlas 800I A3 | 1 | PD Mixed | 64k+1k (90% prefix cache hit rate) | 50ms | BF16 | [Optimal Configuration](#qwen3-6-35b-a3b-1p-in64k-out1k-prefix90-50ms) | +| Qwen3.6-35B-A3B | Atlas 800I A3 | 2 | PD Mixed | 984k+1k | 40.91ms | BF16 | [Optimal Configuration](#qwen3-6-35b-a3b-2p-in984k-out1k) | ## Optimal Configuration @@ -243,7 +243,7 @@ python -m sglang.bench_serving \ **Quantization**: BF16 -**Dataset**: 128K+1K +**Dataset**: 128k+1k **TPOT**: 50ms @@ -342,7 +342,7 @@ python -m sglang.bench_serving \ **Quantization**: BF16 -**Dataset**: 128K+1K (90% prefix cache hit rate) +**Dataset**: 128k+1k (90% prefix cache hit rate) **TPOT**: 50ms @@ -445,7 +445,7 @@ python -m sglang.bench_serving \ **Quantization**: BF16 -**Dataset**: 254K+1K +**Dataset**: 254k+1k **TPOT**: 16.1ms @@ -540,7 +540,7 @@ python -m sglang.bench_serving \ **Quantization**: BF16 -**Dataset**: 3.5K+1.5K +**Dataset**: 3.5k+1.5k **TPOT**: 50ms @@ -638,7 +638,7 @@ python -m sglang.bench_serving \ **Quantization**: BF16 -**Dataset**: 64K+1K +**Dataset**: 64k+1k **TPOT**: 50ms @@ -736,7 +736,7 @@ python -m sglang.bench_serving \ **Quantization**: BF16 -**Dataset**: 64K+1K (90% prefix cache hit rate) +**Dataset**: 64k+1k (90% prefix cache hit rate) **TPOT**: 50ms @@ -840,7 +840,7 @@ python -m sglang.bench_serving \ **Quantization**: BF16 -**Dataset**: 984K+1K +**Dataset**: 984k+1k **TPOT**: 40.91ms diff --git a/docs_new/docs/hardware-platforms/ascend-npus/best_practice/qwen3_next_80b_a3b_instruct.mdx b/docs_new/docs/hardware-platforms/ascend-npus/best_practice/qwen3_next_80b_a3b_instruct.mdx index a54229a87..2e90c26be 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/best_practice/qwen3_next_80b_a3b_instruct.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/best_practice/qwen3_next_80b_a3b_instruct.mdx @@ -14,14 +14,14 @@ On A3 each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration | | --- | --- | --- | --- | --- | --- | --- | --- | -| Qwen3-Next-80B-A3B-Instruct | Atlas 800I A3 | 2 | PD Mixed | 3.5K+1.5K | 20ms | W8A8 INT8 | [Optimal Configuration](#qwen3-next-80b-a3b-instruct-w8a8-2p-in3k5-out1k5-20ms) | -| Qwen3-Next-80B-A3B-Instruct | Atlas 800I A3 | 2 | PD Mixed | 6K+1.5K | 15.62ms | W8A8 INT8 | [Optimal Configuration](#qwen3-next-80b-a3b-instruct-w8a8-2p-in6k-out1k5-bs16) | +| Qwen3-Next-80B-A3B-Instruct | Atlas 800I A3 | 2 | PD Mixed | 3.5k+1.5k | 20ms | W8A8 INT8 | [Optimal Configuration](#qwen3-next-80b-a3b-instruct-w8a8-2p-in3k5-out1k5-20ms) | +| Qwen3-Next-80B-A3B-Instruct | Atlas 800I A3 | 2 | PD Mixed | 6k+1.5k | 15.62ms | W8A8 INT8 | [Optimal Configuration](#qwen3-next-80b-a3b-instruct-w8a8-2p-in6k-out1k5-bs16) | ### High Throughput | Model | Hardware | Cards | Deploy Mode | Dataset | TPOT | Quantization | Configuration | | --- | --- | --- | --- | --- | --- | --- | --- | -| Qwen3-Next-80B-A3B-Instruct | Atlas 800I A3 | 2 | PD Mixed | 3.5K+1.5K | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-next-80b-a3b-instruct-w8a8-2p-in3k5-out1k5-50ms) | +| Qwen3-Next-80B-A3B-Instruct | Atlas 800I A3 | 2 | PD Mixed | 3.5k+1.5k | 50ms | W8A8 INT8 | [Optimal Configuration](#qwen3-next-80b-a3b-instruct-w8a8-2p-in3k5-out1k5-50ms) | ## Optimal Configuration @@ -39,7 +39,7 @@ On A3 each card has 2 dies, so `--tp-size` is twice the card count; see [Ascend **Quantization**: W8A8 INT8 -**Dataset**: 3.5K+1.5K +**Dataset**: 3.5k+1.5k **TPOT**: 20ms @@ -152,7 +152,7 @@ python -m sglang.bench_serving \ **Quantization**: W8A8 INT8 -**Dataset**: 3.5K+1.5K +**Dataset**: 3.5k+1.5k **TPOT**: 50ms @@ -268,7 +268,7 @@ python -m sglang.bench_serving \ **Quantization**: W8A8 INT8 -**Dataset**: 6K+1.5K +**Dataset**: 6k+1.5k **TPOT**: 15.62ms diff --git a/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/deepseek_r1.mdx b/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/deepseek_r1.mdx index ee795dbcc..af02633e8 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/deepseek_r1.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/deepseek_r1.mdx @@ -76,7 +76,7 @@ It is recommended to download the model weights to a shared directory across mul ## Installation -The Docker image requires at least **30 GB** of free space. Ensure sufficient disk space before pulling images. +The Docker image requires at least **30GB** of free space. Ensure sufficient disk space before pulling images. The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the diff --git a/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/deepseek_v3_2.mdx b/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/deepseek_v3_2.mdx index aa314b415..710cd9fd6 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/deepseek_v3_2.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/deepseek_v3_2.mdx @@ -74,7 +74,7 @@ It is recommended to download the model weights to a shared directory across mul ## Installation -Ensure sufficient disk space before pulling images. The Docker image requires at least **30 GB** of free space. +Ensure sufficient disk space before pulling images. The Docker image requires at least **30GB** of free space. The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the diff --git a/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/glm_5_1.mdx b/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/glm_5_1.mdx index c6b2dad62..1a48de030 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/glm_5_1.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/glm_5_1.mdx @@ -11,7 +11,7 @@ active parameters. It uses 256 routed experts (top-8) plus one shared expert, wi and DeepSeek Sparse Attention (DSA), and a built-in multi-token prediction (MTP) head for speculative decoding. The model features built-in bilingual (Chinese-English) capabilities with a unified pre-training framework, excelling at reasoning, math, code, and tool calling tasks. GLM-5.1 supports both Thinking mode (step-by-step reasoning) and Instruct -mode (direct response), with a native context window of approximately 200K tokens. +mode (direct response), with a native context window of approximately 200k tokens. This document demonstrates the deployment of GLM-5.1 on Ascend NPUs using SGLang, including single-node and multi-node deployment, feature configuration, and performance optimization. @@ -78,7 +78,7 @@ It is recommended to download the model weights to a shared directory across mul ## Installation -Ensure sufficient disk space before pulling images. The Docker image requires at least **30 GB** of free space. +Ensure sufficient disk space before pulling images. The Docker image requires at least **30GB** of free space. The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the diff --git a/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/glm_5_2.mdx b/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/glm_5_2.mdx index 344aac75c..a4dbb9f9c 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/glm_5_2.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/glm_5_2.mdx @@ -74,7 +74,7 @@ It is recommended to download the model weights to a shared directory across mul ## Installation -Ensure sufficient disk space before pulling images. The Docker image requires at least **30 GB** of free space. +Ensure sufficient disk space before pulling images. The Docker image requires at least **30GB** of free space. The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the @@ -340,8 +340,8 @@ export HCCL_BUFFSIZE=1000 export HCCL_OP_EXPANSION_MODE=AIV # Run command ifconfig on two nodes, find out which inet addr has same IP with your node IP. That is your public interface, which should be added here -export HCCL_SOCKET_IFNAME=lo -export GLOO_SOCKET_IFNAME=lo +export HCCL_SOCKET_IFNAME= +export GLOO_SOCKET_IFNAME= # DEEPEP export DEEPEP_NORMAL_LONG_SEQ_ROUND=72 @@ -445,8 +445,8 @@ do export DEEPEP_NORMAL_COMBINE_ENABLE_LONG_SEQ=1 export DEEP_NORMAL_MODE_USE_INT8_QUANT=1 export TASK_QUEUE_ENABLE=2 - export HCCL_SOCKET_IFNAME=lo - export GLOO_SOCKET_IFNAME=lo + export HCCL_SOCKET_IFNAME= + export GLOO_SOCKET_IFNAME= # prefill node python -m sglang.launch_server --model-path ${MODEL_PATH} --disaggregation-mode prefill --host ${P_IP[$i]} \ @@ -484,8 +484,8 @@ do export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=32 export TASK_QUEUE_ENABLE=0 - export HCCL_SOCKET_IFNAME=lo - export GLOO_SOCKET_IFNAME=lo + export HCCL_SOCKET_IFNAME= + export GLOO_SOCKET_IFNAME= export SGLANG_NPU_USE_MULTI_STREAM=1 diff --git a/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/kimi_k2_6.mdx b/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/kimi_k2_6.mdx index 5e408b759..2c69eefb9 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/kimi_k2_6.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/kimi_k2_6.mdx @@ -80,7 +80,7 @@ It is recommended to download the model weights to a shared directory across mul ## Installation -Ensure sufficient disk space before pulling images. The Docker image requires at least **30 GB** of free space. +Ensure sufficient disk space before pulling images. The Docker image requires at least **30GB** of free space. The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the diff --git a/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/mimo_v2_flash.mdx b/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/mimo_v2_flash.mdx index ea1ae486e..f96fb1599 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/mimo_v2_flash.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/mimo_v2_flash.mdx @@ -72,7 +72,7 @@ It is recommended to download the model weights to a shared directory across mul ## Installation -The Docker image requires at least **30 GB** of free space. Ensure sufficient disk space before pulling images. +The Docker image requires at least **30GB** of free space. Ensure sufficient disk space before pulling images. The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the diff --git a/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/minimax_m2_5.mdx b/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/minimax_m2_5.mdx index 330e42425..ebfe2627c 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/minimax_m2_5.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/minimax_m2_5.mdx @@ -8,7 +8,7 @@ metatags: MiniMax-M2.5 is a Mixture-of-Experts (MoE) large language model developed by MiniMax, featuring a sparse MoE architecture with approximately 230B total parameters and 10B active parameters. It supports native long-context -processing up to 200K tokens. The model supports EAGLE3 speculative decoding with a +processing up to 200k tokens. The model supports EAGLE3 speculative decoding with a custom eagle model for accelerated inference, and excels at general language understanding, reasoning, and long-context tasks. @@ -74,7 +74,7 @@ It is recommended to download the model weights to a shared directory across mul ## Installation -Ensure sufficient disk space before pulling images. The Docker image requires at least **30 GB** of free space. +Ensure sufficient disk space before pulling images. The Docker image requires at least **30GB** of free space. The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the diff --git a/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/qwen3-8b.mdx b/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/qwen3-8b.mdx index 82747fcd3..0268e071b 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/qwen3-8b.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/qwen3-8b.mdx @@ -7,7 +7,7 @@ metatags: ## Introduction Qwen3-8B is a compact dense model in the Qwen3 series developed by Alibaba, featuring 8B parameters with -Grouped-Query Attention (GQA) and up to 128K context length. It delivers significant improvements in instruction +Grouped-Query Attention (GQA) and up to 128k context length. It delivers significant improvements in instruction following, logical reasoning, text comprehension, mathematics, science, coding, and tool usage. The model supports EAGLE3 speculative decoding for accelerated inference and is available in both standard and thinking/reasoning-enhanced editions. @@ -72,7 +72,7 @@ It is recommended to download the model weights to a shared directory across mul ## Installation -Ensure sufficient disk space before pulling images. The Docker image requires at least **30 GB** of free space. +Ensure sufficient disk space before pulling images. The Docker image requires at least **30GB** of free space. The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the diff --git a/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/qwen3_235b_a22b.mdx b/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/qwen3_235b_a22b.mdx index 5a1da8bf4..2d7a2d7d3 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/qwen3_235b_a22b.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/qwen3_235b_a22b.mdx @@ -13,7 +13,7 @@ logical reasoning, text comprehension, mathematics, science, coding, and tool us and thinking/reasoning-enhanced editions. This document demonstrates the deployment of Qwen3-235B-A22B on Ascend NPUs using SGLang, including single-node PD mixed -mode, multi-node PD mixed mode, multi-node PD disaggregation mode, 256K long-sequence inference, Prefill Context Parallel, +mode, multi-node PD mixed mode, multi-node PD disaggregation mode, 256k long-sequence inference, Prefill Context Parallel, feature configuration, and performance optimization. This document is validated and written based on **SGLang v0.5.13**. The current model (Qwen3-235B-A22B) is fully supported in @@ -78,7 +78,7 @@ It is recommended to download the model weights to a shared directory across mul ## Installation -The Docker image requires at least **30 GB** of free space. Ensure sufficient disk space before pulling images. +The Docker image requires at least **30GB** of free space. Ensure sufficient disk space before pulling images. The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the diff --git a/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/qwen3_30b_a3b.mdx b/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/qwen3_30b_a3b.mdx index 30c63856b..4230ecf5c 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/qwen3_30b_a3b.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/qwen3_30b_a3b.mdx @@ -72,7 +72,7 @@ It is recommended to download the model weights to a shared directory across mul ## Installation -The Docker image requires at least **30 GB** of free space. Ensure sufficient disk space before pulling images. +The Docker image requires at least **30GB** of free space. Ensure sufficient disk space before pulling images. The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the diff --git a/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/qwen3_32b.mdx b/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/qwen3_32b.mdx index d4c92b246..b6a1afcd9 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/qwen3_32b.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/qwen3_32b.mdx @@ -7,7 +7,7 @@ metatags: ## Introduction Qwen3-32B is a dense model in the Qwen3 series developed by Alibaba, featuring 32B parameters with Grouped-Query -Attention (GQA) and up to 128K (131K with YaRN) context length. It delivers significant improvements in instruction following, logical +Attention (GQA) and up to 128k (131k with YaRN) context length. It delivers significant improvements in instruction following, logical reasoning, text comprehension, mathematics, science, coding, and tool usage. The model supports EAGLE3 speculative decoding for accelerated inference and supports both standard and thinking/reasoning modes. @@ -70,7 +70,7 @@ It is recommended to download the model weights to a shared directory across mul ## Installation -Ensure sufficient disk space before pulling images. The Docker image requires at least **30 GB** of free space. +Ensure sufficient disk space before pulling images. The Docker image requires at least **30GB** of free space. The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the diff --git a/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/qwen3_5_397b.mdx b/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/qwen3_5_397b.mdx index b3fd2577d..e8d032125 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/qwen3_5_397b.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/qwen3_5_397b.mdx @@ -73,7 +73,7 @@ It is recommended to download the model weights to a shared directory across mul ## Installation -Ensure sufficient disk space before pulling images. The Docker image requires at least **30 GB** of free space. +Ensure sufficient disk space before pulling images. The Docker image requires at least **30GB** of free space. The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the diff --git a/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/qwen3_6_27b.mdx b/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/qwen3_6_27b.mdx index 4bf78895f..cf9d2c1dc 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/qwen3_6_27b.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/qwen3_6_27b.mdx @@ -70,7 +70,7 @@ It is recommended to download the model weights to a shared directory across mul ## Installation -Ensure sufficient disk space before pulling images. The Docker image requires at least **30 GB** of free space. +Ensure sufficient disk space before pulling images. The Docker image requires at least **30GB** of free space. The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the diff --git a/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/qwen3_6_35b_a3b.mdx b/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/qwen3_6_35b_a3b.mdx index 835ff0d33..eef3048d1 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/qwen3_6_35b_a3b.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/qwen3_6_35b_a3b.mdx @@ -70,7 +70,7 @@ It is recommended to download the model weights to a shared directory across mul ## Installation -Ensure sufficient disk space before pulling images. The Docker image requires at least **30 GB** of free space. +Ensure sufficient disk space before pulling images. The Docker image requires at least **30GB** of free space. The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the diff --git a/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/qwen3_next_80b_a3b_instruct.mdx b/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/qwen3_next_80b_a3b_instruct.mdx index 4817ae85c..9d19e6140 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/qwen3_next_80b_a3b_instruct.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/qwen3_next_80b_a3b_instruct.mdx @@ -77,7 +77,7 @@ It is recommended to download the model weights to a shared directory across mul ## Installation -Ensure sufficient disk space before pulling images. The Docker image requires at least **30 GB** of free space. +Ensure sufficient disk space before pulling images. The Docker image requires at least **30GB** of free space. The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the