diff --git a/docs_new/docs.json b/docs_new/docs.json index 0000ca998..a19eee433 100644 --- a/docs_new/docs.json +++ b/docs_new/docs.json @@ -900,6 +900,7 @@ "docs/hardware-platforms/ascend-npus/model-tutorials/glm_5_1", "docs/hardware-platforms/ascend-npus/model-tutorials/kimi_k2_6", "docs/hardware-platforms/ascend-npus/model-tutorials/minimax_m2_5", + "docs/hardware-platforms/ascend-npus/model-tutorials/mimo_v2_flash", "docs/hardware-platforms/ascend-npus/model-tutorials/qwen3-8b", "docs/hardware-platforms/ascend-npus/model-tutorials/qwen3_32b", "docs/hardware-platforms/ascend-npus/model-tutorials/qwen3_30b_a3b", diff --git a/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/mimo_v2_flash.mdx b/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/mimo_v2_flash.mdx new file mode 100644 index 000000000..e4916b0cf --- /dev/null +++ b/docs_new/docs/hardware-platforms/ascend-npus/model-tutorials/mimo_v2_flash.mdx @@ -0,0 +1,215 @@ +--- +title: "MiMo-V2-Flash" +metatags: + description: "Deploy MiMo-V2-Flash model with SGLang on Ascend NPUs, including multi-node PD disaggregation mode." +--- + +## Introduction + +MiMo-V2-Flash is a Mixture-of-Experts (MoE) large language model developed by Xiaomi. It employs advanced architecture +with speculative decoding capabilities for accelerated inference. The model is optimized for high throughput and low +latency scenarios through PD disaggregation deployment. + +This document demonstrates the deployment of MiMo-V2-Flash on Ascend NPUs using SGLang, including multi-node PD +disaggregation mode, feature configuration, and performance optimization. + +This document is validated and written based on **SGLang v0.5.13**. The current model (MiMo-V2-Flash) is fully supported in +this version. To use the latest features (e.g., PD disaggregation, speculative decoding), it is recommended to use +v0.5.13 or a later version. + +## Supported features + +| Feature | Example usage | +|-------------------------------|-----------------------------------------------------------------------------------------------| +| Tensor Parallelism | `--tp-size 8` (prefill) or `--tp-size 16` (decode) | +| Data Parallelism | `--dp-size 2` | +| Expert Parallelism | `--moe-a2a-backend deepep \`
`--deepep-mode low_latency` | +| PD Disaggregation | `--disaggregation-mode prefill \`
`--disaggregation-transfer-backend ascend` | +| Quantization | `--quantization modelslim` | +| NPU Graph | enabled by default; disable with `--disable-cuda-graph`;
control range via `--cuda-graph-bs` or `--cuda-graph-max-bs`; e.g. `--cuda-graph-bs 1 2 4 8 12 16 20 24 28 32` | +| Speculative Decoding | `--speculative-algorithm EAGLE \`
`--speculative-num-steps 3 \`
`--speculative-eagle-topk 1 \`
`--speculative-num-draft-tokens 4 \`
`--enable-multi-layer-eagle` | +| Overlap Schedule | `export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=0` | +| DP LM Head | `--enable-dp-lm-head` | +| DP Attention | `--enable-dp-attention` | + + +The values in the **Example usage** column are for illustration only. Adjust them according to your hardware, deployment +mode, and workload. For parameter details, see +[Feature descriptions](/docs/hardware-platforms/ascend-npus/ascend_npu_optimization#feature-descriptions); for +recommended configurations for each deployment scenario, see [Best practices](#best-practices). + + +For feature compatibility and conflict information between features, +see [Feature Compatibility](/docs/hardware-platforms/ascend-npus/ascend_npu_optimization#feature-compatibility). + +## Prerequisites + +### Environment + +Before following this tutorial, complete the environment setup in the documents below: + +- [Ascend NPU Quickstart](/docs/hardware-platforms/ascend-npus/ascend_npu_quick_start) — the fastest way to get started. + It walks you through launching the official container image, starting the SGLang server, and sending a test request. + Recommended if you are new to SGLang on Ascend. +- [SGLang Installation with NPU Support](/docs/hardware-platforms/ascend-npus/ascend_npu) — the full installation guide. + It covers the component version mapping (CANN, PyTorch adapter, Triton, kernels, etc.), building from source or from a + Dockerfile, and recommended system settings (CPU power scheme, NUMA, swap). Use it when you need to install or customize + the environment instead of using the official image. + +### Model weights + + +Before downloading model weights, check the model size to reserve enough disk space. + + +- [MiMo-V2-Flash-W8A8](https://modelers.cn/models/Modelers_Park/MiMo-V2-Flash-W8A8) (Quantized version) + +Ensure the available device memory exceeds the model weight size before deployment. For optimal throughput and latency, +refer to the [best practice configurations](#best-practices) which may require additional nodes or cards. + +It is recommended to download the model weights to a shared directory across multiple nodes. + +## Installation + + +The Docker image requires at least **30 GB** of free space. Ensure sufficient disk space before pulling images. + + +The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the +online platform. You can directly pull it. + +Both **stable releases** and **daily builds** are available. The following command is based on the stable release tag. +For details, see [Docker image versions](/docs/hardware-platforms/ascend-npus/ascend_npu_faq#8-docker-image-versions-stable-release-vs-daily-build). + + + + +```bash Command +docker pull quay.io/ascend/sglang:v0.5.13.post1-cann9.0.0-a3 + +docker run -itd --shm-size=16g --name ${NAME} \ +--privileged=true --net=host \ +-v /var/queue_schedule:/var/queue_schedule \ +-v /etc/ascend_install.info:/etc/ascend_install.info \ +-v /usr/local/sbin:/usr/local/sbin \ +-v /usr/local/Ascend/driver:/usr/local/Ascend/driver \ +-v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \ +--device=/dev/davinci0:/dev/davinci0 \ +--device=/dev/davinci1:/dev/davinci1 \ +--device=/dev/davinci2:/dev/davinci2 \ +--device=/dev/davinci3:/dev/davinci3 \ +--device=/dev/davinci4:/dev/davinci4 \ +--device=/dev/davinci5:/dev/davinci5 \ +--device=/dev/davinci6:/dev/davinci6 \ +--device=/dev/davinci7:/dev/davinci7 \ +--device=/dev/davinci8:/dev/davinci8 \ +--device=/dev/davinci9:/dev/davinci9 \ +--device=/dev/davinci10:/dev/davinci10 \ +--device=/dev/davinci11:/dev/davinci11 \ +--device=/dev/davinci12:/dev/davinci12 \ +--device=/dev/davinci13:/dev/davinci13 \ +--device=/dev/davinci14:/dev/davinci14 \ +--device=/dev/davinci15:/dev/davinci15 \ +--device=/dev/davinci_manager:/dev/davinci_manager \ +--device=/dev/hisi_hdc:/dev/hisi_hdc \ +--entrypoint=bash \ +quay.io/ascend/sglang:v0.5.13.post1-cann9.0.0-a3 +``` + + + + +```bash Command +docker pull quay.io/ascend/sglang:v0.5.13.post1-cann9.0.0-910b + +docker run -itd --shm-size=16g --name ${NAME} \ +--privileged=true --net=host \ +-v /var/queue_schedule:/var/queue_schedule \ +-v /etc/ascend_install.info:/etc/ascend_install.info \ +-v /usr/local/sbin:/usr/local/sbin \ +-v /usr/local/Ascend/driver:/usr/local/Ascend/driver \ +-v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \ +--device=/dev/davinci0:/dev/davinci0 \ +--device=/dev/davinci1:/dev/davinci1 \ +--device=/dev/davinci2:/dev/davinci2 \ +--device=/dev/davinci3:/dev/davinci3 \ +--device=/dev/davinci4:/dev/davinci4 \ +--device=/dev/davinci5:/dev/davinci5 \ +--device=/dev/davinci6:/dev/davinci6 \ +--device=/dev/davinci7:/dev/davinci7 \ +--device=/dev/davinci_manager:/dev/davinci_manager \ +--device=/dev/hisi_hdc:/dev/hisi_hdc \ +--entrypoint=bash \ +quay.io/ascend/sglang:v0.5.13.post1-cann9.0.0-910b +``` + + + + + +- If the model weights have already been downloaded to a shared directory, use `-v` to mount the model path into the + container, for example: `-v /path/to/models:/models`. +- Replace `${NAME}` with your own container name or remove `--name` to use default name. + + +## Online service deployment + +### Multi-node PD disaggregation deployment + +PD disaggregation splits the prefill and decode stages onto separate nodes, reducing interference and improving +throughput for high-concurrency scenarios. This scenario is already covered in the best practice. For the complete, optimized +deployment commands and benchmark data, see +[MiMo-V2-Flash Best Practice — W8A8 24P PD Disaggregation On A3](/docs/hardware-platforms/ascend-npus/best_practice/mimo_v2_flash#pd-disaggregation). + +## Functional verification + +After the service is started, you can invoke the model by sending a prompt: + +```shell +# ============================================================ +# Before running, update the following variables: +# HOST: the server host address (e.g., localhost) +# PORT: the server port number (e.g., 9903) +# ============================================================ + +curl http://${HOST}:${PORT}/generate \ + -H "Content-Type: application/json" \ + -d '{ + "text": "What is the capital of France?", + "sampling_params": { + "max_new_tokens": 64, + "temperature": 0 + } + }' +``` + +Expected result: an HTTP 200 response with the generated text containing "Paris". + +Once the server prints `The server is fired up and ready to roll!` in the logs, it is ready to accept requests. For more +testing examples (Health Check, Generate, Chat Completions, and port usage guidance), +see [Testing the Service](/docs/hardware-platforms/ascend-npus/ascend_npu#testing-the-service). + +## Accuracy evaluation + +For accuracy evaluation methods and datasets, see [Accuracy Evaluation on Ascend NPU](/docs/hardware-platforms/ascend-npus/ascend_npu_accuracy_evaluation). + +## Performance + +For performance data and benchmark commands, see [Performance Testing on Ascend NPU](/docs/hardware-platforms/ascend-npus/ascend_npu_performance_testing). + +## Best practices + +### Best practice configuration reference + +For complete optimal configurations with deployment scripts and benchmark commands, see the +[MiMo-V2-Flash Best Practice](/docs/hardware-platforms/ascend-npus/best_practice/mimo_v2_flash) page. + +## Performance tuning + +For the full list of supported features, see [Supported features](#supported-features). For detailed optimization +guidance, see [Optimization on Ascend NPU](/docs/hardware-platforms/ascend-npus/ascend_npu_optimization). + +## FAQ + +For common environment, installation, and general parameter issues, please refer to the [Ascend NPU FAQ](/docs/hardware-platforms/ascend-npus/ascend_npu_faq). +This section only covers model-specific issues.