add mimo best practice (#27665)

2026-06-11 10:15:31 +08:00
parent 475e9d25bf
commit ce0ff154a5
1 changed files with 326 additions and 0 deletions
@@ -776,6 +776,106 @@ you encounter issues or have any questions, please [open an issue](https://githu
  </tbody>
 </table>

+## MiMo-V2-Flash Series Models
+
+### Low Latency
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <colgroup>
+    <col style={{width: "13%"}} />
+    <col style={{width: "13%"}} />
+    <col style={{width: "13%"}} />
+    <col style={{width: "13%"}} />
+    <col style={{width: "12%"}} />
+    <col style={{width: "12%"}} />
+    <col style={{width: "12%"}} />
+    <col style={{width: "12%"}} />
+  </colgroup>
+  <thead>
+    <tr style={{borderBottom: "2px solid #d55816"}}>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Model</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Hardware</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Cards</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Deploy Mode</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Dataset</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>TTFT</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Quantization</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Configuration</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>MiMo-V2-Flash</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Atlas 800I A3</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>24</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>PD Disaggregation</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>16K+1</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>974ms</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>W8A8 INT8</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><a href="#mimo-v2-flash-16k-1-974ms-on-a3-24-cards-disaggregation-mode">Optimal Configuration</a></td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>MiMo-V2-Flash</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Atlas 800I A3</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>24</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>PD Disaggregation</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>32K+1</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>2,147ms</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>W8A8 INT8</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><a href="#mimo-v2-flash-32k-1-2147ms-on-a3-24-cards-disaggregation-mode">Optimal Configuration</a></td>
+    </tr>
+  </tbody>
+</table>
+
+### High Throughput
+
+<table style={{width: "100%", borderCollapse: "collapse", tableLayout: "fixed"}}>
+  <colgroup>
+    <col style={{width: "13%"}} />
+    <col style={{width: "13%"}} />
+    <col style={{width: "13%"}} />
+    <col style={{width: "13%"}} />
+    <col style={{width: "12%"}} />
+    <col style={{width: "12%"}} />
+    <col style={{width: "12%"}} />
+    <col style={{width: "12%"}} />
+  </colgroup>
+  <thead>
+    <tr style={{borderBottom: "2px solid #d55816"}}>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Model</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Hardware</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Cards</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Deploy Mode</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Dataset</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>TPOT</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.02)"}}>Quantization</th>
+      <th style={{textAlign: "left", padding: "10px 12px", fontWeight: 700, whiteSpace: "nowrap", backgroundColor: "rgba(255,255,255,0.05)"}}>Configuration</th>
+    </tr>
+  </thead>
+  <tbody>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>MiMo-V2-Flash</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Atlas 800I A3</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>24</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>PD Disaggregation</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>16K+1K</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>16ms</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>W8A8 INT8</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><a href="#mimo-v2-flash-16k-1k-16ms-on-a3-24-cards-disaggregation-mode">Optimal Configuration</a></td>
+    </tr>
+    <tr>
+      <td style={{padding: "9px 12px", fontWeight: 500, backgroundColor: "rgba(255,255,255,0.02)"}}>MiMo-V2-Flash</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>Atlas 800I A3</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>24</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>PD Disaggregation</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>32K+1K</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}>12ms</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.02)"}}>W8A8 INT8</td>
+      <td style={{padding: "9px 12px", backgroundColor: "rgba(255,255,255,0.05)"}}><a href="#mimo-v2-flash-32k-1k-12ms-on-a3-24-cards-disaggregation-mode">Optimal Configuration</a></td>
+    </tr>
+  </tbody>
+</table>
+
 ## Kimi Series Models

 ### Low Latency
@@ -6655,3 +6755,229 @@ We tested it based on the `RANDOM` dataset (90% cache hit), this dataset is gene
 ```bash Command
 python -m sglang.bench_serving --dataset-name random --backend sglang --host 127.0.0.1 --port 8003 --random-range-ratio 1 --random-output-len 1000 --random-input-len 131072 --num-prompts 192
 ```
+
+### MiMo-V2-Flash 16K+1 974ms on A3 24 Cards Disaggregation Mode
+
+**Model:** MiMo-V2-Flash
+
+**Hardware:** Atlas 800I A3 24 Cards (8 cards for prefill node, 16 cards for decode node)
+
+**Deploy Mode:** PD Disaggregation
+
+**Dataset:** random
+
+**Input/Output Length:** 16K+1
+
+**TTFT:** 974ms (P99: 2,068ms)
+
+**Quantization:** W8A8 INT8
+
+#### Model Deployment
+
+**Prefill Node:**
+
+```bash
+# High performance CPU configuration
+echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
+sysctl -w vm.swappiness=0
+sysctl -w kernel.numa_balancing=0
+sysctl -w kernel.sched_migration_cost_ns=50000
+
+# Bind CPU
+export SGLANG_SET_CPU_AFFINITY=1
+
+# Proxy settings (unset if needed)
+unset https_proxy
+unset http_proxy
+unset HTTPS_PROXY
+unset HTTP_PROXY
+unset ASCEND_LAUNCH_BLOCKING
+
+# Performance environment variables
+export STREAMS_PER_DEVICE=32
+export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=32
+export HCCL_BUFFSIZE=1024
+export HCCL_OP_EXPANSION_MODE=AIV
+export HCCL_SOCKET_IFNAME=lo
+export GLOO_SOCKET_IFNAME=lo
+export SGLANG_NPU_PROFILING=0
+export SGLANG_NPU_PROFILING_STAGE="prefill"
+export DEEPEP_NORMAL_LONG_SEQ_ROUND=32
+export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=3584
+export ASCEND_MF_STORE_URL="tcp://<PREFILL_IP>:24669"
+export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=3600
+export SGLANG_DISAGGREGATION_WAITING_TIMEOUT=3600
+export SGLANG_ENABLE_SPEC_V2=1
+export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=0
+export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
+export SGLANG_DEEPEP_BF16_DISPATCH=0
+export SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANK=1
+export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
+export HCCL_CONNECT_TIMEOUT=1800
+export ASCEND_USE_FIA=1
+
+# Launch prefill server
+python3 -m sglang.launch_server \
+    --model-path /path/to/MiMo-V2-Flash-w8a8-all-0512 \
+    --attention-backend ascend \
+    --device npu \
+    --tp-size 8 --nnodes 1 --node-rank 0 \
+    --chunked-prefill-size 8192 \
+    --trust-remote-code --port 10000 \
+    --host <PREFILL_IP> --max-running-requests 64 \
+    --mem-fraction-static 0.8 \
+    --swa-full-tokens-ratio 0.3 \
+    --disaggregation-mode prefill --disaggregation-transfer-backend ascend \
+    --disaggregation-bootstrap-port 8996 \
+    --disable-radix-cache \
+    --disable-cuda-graph \
+    --disable-piecewise-cuda-graph \
+    --dp-size 2
+```
+
+**Decode Node:**
+
+```bash
+# High performance CPU configuration
+echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
+sysctl -w vm.swappiness=0
+sysctl -w kernel.numa_balancing=0
+sysctl -w kernel.sched_migration_cost_ns=50000
+
+# Bind CPU
+export SGLANG_SET_CPU_AFFINITY=1
+
+# Proxy settings (unset if needed)
+unset https_proxy
+unset http_proxy
+unset HTTPS_PROXY
+unset HTTP_PROXY
+unset ASCEND_LAUNCH_BLOCKING
+
+# Performance environment variables
+export STREAMS_PER_DEVICE=32
+export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=128
+export HCCL_BUFFSIZE=800
+export HCCL_OP_EXPANSION_MODE=AIV
+export HCCL_SOCKET_IFNAME=lo
+export GLOO_SOCKET_IFNAME=lo
+export SGLANG_NPU_PROFILING=0
+export SGLANG_NPU_PROFILING_STAGE="prefill"
+export DEEPEP_NORMAL_LONG_SEQ_ROUND=32
+export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=3584
+export ASCEND_MF_STORE_URL="tcp://<PREFILL_IP>:24669"
+export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=3600
+export SGLANG_DISAGGREGATION_WAITING_TIMEOUT=3600
+export SGLANG_ENABLE_SPEC_V2=1
+export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=0
+export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
+export SGLANG_DEEPEP_BF16_DISPATCH=0
+export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
+export HCCL_CONNECT_TIMEOUT=1800
+export SGLANG_PROFILE_WITH_STACK=True
+export ASCEND_USE_FIA=1
+
+# Launch decode server
+python3 -m sglang.launch_server \
+    --model-path /path/to/MiMo-V2-Flash-w8a8-all-0512/ \
+    --attention-backend ascend \
+    --device npu \
+    --tp-size 16 --nnodes 1 --node-rank 0 \
+    --trust-remote-code --port 10001 \
+    --host <DECODE_IP> --max-running-requests 64 \
+    --mem-fraction-static 0.8 \
+    --swa-full-tokens-ratio 0.3 \
+    --cuda-graph-bs 1 2 4 8 12 16 20 24 28 32 \
+    --disaggregation-mode decode --disaggregation-transfer-backend ascend \
+    --disaggregation-bootstrap-port 8996 \
+    --speculative-algorithm EAGLE --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 \
+    --enable-multi-layer-eagle \
+    --disable-radix-cache \
+    --dp-size 2 --enable-dp-attention --enable-dp-lm-head \
+    --moe-a2a-backend deepep --deepep-mode low_latency
+```
+
+**Router:**
+
+```bash
+python -m sglang_router.launch_router \
+    --pd-disaggregation \
+    --prefill http://<PREFILL_IP>:10000 \
+    --decode  http://<DECODE_IP>:10001 \
+    --host <ROUTER_IP> \
+    --port 9903 \
+    --health-check-interval-secs 3600 \
+    --mini-lb
+```
+
+#### Benchmark
+
+We tested it based on the `RANDOM` dataset with throughput focus (infinite request rate).
+
+```bash
+python3 -m sglang.bench_serving --backend sglang --host <ROUTER_IP> --port 9903  --dataset-path /path/to/ShareGPT_V3_unfiltered_cleaned_split.json --dataset-name random --random-input-len 16000 --random-output-len 1 --request-rate inf --random-range-ratio 1 --num-prompts 128 --max-concurrency 64
+```
+
+### MiMo-V2-Flash 32K+1 2,147ms on A3 24 Cards Disaggregation Mode
+
+**Model:** MiMo-V2-Flash
+
+**Hardware:** Atlas 800I A3 24 Cards (8 cards for prefill node, 16 cards for decode node)
+
+**Deploy Mode:** PD Disaggregation
+
+**Dataset:** random
+
+**Input/Output Length:** 32K+1
+
+**TTFT:** 2,147ms (P99: 4,546ms)
+
+**Quantization:** W8A8 INT8
+
+#### Model Deployment
+
+Same deployment configuration as [MiMo-V2-Flash 16K+1 974ms](#mimo-v2-flash-16k-1-974ms-on-a3-24-cards-disaggregation-mode).
+
+### MiMo-V2-Flash 16K+1K 16ms on A3 24 Cards Disaggregation Mode
+
+**Model:** MiMo-V2-Flash
+
+**Hardware:** Atlas 800I A3 24 Cards (8 cards for prefill node, 16 cards for decode node)
+
+**Deploy Mode:** PD Disaggregation
+
+**Dataset:** random
+
+**Input/Output Length:** 16K+1K
+
+**TTFT:** 11,326ms (P99: 28,160ms)
+
+**TPOT:** 16ms (P99: 23ms)
+
+**Quantization:** W8A8 INT8
+
+#### Model Deployment
+
+Same deployment configuration as [MiMo-V2-Flash 16K+1 974ms](#mimo-v2-flash-16k-1-974ms-on-a3-24-cards-disaggregation-mode).
+
+### MiMo-V2-Flash 32K+1K 12ms on A3 24 Cards Disaggregation Mode
+
+**Model:** MiMo-V2-Flash
+
+**Hardware:** Atlas 800I A3 24 Cards (8 cards for prefill node, 16 cards for decode node)
+
+**Deploy Mode:** PD Disaggregation
+
+**Dataset:** random
+
+**Input/Output Length:** 32K+1K
+
+**TTFT:** 35,144ms (P99: 55,004ms)
+
+**TPOT:** 12ms (P99: 17ms)
+
+**Quantization:** W8A8 INT8
+
+#### Model Deployment
+
+Same deployment configuration as [MiMo-V2-Flash 16K+1 974ms](#mimo-v2-flash-16k-1-974ms-on-a3-24-cards-disaggregation-mode).