minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity

[NPU] [DOC] add performance testing and optimization docs for npu (#25114)

Browse Source
This commit is contained in:
amote-i
2026-05-14 09:47:55 +08:00
committed by GitHub
parent 992fc0d6fe
commit 65e9f81c7d
3 changed files with 1060 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_optimization.mdx
+398
View File
@@ -0,0 +1,398 @@
---
title: "Ascend NPU Optimization"
metatags:
description: "Learn how to optimize SGLang for Ascend NPU using the best practice configurations for mainstream LLM models"
---
This guide explains the role of each parameter used in SGLang deployments on Ascend NPU. It uses
the [DeepSeek-V3.2 best practice configuration](/docs/hardware-platforms/ascend-npus/ascend_npu_best_practice#deepseek-v3-2-128k-1k-26ms-on-a3-32-cards-disaggregation-mode)
as the reference example. For a complete list of tested deployment configurations, see the
[Ascend NPU Best Practice](/docs/hardware-platforms/ascend-npus/ascend_npu_best_practice) page.
<Note>
Parameters in this guide fall into two categories:
- **Required configurations** (marked with `[Required]`): These must be set correctly for the target deployment scenario (e.g., multi-node communication, PD disaggregation). Incorrect values will cause deployment failures or incorrect behavior.
- **Performance optimizations**: These improve throughput, latency, or memory efficiency. The optimal values depend on your specific model, hardware, and workload and may require tuning. Where the optimal value is not obvious, tuning guidance is provided.
</Note>
## System-Level Optimizations
The following system-level tuning steps reduce OS interference and improve CPU scheduling determinism:
<table>
<thead>
<tr>
<th>Command / Variable</th>
<th>Purpose</th>
</tr>
</thead>
<tbody>
<tr>
<td>`echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor`</td>
<td>Locks all CPU cores to the maximum frequency, eliminating DVFS-induced latency jitter during inference-critical paths.</td>
</tr>
<tr>
<td>`sysctl -w vm.swappiness=0`</td>
<td>Minimizes kernel swapping of anonymous pages. Reduces the risk of page faults on NPU memory buffers pinned to host RAM.</td>
</tr>
<tr>
<td>`sysctl -w kernel.numa_balancing=0`</td>
<td>Disables automatic NUMA page migration. Prevents the kernel from moving memory pages between NUMA nodes while inference is running, which would cause latency spikes.</td>
</tr>
<tr>
<td>`sysctl -w kernel.sched_migration_cost_ns=50000`</td>
<td>Sets a minimum task migration cost, discouraging the scheduler from moving inference threads between CPU cores unnecessarily.</td>
</tr>
<tr>
<td>`SGLANG_SET_CPU_AFFINITY=1`</td>
<td>Binds SGLang worker processes to specific CPU cores, avoiding cross-core migration overhead for high-frequency scheduling loops.</td>
</tr>
</tbody>
</table>
## Memory & Device Configuration
<table>
<thead>
<tr>
<th>Variable / Argument</th>
<th>Purpose</th>
<th>Reference Value</th>
</tr>
</thead>
<tbody>
<tr>
<td>`PYTORCH_NPU_ALLOC_CONF=expandable_segments:True`</td>
<td>Enables the expandable NPU memory allocator, allowing the memory pool to grow dynamically. This avoids out-of-memory errors when workloads have variable memory requirements and is essential for large models such as MoE architectures.</td>
<td>`expandable_segments:True`</td>
</tr>
<tr>
<td>`STREAMS_PER_DEVICE=32`</td>
<td>Sets the maximum number of parallel streams per NPU device. More streams allow better overlap between compute and communication operations. The default of `32` is sufficient for most deployments; increase only if profiling reveals stream contention in complex pipeline parallelism setups.</td>
<td>`32`</td>
</tr>
<tr>
<td>`--mem-fraction-static`</td>
<td>Controls the fraction of NPU memory allocated to model weights and the KV cache pool. Lower values leave headroom for intermediate activations; higher values maximize KV cache capacity for serving more concurrent requests. The optimal value depends on your model size, sequence length, and available NPU memory. Start conservatively and increase gradually while monitoring for out-of-memory errors.</td>
<td>Prefill: `0.73`, Decode: `0.79`</td>
</tr>
</tbody>
</table>
## Communication Configuration
<table>
<thead>
<tr>
<th>Variable</th>
<th>Purpose</th>
<th>Reference Value</th>
</tr>
</thead>
<tbody>
<tr>
<td>`HCCL_BUFFSIZE`</td>
<td>Sets the HCCL communication buffer size in MB. Larger buffers increase throughput for bulk transfers but consume more host memory. The optimal value depends on your communication pattern — larger values benefit prefill (bulk token transfers), while smaller values are sufficient for decode (small batches). Tune based on your expected token dispatch volume.</td>
<td>Prefill: `1200`, Decode: `400`</td>
</tr>
<tr>
<td>`HCCL_SOCKET_IFNAME` / `GLOO_SOCKET_IFNAME`</td>
<td><strong>[Required for multi-node]</strong> Specifies the network interface used for HCCL and GLOO distributed communication. Must be set to the high-bandwidth inter-node network interface (e.g., RDMA-capable NIC) for multi-node deployments. Without this, the framework may default to a low-bandwidth interface, severely degrading distributed communication performance. Not needed for single-node deployments.</td>
<td>Set per-cluster</td>
</tr>
</tbody>
</table>
## MoE & Expert Parallelism
<table>
<thead>
<tr>
<th>Variable / Argument</th>
<th>Purpose</th>
<th>Reference Value</th>
</tr>
</thead>
<tbody>
<tr>
<td>`--moe-a2a-backend`</td>
<td>Selects the all-to-all communication backend for MoE expert dispatch and combine. On Ascend NPU, the primary options are `deepep` (DeepEP) and `ascend_fuseep` (Ascend Fused EP). DeepEP is optimized for large-scale models with flexible prefill/decode dispatch paths; `ascend_fuseep` provides a general fused MoE dispatch path.</td>
<td>`deepep`</td>
</tr>
<tr>
<td>`--deepep-mode`</td>
<td>Selects the DeepEP operating mode. Available options: `normal` (optimized for high throughput, long sequences, and large token counts — suitable for prefill), `low_latency` (optimized for low latency, CUDA Graph compatible, small batches — suitable for decode), and `auto` (switches automatically based on the operation type). Use `auto` if unsure; use explicit modes when managing prefill/decode independently in PD disaggregation.</td>
<td>Prefill: `normal`, Decode: `low_latency`</td>
</tr>
<tr>
<td>`SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK`</td>
<td>Sets the maximum number of tokens that a single rank can dispatch in one DeepEP operation (hard upper limit: `1024`). Larger values accommodate more tokens per dispatch but increase buffer allocation overhead. For prefill, use a large value or unbounded (`0`) since many tokens are processed. For decode, set to match your expected tokens per iteration. Must satisfy: `max-running-requests * (1 + draft_tokens) <= this value`.</td>
<td>Prefill: `0` (unbounded), Decode: `8`</td>
</tr>
<tr>
<td>`DEEP_NORMAL_MODE_USE_INT8_QUANT`</td>
<td>When set to `1`, quantizes intermediate activations to INT8 in the DeepEP dispatch operator, reducing communication volume during MoE dispatch. This is most beneficial for large-scale multi-node prefill with many tokens. The trade-off is a small accuracy impact from quantization and additional compute for the quantize/dequantize operations.</td>
<td>Prefill: `1`</td>
</tr>
<tr>
<td>`TASK_QUEUE_ENABLE`</td>
<td>Controls the ASCEND Runtime task queue optimization level: `0` = disabled, `1` = default optimization, `2` = aggressive optimization with greater task fusion and overlap. Higher levels improve throughput but may interfere with CUDA Graph-launched tasks. Start with `1` for general use. Use `2` for throughput-critical prefill workloads; use `0` for decode where CUDA Graph compatibility is needed.</td>
<td>Prefill: `2`, Decode: `0`</td>
</tr>
<tr>
<td>`--moe-dense-tp-size`</td>
<td>Sets the tensor parallelism size for MoE dense (shared) MLP layers. When using DP attention, setting this to `1` avoids an unnecessary all-reduce across the DP group for the dense MLP layers, since each DP shard already has the full weight. In deployments without DP attention, set this to match your TP size.</td>
<td>`1`</td>
</tr>
</tbody>
</table>
## Prefill Optimizations
These arguments and environment variables are critical for tuning prefill performance:
<table>
<thead>
<tr>
<th>Argument / Variable</th>
<th>Purpose</th>
<th>Reference Value</th>
</tr>
</thead>
<tbody>
<tr>
<td>`--chunked-prefill-size`</td>
<td>Sets the maximum number of tokens per prefill chunk. A positive value enables chunked prefill, which interleaves prefill and decode for better concurrency in mixed workloads. Set to `-1` to disable chunking and process each request in a single forward pass, which is preferred for dedicated prefill servers with long-context sequences.</td>
<td>`-1`</td>
</tr>
<tr>
<td>`--max-prefill-tokens`</td>
<td>Limits the total number of tokens the prefill server can process in one batch. The effective bound is `max(this value, model_max_context_length)`. Set this based on your target sequence length and available NPU memory to bound memory usage while maximizing throughput. Tune by increasing until you encounter out-of-memory errors.</td>
<td>`68000`</td>
</tr>
<tr>
<td>`--max-running-requests`</td>
<td>Limits the number of concurrent requests being processed. For prefill, a low value (e.g., `1`) dedicates more compute and memory to each request, achieving higher per-request throughput — ideal for dedicated prefill nodes processing long sequences. For general-purpose serving, use a higher value to support multi-request concurrency.</td>
<td>`1`</td>
</tr>
<tr>
<td>`--disable-radix-cache`</td>
<td>Disables prefix caching via RadixAttention. Set this flag when processing non-overlapping long sequences where prefix caching provides no benefit and only consumes memory. Leave unset (radix cache enabled) for chat/conversation workloads with shared system prompts.</td>
<td>true</td>
</tr>
<tr>
<td>`--disable-cuda-graph`</td>
<td>Disables CUDA Graph capture. CUDA Graphs reduce kernel launch overhead for small, predictable batch sizes, making them ideal for decode. For prefill with large and variable batch sizes, CUDA Graphs provide minimal benefit and can cause issues with dynamic shapes. Set this flag on prefill nodes; leave unset on decode nodes.</td>
<td>true</td>
</tr>
<tr>
<td>`--enable-nsa-prefill-context-parallel`</td>
<td><strong>(DeepSeek V3.2 NSA-specific)</strong> Enables context parallelism for the long-sequence prefill phase of DeepSeek V3.2 with NSA (Native Sparse Attention). Distributes the sequence across CP ranks to parallelize the computationally expensive NSA prefill for ultra-long contexts.</td>
<td>Enabled</td>
</tr>
<tr>
<td>`--nsa-prefill-cp-mode`</td>
<td><strong>(DeepSeek V3.2 NSA-specific)</strong> Controls how the long sequence is split across context parallel ranks: `in-seq-split` divides each sequence uniformly across CP ranks, optimal for single-request prefill. `round-robin-split` (code default) distributes tokens by index mod CP size, supporting multi-batch prefill. Only effective when `--enable-nsa-prefill-context-parallel` is enabled.</td>
<td>`in-seq-split`</td>
</tr>
<tr>
<td>`--attn-cp-size`</td>
<td>Specifies the context parallelism group size for attention computation. Larger values distribute the sequence across more ranks, reducing per-rank memory and compute at the cost of increased communication. For models with NSA, this controls the CP size for sparse attention prefill. Set to the number of available devices for maximum parallelization.</td>
<td>`32`</td>
</tr>
</tbody>
</table>
## Decode Optimizations
These arguments and environment variables are critical for tuning decode performance:
<table>
<thead>
<tr>
<th>Argument / Variable</th>
<th>Purpose</th>
<th>Reference Value</th>
</tr>
</thead>
<tbody>
<tr>
<td>`--dp-size`</td>
<td>Sets the data parallelism degree for the decode server. With DP attention enabled, attention layers are sharded across DP ranks while FFN/MoE layers use tensor parallelism. Higher values create more independent decode instances, increasing throughput through parallel request processing. Choose a value that divides evenly into your total card count, with remaining cards used for TP/EP.</td>
<td>`8`</td>
</tr>
<tr>
<td>`--ep`</td>
<td>Sets the expert parallelism degree. For MoE models, this distributes experts across cards, reducing per-card expert loading overhead and enabling all-to-all dispatch. The code default is `1`; set explicitly for MoE models. The optimal value depends on your model's expert count and architecture. DeepSeek V3.2 with 256 routed experts uses `ep=32`. For models with fewer experts, use a proportionally smaller value.</td>
<td>`32`</td>
</tr>
<tr>
<td>`--enable-dp-attention`</td>
<td>Enables data parallelism for attention layers while keeping tensor parallelism for FFN/MoE layers. This is a key optimization for decode throughput — attention is DP-sharded to reduce KV cache duplication, while MoE layers remain TP-sharded to leverage expert parallelism. Best suited for MoE models where attention is not the compute bottleneck.</td>
<td>Enabled</td>
</tr>
<tr>
<td>`--enable-dp-lm-head`</td>
<td>Enables vocabulary parallelism across the DP attention group, sharding the LM head weight across ranks. Each rank only computes logits for its vocabulary shard, avoiding a costly all-gather of logits across the DP group. This is essential when DP attention is enabled to maintain throughput.</td>
<td>Enabled</td>
</tr>
<tr>
<td>`--cuda-graph-max-bs`</td>
<td>Caps the maximum batch size for which CUDA Graphs are captured. Larger values cover more batch sizes but increase graph capture time and memory overhead. If your `max-running-requests` is high but typical batch sizes are lower, use a smaller value to reduce capture overhead. Tune based on your observed batch size distribution during serving.</td>
<td>`4`</td>
</tr>
<tr>
<td>`SGLANG_SCHEDULER_SKIP_ALL_GATHER=1`</td>
<td>When DP attention is enabled, the scheduler normally performs an all-gather across DP ranks to determine the full set of ready requests. Setting this to `1` skips that operation, reducing decode scheduling latency. Only safe when load is balanced across DP ranks (e.g., via a round-robin load balancing policy). Disable if you observe uneven load distribution across DP ranks.</td>
<td>`1`</td>
</tr>
</tbody>
</table>
## Speculative Decoding
Speculative decoding reduces per-token latency by generating draft tokens that are then verified by the target model:
<table>
<thead>
<tr>
<th>Argument / Variable</th>
<th>Purpose</th>
<th>Reference Value</th>
</tr>
</thead>
<tbody>
<tr>
<td>`--speculative-algorithm`</td>
<td>Selects the speculative decoding algorithm. `NEXTN` (aliased to `EAGLE`) uses the model's built-in MTP (Multi-Token Prediction) heads, requiring no separate draft model. `EAGLE3` uses an external draft model, which can achieve higher acceptance rates at the cost of additional memory. Other built-in options include `STANDALONE`, `NGRAM`, and `DFLASH`, plus any plugin-registered name via `SpeculativeAlgorithm.register`. Choose `NEXTN` for models with native MTP support (e.g., DeepSeek V3.2/R1); choose `EAGLE3` for models without MTP (e.g., Qwen MoE).</td>
<td>`NEXTN`</td>
</tr>
<tr>
<td>`--speculative-num-steps`</td>
<td>Number of speculative forward passes per iteration. More steps can increase the acceptance length and throughput but add latency. For prefill, use a small value (`1`) to minimize prefill latency impact. For decode, use a larger value (`2`–`4`) to maximize throughput. Tune based on your latency vs throughput requirements.</td>
<td>Prefill: `1`, Decode: `3`</td>
</tr>
<tr>
<td>`--speculative-eagle-topk`</td>
<td>Limits the number of draft tokens considered per position. Lower values reduce compute on unlikely tokens and are required for the experimental SpecV2 overlap scheduler. Higher values may increase acceptance rates but add overhead. Start with `1` if using SpecV2; otherwise, `4`–`8` is typical.</td>
<td>`1`</td>
</tr>
<tr>
<td>`--speculative-num-draft-tokens`</td>
<td>Number of draft tokens generated per speculative step. Higher values increase potential acceptance length and throughput but add per-step computation. Balance against your latency budget — prefill typically uses fewer draft tokens (`2`) to minimize overhead; decode can use more (`4`) to maximize throughput.</td>
<td>Prefill: `2`, Decode: `4`</td>
</tr>
<tr>
<td>`SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1`</td>
<td>Enables the overlap plan stream feature for EAGLE v2/v3 speculative decoding workers. This overlaps draft model computation with target model verification, effectively hiding draft latency. Enable when using EAGLE-based speculative decoding; not applicable for NEXTN.</td>
<td>`1`</td>
</tr>
<tr>
<td>`SGLANG_ENABLE_SPEC_V2=1`</td>
<td>Enables the experimental SpecV2 overlap scheduler for speculative decoding. Works with `--speculative-eagle-topk 1` to overlap the draft generation and verification stages. Requires `SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1`.</td>
<td>`1`</td>
</tr>
</tbody>
</table>
## Quantization
<table>
<thead>
<tr>
<th>Argument</th>
<th>Purpose</th>
<th>Reference Value</th>
</tr>
</thead>
<tbody>
<tr>
<td>`--quantization modelslim`</td>
<td>Uses the Ascend ModelSlim quantization tool to load W8A8 pre-quantized model weights. This reduces model weight footprint by approximately 50% compared to BF16, allowing larger models to fit in NPU memory with minimal accuracy degradation. The quantization method is auto-detected from the model's `quant_model_description.json` file.</td>
<td>`modelslim`</td>
</tr>
</tbody>
</table>
## Throughput Configuration
<table>
<thead>
<tr>
<th>Argument / Variable</th>
<th>Purpose</th>
<th>Reference Value</th>
</tr>
</thead>
<tbody>
<tr>
<td>`--tokenizer-worker-num`</td>
<td>Sets the number of parallel tokenizer worker processes. Increasing this allows concurrent tokenization of multiple input/output streams, preventing the tokenizer from becoming a bottleneck under high request concurrency. Set based on your CPU core count and expected request rate.</td>
<td>`4`</td>
</tr>
<tr>
<td>`--load-balance-method`</td>
<td>Selects the DP load balancing strategy. Available options: `auto` (default, automatically selects the best strategy), `round_robin` (assigns requests to DP ranks in rotation for even distribution), `total_tokens` (balances by token load), `total_requests` (balances by request count), `follow_bootstrap_room` (follows the bootstrap room assignment). Start with `round_robin` for simple even distribution; use `total_tokens` if your requests have highly variable lengths.</td>
<td>`round_robin`</td>
</tr>
<tr>
<td>`ASCEND_MF_STORE_URL`</td>
<td><strong>[Required for PD disaggregation]</strong> Sets the MemFabric config store address for PD disaggregation. This is the prefill primary node's IP with an arbitrary port, used by the decode nodes to discover and connect to the MemFabric-based KV cache transfer service. Omit this for non-disaggregated deployments.</td>
<td>Prefill IP with port</td>
</tr>
</tbody>
</table>
## Additional Ascend NPU-Specific Parameters
The following environment variables are used in other best practice configurations and may be applicable depending on your model and deployment:
<table>
<thead>
<tr>
<th>Variable</th>
<th>Purpose</th>
<th>Typical Usage</th>
</tr>
</thead>
<tbody>
<tr>
<td>`HCCL_OP_EXPANSION_MODE=AIV`</td>
<td>Configures the HCCL communication algorithm scheduling to use AIV (Ascend Intelligent Vision) expansion mode, which can improve communication efficiency for certain collective operations.</td>
<td>Used in Qwen MoE and R1 non-NSA configurations</td>
</tr>
<tr>
<td>`SGLANG_NPU_FUSED_MOE_MODE`</td>
<td>Controls the fused MoE optimization mode on Ascend NPU. `1` is default; `2` enables a more aggressive fusion strategy (`DISPATCH_FFN_COMBINE`) that can improve MoE dispatch throughput. Mode `2` requires `--quantization modelslim`. Used primarily with DeepSeek R1 models.</td>
<td>`1` or `2`</td>
</tr>
<tr>
<td>`SGLANG_NPU_USE_MLAPO=1`</td>
<td><strong>(DeepSeek MLA-specific)</strong> Adopts the `MLAPO` fusion operator in the MLA (Multi-Head Latent Attention) preprocessing stage for DeepSeek models with MLA architecture.</td>
<td>Used with DeepSeek R1</td>
</tr>
<tr>
<td>`SGLANG_USE_FIA_NZ=1`</td>
<td><strong>(DeepSeek MLA-specific)</strong> Reshapes the KV Cache into FIA NZ format for improved memory access efficiency. Must be used together with `SGLANG_NPU_USE_MLAPO=1`.</td>
<td>Used with DeepSeek R1</td>
</tr>
<tr>
<td>`SGLANG_NPU_USE_MULTI_STREAM=1`</td>
<td><strong>(DeepSeek MoE-specific)</strong> Enables dual-stream computation for shared experts and routing experts in DeepSeek MoE models, allowing the two expert types to execute concurrently on separate streams.</td>
<td>Used with DeepSeek R1</td>
</tr>
<tr>
<td>`SGLANG_USE_AG_AFTER_QLORA=1`</td>
<td>Delays the all-gather operation until after Q-LoRA processing. This reduces communication overhead by performing Q-LoRA projection before the all-gather, requiring fewer bytes to be transferred.</td>
<td>Used with DeepSeek V3.2/R1 prefill</td>
</tr>
</tbody>
</table>
## See Also
- [Ascend NPU Best Practice](/docs/hardware-platforms/ascend-npus/ascend_npu_best_practice) — Complete deployment configurations and benchmark results for all supported models
- [Ascend NPU Environment Variables](/docs/hardware-platforms/ascend-npus/ascend_npu_environment_variables) — Reference for all Ascend NPU-related environment variables
- [DeepSeek V3.2 Guide](/docs/basic_usage/deepseek_v32) — Detailed usage guide for DeepSeek V3.2 deployment
- [Expert Parallelism](/docs/advanced_features/expert_parallelism) — DeepEP configuration and tuning guide
docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_performance_testing.mdx
+660
View File
@@ -0,0 +1,660 @@
---
title: "Ascend NPU Performance Testing"
metatags:
description: "Performance testing for SGLang on Ascend NPUs for text generation, multimodal, and embedding models in both online and offline serving modes."
---
This page walks through performance testing your SGLang deployment on Ascend NPUs. We cover three model types — text generation (`Qwen/Qwen2.5-7B-Instruct`), multimodal vision (`Qwen/Qwen2.5-VL-7B-Instruct`), and embedding (`Qwen/Qwen3-Embedding-8B`) — in both online and offline serving modes. You can use [Evalscope](https://evalscope.readthedocs.io/en/latest/), [AISBench](https://ais-bench-benchmark.readthedocs.io/en/latest/), or SGLang's built-in benchmarking tools.
<Note>The benchmark output examples in this guide are for illustration only. Actual performance depends on your hardware (e.g., Atlas 800I A2 vs A3), model version, SGLang version, and deployment configuration. Always run benchmarks on your own hardware to obtain accurate performance data.</Note>
## 1. Prepare
### 1.1 Start SGLang server
Launch the server with the appropriate flags for each model type. Make sure SGLang is installed first — see [Ascend NPU Quickstart](/docs/hardware-platforms/ascend-npus/ascend_npu_quick_start) for environment setup.
<Tabs>
<Tab title="Text Generation">
```shell Command
# The model will be automatically downloaded by sglang or set --model-path to the local path if the model is already downloaded.
sglang serve --model-path Qwen/Qwen2.5-7B-Instruct
```
</Tab>
<Tab title="Multimodal">
```shell Command
# The model will be automatically downloaded by sglang or set --model-path to the local path if the model is already downloaded.
sglang serve --model-path Qwen/Qwen2.5-VL-7B-Instruct --mm-attention-backend ascend_attn
```
</Tab>
<Tab title="Embedding">
```shell Command
# The model will be automatically downloaded by sglang or set --model-path to the local path if the model is already downloaded.
sglang serve --model-path Qwen/Qwen3-Embedding-8B --is-embedding
```
</Tab>
</Tabs>
<Tip>Add `&` at the end of the command to run the server in the background, or open a new terminal to run the benchmark commands in the following sections.</Tip>
<Note>The server binds to `http://127.0.0.1:30000` by default. All online benchmarks below assume the server is running at that address. The `--is-embedding` flag is required for embedding models.</Note>
### 1.2 Install benchmarking tools
`bench_serving` and `bench_offline_throughput` are built into SGLang and require no extra installation. For Evalscope and AISBench, set up each in its own virtual environment:
<Tabs>
<Tab title="Evalscope">
```shell Command
python3 -m venv .evalscope_venv
source .evalscope_venv/bin/activate
pip install evalscope[perf] -U
```
</Tab>
<Tab title="AISBench">
```shell Command
python3 -m venv .aisbench_venv
source .aisbench_venv/bin/activate
git clone https://github.com/AISBench/benchmark.git
cd benchmark/
pip3 install -e ./ --use-pep517
pip3 install -r requirements/api.txt
pip3 install -r requirements/extra.txt
```
Run `ais_bench -h` to verify.
<Note>AISBench requires Python 3.10-3.12. After installation, all AISBench commands must be run from the `benchmark/` directory (the cloned repo root). Set `stream=True` and `ignore_eos=True` in the model config for accurate results.</Note>
</Tab>
</Tabs>
## 2. Online Service: Text Generation Model
Test `Qwen/Qwen2.5-7B-Instruct` via the online serving endpoint.
<Note>Before running any benchmark in this section, make sure the SGLang text-generation server is running at `http://127.0.0.1:30000`. See [Start SGLang server](#1-1-start-sglang-server) for the launch command.</Note>
<Tip>For performance testing, prefer random datasets (`--dataset random`, `--dataset-name random`) over real datasets. Random datasets let you pin `--min-prompt-length` / `--max-prompt-length` and `--min-tokens` / `--max-tokens` to fixed values, producing consistent, repeatable results. Real datasets (ShareGPT, openqa, etc.) have variable input lengths that add noise and make cross-run comparisons unreliable.</Tip>
### 2.1 Using Evalscope
<Note>Prerequisites: [Evalscope installed](#1-2-install-benchmarking-tools) and its virtual environment activated (`source .evalscope_venv/bin/activate`). SGLang server running at `http://127.0.0.1:30000`.</Note>
Run the following command to run a performance test against the server:
```shell Command
evalscope perf \
--parallel 10 \
--number 20 \
--model Qwen/Qwen2.5-7B-Instruct \
--url http://127.0.0.1:30000/v1/chat/completions \
--api openai \
--dataset random \
--max-tokens 1024 \
--min-tokens 1024 \
--prefix-length 0 \
--min-prompt-length 1024 \
--max-prompt-length 1024 \
--tokenizer-path Qwen/Qwen2.5-7B-Instruct \
--extra-args '{"ignore_eos": true}'
```
<Tip>If the model has already been downloaded, you can point `--tokenizer-path` to the local model path instead of the model id.</Tip>
Example output (for illustration only — actual results depend on your hardware and configuration):
```text
Benchmarking summary:
┌────────────────────────────┬─────────────┐
│ Metric │ Value │
├────────────────────────────┼─────────────┤
│ ── General ── │ │
│ Test Duration (s) │ 89.34 │
│ Concurrency │ 10 │
│ Request Rate (req/s) │ -1.00 │
│ Total / Success / Failed │ 20 / 20 / 0 │
│ Req Throughput (req/s) │ 0.22 │
│ ── Latency ── │ │
│ Avg Latency (s) │ 44.67 │
│ TTFT (ms) │ 578.51 │
│ TPOT (ms) │ 43.10 │
│ ITL (ms) │ 43.12 │
│ ── Tokens ── │ │
│ Avg Input Tokens │ 1024.00 │
│ Avg Output Tokens │ 1024.00 │
│ Output Throughput (tok/s) │ 229.24 │
│ Total Throughput (tok/s) │ 458.49 │
│ ── Speculative Decoding ── │ │
│ Decoded Tok/Iter │ 1.00 │
│ Spec. Accept Rate │ 0.00 │
└────────────────────────────┴─────────────┘
Percentile results:
┌────────────────┬─────────┬─────────┬─────────┬─────────┬─────────┬─────────┬─────────┬─────────┬─────────┐
│ Metric │ 1% │ 5% │ 10% │ 25% │ 50% │ 75% │ 90% │ 95% │ 99% │
├────────────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┤
│ Latency (s) │ 44.47 │ 44.47 │ 44.47 │ 44.47 │ 44.86 │ 44.86 │ 44.86 │ 44.86 │ 44.86 │
│ TTFT (ms) │ 138.12 │ 142.07 │ 426.17 │ 426.87 │ 783.67 │ 785.26 │ 786.85 │ 787.97 │ 787.97 │
│ ITL (ms) │ 41.84 │ 42.14 │ 42.22 │ 42.36 │ 42.57 │ 42.80 │ 42.99 │ 49.24 │ 49.84 │
│ TPOT (ms) │ 42.71 │ 42.71 │ 42.71 │ 43.05 │ 43.08 │ 43.43 │ 43.43 │ 43.71 │ 43.71 │
│ Input tokens │ 1024.00 │ 1024.00 │ 1024.00 │ 1024.00 │ 1024.00 │ 1024.00 │ 1024.00 │ 1024.00 │ 1024.00 │
│ Output tokens │ 1024.00 │ 1024.00 │ 1024.00 │ 1024.00 │ 1024.00 │ 1024.00 │ 1024.00 │ 1024.00 │ 1024.00 │
│ Output (tok/s) │ 22.83 │ 22.83 │ 22.83 │ 22.83 │ 23.02 │ 23.03 │ 23.03 │ 23.03 │ 23.03 │
│ Total (tok/s) │ 45.65 │ 45.65 │ 45.65 │ 45.65 │ 46.05 │ 46.05 │ 46.05 │ 46.05 │ 46.05 │
│ Decode (tok/s) │ 22.88 │ 23.03 │ 23.03 │ 23.07 │ 23.21 │ 23.42 │ 23.42 │ 23.42 │ 23.42 │
└────────────────┴─────────┴─────────┴─────────┴─────────┴─────────┴─────────┴─────────┴─────────┴─────────┘
...
```
<Note>See the [Evalscope Performance Testing Guide](https://evalscope.readthedocs.io/en/latest/user_guides/stress_test/quick_start.html) for full details.</Note>
### 2.2 Using AISBench
<Note>Prerequisites: [AISBench installed](#1-2-install-benchmarking-tools) and its virtual environment activated (`source .aisbench_venv/bin/activate`). All commands must be run from the `benchmark/` directory. SGLang server running at `http://127.0.0.1:30000`. Set `stream=True` and `ignore_eos=True` in the model config for accurate results.</Note>
Two files need to be configured for performance testing.
First, describe the model and server settings in `ais_bench/benchmark/configs/models/vllm_api/vllm_api_stream_chat.py`:
```python vllm_api_stream_chat.py
# more details: https://ais-bench-benchmark.readthedocs.io/en/latest/base_tutorials/scenes_intro/performance_benchmark.html
from ais_bench.benchmark.models import VLLMCustomAPIChat
from ais_bench.benchmark.utils.postprocess.model_postprocessors import extract_non_reasoning_content
models = [
dict(
attr="service",
type=VLLMCustomAPIChat,
abbr="vllm-api-stream-chat",
path="Qwen/Qwen2.5-7B-Instruct",
model="Qwen/Qwen2.5-7B-Instruct",
stream=True,
request_rate=0,
use_timestamp=False,
retry=2,
api_key="",
host_ip="127.0.0.1",
host_port=30000,
url="",
max_out_len=512,
batch_size=32,
trust_remote_code=False,
generation_kwargs=dict(
temperature=0.01,
ignore_eos=True,
),
pred_postprocessor=dict(type=extract_non_reasoning_content),
)
]
```
<Tip>If the model has already been downloaded, point `path` to the local model path instead of the model id.</Tip>
Second, configure random prompt lengths in `ais_bench/datasets/synthetic/synthetic_config.py`:
```python synthetic_config.py
# more details: https://ais-bench-benchmark.readthedocs.io/en/latest/advanced_tutorials/synthetic_dataset.html
synthetic_config = {
"Type":"tokenid",
"RequestCount": 10,
"TrustRemoteCode": False,
"StringConfig" : {
"Input" : {
"Method": "uniform",
"Params": {"MinValue": 1, "MaxValue": 200}
},
"Output" : {
"Method": "gaussian",
"Params": {"Mean": 100, "Var": 200, "MinValue": 1, "MaxValue": 100}
}
},
"TokenIdConfig" : {
"RequestSize": 10,
"PrefixLen": 0
}
}
```
Run with a synthetic dataset:
```shell Command
ais_bench --models vllm_api_stream_chat --datasets synthetic_gen_string -m perf
```
Example output (for illustration only — actual results depend on your hardware and configuration):
```text
╒══════════════════════════╤═════════╤═════════════════╤═════════════════╤═════════════════╤═════════════════╤═════════════════╤═════════════════╤═════════════════╤═════╕
│ Performance Parameters │ Stage │ Average │ Min │ Max │ Median │ P75 │ P90 │ P99 │ N │
╞══════════════════════════╪═════════╪═════════════════╪═════════════════╪═════════════════╪═════════════════╪═════════════════╪═════════════════╪═════════════════╪═════╡
│ E2EL │ total │ 3896.4 ms │ 3081.6 ms │ 4175.3 ms │ 4013.8 ms │ 4123.4 ms │ 4137.1 ms │ 4171.5 ms │ 10 │
├──────────────────────────┼─────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────┤
│ TTFT │ total │ 411.6 ms │ 346.7 ms │ 439.7 ms │ 416.3 ms │ 426.6 ms │ 434.4 ms │ 439.2 ms │ 10 │
├──────────────────────────┼─────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────┤
│ TPOT │ total │ 38.3 ms │ 37.4 ms │ 39.0 ms │ 38.3 ms │ 38.7 ms │ 38.9 ms │ 39.0 ms │ 10 │
├──────────────────────────┼─────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────┤
│ ITL │ total │ 38.7 ms │ 0.0 ms │ 156.5 ms │ 38.9 ms │ 39.0 ms │ 39.2 ms │ 117.1 ms │ 10 │
├──────────────────────────┼─────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────┤
│ InputTokens │ total │ 123.4 │ 34.0 │ 228.0 │ 130.5 │ 170.5 │ 217.2 │ 226.92 │ 10 │
├──────────────────────────┼─────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────┤
│ OutputTokens │ total │ 92.1 │ 69.0 │ 100.0 │ 95.0 │ 99.75 │ 100.0 │ 100.0 │ 10 │
├──────────────────────────┼─────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────┤
│ OutputTokenThroughput │ total │ 23.5937 token/s │ 22.3912 token/s │ 24.2616 token/s │ 23.7399 token/s │ 23.9919 token/s │ 24.2027 token/s │ 24.2557 token/s │ 10 │
╘══════════════════════════╧═════════╧═════════════════╧═════════════════╧═════════════════╧═════════════════╧═════════════════╧═════════════════╧═════════════════╧═════╛
╒══════════════════════════╤═════════╤══════════════════╕
│ Common Metric │ Stage │ Value │
╞══════════════════════════╪═════════╪══════════════════╡
│ Benchmark Duration │ total │ 4175.4485 ms │
├──────────────────────────┼─────────┼──────────────────┤
│ Total Requests │ total │ 10 │
├──────────────────────────┼─────────┼──────────────────┤
│ Failed Requests │ total │ 0 │
├──────────────────────────┼─────────┼──────────────────┤
│ Success Requests │ total │ 10 │
├──────────────────────────┼─────────┼──────────────────┤
│ Concurrency │ total │ 9.3317 │
├──────────────────────────┼─────────┼──────────────────┤
│ Max Concurrency │ total │ 32 │
├──────────────────────────┼─────────┼──────────────────┤
│ Request Throughput │ total │ 2.395 req/s │
├──────────────────────────┼─────────┼──────────────────┤
│ Total Input Tokens │ total │ 1234 │
├──────────────────────────┼─────────┼──────────────────┤
│ Prefill Token Throughput │ total │ 299.8329 token/s │
├──────────────────────────┼─────────┼──────────────────┤
│ Total Generated Tokens │ total │ 921 │
├──────────────────────────┼─────────┼──────────────────┤
│ Input Token Throughput │ total │ 295.5371 token/s │
├──────────────────────────┼─────────┼──────────────────┤
│ Output Token Throughput │ total │ 220.5751 token/s │
├──────────────────────────┼─────────┼──────────────────┤
│ Total Token Throughput │ total │ 516.1122 token/s │
╘══════════════════════════╧═════════╧══════════════════╛
```
<Note>See the [AISBench Documentation](https://ais-bench-benchmark.readthedocs.io/en/latest/) for details.</Note>
### 2.3 Using bench_serving
SGLang's built-in `bench_serving` requires no extra installation. Make sure the server is running at `http://127.0.0.1:30000` before running the benchmark.
<Note>See the [Bench Serving Guide](/docs/developer_guide/bench_serving) for all backends, datasets, and advanced options.</Note>
```shell Command
python -m sglang.bench_serving \
--backend sglang-oai \
--base-url http://127.0.0.1:30000 \
--model Qwen/Qwen2.5-7B-Instruct \
--dataset-name random \
--random-input-len 1024 \
--random-output-len 512 \
--random-range-ratio 1 \
--num-prompts 100 \
--max-concurrency 32
```
<Note>`--dataset-name random` samples token IDs from the ShareGPT dataset to generate realistic input; the first run downloads ShareGPT from Hugging Face automatically. Set `export HF_ENDPOINT=https://hf-mirror.com` if network is not available.</Note>
<Tip>Set `--random-range-ratio 1` for fixed input/output lengths (recommended for consistent comparisons) or `0` (default) for uniform distribution. Add `--request-rate` to control the request rate. For all backends, datasets, and advanced options, see the full [Bench Serving Guide](/docs/developer_guide/bench_serving).</Tip>
Example output (for illustration only — actual results depend on your hardware and configuration):
```text
============ Serving Benchmark Result ============
Backend: sglang-oai
Traffic request rate: inf
Max request concurrency: 32
Successful requests: 100
Benchmark duration (s): 47.51
Total input tokens: 102400
Total input text tokens: 102400
Total generated tokens: 51200
Total generated tokens (retokenized): 51195
Request throughput (req/s): 2.10
Input token throughput (tok/s): 2155.35
Output token throughput (tok/s): 1077.68
Peak output token throughput (tok/s): 1587.00
Peak concurrent requests: 64
Total token throughput (tok/s): 3233.03
Concurrency: 26.93
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 12793.49
Median E2E Latency (ms): 12940.17
P90 E2E Latency (ms): 13049.86
P99 E2E Latency (ms): 13051.61
---------------Time to First Token----------------
Mean TTFT (ms): 1423.99
Median TTFT (ms): 1489.29
P99 TTFT (ms): 2325.56
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 22.25
Median TPOT (ms): 22.22
P99 TPOT (ms): 25.08
---------------Inter-Token Latency----------------
Mean ITL (ms): 22.26
Median ITL (ms): 20.74
P95 ITL (ms): 21.40
P99 ITL (ms): 23.62
Max ITL (ms): 2229.30
==================================================
```
## 3. Online Service: Multimodal Model
Test `Qwen/Qwen2.5-VL-7B-Instruct` for vision-language tasks.
<Note>Before running any benchmark in this section, make sure the SGLang multimodal server is running at `http://127.0.0.1:30000`. See [Start SGLang server](#1-1-start-sglang-server) and use the Multimodal tab for the launch command.</Note>
<Tip>For consistent, repeatable results, set `--random-range-ratio 1` to fix input/output lengths, or `0` (default) for uniform distribution.</Tip>
### 3.1 Using Evalscope
<Note>Prerequisites: [Evalscope installed](#1-2-install-benchmarking-tools) and its virtual environment activated (`source .evalscope_venv/bin/activate`). SGLang multimodal server running at `http://127.0.0.1:30000`.</Note>
Evalscope's `perf` tool uses the OpenAI-compatible `/v1/chat/completions` endpoint. Use `--dataset random_vl` for randomized multimodal data with image generation:
```shell Command
evalscope perf \
--parallel 10 \
--number 20 \
--model Qwen/Qwen2.5-VL-7B-Instruct \
--url http://127.0.0.1:30000/v1/chat/completions \
--api openai \
--dataset random_vl \
--min-tokens 1024 \
--max-tokens 1024 \
--prefix-length 0 \
--min-prompt-length 1024 \
--max-prompt-length 1024 \
--image-width 512 \
--image-height 512 \
--image-format RGB \
--image-num 1 \
--tokenizer-path Qwen/Qwen2.5-VL-7B-Instruct \
--extra-args '{"ignore_eos": true}'
```
<Tip>If the model has already been downloaded, you can point `--tokenizer-path` to the local model path instead of the model id.</Tip>
### 3.2 Using AISBench
<Note>Prerequisites: [AISBench installed](#1-2-install-benchmarking-tools) and its virtual environment activated (`source .aisbench_venv/bin/activate`). All commands run from the `benchmark/` directory. SGLang multimodal server running at `http://127.0.0.1:30000`. AISBench does not include a built-in multimodal dataset — you must provide your own.</Note>
First, edit `ais_bench/benchmark/configs/models/vllm_api/vllm_api_stream_chat.py` to configure the vision model:
```python vllm_api_stream_chat.py
from ais_bench.benchmark.models import VLLMCustomAPIChat
from ais_bench.benchmark.utils.postprocess.model_postprocessors import extract_non_reasoning_content
models = [
dict(
attr="service",
type=VLLMCustomAPIChat,
abbr="vllm-api-stream-chat",
path="Qwen/Qwen2.5-VL-7B-Instruct",
model="Qwen/Qwen2.5-VL-7B-Instruct",
stream=True,
request_rate=0,
use_timestamp=False,
retry=2,
api_key="",
host_ip="127.0.0.1",
host_port=30000,
url="",
max_out_len=256,
batch_size=16,
trust_remote_code=False,
generation_kwargs=dict(
temperature=0.01,
ignore_eos=True,
),
pred_postprocessor=dict(type=extract_non_reasoning_content),
)
]
```
<Tip>If the model has already been downloaded, point `path` to the local model path instead of the model id.</Tip>
Next, download a multimodal dataset such as mmstar:
```shell Command
# Download the mmstar dataset (from within the benchmark/ directory)
cd ais_bench/datasets
mkdir mmstar
cd mmstar
wget https://www.modelscope.cn/datasets/evalscope/MMStar/resolve/master/MMStar.tsv
```
Run the performance test:
```shell Command
ais_bench --models vllm_api_stream_chat --datasets mmstar_gen -m perf
```
Example output (for illustration only — actual results depend on your hardware and configuration):
```text
╒══════════════════════════╤═════════╤═════════════════╤════════════════╤═════════════════╤═════════════════╤═════════════════╤═════════════════╤═════════════════╤══════╕
│ Performance Parameters │ Stage │ Average │ Min │ Max │ Median │ P75 │ P90 │ P99 │ N │
╞══════════════════════════╪═════════╪═════════════════╪════════════════╪═════════════════╪═════════════════╪═════════════════╪═════════════════╪═════════════════╪══════╡
│ E2EL │ total │ 6190.9 ms │ 5071.4 ms │ 8464.8 ms │ 6126.6 ms │ 6475.2 ms │ 6833.5 ms │ 7897.9 ms │ 1500 │
├──────────────────────────┼─────────┼─────────────────┼────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼──────┤
│ TTFT │ total │ 693.3 ms │ 96.0 ms │ 2161.5 ms │ 747.4 ms │ 870.9 ms │ 1032.3 ms │ 1620.8 ms │ 1500 │
├──────────────────────────┼─────────┼─────────────────┼────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼──────┤
│ TPOT │ total │ 21.6 ms │ 17.8 ms │ 32.1 ms │ 21.3 ms │ 23.1 ms │ 24.5 ms │ 29.1 ms │ 1500 │
├──────────────────────────┼─────────┼─────────────────┼────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼──────┤
│ ITL │ total │ 25.5 ms │ 0.0 ms │ 1951.1 ms │ 18.8 ms │ 19.7 ms │ 37.3 ms │ 121.8 ms │ 1500 │
├──────────────────────────┼─────────┼─────────────────┼────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼──────┤
│ InputTokens │ total │ 0.0 │ 0.0 │ 0.0 │ 0.0 │ 0.0 │ 0.0 │ 0.0 │ 1500 │
├──────────────────────────┼─────────┼─────────────────┼────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼──────┤
│ OutputTokens │ total │ 256.0 │ 256.0 │ 256.0 │ 256.0 │ 256.0 │ 256.0 │ 256.0 │ 1500 │
├──────────────────────────┼─────────┼─────────────────┼────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┼──────┤
│ OutputTokenThroughput │ total │ 41.6779 token/s │ 30.243 token/s │ 50.4791 token/s │ 41.7847 token/s │ 44.6424 token/s │ 45.6484 token/s │ 46.0932 token/s │ 1500 │
╘══════════════════════════╧═════════╧═════════════════╧════════════════╧═════════════════╧═════════════════╧═════════════════╧═════════════════╧═════════════════╧══════╛
╒═════════════════════════╤═════════╤══════════════════╕
│ Common Metric │ Stage │ Value │
╞═════════════════════════╪═════════╪══════════════════╡
│ Benchmark Duration │ total │ 582099.6816 ms │
├─────────────────────────┼─────────┼──────────────────┤
│ Total Requests │ total │ 1500 │
├─────────────────────────┼─────────┼──────────────────┤
│ Failed Requests │ total │ 0 │
├─────────────────────────┼─────────┼──────────────────┤
│ Success Requests │ total │ 1500 │
├─────────────────────────┼─────────┼──────────────────┤
│ Concurrency │ total │ 15.9532 │
├─────────────────────────┼─────────┼──────────────────┤
│ Max Concurrency │ total │ 16 │
├─────────────────────────┼─────────┼──────────────────┤
│ Request Throughput │ total │ 2.5769 req/s │
├─────────────────────────┼─────────┼──────────────────┤
│ Total Input Tokens │ total │ 0 │
├─────────────────────────┼─────────┼──────────────────┤
│ Total Generated Tokens │ total │ 384000 │
├─────────────────────────┼─────────┼──────────────────┤
│ Input Token Throughput │ total │ 0.0 token/s │
├─────────────────────────┼─────────┼──────────────────┤
│ Output Token Throughput │ total │ 659.6808 token/s │
├─────────────────────────┼─────────┼──────────────────┤
│ Total Token Throughput │ total │ 659.6808 token/s │
╘═════════════════════════╧═════════╧══════════════════╛
```
<Note>See the [AISBench Documentation](https://ais-bench-benchmark.readthedocs.io/en/latest/) for details.</Note>
### 3.3 Using bench_serving (image dataset)
Set `--dataset-name image` for image datasets. `bench_serving` will generate random prompts with image inputs. Make sure the server is running at `http://127.0.0.1:30000` before running the benchmark.
<Note>See the [Bench Serving Guide](/docs/developer_guide/bench_serving) for the full list of image-related flags.</Note>
```shell Command
python -m sglang.bench_serving \
--backend sglang \
--base-url http://127.0.0.1:30000 \
--model Qwen/Qwen2.5-VL-7B-Instruct \
--dataset-name image \
--random-input-len 1024 \
--random-output-len 512 \
--random-range-ratio 1 \
--num-prompts 32 \
--max-concurrency 16 \
--image-count 1 \
--image-resolution 720p
```
Example output (for illustration only — actual results depend on your hardware and configuration):
```text
============ Serving Benchmark Result ============
Backend: sglang
Traffic request rate: inf
Max request concurrency: 16
Successful requests: 32
Benchmark duration (s): 51.74
Total input tokens: 73464
Total input text tokens: 35128
Total input vision tokens: 38336
Total generated tokens: 16384
Total generated tokens (retokenized): 9300
Request throughput (req/s): 0.62
Input token throughput (tok/s): 1419.96
Output token throughput (tok/s): 316.68
Peak output token throughput (tok/s): 800.00
Peak concurrent requests: 32
Total token throughput (tok/s): 1736.64
Concurrency: 15.98
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 25841.84
Median E2E Latency (ms): 25842.85
P90 E2E Latency (ms): 26296.42
P99 E2E Latency (ms): 26303.13
---------------Time to First Token----------------
Mean TTFT (ms): 12211.59
Median TTFT (ms): 14405.77
P99 TTFT (ms): 15837.60
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 26.67
Median TPOT (ms): 21.75
P99 TPOT (ms): 41.89
---------------Inter-Token Latency----------------
Mean ITL (ms): 26.67
Median ITL (ms): 20.34
P95 ITL (ms): 20.85
P99 ITL (ms): 21.70
Max ITL (ms): 11309.91
==================================================
```
## 4. Online Service: Embedding Model
Test `Qwen/Qwen3-Embedding-8B` on the embedding API endpoint.
<Note>Before running any benchmark in this section, make sure the SGLang embedding server is running with `--is-embedding` at `http://127.0.0.1:30000`. See [Start SGLang server](#1-1-start-sglang-server) and use the Embedding tab for the launch command. AISBench does not support embedding endpoints — use `bench_serving` or Evalscope instead.</Note>
### 4.1 Using Evalscope
<Note>Prerequisites: [Evalscope installed](#1-2-install-benchmarking-tools) and its virtual environment activated (`source .evalscope_venv/bin/activate`). SGLang embedding server running with `--is-embedding` at `http://127.0.0.1:30000`.</Note>
Evalscope supports embedding evaluation. For performance testing the embedding API directly:
```shell Command
evalscope perf \
--parallel 10 \
--number 20 \
--model Qwen/Qwen3-Embedding-8B \
--url http://127.0.0.1:30000/v1/embeddings \
--api openai_embedding \
--dataset random_embedding \
--min-prompt-length 1024 \
--max-prompt-length 1024 \
--tokenizer-path Qwen/Qwen3-Embedding-8B
```
<Tip>If the model has already been downloaded, you can point `--tokenizer-path` to the local model path instead of the model id.</Tip>
<Note>Evalscope's embedding performance testing support may vary by version. If the `perf` command does not accept the embeddings endpoint, use [`bench_serving` with `--backend sglang-embedding`](#4-2-using-bench_serving-embedding-backend) as the primary option.</Note>
### 4.2 Using bench_serving (embedding backend)
`bench_serving` is built into SGLang. Use `--backend sglang-embedding` to target the `/v1/embeddings` endpoint. Make sure the server is running with `--is-embedding` at `http://127.0.0.1:30000`.
```shell Command
python -m sglang.bench_serving \
--backend sglang-embedding \
--base-url http://127.0.0.1:30000 \
--model Qwen/Qwen3-Embedding-8B \
--dataset-name random \
--random-input-len 512 \
--random-output-len 0 \
--num-prompts 1000 \
--max-concurrency 64 \
--request-rate 32
```
<Note>`--dataset-name random` samples token IDs from the ShareGPT dataset; the first run downloads ShareGPT from Hugging Face automatically. Set `export HF_ENDPOINT=https://hf-mirror.com` if network is not available. Set `--random-output-len 0` for embedding benchmarks — no output tokens are generated.</Note>
Example output (for illustration only — actual results depend on your hardware and configuration):
```text
============ Serving Benchmark Result ============
Backend: sglang-embedding
Traffic request rate: 32.0
Max request concurrency: 64
Successful requests: 1000
Benchmark duration (s): 31.86
Total input tokens: 257891
Total input text tokens: 257891
Request throughput (req/s): 31.39
Input token throughput (tok/s): 8094.67
Peak concurrent requests: 62
Concurrency: 6.67
----------------End-to-End Latency----------------
Mean E2E Latency (ms): 212.34
Median E2E Latency (ms): 160.97
P90 E2E Latency (ms): 267.31
P99 E2E Latency (ms): 1445.94
==================================================
```
## 5. Offline Performance Testing
SGLang's `Engine` API runs inference in-process, without an HTTP server, letting you measure maximum throughput. `bench_offline_throughput` is built into SGLang and requires no extra installation or running server.
<Note>`bench_offline_throughput` currently only supports text-generation (LLM) benchmarks. Multimodal and embedding models are not supported.</Note>
### 5.1 Using bench_offline_throughput
`bench_offline_throughput` uses the `Engine` API internally and measures pure inference throughput without HTTP overhead:
```shell Command
python -m sglang.bench_offline_throughput \
--model-path Qwen/Qwen2.5-7B-Instruct \
--dataset-name random \
--random-input-len 1024 \
--random-output-len 512 \
--num-prompts 500
```
<Note>`--dataset-name random` samples token IDs from the ShareGPT dataset; the first run downloads ShareGPT from Hugging Face automatically. Set `export HF_ENDPOINT=https://hf-mirror.com` if network is not available.</Note>
<Tip>`--dataset-name random` with `--random-input-len` and `--random-output-len` gives you full control over input/output token counts. Fixed-length random data eliminates variance from real datasets, making throughput comparisons across runs deterministic and reliable.</Tip>
## See also
- [Bench Serving Guide](/docs/developer_guide/bench_serving) — all backends, datasets, and advanced options for `bench_serving`
- [Ascend NPU Quickstart](/docs/hardware-platforms/ascend-npus/ascend_npu_quick_start) — environment setup for Ascend NPUs
- [Evalscope Performance Testing Guide](https://evalscope.readthedocs.io/en/latest/user_guides/stress_test/quick_start.html) — full Evalscope documentation
- [AISBench Documentation](https://ais-bench-benchmark.readthedocs.io/en/latest/) — full AISBench documentation