minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 13 Packages Projects Releases Wiki Activity

[docs][NPU]Update model and feature docs support (#16124)

Browse Source
This commit is contained in:
husf
2025-12-30 20:05:40 +08:00
committed by GitHub
parent 664f611e83
commit 7f9a3d0609
5 changed files with 165 additions and 145 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/platforms/ascend_npu_support_features.md
+31 -30
View File
@@ -5,30 +5,31 @@ questions, please [open an issue](https://github.com/sgl-project/sglang/issues).
## Model and tokenizer
| Argument | Description | Defaults | Options | A2 Supported | A3 Supported |
|-------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------|----------------------------------|:----------------------------------------:|:----------------------------------------:|
| `--model-path`<br>`--model` | The path of the model weights. This can be a local folder or a Hugging Face repo ID. | `None` | Type: str | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--tokenizer-path` | The path of the tokenizer. | `None` | Type: str | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--tokenizer-mode` | Tokenizer mode. 'auto' will use the fast tokenizer if available, and 'slow' will always use the slow tokenizer. | `auto` | `auto`, `slow` | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--tokenizer-worker-num` | The worker num of the tokenizer manager. | `1` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--skip-tokenizer-init` | If set, skip init tokenizer and pass input_ids in generate request. | `False` | bool flag (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--trust-remote-code` | Whether or not to allow for custom models defined on the Hub in their own modeling files. | `False` | bool flag (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--context-length` | The model's maximum context length. Defaults to None (will use the value from the model's config.json instead). | `None` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--is-embedding` | Whether to use a CausalLM as an embedding model. | `False` | bool flag (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--enable-multimodal` | Enable the multimodal functionality for the served model. If the model being served is not multimodal, nothing will happen | `None` | bool flag (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--revision` | The specific model version to use. It can be a branch name, a tag name, or a commit id. If unspecified, will use the default version. | `None` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--model-impl` | Which implementation of the model to use. * “auto” will try to use the SGLang implementation if it exists and fall back to the Transformers implementation if no SGLang implementation is available. * “sglang” will use the SGLang model implementation. * “transformers” will use the Transformers model implementation.* "mindspore" will use the MindSpore model implementation. | `None` | 'auto', 'sglang', 'transformers' | **<span style="color: red;">×</span>** | **<span style="color: green;">√</span>** |
| `--model-loader-extra-config` | Extra config for model loader. This will be passed to the model loader corresponding to the chosen load_format. | {} | Type: str' | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| Argument | Description | Defaults | Options | A2 Supported | A3 Supported |
|-------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------|----------------------------------------------------------------------------------|:----------------------------------------:|:----------------------------------------:|
| `--model-path`<br/>`--model` | The path of the model weights. This can be a local folder or a Hugging Face repo ID. | `None` | Type: str | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--tokenizer-path` | The path of the tokenizer. | `None` | Type: str | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--tokenizer-mode` | Tokenizer mode. 'auto' will use the fast tokenizer if available, and 'slow' will always use the slow tokenizer. | `auto` | `auto`, `slow` | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--tokenizer-worker-num` | The worker num of the tokenizer manager. | `1` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--skip-tokenizer-init` | If set, skip init tokenizer and pass input_ids in generate request. | `False` | bool flag (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--load-format` | The format of the model weights to load. <br/> `auto` will try to load the weights in the safetensors format and fall back to the pytorch bin format if safetensors format is not available. <br/> `pt` will load the weights in the pytorch bin format. <br/> `safetensors` will load the weights in the safetensors format. <br/> `npcache` will load the weights in pytorch format and store a numpy cache to speed up the loading. <br/> `dummy` will initialize the weights with random values, which is mainly for profiling. <br/> `gguf` will load the weights in the gguf format. <br/> `bitsandbytes` will load the weights using bitsandbytes quantization.<br/>`layered` loads weights layer by layer so that one can quantize a layer before loading another to make the peak memory envelope smaller. | `auto` | `auto`, `safetensors` | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--model-loader-extra-config` | Extra config for model loader. This will be passed to the model loader corresponding to the chosen load_format. | {} | Type: str <br/> for example: {"enable_multithread_load": true,"num_threads": 64} | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--trust-remote-code` | Whether or not to allow for custom models defined on the Hub in their own modeling files. | `False` | bool flag (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--context-length` | The model's maximum context length. Defaults to None (will use the value from the model's config.json instead). | `None` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--is-embedding` | Whether to use a CausalLM as an embedding model. | `False` | bool flag (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--enable-multimodal` | Enable the multimodal functionality for the served model. If the model being served is not multimodal, nothing will happen | `None` | bool flag (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--revision` | The specific model version to use. It can be a branch name, a tag name, or a commit id. If unspecified, will use the default version. | `None` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--model-impl` | Which implementation of the model to use. <br/> `auto` will try to use the SGLang implementation if it exists and fall back to the Transformers implementation if no SGLang implementation is available. <br/> `sglang` will use the sglang model implementation. <br/> `transformers` will use the transformers model implementation. <br/> `mindspore` will use the MindSpore model implementation. | `None` | `auto`, `sglang`, `transformers` | **<span style="color: red;">×</span>** | **<span style="color: green;">√</span>** |
## HTTP server
| Argument | Description | Defaults | Options | A2 Supported | A3 Supported |
|------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------|---------------------------|:----------------------------------------:|:----------------------------------------:|
| `--host` | The host of the HTTP server. | `127.0.0.1` | Type: str | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--port` | The port of the HTTP server. | `30000` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--skip-server-warmup` | If set, skip warmup. | `False` | bool flag (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--warmups` | Specify custom warmup functions (csv) to run before server starts eg. --warmups=warmup_name1,warmup_name2 will run the functions `warmup_name1` and `warmup_name2` specified in warmup.py before the server starts listening for requests | `None` | Type: str | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--nccl-port` | The port for NCCL distributed environment setup. Defaults to a random port. | `None` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| Argument | Description | Defaults | Options | A2 Supported | A3 Supported |
|------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------|---------------------------|:----------------------------------------:|:----------------------------------------:|
| `--host` | The host of the HTTP server. | `127.0.0.1` | Type: str | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--port` | The port of the HTTP server. | `30000` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--skip-server-warmup` | If set, skip warmup. | `False` | bool flag (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--warmups` | Specify custom warmup functions (csv) to run before server starts eg. `--warmups=warmup_name1,warmup_name2` will run the functions `warmup_name1` and `warmup_name2` specified in warmup.py before the server starts listening for requests | `None` | Type: str | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--nccl-port` | The port for NCCL distributed environment setup. Defaults to a random port. | `None` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
## Quantization and data type
@@ -65,8 +66,8 @@ questions, please [open an issue](https://github.com/sgl-project/sglang/issues).
| Argument | Description | Defaults | Options | A2 Supported | A3 Supported |
|---------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------|---------------------------|:----------------------------------------:|:----------------------------------------:|
| `--device` | The device to use ('cuda', 'xpu', 'hpu', 'npu', 'cpu'). Defaults to auto-detection if not specified. | `None` | Type: str | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--tensor-parallel-size`<br>`--tp-size` | The tensor parallelism size. | `1` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--pipeline-parallel-size`<br>`--pp-size` | The pipeline parallelism size. | `1` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--tensor-parallel-size`<br/>`--tp-size` | The tensor parallelism size. | `1` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--pipeline-parallel-size`<br/>`--pp-size` | The pipeline parallelism size. | `1` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--pp-max-micro-batch-size` | The maximum micro batch size in pipeline parallelism. | `None` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--stream-interval` | The interval (or buffer size) for streaming in terms of the token length. A smaller value makes streaming smoother, while a larger value makes the throughput higher | `1` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--stream-output` | Whether to output as a sequence of disjoint segments. | `False` | bool flag (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
@@ -119,7 +120,7 @@ questions, please [open an issue](https://github.com/sgl-project/sglang/issues).
| `--completion-template` | The buliltin completion template name or the path of the completion template file. This is only used for OpenAI-compatible API server. only for code completion currently. | `None` | Type: str | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--enable-cache-report` | Return number of cached tokens in usage.prompt_tokens_details for each openai request. | `True` | bool flag (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--reasoning-parser` | Specify the parser for reasoning models. Supported parsers: [deepseek-r1, deepseek-v3, glm45, gpt-oss, kimi, qwen3, qwen3-thinking, step3]. | `None` | `deepseek-r1`, `deepseek-v3`, `glm45`, `gpt-oss`, `kimi`, `qwen3`, `qwen3-thinking`, `step3` | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--tool-call-parser` | Specify the parser for handling tool-call interactions. Supported parsers: [ llama3,qwen]. | `None` | `llama3`,`qwen` | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--tool-call-parser` | Specify the parser for handling tool-call interactions. Supported parsers: [ llama3, qwen]. | `None` | `llama3`,`qwen` | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--sampling-defaults` | Where to get default sampling parameters. 'openai' uses SGLang/OpenAI defaults (temperature=1.0, top_p=1.0, etc.). 'model' uses the model's generation_config.json to get the recommended sampling parameters if available. Default is 'model'. | `model` | `openai`, `model` | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--tool-server` | Either 'demo' or a comma-separated list of tool server urls to use for the model. If not specified, no tool server will be used. | `None` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
@@ -127,7 +128,7 @@ questions, please [open an issue](https://github.com/sgl-project/sglang/issues).
| Argument | Description | Defaults | Options | A2 Supported | A3 Supported |
|---------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------|---------------------------------------------------|:----------------------------------------:|:----------------------------------------:|
| `--data-parallel-size`<br>`--dp-size` | The data parallelism size. | `1` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--data-parallel-size`<br/>`--dp-size` | The data parallelism size. | `1` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--load-balance-method` | The load balancing strategy for data parallelism. The Minimum Token algorithm can only be used when DP attention is applied. This algorithm performs load balancing based on the real-time token load of the DP workers. | `round_robin` | `round_robin`, `shortest_queue`, `minimum_tokens` | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--prefill-round-robin-balance` | Prefill is round robin balanced. This is used to promise decode server can get the correct dp rank. | `False` | bool flag (set to enable) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
@@ -135,7 +136,7 @@ questions, please [open an issue](https://github.com/sgl-project/sglang/issues).
| Argument | Description | Defaults | Options | A2 Supported | A3 Supported |
|------------------------------------------|------------------------------------------------------------------------------------|----------|-----------|:----------------------------------------:|:----------------------------------------:|
| `--dist-init-addr`<br>`--nccl-init-addr` | The host address for initializing distributed backend (e.g., `192.168.0.2:25000`). | `None` | Type: str | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--dist-init-addr`<br/>`--nccl-init-addr` | The host address for initializing distributed backend (e.g., `192.168.0.2:25000`). | `None` | Type: str | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--nnodes` | The number of nodes. | `1` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--node-rank` | The node rank. | `0` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
@@ -180,7 +181,7 @@ questions, please [open an issue](https://github.com/sgl-project/sglang/issues).
| Argument | Description | Defaults | Options | A2 Supported | A3 Supported |
|-----------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------|-----------|---------------------------------------------------|:----------------------------------------:|:----------------------------------------:|
| `--speculative-algorithm` | Speculative algorithm. | `None` | `EAGLE`, `EAGLE3`, `NEXTN`, `STANDALONE`, `NGRAM` | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--speculative-draft-model-path`<br>`--speculative-draft-model` | The path of the draft model weights. This can be a local folder or a Hugging Face repo ID. | `None` | Type: str | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--speculative-draft-model-path`<br/>`--speculative-draft-model` | The path of the draft model weights. This can be a local folder or a Hugging Face repo ID. | `None` | Type: str | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--speculative-draft-model-revision` | The specific draft model version to use. It can be a branch name, a tag name, or a commit id. If unspecified, will use the default version. | `None` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--speculative-num-steps` | The number of steps sampled from draft model in Speculative Decoding. | `None` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--speculative-eagle-topk` | The number of tokens sampled from the draft model in eagle2 each step. | `None` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
@@ -208,12 +209,13 @@ questions, please [open an issue](https://github.com/sgl-project/sglang/issues).
| Argument | Description | Defaults | Options | A2 Supported | A3 Supported |
|---------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------|-----------------------------------|:----------------------------------------:|:----------------------------------------:|
| `--expert-parallel-size`<br>`--ep-size`<br>`--ep` | The expert parallelism size. Default equal to tp size. | `tp-size` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: green;">√</span>** |
| `--expert-parallel-size`<br/>`--ep-size`<br/>`--ep` | The expert parallelism size. Default equal to tp size. | `tp-size` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: green;">√</span>** |
| `--moe-a2a-backend` | Select the backend for all-to-all communication for expert parallelism. | `none` | `none`, `deepep`, `ascend_fuseep` | **<span style="color: red;">×</span>** | **<span style="color: green;">√</span>** |
| `--moe-runner-backend` | Choose the runner backend for MoE. | `auto` | `auto` | **<span style="color: red;">×</span>** | **<span style="color: green;">√</span>** |
| `--flashinfer-mxfp4-moe-precision` | Choose the computation precision of flashinfer mxfp4 moe | `default` | `default`, `bf16` | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--enable-flashinfer-allreduce-fusion` | Enable FlashInfer allreduce fusion with Residual RMSNorm. | `False` | bool flag (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--deepep-mode` | Select the mode when enable DeepEP MoE, could be `normal`, `low_latency` or `auto`. Default is `auto`, which means `low_latency` for decode batch and `normal` for prefill batch. | `auto` | `normal`, `low_latency`, `auto` | **<span style="color: red;">×</span>** | **<span style="color: green;">√</span>** |
| `--deepep-config` | Tuned DeepEP config suitable for your own cluster. It can be either a string with JSON content or a file path. | `None` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--ep-num-redundant-experts` | Allocate this number of redundant experts in expert parallel. | `0` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--ep-dispatch-algorithm` | The algorithm to choose ranks for redundant experts in expert parallel. | `None` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--init-expert-location` | Initial location of EP experts. | `trivial` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
@@ -224,7 +226,6 @@ questions, please [open an issue](https://github.com/sgl-project/sglang/issues).
| `--expert-distribution-recorder-mode` | Mode of expert distribution recorder. | `None` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--expert-distribution-recorder-buffer-size` | Circular buffer size of expert distribution recorder. Set to -1 to denote infinite buffer. | `None` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--enable-expert-distribution-metrics` | Enable logging metrics for expert balancedness | `False` | bool flag (set to enable) | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--deepep-config` | Tuned DeepEP config suitable for your own cluster. It can be either a string with JSON content or a file path. | `None` | Type: str | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--moe-dense-tp-size` | TP size for MoE dense MLP layers. This flag is useful when, with large TP size, there are errors caused by weights in MLP layers having dimension smaller than the min dimension GEMM supports. | `None` | Type: int | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
| `--elastic-ep-backend` | Select the collective communication backend for elastic EP. Currently supports 'mooncake'. | None | N/A | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--mooncake-ib-device` | The InfiniBand devices for Mooncake Backend, accepts multiple comma-separated devices. Default is None, which triggers automatic device detection when Mooncake Backend is enabled. | None | N/A | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
@@ -236,7 +237,7 @@ questions, please [open an issue](https://github.com/sgl-project/sglang/issues).
| `--max-mamba-cache-size` | The maximum size of the mamba cache. | `None` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--mamba-ssm-dtype` | The data type of the SSM states in mamba cache. | `float32` | `float32`, `bfloat16` | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--mamba-full-memory-ratio` | The ratio of mamba state memory to full kv cache memory. | `0.2` | Type: float | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--mamba-scheduler-strategy` | The strategy to use for mamba scheduler. `auto` currently defaults to `no_buffer`. 1. `no_buffer` does not support overlap scheduler due to not allocating extra mamba state buffers. Branching point caching support is feasible but not implemented. 2. `extra_buffer` supports overlap schedule by allocating extra mamba state buffers to track mamba state for caching (mamba state usage per running req becomes `2x` for non-spec; `1+(1/(2+speculative_num_draft_tokens))x` for spec dec (e.g. 1.16x if speculative_num_draft_tokens==4)). 2a. `extra_buffer` is strictly better for non-KV-cache-bound cases; for KV-cache-bound cases, the tradeoff depends on whether enabling overlap outweighs reduced max running requests. 2b. mamba caching at radix cache branching point is strictly better than non-branch but requires kernel support (currently only FLA backend), currently only extra_buffer supports branching. | `auto` | `auto`, `no_buffer`, `extra_buffer` | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--mamba-scheduler-strategy` | The strategy to use for mamba scheduler. <br/>`auto` currently defaults to `no_buffer`. <br/>1. `no_buffer` does not support overlap scheduler due to not allocating extra mamba state buffers. Branching point caching support is feasible but not implemented. <br/>2. `extra_buffer` supports overlap schedule by allocating extra mamba state buffers to track mamba state for caching (mamba state usage per running req becomes `2x` for non-spec; `1+(1/(2+speculative_num_draft_tokens))x` for spec dec (e.g. 1.16x if speculative_num_draft_tokens==4)). <br/>2a. `extra_buffer` is strictly better for non-KV-cache-bound cases; for KV-cache-bound cases, the tradeoff depends on whether enabling overlap outweighs reduced max running requests. <br/>2b. mamba caching at radix cache branching point is strictly better than non-branch but requires kernel support (currently only FLA backend), currently only extra_buffer supports branching. | `auto` | `auto`, `no_buffer`, `extra_buffer` | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
| `--mamba-track-interval` | The interval (in tokens) to track the mamba state during decode. Only used when `--mamba-scheduler-strategy` is `extra_buffer`. Must be divisible by page_size if set, and must be >= speculative_num_draft_tokens when using speculative decoding. | `256` | Type: int | **<span style="color: red;">×</span>** | **<span style="color: red;">×</span>** |
## Args for multi-item scoring