diff --git a/docs_new/cookbook/autoregressive/InclusionAI/Ling-2.6.mdx b/docs_new/cookbook/autoregressive/InclusionAI/Ling-2.6.mdx index 5bbb2343f..c6cd1ebe9 100644 --- a/docs_new/cookbook/autoregressive/InclusionAI/Ling-2.6.mdx +++ b/docs_new/cookbook/autoregressive/InclusionAI/Ling-2.6.mdx @@ -45,7 +45,7 @@ import { Ling26FlashDeployment } from '/src/snippets/autoregressive/ling-26-flas - Native context is 128K. Enable YaRN (`--json-model-override-args '{"rope_scaling": {"rope_type": "yarn", "factor": 2.0, ...}}'`) to extend to 256K — the snippet does this for you. - `--tool-call-parser qwen25` matches the model's `...` schema. - The recommended baseline does **not** include `--reasoning-parser qwen3`. Ling-2.6 is a controllable-reasoning model whose chat template defaults to `detailed thinking off`; the SGLang `qwen3` reasoning parser, in contrast, assumes default-thinking semantics and would mis-route normal output into `reasoning_content`. Only enable it if you specifically want `...` blocks split out — see [§4.3 Thinking Mode](#4-3-thinking-mode). -- **MTP (multi-token prediction)** is supported. Add `--speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 --mamba-scheduler-strategy extra_buffer` to enable it — see the [model card](https://huggingface.co/inclusionAI/Ling-2.6-flash#run-inference) for the full example. +- **MTP (multi-token prediction)** is supported. Add `--speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4 --mamba-radix-cache-strategy extra_buffer` to enable it — see the [model card](https://huggingface.co/inclusionAI/Ling-2.6-flash#run-inference) for the full example. ### 3.2 Ling-2.6-1T diff --git a/docs_new/cookbook/autoregressive/InternLM/Intern-S2-Preview.mdx b/docs_new/cookbook/autoregressive/InternLM/Intern-S2-Preview.mdx index bcac878ac..f7c7a79d7 100644 --- a/docs_new/cookbook/autoregressive/InternLM/Intern-S2-Preview.mdx +++ b/docs_new/cookbook/autoregressive/InternLM/Intern-S2-Preview.mdx @@ -54,7 +54,7 @@ import { InternS2PreviewDeployment } from "/src/snippets/autoregressive/intern-s - Use `tp>=2` for the NVIDIA deployment commands. - Use `--reasoning-parser qwen3` to separate reasoning content from final content in streaming responses. - Use `--tool-call-parser qwen3_coder` when serving tool-calling workloads. -- Add `--mamba-scheduler-strategy extra_buffer` with `--speculative-algo 'NEXTN'` to enable MTP. +- Add `--mamba-radix-cache-strategy extra_buffer` with `--speculative-algo 'NEXTN'` to enable MTP. - If weight loading is slow, add `--model-loader-extra-config='{"enable_multithread_load": "true", "num_threads": 64}'`. ## 4. Model Invocation diff --git a/docs_new/cookbook/autoregressive/LiquidAI/LFM2.5.mdx b/docs_new/cookbook/autoregressive/LiquidAI/LFM2.5.mdx index e0a5d5301..6b9abb1f1 100644 --- a/docs_new/cookbook/autoregressive/LiquidAI/LFM2.5.mdx +++ b/docs_new/cookbook/autoregressive/LiquidAI/LFM2.5.mdx @@ -217,7 +217,7 @@ The Deploy panel above covers the eight serving variants; **LFM2.5-1.2B-JP** (or - **VL vision tower (`--mm-attention-backend`)**: on sm100 the `trtllm_mha` default is fastest for text but applies *causal* attention to image tokens. For the VL model, pass `--mm-attention-backend fa4` on B200/B300 (or `fa3` on H100/H200) to restore bidirectional image-token attention and full vision quality. - **VL multimodal feature transport**: the generator launches the VL models with `SGLANG_USE_CUDA_IPC_TRANSPORT=1 SGLANG_USE_IPC_POOL_HANDLE_CACHE=1`. The first moves the processor→scheduler image-feature handoff onto CUDA IPC instead of serializing tensors between processes; the second ships the pool handle so the scheduler opens it once and caches it, instead of opening a per-item handle on every request. On the image serving workload (1 image @ 720p, measured on VL-1.6B on H100 and B200) this pair is worth roughly 30–50% higher image throughput and 30–40% lower image TTFT vs running without them (measured on VL-1.6B, H100 and B200); decode speed (TPOT) is unaffected. - **VL-450M memory headroom (`--mem-fraction-static 0.8`)**: with the default memory fraction, the 450M's small weights make SGLang size its static KV/mamba pools to nearly the whole GPU, leaving no headroom for image-feature tensors — under sustained concurrent image load the scheduler can crash with a CUDA OOM in the radix-cache free path. The generator caps `--mem-fraction-static 0.8` for VL-450M; the pool is still far larger than this model ever needs. -- **Mamba scheduling**: LFM2.5 runs on the default `no_buffer` mamba scheduler strategy — no `--mamba-scheduler-strategy` flag is needed. The `extra_buffer` strategy (an overlap-scheduling throughput optimization available for some Gated-DeltaNet hybrids) does not apply to LFM2.5, whose convolution blocks use `mamba_chunk_size=1`. +- **Mamba scheduling**: LFM2.5 runs on the default `no_buffer` mamba scheduler strategy — no `--mamba-radix-cache-strategy` flag is needed. The `extra_buffer` strategy (an overlap-scheduling throughput optimization available for some Gated-DeltaNet hybrids) does not apply to LFM2.5, whose convolution blocks use `mamba_chunk_size=1`. - **Hardware requirements**: all LFM2.5 models run on a single GPU (TP=1) on either Hopper or Blackwell. The 1.2B / 350M dense models fit in a few GB; the 8B-A1B MoE needs roughly 16 GB for bf16 weights plus KV cache. Multi-GPU tensor parallelism is not required for any variant. **Recommended sampling parameters** — pass these explicitly on every request. Some LFM2.5 checkpoints do not ship sampling defaults in `generation_config.json`, so the server will not apply them for you. `top_k`, `min_p`, and `repetition_penalty` are not standard OpenAI `chat.completions` fields — pass them through **`extra_body`** and SGLang forwards them to its sampler. Do not set `max_tokens` unless you intend to cap output, as it can truncate a response (or a reasoning model's chain-of-thought) mid-stream. diff --git a/docs_new/cookbook/autoregressive/NVIDIA/Nemotron3-Ultra.mdx b/docs_new/cookbook/autoregressive/NVIDIA/Nemotron3-Ultra.mdx index ab379faab..66e6525e2 100644 --- a/docs_new/cookbook/autoregressive/NVIDIA/Nemotron3-Ultra.mdx +++ b/docs_new/cookbook/autoregressive/NVIDIA/Nemotron3-Ultra.mdx @@ -34,14 +34,10 @@ Available model variants on HuggingFace: ## 2. SGLang Installation -Nemotron3-Ultra support has not yet propagated to `lmsysorg/sglang:latest` or any stable release. Pull one of the two dedicated images below — matching your CUDA version — to get a runtime with Nemotron3-Ultra support. +Nemotron3-Ultra support is included in the latest stable release. ```bash Command -# CUDA 13 -docker pull lmsysorg/sglang:dev-nemotron3-ultra - -# CUDA 12 -docker pull lmsysorg/sglang:dev-cu12-nemotron3-ultra +docker pull lmsysorg/sglang:latest ``` ## 3. Model Deployment @@ -64,7 +60,7 @@ The generator only emits a runnable command for combinations that NVIDIA / SGLan - **Mamba scheduler strategy**: - Always launch with `--mamba-scheduler-strategy extra_buffer`. This hybrid Transformer-Mamba model requires the `extra_buffer` strategy for correct scheduling of its Mamba state. + Always launch with `--mamba-radix-cache-strategy extra_buffer`. This hybrid Transformer-Mamba model requires the `extra_buffer` strategy for correct scheduling of its Mamba state. - **Mamba backend**: @@ -119,7 +115,7 @@ python3 -m sglang.launch_server \ --model-path nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16 \ --trust-remote-code \ --tp 8 \ - --mamba-scheduler-strategy extra_buffer \ + --mamba-radix-cache-strategy extra_buffer \ --attention-backend trtllm_mha \ --tool-call-parser qwen3_coder \ --reasoning-parser nemotron_3 @@ -449,7 +445,7 @@ python3 -m sglang.launch_server \ --model-path nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4 \ --trust-remote-code \ --tp 4 \ - --mamba-scheduler-strategy extra_buffer \ + --mamba-radix-cache-strategy extra_buffer \ --attention-backend trtllm_mha \ --max-running-requests 1024 ``` @@ -525,7 +521,7 @@ python3 -m sglang.launch_server \ --model-path nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4 \ --trust-remote-code \ --tp 4 \ - --mamba-scheduler-strategy extra_buffer \ + --mamba-radix-cache-strategy extra_buffer \ --attention-backend trtllm_mha \ --reasoning-parser nemotron_3 ``` diff --git a/docs_new/cookbook/autoregressive/OpenBMB/MiniCPM-V-4_6.mdx b/docs_new/cookbook/autoregressive/OpenBMB/MiniCPM-V-4_6.mdx index fabed8ada..5d68802ab 100644 --- a/docs_new/cookbook/autoregressive/OpenBMB/MiniCPM-V-4_6.mdx +++ b/docs_new/cookbook/autoregressive/OpenBMB/MiniCPM-V-4_6.mdx @@ -51,9 +51,9 @@ import { MiniCPMV46Deployment } from '/src/snippets/autoregressive/minicpm-v-4_6 ### 3.2 Configuration Tips -- **Mamba Radix Cache**: Qwen3.5's hybrid Gated Delta Networks architecture supports two mamba scheduling strategies via `--mamba-scheduler-strategy`: +- **Mamba Radix Cache**: Qwen3.5's hybrid Gated Delta Networks architecture supports two mamba scheduling strategies via `--mamba-radix-cache-strategy`: - **V1 (`no_buffer`)**: Default. No overlap scheduler, lower memory usage. Required for AMD MI GPUs. - - **V2 (`extra_buffer`)**: Enables overlap scheduling and branching point caching with `--mamba-scheduler-strategy extra_buffer --page-size 64`. Requires FLA kernel backend (NVIDIA GPUs only). Trades higher mamba state memory for better throughput. Strictly superior in non-KV-cache-bound scenarios; in KV-cache-bound cases, weigh the overlap scheduling benefit against reduced max concurrency. `--page-size` must satisfy `FLA_CHUNK_SIZE % page_size == 0` or `page_size % FLA_CHUNK_SIZE == 0` (`FLA_CHUNK_SIZE` is currently 64). + - **V2 (`extra_buffer`)**: Enables overlap scheduling and branching point caching with `--mamba-radix-cache-strategy extra_buffer --page-size 64`. Requires FLA kernel backend (NVIDIA GPUs only). Trades higher mamba state memory for better throughput. Strictly superior in non-KV-cache-bound scenarios; in KV-cache-bound cases, weigh the overlap scheduling benefit against reduced max concurrency. `--page-size` must satisfy `FLA_CHUNK_SIZE % page_size == 0` or `page_size % FLA_CHUNK_SIZE == 0` (`FLA_CHUNK_SIZE` is currently 64). - The `--mem-fraction-static` flag is recommended for optimal memory utilization, adjust it based on your hardware and workload. - Context length defaults to 262,144 tokens. If you encounter OOM errors, consider reducing it, but maintain at least 128K to preserve thinking capabilities. - To speed up weight loading for this large model, add `--model-loader-extra-config='{"enable_multithread_load": "true","num_threads": 64}'` to the launch command. @@ -73,7 +73,7 @@ sglang serve --model-path openbmb/MiniCPM-V-4.6 \ --trust-remote-code \ --dtype bfloat16 \ --mem-fraction-static 0.15 \ - --mamba-scheduler-strategy extra_buffer \ + --mamba-radix-cache-strategy extra_buffer \ --page-size 64 \ --host 0.0.0.0 --port 30000 ``` @@ -307,7 +307,7 @@ CUDA_VISIBLE_DEVICES=0 python -m sglang.launch_server \ --trust-remote-code \ --dtype bfloat16 \ --mem-fraction-static 0.5 \ - --mamba-scheduler-strategy extra_buffer \ + --mamba-radix-cache-strategy extra_buffer \ --chunked-prefill-size -1 \ --host 0.0.0.0 --port 30000 ``` diff --git a/docs_new/cookbook/autoregressive/Qwen/Qwen3-Coder-Next.mdx b/docs_new/cookbook/autoregressive/Qwen/Qwen3-Coder-Next.mdx index 241cd256a..a6d900d5b 100644 --- a/docs_new/cookbook/autoregressive/Qwen/Qwen3-Coder-Next.mdx +++ b/docs_new/cookbook/autoregressive/Qwen/Qwen3-Coder-Next.mdx @@ -45,9 +45,9 @@ This section provides a progressive guide from quick deployment to performance o - **Context Length**: The model supports up to 256K tokens natively. If you encounter OOM issues, try `--context-length 32768`. - **Tool Use**: To enable tool calling capabilities, use the `--tool-call-parser qwen3_coder` flag. - **Sampling Parameters**: SGLang automatically applies the recommended sampling parameters from the model's `generation_config.json`. No manual configuration is needed. -- **Mamba Radix Cache**: Qwen3-Coder-Next's hybrid Gated Delta Networks architecture supports two mamba scheduling strategies via `--mamba-scheduler-strategy`: +- **Mamba Radix Cache**: Qwen3-Coder-Next's hybrid Gated Delta Networks architecture supports two mamba scheduling strategies via `--mamba-radix-cache-strategy`: - **V1 (`no_buffer`)**: Default. No overlap scheduler, lower memory usage. - - **V2 (`extra_buffer`)**: Enables overlap scheduling and branching point caching with `--mamba-scheduler-strategy extra_buffer --page-size 64`. Requires FLA kernel backend. Trades higher mamba state memory for better throughput. Strictly superior in non-KV-cache-bound scenarios; in KV-cache-bound cases, weigh the overlap scheduling benefit against reduced max concurrency. `--page-size` must satisfy `FLA_CHUNK_SIZE % page_size == 0` or `page_size % FLA_CHUNK_SIZE == 0` (`FLA_CHUNK_SIZE` is currently 64). + - **V2 (`extra_buffer`)**: Enables overlap scheduling and branching point caching with `--mamba-radix-cache-strategy extra_buffer --page-size 64`. Requires FLA kernel backend. Trades higher mamba state memory for better throughput. Strictly superior in non-KV-cache-bound scenarios; in KV-cache-bound cases, weigh the overlap scheduling benefit against reduced max concurrency. `--page-size` must satisfy `FLA_CHUNK_SIZE % page_size == 0` or `page_size % FLA_CHUNK_SIZE == 0` (`FLA_CHUNK_SIZE` is currently 64). - **Xeon CPU service configuration**: Please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings. ## 4. Model Invocation diff --git a/docs_new/cookbook/autoregressive/Qwen/Qwen3-Next.mdx b/docs_new/cookbook/autoregressive/Qwen/Qwen3-Next.mdx index f6941cb44..7577d7cea 100644 --- a/docs_new/cookbook/autoregressive/Qwen/Qwen3-Next.mdx +++ b/docs_new/cookbook/autoregressive/Qwen/Qwen3-Next.mdx @@ -54,9 +54,9 @@ The Qwen3-Next series comes in only one size but offers different thinking modes - `--mamba-full-memory-ratio`: Adjust `--mamba-full-memory-ratio` to set the ratio of mamba state memory to full kv cache memory. The default setting is `0.9`. -- **Mamba Radix Cache**: Qwen3-Next's hybrid Gated Delta Networks architecture supports two mamba scheduling strategies via `--mamba-scheduler-strategy`: +- **Mamba Radix Cache**: Qwen3-Next's hybrid Gated Delta Networks architecture supports two mamba scheduling strategies via `--mamba-radix-cache-strategy`: - **V1 (`no_buffer`)**: Default. No overlap scheduler, lower memory usage. - - **V2 (`extra_buffer`)**: Enables overlap scheduling and branching point caching with `--mamba-scheduler-strategy extra_buffer --page-size 64`. Requires FLA kernel backend. Trades higher mamba state memory for better throughput. Strictly superior in non-KV-cache-bound scenarios; in KV-cache-bound cases, weigh the overlap scheduling benefit against reduced max concurrency. `--page-size` must satisfy `FLA_CHUNK_SIZE % page_size == 0` or `page_size % FLA_CHUNK_SIZE == 0` (`FLA_CHUNK_SIZE` is currently 64). + - **V2 (`extra_buffer`)**: Enables overlap scheduling and branching point caching with `--mamba-radix-cache-strategy extra_buffer --page-size 64`. Requires FLA kernel backend. Trades higher mamba state memory for better throughput. Strictly superior in non-KV-cache-bound scenarios; in KV-cache-bound cases, weigh the overlap scheduling benefit against reduced max concurrency. `--page-size` must satisfy `FLA_CHUNK_SIZE % page_size == 0` or `page_size % FLA_CHUNK_SIZE == 0` (`FLA_CHUNK_SIZE` is currently 64). - **Xeon CPU service configuration**: Please refer to the `Notes` part in the serving engine launching section in [the SGLang CPU server document](../../../docs/hardware-platforms/cpu_server#launch-of-the-serving-engine) to better understand how to configure the arguments, especially for TP (tensor parallel) and NUMA binding settings. diff --git a/docs_new/cookbook/autoregressive/Qwen/Qwen3.5.mdx b/docs_new/cookbook/autoregressive/Qwen/Qwen3.5.mdx index 5ab2e64c5..895360dc5 100644 --- a/docs_new/cookbook/autoregressive/Qwen/Qwen3.5.mdx +++ b/docs_new/cookbook/autoregressive/Qwen/Qwen3.5.mdx @@ -129,9 +129,9 @@ This section provides deployment configurations optimized for different hardware - **H100 FP8:** Add `--enable-symm-mem` to enable NCCL symmetric memory for faster collectives and better performance under multi-GPU settings. - **AMD GPUs (MI300X / MI325X / MI355X):** Use `SGLANG_USE_AITER=1` and `SGLANG_USE_AITER_UNIFIED_ATTN=1` with `--attention-backend aiter`, which requires `--page-size 16` and can also enable `--enable-aiter-allreduce-fusion`. Additionally set `AITER_FLYDSL_FORCE=1` to force the AITER FlyDSL MoE kernels and `SGLANG_MAMBA_SSM_DTYPE=bfloat16` to store the Mamba SSM state in bfloat16 (instead of the default float32). - **Watchdog timeout:** Increase `--watchdog-timeout` to `1200` or higher for this large model, as weight loading can take significant time. -- **Mamba Radix Cache**: Qwen3.5's hybrid Gated Delta Networks architecture supports two mamba scheduling strategies via `--mamba-scheduler-strategy`: +- **Mamba Radix Cache**: Qwen3.5's hybrid Gated Delta Networks architecture supports two mamba scheduling strategies via `--mamba-radix-cache-strategy`: - **V1 (`no_buffer`)**: Default. No overlap scheduler, lower memory usage. Required for AMD MI GPUs. - - **V2 (`extra_buffer`)**: Enables overlap scheduling and branching point caching with `--mamba-scheduler-strategy extra_buffer --page-size 64`. Requires FLA kernel backend (NVIDIA GPUs only). Trades higher mamba state memory for better throughput. Strictly superior in non-KV-cache-bound scenarios; in KV-cache-bound cases, weigh the overlap scheduling benefit against reduced max concurrency. `--page-size` must satisfy `FLA_CHUNK_SIZE % page_size == 0` or `page_size % FLA_CHUNK_SIZE == 0` (`FLA_CHUNK_SIZE` is currently 64). + - **V2 (`extra_buffer`)**: Enables overlap scheduling and branching point caching with `--mamba-radix-cache-strategy extra_buffer --page-size 64`. Requires FLA kernel backend (NVIDIA GPUs only). Trades higher mamba state memory for better throughput. Strictly superior in non-KV-cache-bound scenarios; in KV-cache-bound cases, weigh the overlap scheduling benefit against reduced max concurrency. `--page-size` must satisfy `FLA_CHUNK_SIZE % page_size == 0` or `page_size % FLA_CHUNK_SIZE == 0` (`FLA_CHUNK_SIZE` is currently 64). - The `--mem-fraction-static` flag is recommended for optimal memory utilization, adjust it based on your hardware and workload. - Context length defaults to 262,144 tokens. If you encounter OOM errors, consider reducing it, but maintain at least 128K to preserve thinking capabilities. - To speed up weight loading for this large model, add `--model-loader-extra-config='{"enable_multithread_load": "true","num_threads": 64}'` to the launch command. diff --git a/docs_new/cookbook/autoregressive/Qwen/Qwen3.6.mdx b/docs_new/cookbook/autoregressive/Qwen/Qwen3.6.mdx index e48a6520e..58ffa02be 100644 --- a/docs_new/cookbook/autoregressive/Qwen/Qwen3.6.mdx +++ b/docs_new/cookbook/autoregressive/Qwen/Qwen3.6.mdx @@ -103,9 +103,9 @@ This section provides deployment configurations optimized for different hardware ### 3.2 Configuration Tips - Speculative decoding (MTP) can significantly reduce latency for interactive use cases. -- **Mamba Radix Cache**: Qwen3.6's hybrid Gated Delta Networks architecture supports two mamba scheduling strategies via `--mamba-scheduler-strategy`: +- **Mamba Radix Cache**: Qwen3.6's hybrid Gated Delta Networks architecture supports two mamba scheduling strategies via `--mamba-radix-cache-strategy`: - **V1 (`no_buffer`)**: Default. No overlap scheduler, lower memory usage. - - **V2 (`extra_buffer`)**: Enables overlap scheduling and branching point caching with `--mamba-scheduler-strategy extra_buffer --page-size 64`. Requires FLA kernel backend (NVIDIA GPUs only). Trades higher mamba state memory for better throughput. + - **V2 (`extra_buffer`)**: Enables overlap scheduling and branching point caching with `--mamba-radix-cache-strategy extra_buffer --page-size 64`. Requires FLA kernel backend (NVIDIA GPUs only). Trades higher mamba state memory for better throughput. - The `--mem-fraction-static` flag is recommended for optimal memory utilization, adjust it based on your hardware and workload. - Context length defaults to 262,144 tokens. If you encounter OOM errors, consider reducing it, but maintain at least 128K to preserve thinking capabilities. - **CUDA IPC Transport**: Add `SGLANG_USE_CUDA_IPC_TRANSPORT=1` as an environment variable to use CUDA IPC for transferring multimodal features, significantly improving TTFT (Time To First Token). Note: this consumes additional memory proportional to image size, so you may need to lower `--mem-fraction-static` or `--max-running-requests`. diff --git a/docs_new/docs/advanced_features/quantization.mdx b/docs_new/docs/advanced_features/quantization.mdx index fd2f6df24..d763d70ae 100644 --- a/docs_new/docs/advanced_features/quantization.mdx +++ b/docs_new/docs/advanced_features/quantization.mdx @@ -248,7 +248,7 @@ Backend selection applies to **blockwise FP8**, **MXFP8** (dense linear), and ** cutlass - SM90, SM100/120 + SM120 sgl-kernel CUTLASS @@ -264,7 +264,7 @@ Backend selection applies to **blockwise FP8**, **MXFP8** (dense linear), and ** -**`auto` selection order:** 1) DeepGEMM (SM90/SM100, installed); 2) FlashInfer TRTLLM (SM100, FlashInfer available); 3) CUTLASS (SM90/SM100/120); 4) AITER (AMD); 5) Triton. **Exception:** SM120 always resolves to Triton. +**`auto` selection order:** 1) DeepGEMM (SM90/SM100, installed); 2) FlashInfer TRTLLM (SM100, FlashInfer available); 3) CUTLASS (SM120); 4) AITER (AMD); 5) Triton (fallback). **MXFP8 dense linear:** `auto` uses `flashinfer_cutlass` on SM100 (else `triton`). `flashinfer_cutlass` is fastest on most shapes; `flashinfer_trtllm` is faster only at small M. diff --git a/docs_new/docs/advanced_features/server_arguments.mdx b/docs_new/docs/advanced_features/server_arguments.mdx index b85ee4cc3..c9815da81 100644 --- a/docs_new/docs/advanced_features/server_arguments.mdx +++ b/docs_new/docs/advanced_features/server_arguments.mdx @@ -1430,7 +1430,7 @@ Please consult the documentation below and [server_args.py](https://github.com/s `--fp8-gemm-backend` - Choose the runner backend for Blockwise FP8 GEMM operations. Options: 'auto' (default, auto-selects based on hardware), 'deep_gemm' (JIT-compiled; enabled by default on NVIDIA Hopper (SM90) and Blackwell (SM100) when DeepGEMM is installed), 'flashinfer_trtllm' (FlashInfer TRTLLM backend; SM100/SM103 only), 'flashinfer_cutlass' (FlashInfer CUTLASS backend, SM120 only), 'flashinfer_deepgemm' (Hopper SM90 only, uses swapAB optimization for small M dimensions in decoding), 'cutlass' (optimal for Hopper/Blackwell GPUs and high-throughput), 'triton' (fallback, widely compatible), 'aiter' (ROCm only). + Choose the runner backend for Blockwise FP8 GEMM operations. Options: 'auto' (default, auto-selects based on hardware), 'deep_gemm' (JIT-compiled; enabled by default on NVIDIA Hopper (SM90) and Blackwell (SM100) when DeepGEMM is installed), 'flashinfer_trtllm' (FlashInfer TRTLLM backend; SM100/SM103 only), 'flashinfer_cutlass' (FlashInfer CUTLASS backend, SM120 only), 'flashinfer_deepgemm' (Hopper SM90 only, uses swapAB optimization for small M dimensions in decoding), 'cutlass' (optimal for SM120 GPUs), 'triton' (fallback, widely compatible), 'aiter' (ROCm only). `auto` auto, deep_gemm, flashinfer_trtllm, flashinfer_cutlass, flashinfer_deepgemm, cutlass, triton, aiter @@ -1943,14 +1943,14 @@ Please consult the documentation below and [server_args.py](https://github.com/s Type: float - `--mamba-scheduler-strategy` + `--mamba-radix-cache-strategy` The strategy to use for mamba scheduler. auto currently defaults to no_buffer. 1. no_buffer does not support overlap scheduler due to not allocating extra mamba state buffers. Branching point caching support is feasible but not implemented. 2. extra_buffer supports overlap schedule by allocating extra mamba state buffers to track mamba state for caching (mamba state usage per running req becomes 2x for non-spec; 1+(1/(2+speculative_num_draft_tokens))x for spec dec (e.g. 1.16x if speculative_num_draft_tokens==4)). 2a. extra_buffer is strictly better for non-KV-cache-bound cases; for KV-cache-bound cases, the tradeoff depends on whether enabling overlap outweighs reduced max running requests. 2b. mamba caching at radix cache branching point is strictly better than non-branch but requires kernel support, currently only extra_buffer supports branching. `auto` auto, no_buffer, extra_buffer, extra_buffer_lazy `--mamba-track-interval` - The interval (in tokens) to track the mamba state during decode. Only used when --mamba-scheduler-strategy is extra_buffer. Must be divisible by page_size if set, and must be >= speculative_num_draft_tokens when using speculative decoding. + The interval (in tokens) to track the mamba state during decode. Only used when --mamba-radix-cache-strategy is extra_buffer. Must be divisible by page_size if set, and must be >= speculative_num_draft_tokens when using speculative decoding. `256` Type: int diff --git a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_support_features.mdx b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_support_features.mdx index 4c5b88f6b..513ccd98b 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_support_features.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/ascend_npu_support_features.mdx @@ -1779,7 +1779,7 @@ If the value is int8, you must also set the environment variable:DEEP_NORMAL_MOD A2, A3 - `--mamba-scheduler-strategy` + `--mamba-radix-cache-strategy` `auto` `auto`,
`no_buffer`,
`extra_buffer` A2, A3 diff --git a/docs_new/docs/hardware-platforms/ascend-npus/best_practice/qwen3_5_397b.mdx b/docs_new/docs/hardware-platforms/ascend-npus/best_practice/qwen3_5_397b.mdx index f1e3e5680..58947d8a1 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/best_practice/qwen3_5_397b.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/best_practice/qwen3_5_397b.mdx @@ -320,7 +320,7 @@ python3 -m sglang.launch_server \ --max-prefill-tokens 131072 \ --max-mamba-cache-size 320 \ --prefill-max-requests 10 \ - --mamba-scheduler-strategy extra_buffer \ + --mamba-radix-cache-strategy extra_buffer \ --trust-remote-code \ --max-running-requests 64 \ --mem-fraction-static 0.6 \ @@ -1110,7 +1110,7 @@ python3 -m sglang.launch_server \ --chunked-prefill-size -1 \ --max-prefill-tokens 65536 \ --max-mamba-cache-size 640 \ - --mamba-scheduler-strategy extra_buffer \ + --mamba-radix-cache-strategy extra_buffer \ --trust-remote-code \ --max-running-requests 128 \ --mem-fraction-static 0.6 \ diff --git a/docs_new/docs/hardware-platforms/ascend-npus/best_practice/qwen3_6_27b.mdx b/docs_new/docs/hardware-platforms/ascend-npus/best_practice/qwen3_6_27b.mdx index c39333847..687f09e3a 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/best_practice/qwen3_6_27b.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/best_practice/qwen3_6_27b.mdx @@ -284,7 +284,7 @@ python3 -m sglang.launch_server \ --device npu \ --chunked-prefill-size 32768 \ --max-prefill-tokens 32768 \ - --mamba-scheduler-strategy extra_buffer \ + --mamba-radix-cache-strategy extra_buffer \ --trust-remote-code \ --max-running-requests 20 \ --max-mamba-cache-size 160 \ diff --git a/docs_new/docs/hardware-platforms/ascend-npus/best_practice/qwen3_6_35b_a3b.mdx b/docs_new/docs/hardware-platforms/ascend-npus/best_practice/qwen3_6_35b_a3b.mdx index 0bf928ebd..d21063828 100644 --- a/docs_new/docs/hardware-platforms/ascend-npus/best_practice/qwen3_6_35b_a3b.mdx +++ b/docs_new/docs/hardware-platforms/ascend-npus/best_practice/qwen3_6_35b_a3b.mdx @@ -389,7 +389,7 @@ python3 -m sglang.launch_server \ --max-prefill-tokens 65536 \ --trust-remote-code \ --enable-prefill-delayer \ - --mamba-scheduler-strategy extra_buffer \ + --mamba-radix-cache-strategy extra_buffer \ --max-running-requests 103 \ --max-mamba-cache-size 85 \ --mem-fraction-static 0.85 \ @@ -780,7 +780,7 @@ python3 -m sglang.launch_server \ --max-total-tokens 470784 \ --max-prefill-tokens 65536 \ --trust-remote-code \ - --mamba-scheduler-strategy extra_buffer \ + --mamba-radix-cache-strategy extra_buffer \ --max-running-requests 40 \ --max-mamba-cache-size 200 \ --mem-fraction-static 0.9 \ diff --git a/docs_new/src/snippets/autoregressive/intern-s2-preview-deployment.jsx b/docs_new/src/snippets/autoregressive/intern-s2-preview-deployment.jsx index 7fa424207..1564afbb0 100644 --- a/docs_new/src/snippets/autoregressive/intern-s2-preview-deployment.jsx +++ b/docs_new/src/snippets/autoregressive/intern-s2-preview-deployment.jsx @@ -77,7 +77,7 @@ export const InternS2PreviewDeployment = () => { if (reasoning === 'enabled') flags.push(' --reasoning-parser qwen3'); if (toolcall === 'enabled') flags.push(' --tool-call-parser qwen3_coder'); if (mtp === 'enabled') { - flags.push(' --mamba-scheduler-strategy extra_buffer'); + flags.push(' --mamba-radix-cache-strategy extra_buffer'); flags.push(" --speculative-algo 'NEXTN'"); flags.push(' --speculative-eagle-topk 1'); flags.push(' --speculative-num-steps 3'); diff --git a/docs_new/src/snippets/autoregressive/minicpm-v-4_6-deployment.jsx b/docs_new/src/snippets/autoregressive/minicpm-v-4_6-deployment.jsx index 23f8f0ca4..901f9bbbf 100644 --- a/docs_new/src/snippets/autoregressive/minicpm-v-4_6-deployment.jsx +++ b/docs_new/src/snippets/autoregressive/minicpm-v-4_6-deployment.jsx @@ -98,7 +98,7 @@ export const MiniCPMV46Deployment = () => { cmd += ` \\\n --tool-call-parser qwen3_coder`; } if (mambaCache === 'v2') { - cmd += ` \\\n --mamba-scheduler-strategy extra_buffer`; + cmd += ` \\\n --mamba-radix-cache-strategy extra_buffer`; } cmd += ` \\\n --host 0.0.0.0 --port 30000`; diff --git a/docs_new/src/snippets/autoregressive/nemotron3-super-deployment.jsx b/docs_new/src/snippets/autoregressive/nemotron3-super-deployment.jsx index 47cbd53ac..7c0f8c447 100644 --- a/docs_new/src/snippets/autoregressive/nemotron3-super-deployment.jsx +++ b/docs_new/src/snippets/autoregressive/nemotron3-super-deployment.jsx @@ -43,7 +43,7 @@ export const Nemotron3SuperDeployment = () => { // trtllm_mha is Blackwell-only; on B200 it replaces the flashinfer default, // whose per-step plan() host-sync breaks the spec-v2 overlap scheduler. H200 // defaults to fa3 (no such sync), so no override is needed there. - commandRule: (value, state) => value === 'enabled' ? '--speculative-algorithm EAGLE \\\n --speculative-num-steps 3 \\\n --speculative-eagle-topk 1 \\\n --speculative-num-draft-tokens 4 \\\n --mamba-scheduler-strategy extra_buffer' + (state.hardware === 'b200' ? ' \\\n --attention-backend trtllm_mha' : '') : null + commandRule: (value, state) => value === 'enabled' ? '--speculative-algorithm EAGLE \\\n --speculative-num-steps 3 \\\n --speculative-eagle-topk 1 \\\n --speculative-num-draft-tokens 4 \\\n --mamba-radix-cache-strategy extra_buffer' + (state.hardware === 'b200' ? ' \\\n --attention-backend trtllm_mha' : '') : null }, kvcache: { name: 'kvcache', diff --git a/docs_new/src/snippets/autoregressive/nemotron3-ultra-deployment.jsx b/docs_new/src/snippets/autoregressive/nemotron3-ultra-deployment.jsx index 0b4755eee..537f2b54c 100644 --- a/docs_new/src/snippets/autoregressive/nemotron3-ultra-deployment.jsx +++ b/docs_new/src/snippets/autoregressive/nemotron3-ultra-deployment.jsx @@ -251,7 +251,7 @@ export const Nemotron3UltraDeployment = () => { } } - cmd += ` --mamba-scheduler-strategy extra_buffer \\\n`; + cmd += ` --mamba-radix-cache-strategy extra_buffer \\\n`; if (['b200', 'gb200', 'b300', 'gb300'].includes(hardware)) { cmd += ` --attention-backend trtllm_mha \\\n`; } diff --git a/docs_new/src/snippets/autoregressive/qwen3-coder-next-deployment.jsx b/docs_new/src/snippets/autoregressive/qwen3-coder-next-deployment.jsx index f36c588fd..c8104cbcb 100644 --- a/docs_new/src/snippets/autoregressive/qwen3-coder-next-deployment.jsx +++ b/docs_new/src/snippets/autoregressive/qwen3-coder-next-deployment.jsx @@ -39,7 +39,7 @@ export const Qwen3CoderNextDeployment = () => { { id: 'v1', label: 'V1', default: true }, { id: 'v2', label: 'V2', default: false } ], - commandRule: (value) => value === 'v2' ? '--mamba-scheduler-strategy extra_buffer \\\n --page-size 64' : null + commandRule: (value) => value === 'v2' ? '--mamba-radix-cache-strategy extra_buffer \\\n --page-size 64' : null } }; diff --git a/docs_new/src/snippets/autoregressive/qwen3-next-deployment.jsx b/docs_new/src/snippets/autoregressive/qwen3-next-deployment.jsx index dd886e4cf..17b0c61d7 100644 --- a/docs_new/src/snippets/autoregressive/qwen3-next-deployment.jsx +++ b/docs_new/src/snippets/autoregressive/qwen3-next-deployment.jsx @@ -65,7 +65,7 @@ export const Qwen3NextDeployment = () => { { id: 'v1', label: 'V1', default: true }, { id: 'v2', label: 'V2', default: false } ], - commandRule: (value) => value === 'v2' ? '--mamba-scheduler-strategy extra_buffer \\\n --page-size 64' : null + commandRule: (value) => value === 'v2' ? '--mamba-radix-cache-strategy extra_buffer \\\n --page-size 64' : null } }; diff --git a/docs_new/src/snippets/autoregressive/qwen35-deployment.jsx b/docs_new/src/snippets/autoregressive/qwen35-deployment.jsx index 879dcab4a..69795a7a0 100644 --- a/docs_new/src/snippets/autoregressive/qwen35-deployment.jsx +++ b/docs_new/src/snippets/autoregressive/qwen35-deployment.jsx @@ -359,7 +359,7 @@ export const Qwen35Deployment = () => { // Dense models with MTP off: force V1 — values.mambaCache is not // re-resolved on a speculative toggle (useEffect deps are hardware/model), // so it can stay at 'v2' from a prior MTP-on state. Reading it directly - // would emit a spurious --mamba-scheduler-strategy extra_buffer. The UI + // would emit a spurious --mamba-radix-cache-strategy extra_buffer. The UI // radio is hidden for dense models, so users can't manually correct it. // MoE keeps the old behavior — the UI radio is the recovery path there. const mamba_v1_dev = ['mi300x', 'mi325x', 'mi355x', 'xeon']; @@ -373,7 +373,7 @@ export const Qwen35Deployment = () => { reasoning: (value) => value === 'enabled' ? '--reasoning-parser qwen3' : null, toolcall: (value) => value === 'enabled' ? '--tool-call-parser qwen3_coder' : null, speculative: (value) => value === 'enabled' ? '--speculative-algorithm NEXTN \\\n --speculative-num-steps 3 \\\n --speculative-eagle-topk 1 \\\n --speculative-num-draft-tokens 4' : null, - mambaCache: (value) => value === 'v2' ? '--mamba-scheduler-strategy extra_buffer' : null, + mambaCache: (value) => value === 'v2' ? '--mamba-radix-cache-strategy extra_buffer' : null, }; // Iterate options in order, applying commandRules @@ -382,7 +382,7 @@ export const Qwen35Deployment = () => { // Skip options that don't pass their condition. mambaCache is special: // its condition gates only the UI radio (hidden for dense models), but // the rule still fires for dense models on NVIDIA + MTP to emit - // --mamba-scheduler-strategy extra_buffer. + // --mamba-radix-cache-strategy extra_buffer. if (option.condition && !option.condition(values) && (key !== 'mambaCache' || speculative !== 'enabled')) continue; const rule = commandRules[key]; if (rule) { diff --git a/docs_new/src/snippets/autoregressive/qwen36-deployment.jsx b/docs_new/src/snippets/autoregressive/qwen36-deployment.jsx index bcb967007..427ee8e06 100644 --- a/docs_new/src/snippets/autoregressive/qwen36-deployment.jsx +++ b/docs_new/src/snippets/autoregressive/qwen36-deployment.jsx @@ -85,7 +85,7 @@ export const Qwen36Deployment = () => { { id: 'v2', label: 'V2', default: false }, ]; }, - commandRule: (value) => value === 'v2' ? '--mamba-scheduler-strategy extra_buffer' : null, + commandRule: (value) => value === 'v2' ? '--mamba-radix-cache-strategy extra_buffer' : null, }, }; diff --git a/python/sglang/srt/server_args.py b/python/sglang/srt/server_args.py index 05ef12de9..d84093c4f 100644 --- a/python/sglang/srt/server_args.py +++ b/python/sglang/srt/server_args.py @@ -1427,7 +1427,7 @@ class ServerArgs: fp8_gemm_runner_backend: A[ str, Arg( - help="Choose the runner backend for Blockwise FP8 GEMM operations. Options: 'auto' (default, auto-selects based on hardware), 'deep_gemm' (JIT-compiled; enabled by default on NVIDIA Hopper (SM90) and Blackwell (SM100) when DeepGEMM is installed), 'flashinfer_trtllm' (optimal for Blackwell and low-latency), 'flashinfer_cutlass' (FlashInfer CUTLASS groupwise FP8 GEMM), 'flashinfer_deepgemm' (Hopper SM90 only; uses swapAB optimization for small M dimensions in decoding), 'cutlass' (optimal for Hopper/Blackwell GPUs and high-throughput), 'triton' (fallback, widely compatible), 'aiter' (ROCm only). ", + help="Choose the runner backend for Blockwise FP8 GEMM operations. Options: 'auto' (default, auto-selects based on hardware), 'deep_gemm' (JIT-compiled; enabled by default on NVIDIA Hopper (SM90) and Blackwell (SM100) when DeepGEMM is installed), 'flashinfer_trtllm' (optimal for Blackwell and low-latency), 'flashinfer_cutlass' (FlashInfer CUTLASS groupwise FP8 GEMM), 'flashinfer_deepgemm' (Hopper SM90 only; uses swapAB optimization for small M dimensions in decoding), 'cutlass' (optimal for SM120 GPUs), 'triton' (fallback, widely compatible), 'aiter' (ROCm only). ", cli_name="--fp8-gemm-backend", choices=FP8_GEMM_RUNNER_BACKEND_CHOICES, resolvable=True, @@ -2136,7 +2136,7 @@ class ServerArgs: "baseline (the per-K g_cache is K x larger and the reconstruction " "refolds the per-K decay every step), so it is not recommended for KDA " "models. Requires the Triton linear-attn decode backend and " - "--mamba-scheduler-strategy no_buffer (the default).", + "--mamba-radix-cache-strategy no_buffer (the default).", ] = False linear_replayssm_cache_len: A[ int, @@ -5221,10 +5221,10 @@ class ServerArgs: if mamba_extra_buffer_of(resolved_view(self)): raise ValueError( - "--enable-linear-replayssm requires --mamba-scheduler-strategy " + "--enable-linear-replayssm requires --mamba-radix-cache-strategy " "no_buffer (the default); the extra_buffer ping-pong " "donation path is not yet supported (follow-up). Got " - f"--mamba-scheduler-strategy={self.mamba_scheduler_strategy!r}." + f"--mamba-radix-cache-strategy={self.mamba_radix_cache_strategy!r}." ) if self.disaggregation_mode != "null": # The disaggregated decode pool (HybridMambaDecodeReqToTokenPool)