diff --git a/docs_new/docs.json b/docs_new/docs.json index 2780ce2c5..8a7a8541f 100644 --- a/docs_new/docs.json +++ b/docs_new/docs.json @@ -711,8 +711,7 @@ "docs/basic_usage/ollama_api", "docs/basic_usage/offline_engine_api", "docs/basic_usage/native_api", - "docs/basic_usage/sampling_params", - "docs/basic_usage/aws_sagemaker" + "docs/basic_usage/sampling_params" ] }, { @@ -721,7 +720,6 @@ "pages": [ "docs/advanced_features/overview", "docs/advanced_features/server_arguments", - "docs/advanced_features/object_storage", "docs/advanced_features/hyperparameter_tuning", "docs/advanced_features/attention_backend", "docs/advanced_features/hisparse_guide", @@ -757,6 +755,8 @@ "docs/advanced_features/llm-d", "docs/advanced_features/deterministic_inference", "docs/advanced_features/observability", + "docs/advanced_features/model_loading", + "docs/advanced_features/object_storage", "docs/advanced_features/checkpoint_engine", "docs/advanced_features/sglang_for_rl" ] @@ -833,13 +833,14 @@ "docs/references/production_metrics", "docs/references/production_request_trace", { - "group": "Multi-Node Deployment", + "group": "Deployment", "pages": [ "docs/references/multi_node_deployment/multi_node_index", "docs/references/multi_node_deployment/multi_node", "docs/references/multi_node_deployment/deploy_on_k8s", "docs/references/multi_node_deployment/lws_pd/lws_pd_deploy", - "docs/references/multi_node_deployment/rbg_pd/deepseekv32_pd" + "docs/references/multi_node_deployment/rbg_pd/deepseekv32_pd", + "docs/basic_usage/aws_sagemaker" ] }, "docs/references/custom_chat_template", diff --git a/docs_new/docs/advanced_features/attention_backend.mdx b/docs_new/docs/advanced_features/attention_backend.mdx index abcb115de..966234b7d 100644 --- a/docs_new/docs/advanced_features/attention_backend.mdx +++ b/docs_new/docs/advanced_features/attention_backend.mdx @@ -244,6 +244,15 @@ The support matrix is split into two parts: MHA (standard attention) and MLA (mu ✅ ❌ + + **TokenSpeed MLA (Blackwell)** + 32 or 64 + ✅ (required) + ❌ + ✅ + ✅ + ❌ + **FA3 (FlashAttention 3)** n/a @@ -288,14 +297,9 @@ Multimodal attention is selected by `--mm-attention-backend`. The "MultiModal" c -- FlashAttention 4 supports both prefill and decode on SM90 (Hopper) and SM100 (Blackwell). FA4 MLA supports `page_size = 1`; FA4 MHA requires `page_size = 128`. On SM100, this is auto-enforced by the server; on SM90, users must set `--page-size 128` manually. - DSA is specifically designed for [DeepSeek V3.2](https://lmsys.org/blog/2025-09-29-deepseek-V32/). See the [DSA Attention Backend](#dsa-attention-backend) section and [DeepSeek V3.2 deployment guide](/cookbook/autoregressive/DeepSeek/DeepSeek-V3_2) for details. - -**FA4 on Hopper (SM90):** FA4 decode speed decreases as sequence length grows due to lack of SplitKV support. At batch=1 compared to FA3 on H100: ~-10% at 2K tokens, ~-18% at 4K, ~-31% at 8K, ~-49% at 16K. Larger batch sizes reduce the gap (e.g., batch=8: ~-2% at 2K, ~-8% at 4K). Blackwell (SM100) is not affected. - - For the KV4 FA4 scenario, FA4 requires using a different --decode-attention-backend to run. Except for trtllm_mha being incompatible with FA4, all other decode backends behave as shown in the table. @@ -304,14 +308,6 @@ For the KV4 FA4 scenario, FA4 requires using a different --decode-attention-back Speculative decoding topk: `topk` is the number of draft tokens sampled per step from the draft model. `topk = 1` follows classic EAGLE; `topk > 1` explores multiple branches and requires backend support in both draft and verification paths. - -**Speculative Decoding V2 (Spec V2):** speculative decoding runs the V2 workers with overlap scheduling by default, which benefits various attention backends. Requires `--speculative-eagle-topk 1`. - -**Verified backends:** TRTLLM MLA, TRTLLM MHA, FA3, Ascend (NPU), Triton. - -**Limited support:** FlashInfer can run under Spec V2, but its plan stream (used for split-KV optimization) introduces a synchronization point that limits overlap benefits. - - Page size controls how many tokens are grouped into a KV cache block. For the prefix cache to take effect, the number of tokens must fill at least one complete page. For example, if your prompt is only 32 tokens and `page_size = 64`, it won't fill a complete page and cannot be matched in the prefix cache (pages cannot be padded). With 65 tokens and `page_size = 64`, only the first page of 64 tokens will be cached and matched; the remaining 1 token is discarded. Use `page_size = 1` for maximum prefix reuse (token-level matching). Note that higher page sizes generally improve attention kernel performance, so prefer `page_size > 1` when prefix cache reuse is not critical. @@ -324,6 +320,7 @@ MLA page-size constraints: - Cutlass MLA: page_size = 128. - TRTLLM MLA: page_size ∈ {32, 64}. - CuteDSL MLA: page_size ∈ {32, 64} (decode-only; prefill falls back to `trtllm_mla` when unset). +- TokenSpeed MLA: page_size ∈ {32, 64} (Blackwell SM100/SM12x only; requires `--kv-cache-dtype fp8_e4m3`). ### GDN Attention Backends @@ -414,31 +411,31 @@ Internally, the DSA backend dispatches to different sub-backends for prefill and flashmla_sparse ✅ ✅ - Default prefill on Hopper and Blackwell (bf16) + Default prefill on Hopper and Blackwell (BF16) flashmla_kv ✅ ✅ - Default decode for FP8 on Blackwell with DP + Default for FP8 on Hopper (prefill + decode) flashmla_auto ✅ ❌ - Auto-selects flashmla_sparse or flashmla_kv based on kv_cache_dtype + Picks flashmla_sparse or flashmla_kv by KV cache dtype fa3 ✅ ✅ - Default decode on Hopper (bf16) + Default decode on Hopper (BF16) trtllm ✅ ✅ - Default decode on Blackwell (bf16); default for both on Blackwell without DP + Default decode on Blackwell (BF16); default for FP8 on Blackwell (prefill + decode) tilelang diff --git a/docs_new/docs/advanced_features/expert_parallelism.mdx b/docs_new/docs/advanced_features/expert_parallelism.mdx index 1e0e19655..0f012640d 100644 --- a/docs_new/docs/advanced_features/expert_parallelism.mdx +++ b/docs_new/docs/advanced_features/expert_parallelism.mdx @@ -241,20 +241,6 @@ To enable EPLB, use the flags `--enable-eplb`. For optimal performance, increase For more details, refer to the [EPLB Section in the Large-Scale EP Blog](https://lmsys.org/blog/2025-05-05-large-scale-ep/#expert-parallelism-load-balancer) and the [EPLB Repository](https://github.com/deepseek-ai/eplb). -## EP with Spectulative Decoding - - -When utilizing speculative decoding with MTP on MoE architectures, use the `--speculative-moe-runner-backend` and `--speculative-moe-a2a-backend` arguments to customize the MoE layer behavior for the draft model. While they default to the target model’s settings, users can differentiate them for varying precisions between target and draft models. - -For model like `nvidia/DeepSeek-R1-0528-NVFP4-v2`, the target model uses NVFP4 precision while the draft model uses BF16. To apply `flashinfer_trtllm` kernel for target MoE layer while falling back to triton fused MoE kernel for draft MoE layer, users can set the arguments as follows: -```text Output -... ---moe-runner-backend flashinfer_trtllm \ ---speculative-moe-runner-backend triton \ -... -``` - - ## Ascend NPU Guidance ### Guidance on SGLang configuration in Ascend NPU - `--moe-a2a-backend` only supports `deepep` and `ascend_fuseep` backends, diff --git a/docs_new/docs/advanced_features/model_loading.mdx b/docs_new/docs/advanced_features/model_loading.mdx new file mode 100644 index 000000000..173b53568 --- /dev/null +++ b/docs_new/docs/advanced_features/model_loading.mdx @@ -0,0 +1,218 @@ +--- +title: "Model Loading" +description: "Control how SGLang loads model weights: load formats, model loader extra config, multithreaded loading, prefetching, and remote/streaming loaders." +--- + +`--model-path` selects the checkpoint to serve; `--load-format` and the weight-loading flags below control how those weights are read into memory. To stream weights from cloud object storage (S3/GCS/Azure), see [Loading Models from Object Storage](./object_storage). + +## How loading works + +SGLang picks a loader from `--load-format`, falling back to auto-detection from the checkpoint or model path. The default `auto` loader reads `safetensors` and falls back to PyTorch `.bin`. + +```bash +python -m sglang.launch_server \ + --model-path Qwen/Qwen3.6-35B-A3B \ + --load-format auto +``` + +Some formats are auto-detected and override `auto`: + +- A Mistral native checkpoint is detected and loaded with `mistral`. +- A `.gguf` model path is detected and loaded with `gguf`. +- An object storage URI (`s3://`, `gs://`, `az://`) is loaded with `runai_streamer`. +- A remote URI is loaded with `remote`. + +## Load formats + +Set with `--load-format`: + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
FormatDescription
autoDefault. Load safetensors if available, otherwise fall back to the PyTorch .bin format.
safetensorsLoad weights in the safetensors format.
ptLoad weights in the PyTorch .bin format.
npcacheLoad PyTorch-format weights and store a numpy cache to speed up subsequent loads. Only supports .bin checkpoints.
dummyInitialize weights with random values, for profiling.
sharded_stateEach tensor-parallel worker reads only its own pre-sharded shard rather than the full checkpoint, giving a fast load path for large TP models. See examples/runtime/engine/save_sharded_state.py for creating a sharded checkpoint.
fastsafetensorsLoad safetensors using the fastsafetensors iterator.
layeredLoad weights layer by layer, so a layer can be quantized before the next is loaded, lowering the peak memory envelope.
ggufLoad weights in the GGUF format. Auto-detected from a .gguf model path.
bitsandbytesLoad weights using bitsandbytes quantization.
mistralLoad a Mistral native-format checkpoint. Auto-detected for such checkpoints.
flash_rlLoad a BF16/FP16 checkpoint with native SGLang FP8 quantization for RL training. Requires --rl-quant-profile.
runai_streamerStream weights from SSDs, shared filesystems, or object storage. See Loading Models from Object Storage.
remoteLoad tensors from a remote KV/filesystem connector. Auto-detected for remote URIs.
remote_instancePull weights over the network from another running SGLang instance (the "seed") rather than from disk. Configured with the --remote-instance-weight-loader-* flags.
+ +## Model loader extra config + +`--model-loader-extra-config` takes a JSON string passed to the loader selected by `--load-format`. + +```bash +python -m sglang.launch_server \ + --model-path Qwen/Qwen3.6-35B-A3B \ + --model-loader-extra-config '{"enable_multithread_load": true, "num_threads": 16}' +``` + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Load formatKeyDescriptionDefault
auto / safetensors / pt / npcacheenable_multithread_load (bool)Read weight shards with a thread pool instead of sequentially.true
auto / safetensors / pt / npcachenum_threads (int)Number of worker threads when multithreaded loading is enabled.8
sharded_statepattern (str)Filename pattern for per-rank shards.model-rank-{rank}-part-{part}.safetensors
bitsandbytesqlora_adapter_name_or_path (str)QLoRA adapter to apply on top of the bitsandbytes-quantized base weights.—
runai_streamerdistributed, concurrency, memory_limitStreaming controls. See Loading Models from Object Storage.See linked page
+ +## Weight-loading performance flags + +Top-level arguments that tune how safetensors weights are read, independent of `--load-format`. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
FlagDescriptionDefault
--download-dirDirectory used to download and cache Hugging Face model files.HF default
--weight-loader-disable-mmapDisable mmap while loading safetensors. Can help on filesystems where mmap is slow.off
--weight-loader-prefetch-checkpointsPrefetch checkpoint files into the OS page cache before loading. Each rank prefetches a fraction of the shards, cutting total network I/O on shared filesystems (NFS/Lustre) from N×checkpoint to 1×checkpoint. Recommended for models on network storage.off
--weight-loader-prefetch-num-threadsThreads per rank for checkpoint prefetching.4
--weight-loader-drop-cache-after-loadCall posix_fadvise(DONTNEED) on each safetensors shard after loading it, freeing page cache.off
--custom-weight-loaderImport path(s) of a custom weight-loading function, e.g. my_package.weight_load_func.—
+ +## See also + +- [Loading Models from Object Storage](./object_storage) +- [Quantization](./quantization) +- [Server Arguments](./server_arguments) diff --git a/docs_new/docs/advanced_features/object_storage.mdx b/docs_new/docs/advanced_features/object_storage.mdx index a6de5a206..3acf1e0ed 100644 --- a/docs_new/docs/advanced_features/object_storage.mdx +++ b/docs_new/docs/advanced_features/object_storage.mdx @@ -4,7 +4,7 @@ metatags: description: "Load SGLang models directly from S3, Google Cloud Storage, Azure Blob, and S3-compatible object storage with runai_streamer." --- -SGLang supports direct loading of models from object storage (S3 and Google Cloud Storage) without requiring a full local download. This feature uses the `runai_streamer` load format to stream model weights directly from cloud storage, significantly reducing startup time and local storage requirements. +SGLang can load models directly from object storage without a full local download. It uses the `runai_streamer` load format to stream model weights from cloud storage, reducing startup time and local storage requirements. ## Overview @@ -58,7 +58,7 @@ python -m sglang.launch_server \ ### Load Format -The `runai_streamer` load format is specifically designed for object storage, ssd and shared file systems +The `runai_streamer` load format is designed for object storage, SSDs, and shared filesystems. ```bash python -m sglang.launch_server \ @@ -101,7 +101,7 @@ python -m sglang.launch_server \ distributed bool - Enable distributed streaming for multi-GPU setups. Automatically set to true for object storage paths and cuda alike devices. + Enable distributed streaming for multi-GPU setups. Automatically set to true for object storage paths on CUDA-like devices. Auto-detected @@ -123,7 +123,7 @@ python -m sglang.launch_server \ ### Distributed Streaming -For multi-GPU setups, enable distributed streaming to parallelize weight loading between the processes: +For multi-GPU setups, enable distributed streaming to parallelize weight loading across processes: ```bash python -m sglang.launch_server \ @@ -134,8 +134,8 @@ python -m sglang.launch_server \ ## Limitations -- **Supported Formats**: Currently only supports `.safetensors` weight format (recommended format) -- **Supported Device**: Distributed streaming is supported on cuda alike devices. Otherwise fallback to non distributed streaming +- **Supported formats**: Only the `.safetensors` weight format is supported. +- **Supported devices**: Distributed streaming is supported on CUDA-like devices; otherwise it falls back to non-distributed streaming. ## See Also diff --git a/docs_new/docs/advanced_features/server_arguments.mdx b/docs_new/docs/advanced_features/server_arguments.mdx index 5d7e1ddd1..ddf20a9c0 100644 --- a/docs_new/docs/advanced_features/server_arguments.mdx +++ b/docs_new/docs/advanced_features/server_arguments.mdx @@ -96,91 +96,109 @@ Please consult the documentation below and [server_args.py](https://github.com/s - - ` --model-path`
`--model` + + `--model-path`
`--model` The path of the model weights. This can be a local folder or a Hugging Face repo ID. ` None` Type: str - - ` --tokenizer-path` + + `--tokenizer-path` The path of the tokenizer. ` None` Type: str - - ` --tokenizer-mode` + + `--tokenizer-mode` Tokenizer mode. 'auto' will use the fast tokenizer if available, and 'slow' will always use the slow tokenizer. - ` auto` + `auto` auto, slow - - ` --tokenizer-backend` + + `--tokenizer-backend` Tokenizer backend. 'huggingface' uses the default HuggingFace tokenizers library; 'fastokens' uses the fastokens library for faster tokenization. Requires the fastokens package to be installed. - ` huggingface` + `huggingface` huggingface, fastokens - - ` --tokenizer-worker-num` + + `--tokenizer-worker-num` The worker num of the tokenizer manager. - ` 1` + `1` Type: int - - ` --skip-tokenizer-init` + + `--skip-tokenizer-init` If set, skip init tokenizer and pass input_ids in generate request. - ` False` + `False` bool flag (set to enable) - - ` --load-format` + + `--load-format` The format of the model weights to load. "auto" will try to load the weights in the safetensors format and fall back to the pytorch bin format if safetensors format is not available. "pt" will load the weights in the pytorch bin format. "safetensors" will load the weights in the safetensors format. "npcache" will load the weights in pytorch format and store a numpy cache to speed up the loading. "dummy" will initialize the weights with random values, which is mainly for profiling."gguf" will load the weights in the gguf format. "bitsandbytes" will load the weights using bitsandbytes quantization."layered" loads weights layer by layer so that one can quantize a layer before loading another to make the peak memory envelope smaller. "flash_rl" will load the weights in flash_rl format. "fastsafetensors" and "private" are also supported. "runai_streamer" enables direct model loading from object storage and shared file systems. - ` auto` - auto, pt, safetensors, npcache, dummy, sharded_state, gguf, bitsandbytes, layered, flash_rl, remote, remote_instance, fastsafetensors, private, runai_streamer + `auto` + auto, pt, safetensors, npcache, dummy, sharded_state, gguf, bitsandbytes, mistral, layered, flash_rl, remote, remote_instance, fastsafetensors, private, runai_streamer - - ` --model-loader-extra-config` + + `--model-loader-extra-config` Extra config for model loader. This will be passed to the model loader corresponding to the chosen load_format. - ` {}` + `{}` Type: str - - ` --trust-remote-code` + + `--trust-remote-code` Whether or not to allow for custom models defined on the Hub in their own modeling files. - ` False` + `False` bool flag (set to enable) - - ` --context-length` + + `--context-length` The model's maximum context length. Defaults to None (will use the value from the model's config.json instead). ` None` Type: int - - ` --is-embedding` + + `--is-embedding` Whether to use a CausalLM as an embedding model. - ` False` + `False` bool flag (set to enable) - - ` --enable-multimodal` + + `--enable-multimodal` Enable the multimodal functionality for the served model. If the model being served is not multimodal, nothing will happen. ` None` bool flag (set to enable) - - ` --revision` + + `--revision` The specific model version to use. It can be a branch name, a tag name, or a commit id. If unspecified, will use the default version. ` None` Type: str - - ` --model-impl` + + `--model-impl` Which implementation of the model to use. "auto" will try to use the SGLang implementation if it exists and fall back to the Transformers implementation if no SGLang implementation is available. "sglang" will use the SGLang model implementation. "transformers" will use the Transformers model implementation. - ` auto` + `auto` Type: str - + + `--detokenizer-worker-num` + The worker num of the detokenizer manager. + `1` + Type: int + + + `--prefill-only-disable-kv-cache` + Skip the physical KV cache allocation for embedding-mode prefill-only workloads. Currently only valid with --is-embedding, --chunked-prefill-size=-1, --disable-radix-cache, an FA prefill backend, and non-FP4 KV cache so the fa_skip_kv_cache path is active (no layer reads or writes the cache). Other prefill-only workloads such as scoring/MIS may benefit from this later once their attention paths stop using paged KV. Scheduler admission accounting is unchanged; per-layer K/V tensors are sized to (page_size, head_num, head_dim) placeholders so GPU memory is not wasted. + `False` + bool flag (set to enable) + + + `--model-config-parser` + Which model-config parser to use. "auto" picks "mistral" via the is_mistral_model name heuristic, else "hf" (AutoConfig over config.json). Plugins can register additional parsers via @register_model_config_parser. + `auto` + Type: str + + ## HTTP server @@ -200,55 +218,91 @@ Please consult the documentation below and [server_args.py](https://github.com/s - + `--host` The host of the HTTP server. `127.0.0.1` Type: str - + `--port` The port of the HTTP server. `30000` Type: int - + `--fastapi-root-path` App is behind a path based routing proxy. `""` Type: str - + `--grpc-mode` If set, use gRPC server instead of HTTP server. `False` bool flag (set to enable) - + `--skip-server-warmup` If set, skip warmup. `False` bool flag (set to enable) - + `--warmups` Specify custom warmup functions (csv) to run before server starts eg. --warmups=warmup_name1,warmup_name2 will run the functions `warmup_name1` and `warmup_name2` specified in warmup.py before the server starts listening for requests `None` Type: str - + `--nccl-port` The port for NCCL distributed environment setup. Defaults to a random port. `None` Type: int - + `--checkpoint-engine-wait-weights-before-ready` If set, the server will wait for initial weights to be loaded via checkpoint-engine or other update methods before serving inference requests. `False` bool flag (set to enable) - + + `--ssl-keyfile` + The file path to the SSL key file. + `None` + Type: str + + + `--ssl-certfile` + The file path to the SSL certificate file. + `None` + Type: str + + + `--ssl-ca-certs` + The CA certificates file. + `None` + Type: str + + + `--ssl-keyfile-password` + The password to decrypt the SSL keyfile. + `None` + Type: str + + + `--enable-ssl-refresh` + Enable automatic SSL certificate hot-reloading when cert/key files change on disk. Requires --ssl-certfile and --ssl-keyfile. + `False` + bool flag (set to enable) + + + `--enable-http2` + Use Granian instead of Uvicorn as the ASGI server, enabling HTTP/1.1 and HTTP/2 auto-negotiation. Clients may use h2c (cleartext HTTP/2) or plain HTTP/1.1. Requires 'pip install sglang[http2]'. + `False` + bool flag (set to enable) + + ## Quantization and data type @@ -268,73 +322,73 @@ Please consult the documentation below and [server_args.py](https://github.com/s - + `--dtype` Data type for model weights and activations. * "auto" will use FP16 precision for FP32 and FP16 models, and BF16 precision for BF16 models. * "half" for FP16. Recommended for AWQ quantization. * "float16" is the same as "half". * "bfloat16" for a balance between precision and range. * "float" is shorthand for FP32 precision. * "float32" for FP32 precision. `auto` auto, half, float16, bfloat16, float, float32 - + `--quantization` The quantization method. `None` - awq, fp8, gptq, marlin, gptq_marlin, awq_marlin, bitsandbytes, gguf, modelopt, modelopt_fp8, modelopt_fp4, nvfp4_online, petit_nvfp4, w8a8_int8, w8a8_fp8, moe_wna16, qoq, w4afp8, mxfp4, mxfp8, auto-round, compressed-tensors, modelslim, quark_int4fp8_moe + awq, fp8, mxfp8, gptq, marlin, gptq_marlin, awq_marlin, bitsandbytes, gguf, modelopt, modelopt_fp8, modelopt_fp4, nvfp4_online, modelopt_mixed, petit_nvfp4, w8a8_int8, w8a8_fp8, moe_wna16, qoq, w4afp8, mxfp4, auto-round, compressed-tensors, modelslim, quark, quark_int4fp8_moe, quark_mxfp4, mlx_q4, mlx_q8, unquant - + `--quantization-param-path` Path to the JSON file containing the KV cache scaling factors. This should generally be supplied, when KV cache dtype is FP8. Otherwise, KV cache scaling factors default to 1.0, which may cause accuracy issues. `None` Type: Optional[str] - + `--kv-cache-dtype` Data type for kv cache storage. "auto" will use model data type. "bf16" or "bfloat16" for BF16 KV cache. "fp8_e5m2" and "fp8_e4m3" are supported for CUDA 11.8+. "fp4_e2m1" (only mxfp4) is supported for CUDA 12.8+ and PyTorch 2.8.0+ `auto` auto, fp8_e5m2, fp8_e4m3, bf16, bfloat16, fp4_e2m1 - + `--enable-fp32-lm-head` If set, the LM head outputs (logits) are in FP32. `False` bool flag (set to enable) - + `--modelopt-quant` The ModelOpt quantization configuration. Supported values: 'fp8', 'int4_awq', 'w4a8_awq', 'nvfp4', 'nvfp4_awq'. This requires the NVIDIA Model Optimizer library to be installed: pip install nvidia-modelopt `None` Type: str - + `--modelopt-checkpoint-restore-path` Path to restore a previously saved ModelOpt quantized checkpoint. If provided, the quantization process will be skipped and the model will be loaded from this checkpoint. `None` Type: str - + `--modelopt-checkpoint-save-path` Path to save the ModelOpt quantized checkpoint after quantization. This allows reusing the quantized model in future runs. `None` Type: str - + `--modelopt-export-path` Path to export the quantized model in HuggingFace format after ModelOpt quantization. The exported model can then be used directly with SGLang for inference. If not provided, the model will not be exported. `None` Type: str - + `--quantize-and-serve` Quantize the model with ModelOpt and immediately serve it without exporting. This is useful for development and prototyping. For production, it's recommended to use separate quantization and deployment steps. `False` bool flag (set to enable) - + `--rl-quant-profile` Path to the FlashRL quantization profile. Required when using --load-format flash_rl. `None` Type: str - + `--enable-quant-communications` Enable INT8 quantization of TP communications (Supported only for NPU for Qwen3 series). `False` @@ -360,157 +414,169 @@ Please consult the documentation below and [server_args.py](https://github.com/s - + `--mem-fraction-static` The fraction of the memory used for static allocation (model weights and KV cache memory pool). Use a smaller value if you see out-of-memory errors. `None` Type: float - + `--max-running-requests` The maximum number of running requests. `None` Type: int - + `--max-queued-requests` The maximum number of queued requests. This option is ignored when using disaggregation-mode. `None` Type: int - + `--max-total-tokens` The maximum number of tokens in the memory pool. If not specified, it will be automatically calculated based on the memory usage fraction. This option is typically used for development and debugging purposes. `None` Type: int - + `--chunked-prefill-size` The maximum number of tokens in a chunk for the chunked prefill. Setting this to -1 means disabling chunked prefill. `None` Type: int - + `--prefill-max-requests` The maximum number of requests in a prefill batch. If not specified, there is no limit. `None` Type: int - + `--enable-dynamic-chunking` Enable dynamic chunk size adjustment for pipeline parallelism. When enabled, chunk sizes are dynamically calculated based on fitted function to maintain consistent execution time across chunks. `False` bool flag (set to enable) - + `--max-prefill-tokens` The maximum number of tokens in a prefill batch. The real bound will be the maximum of this value and the model's maximum context length. `16384` Type: int - + `--schedule-policy` The scheduling policy of the requests. `fcfs` - `lpm`, `random`, `fcfs`, `dfs-weight`, `lof`, `priority`, `routing-key` + lpm, random, fcfs, dfs-weight, lof, priority, routing-key - + `--enable-priority-scheduling` Enable priority scheduling. Requests with higher priority integer values will be scheduled first by default. `False` bool flag (set to enable) - + `--abort-on-priority-when-disabled` If set, abort requests that specify a priority when priority scheduling is disabled. `False` bool flag (set to enable) - + `--schedule-low-priority-values-first` If specified with --enable-priority-scheduling, the scheduler will schedule requests with lower priority integer values first. `False` bool flag (set to enable) - + `--priority-scheduling-preemption-threshold` Minimum difference in priorities for an incoming request to have to preempt running request(s). `10` Type: int - + `--schedule-conservativeness` How conservative the schedule policy is. A larger value means more conservative scheduling. Use a larger value if you see requests being retracted frequently. `1.0` Type: float - + `--page-size` The number of tokens in a page. `1` Type: int - + `--swa-full-tokens-ratio` The ratio of SWA layer KV tokens / full layer KV tokens, regardless of the number of swa:full layers. It should be between 0 and 1. E.g. 0.5 means if each swa layer has 50 tokens, then each full layer has 100 tokens. `0.8` Type: float - + `--disable-hybrid-swa-memory` Disable the hybrid SWA memory. `False` bool flag (set to enable) - + `--radix-eviction-policy` The eviction policy of radix trees. 'lru' stands for Least Recently Used, 'lfu' stands for Least Frequently Used. `lru` - `lru`, `lfu` + lru, lfu, slru, priority - + `--enable-prefill-delayer` Enable prefill delayer for DP attention to reduce idle time. `False` bool flag (set to enable) - + `--prefill-delayer-max-delay-passes` Maximum forward passes to delay prefill. `30` Type: int - + `--prefill-delayer-token-usage-low-watermark` Token usage low watermark for prefill delayer. `None` Type: float - + `--prefill-delayer-queue-min-ratio` Opt-in to the adaptive queue-based delay trigger (independent of the slot-based one). Defers prefill until the waiting queue reaches `min(running_req * ratio, max_prefill_bs)` so small fragments batch into a larger prefill. Unset keeps the original slot-only behavior. Typical: `0.1`–`0.5`. `None` Type: float - + `--prefill-delayer-max-delay-ms` Wall-clock cap (ms) on a single queue-trigger delay; once exceeded, prefill is force-released to bound worst-case TTFT. Only consulted when `--prefill-delayer-queue-min-ratio` is set. Typical: `1000`–`5000`. - `5000` + `None` Type: float - + `--prefill-delayer-forward-passes-buckets` Custom buckets for prefill delayer forward passes histogram. 0 and max_delay_passes-1 will be auto-added. `None` List[float] - + `--prefill-delayer-wait-seconds-buckets` Custom buckets for prefill delayer wait seconds histogram. 0 will be auto-added. `None` List[float] - + + `--disable-priority-preemption` + Disable priority scheduling preemption. + `False` + bool flag (set to enable) + + + `--default-priority-value` + Default priority for requests without explicit priority. + `None` + Type: int + + ## Runtime options @@ -530,132 +596,168 @@ Please consult the documentation below and [server_args.py](https://github.com/s - + `--device` The device to use ('cuda', 'xpu', 'hpu', 'npu', 'cpu'). Defaults to auto-detection if not specified. `None` Type: str - - `--tensor-parallel-size` `--tp-size` + + `--tensor-parallel-size`
`--tp-size` The tensor parallelism size. `1` Type: int - - `--pipeline-parallel-size` `--pp-size` + + `--pipeline-parallel-size`
`--pp-size` The pipeline parallelism size. `1` Type: int - - --attention-context-parallel-size<br>--attn-cp-size + + `--attention-context-parallel-size`
`--attn-cp-size` The attention context parallelism size. - 1 + `1` Type: int - - --moe-data-parallel-size<br>--moe-dp-size + + `--moe-data-parallel-size`
`--moe-dp-size` The moe data parallelism size. - 1 + `1` Type: int - - --pp-max-micro-batch-size + + `--pp-max-micro-batch-size` The maximum micro batch size in pipeline parallelism. None Type: int - - --pp-async-batch-depth + + `--pp-async-batch-depth` The async batch depth of pipeline parallelism. - 0 + `0` Type: int - - --stream-interval + + `--stream-interval` The interval (or buffer size) for streaming in terms of the token length. A smaller value makes streaming smoother, while a larger value makes the throughput higher - 1 + `1` Type: int - - --incremental-streaming-output + + `--incremental-streaming-output` Whether to output as a sequence of disjoint segments. - False + `False` bool flag (set to enable) - - --random-seed + + `--random-seed` The random seed. None Type: int - - --constrained-json-whitespace-pattern + + `--constrained-json-whitespace-pattern` (outlines and llguidance backends only) Regex pattern for syntactic whitespaces allowed in JSON constrained output. For example, to allow the model to generate consecutive whitespaces, set the pattern to [\n\t ]* None Type: str - - --constrained-json-disable-any-whitespace + + `--constrained-json-disable-any-whitespace` (xgrammar and llguidance backends only) Enforce compact representation in JSON constrained output. - False + `False` bool flag (set to enable) - - --watchdog-timeout + + `--watchdog-timeout` Set watchdog timeout in seconds. If a forward batch takes longer than this, the server will crash to prevent hanging. - 300 + `300` Type: float - - --soft-watchdog-timeout + + `--soft-watchdog-timeout` Set soft watchdog timeout in seconds. If a forward batch takes longer than this, the server will dump information for debugging. `None` Type: float - - --dist-timeout + + `--dist-timeout` Set timeout for torch.distributed initialization. `None` Type: int - - --download-dir + + `--download-dir` Model download directory for huggingface. None Type: str - - --model-checksum + + `--model-checksum` Model file integrity verification. If provided without value, uses model-path as HF repo ID. Otherwise, provide checksums JSON file path or HuggingFace repo ID. None Type: str - - --base-gpu-id + + `--base-gpu-id` The base GPU ID to start allocating GPUs from. Useful when running multiple instances on the same machine. - 0 + `0` + Type: int + + + `--gpu-id-step` + The delta between consecutive GPU IDs that are used. For example, setting it to 2 will use GPU 0,2,4,... + `1` Type: int - --gpu-id-step - The delta between consecutive GPU IDs that are used. For example, setting it to 2 will use GPU 0,2,4,... - 1 - Type: int - - - --sleep-on-idle + `--sleep-on-idle` Reduce CPU usage when sglang is idle. - False + `False` bool flag (set to enable) - + `--custom-sigquit-handler` Register a custom sigquit handler so you can do additional cleanup after the server is shutdown. This is only available for Engine, not for CLI. `None` Type: str + + `--batch-notify-size` + Number of streaming notifications to batch before yielding to the event loop. Reduces asyncio wakeup overhead under high concurrency. + `16` + Type: int + + + `--stream-response-default-include-usage` + Include usage in every streaming response (even when stream_options is not specified). + `False` + bool flag (set to enable) + + + `--stream-output` + [Deprecated] Use --incremental-streaming-output instead. + — + Type: str + + + `--enable-streaming-session` + Enable streaming session mode and StreamingSession wrapper. + `False` + bool flag (set to enable) + + + `--load-snapshot-publish-interval` + Publish load snapshot to shared memory every N decode iterations. Prefill and idle always publish immediately. + `15` + Type: int + + + `--use-ray` + Use Ray actors for scheduler process management. + `False` + bool flag (set to enable) + @@ -676,168 +778,198 @@ Please consult the documentation below and [server_args.py](https://github.com/s - + `--log-level` The logging level of all loggers. `info` Type: str - + `--log-level-http` The logging level of HTTP server. If not set, reuse --log-level by default. `None` Type: str - + `--log-requests` Log metadata, inputs, outputs of all requests. The verbosity is decided by --log-requests-level `False` bool flag (set to enable) - + `--log-requests-level` 0: Log metadata (no sampling parameters). 1: Log metadata and sampling parameters. 2: Log metadata, sampling parameters and partial input/output. 3: Log every input/output. `2` 0, 1, 2, 3 - + `--log-requests-format` Format for request logging: 'text' (human-readable) or 'json' (structured) `text` text, json - + `--log-requests-target` Target(s) for request logging: 'stdout' and/or directory path(s) for file output. Can specify multiple targets, e.g., '--log-requests-target stdout /my/path'. `None` List[str] - + `--uvicorn-access-log-exclude-prefixes` Exclude uvicorn access logs whose request path starts with any of these prefixes. Defaults to empty (disabled). `[]` List[str] - + `--crash-dump-folder` Folder path to dump requests from the last 5 min before a crash (if any). If not specified, crash dumping is disabled. `None` Type: str - + `--show-time-cost` Show time cost of custom marks. `False` bool flag (set to enable) - + `--enable-metrics` Enable log prometheus metrics. `False` bool flag (set to enable) - - --enable-mfu-metrics + + `--enable-mfu-metrics` Enable estimated MFU-related prometheus metrics. `False` bool flag (set to enable) - - --enable-metrics-for-all-schedulers + + `--enable-metrics-for-all-schedulers` Enable --enable-metrics-for-all-schedulers when you want schedulers on all TP ranks (not just TP 0) to record request metrics separately. This is especially useful when dp_attention is enabled, as otherwise all metrics appear to come from TP 0. - False + `False` bool flag (set to enable) - - --tokenizer-metrics-custom-labels-header + + `--tokenizer-metrics-custom-labels-header` Specify the HTTP header for passing custom labels for tokenizer metrics. - x-custom-labels + `x-custom-labels` Type: str - - --tokenizer-metrics-allowed-custom-labels + + `--tokenizer-metrics-allowed-custom-labels` The custom labels allowed for tokenizer metrics. The labels are specified via a dict in '--tokenizer-metrics-custom-labels-header' field in HTTP requests, e.g., {'label1': 'value1', 'label2': 'value2'} is allowed if '--tokenizer-metrics-allowed-custom-labels label1 label2' is set. `None` List[str] - - --bucket-time-to-first-token + + `--bucket-time-to-first-token` The buckets of time to first token, specified as a list of floats. `None` List[float] - - --bucket-inter-token-latency + + `--bucket-inter-token-latency` The buckets of inter-token latency, specified as a list of floats. `None` List[float] - - --bucket-e2e-request-latency + + `--bucket-e2e-request-latency` The buckets of end-to-end request latency, specified as a list of floats. None List[float] - - --collect-tokens-histogram + + `--collect-tokens-histogram` Collect prompt/generation tokens histogram. False bool flag (set to enable) - - --prompt-tokens-buckets + + `--prompt-tokens-buckets` The buckets rule of prompt tokens. Supports 3 rule types: 'default' uses predefined buckets; 'tse <middle> <base> <count>' generates two sides exponential distributed buckets (e.g., 'tse 1000 2 8' generates buckets [984.0, 992.0, 996.0, 998.0, 1000.0, 1002.0, 1004.0, 1008.0, 1016.0]).); 'custom <value1> <value2> ...' uses custom bucket values (e.g., 'custom 10 50 100 500'). `None` List[str] - - --generation-tokens-buckets + + `--generation-tokens-buckets` The buckets rule for generation tokens histogram. Supports 3 rule types: 'default' uses predefined buckets; 'tse <middle> <base> <count>' generates two sides exponential distributed buckets (e.g., 'tse 1000 2 8' generates buckets [984.0, 992.0, 996.0, 998.0, 1000.0, 1002.0, 1004.0, 1008.0, 1016.0]).); 'custom <value1> <value2> ...' uses custom bucket values (e.g., 'custom 10 50 100 500'). None List[str] - - --gc-warning-threshold-secs + + `--gc-warning-threshold-secs` The threshold for long GC warning. If a GC takes longer than this, a warning will be logged. Set to 0 to disable. - 0.0 + `0.0` Type: float - - --decode-log-interval + + `--decode-log-interval` The log interval of decode batch. - 40 + `40` Type: int - - --enable-request-time-stats-logging + + `--enable-request-time-stats-logging` Enable per request time stats logging - False + `False` bool flag (set to enable) - - --kv-events-config + + `--kv-events-config` Config in json format for NVIDIA dynamo KV event publishing. Publishing will be enabled if this flag is used. None Type: str - - --enable-trace + + `--enable-trace` Enable opentelemetry trace - False + `False` bool flag (set to enable) - - --trace-modules + + `--trace-modules` Select the components to trace. Available options are 'request' and 'mooncake'. Format: <module1 name>,<module2 name>,...... - request + `request` Type: str - + `--otlp-traces-endpoint` Config opentelemetry collector endpoint if --enable-trace is set. format: <ip>:<port> `localhost:4317` Type: str + + `--grpc-http-sidecar-port` + Port for the HTTP sidecar server in gRPC mode (--grpc-mode). Serves Prometheus metrics and profiling endpoints. Defaults to --port + 1. Not used in HTTP mode. + `None` + Type: int + + + `--extra-metric-labels` + The custom labels for metrics. e.g. '{"label1": "value1", "label2": "value2"}' + `None` + Type: str + + + `--enable-forward-pass-metrics` + Enable per-iteration forward pass metrics via ZMQ IPC. External consumers (e.g. Dynamo planner) subscribe to the IPC endpoint exposed in server_args.forward_pass_metrics_ipc_name. + `False` + bool flag (set to enable) + + + `--forward-pass-metrics-worker-id` + — + `""` + Type: str + + + `--forward-pass-metrics-ipc-name` + — + `None` + Type: str + @@ -858,13 +990,13 @@ Please consult the documentation below and [server_args.py](https://github.com/s - + `--export-metrics-to-file` Export performance metrics for each request to local file (e.g. for forwarding to external systems). `False` bool flag (set to enable) - + `--export-metrics-to-file-dir` Directory path for writing performance metrics files (required when --export-metrics-to-file is enabled). `None` @@ -890,85 +1022,109 @@ Please consult the documentation below and [server_args.py](https://github.com/s - + `--api-key` Set API key of the server. It is also used in the OpenAI API compatible server. `None` Type: str - + `--admin-api-key` Set admin API key for administrative/control endpoints (e.g., weights update, cache flush, /server_info). Endpoints marked as admin-only require Authorization: Bearer <admin_api_key> when this is set. `None` Type: str - + `--served-model-name` Override the model name returned by the v1/models endpoint in OpenAI API server. `None` Type: str - + `--weight-version` Version identifier for the model weights. Defaults to 'default' if not specified. `default` Type: str - + `--chat-template` The builtin chat template name or the path of the chat template file. This is only used for OpenAI-compatible API server. `None` Type: str - + `--hf-chat-template-name` When the HuggingFace tokenizer has multiple chat templates (e.g., 'default', 'tool_use', 'rag'), specify which named template to use. If not set, the first available template is used. `None` Type: str - + `--completion-template` The builtin completion template name or the path of the completion template file. This is only used for OpenAI-compatible API server. only for code completion currently. `None` Type: str - + `--file-storage-path` The path of the file storage in backend. `sglang_storage` Type: str - + `--enable-cache-report` Return number of cached tokens in usage.prompt_tokens_details for each openai request. `False` bool flag (set to enable) - + `--reasoning-parser` Specify the parser for reasoning models. Supported parsers: [deepseek-r1, deepseek-v3, glm45, gpt-oss, kimi, qwen3, qwen3-thinking, step3]. `None` deepseek-r1, deepseek-v3, glm45, gpt-oss, kimi, qwen3, qwen3-thinking, step3 - + `--tool-call-parser` Specify the parser for handling tool-call interactions. Supported parsers: [deepseekv3, deepseekv31, glm, glm45, glm47, gpt-oss, kimi_k2, llama3, mistral, pythonic, qwen, qwen25, qwen3_coder, step3]. `None` deepseekv3, deepseekv31, glm, glm45, glm47, gpt-oss, kimi_k2, llama3, mistral, pythonic, qwen, qwen25, qwen3_coder, step3, gigachat3 - + `--tool-server` Either 'demo' or a comma-separated list of tool server urls to use for the model. If not specified, no tool server will be used. `None` Type: str - + `--sampling-defaults` Where to get default sampling parameters. 'openai' uses SGLang/OpenAI defaults (temperature=1.0, top_p=1.0, etc.). 'model' uses the model's generation_config.json to get the recommended sampling parameters if available. Default is 'model'. `model` openai, model - + + `--strip-thinking-cache` + Skip caching reasoning-model output (thinking + answer) in the radix tree on finish; keep only the prompt prefix. Opt-in: changes cache contents. + `False` + bool flag (set to enable) + + + `--enable-strict-thinking` + Enable strict token filtering during the thinking phase. Blocks model-specific excluded tokens (e.g., tool call markers) during reasoning. Requires a grammar backend that supports token filtering. + `False` + bool flag (set to enable) + + + `--asr-max-buffer-seconds` + Maximum seconds of PCM audio the streaming ASR WebSocket handler will accumulate before closing the session with a buffer_overflow error. Guards against OOM when a client streams audio faster than inference can consume it. Default 60s. + `60` + Type: int + + + `--asr-max-concurrent-sessions` + Maximum number of concurrent realtime ASR WebSocket sessions served by /v1/realtime. New connections beyond this cap are accepted, sent an error{code:too_many_sessions} frame, and closed. Default 32. + `32` + Type: int + + ## Data parallelism @@ -988,17 +1144,17 @@ Please consult the documentation below and [server_args.py](https://github.com/s - - ` --data-parallel-size`
`--dp-size` + + `--data-parallel-size`
`--dp-size` The data parallelism size. - ` 1` + `1` Type: int - - ` --load-balance-method` + + `--load-balance-method` The load balancing strategy for data parallelism. The `total_tokens` algorithm can only be used when DP attention is applied. This algorithm performs load balancing based on the real-time token load of the DP workers. - ` auto` - ` auto`, `round_robin`, `follow_bootstrap_room`, `total_requests`, `total_tokens` + `auto` + auto, round_robin, follow_bootstrap_room, total_requests, total_tokens @@ -1020,22 +1176,22 @@ Please consult the documentation below and [server_args.py](https://github.com/s - - ` --dist-init-addr`
`--nccl-init-addr` + + `--dist-init-addr`
`--nccl-init-addr` The host address for initializing distributed backend (e.g., `192.168.0.2:25000`). ` None` Type: str - - ` --nnodes` + + `--nnodes` The number of nodes. - ` 1` + `1` Type: int - - ` --node-rank` + + `--node-rank` The node rank. - ` 0` + `0` Type: int @@ -1058,13 +1214,13 @@ Please consult the documentation below and [server_args.py](https://github.com/s - + `--json-model-override-args` A dictionary in JSON string format used to override default model configurations. `{}` Type: str - + `--preferred-sampling-params` json-formatted sampling settings that will be returned in /get_model_info `None` @@ -1090,73 +1246,91 @@ Please consult the documentation below and [server_args.py](https://github.com/s - + `--enable-lora` Enable LoRA support for the model. This argument is automatically set to `True` if `--lora-paths` is provided for backward compatibility. `False` Bool flag (set to enable) - + `--enable-lora-overlap-loading` Enable asynchronous LoRA weight loading in order to overlap H2D transfers with GPU compute. This should be enabled if you find that your LoRA workloads are bottlenecked by adapter weight loading, for example when frequently loading large LoRA adapters. `False` Bool flag (set to enable) - + `--max-lora-rank` The maximum LoRA rank that should be supported. If not specified, it will be automatically inferred from the adapters provided in --lora-paths. This argument is needed when you expect to dynamically load adapters of larger LoRA rank after server startup. `None` Type: int - + `--lora-target-modules` The union set of all target modules where LoRA should be applied (e.g., q_proj, k_proj, gate_proj). If not specified, it will be automatically inferred from the adapters provided in --lora-paths. You can also set it to all to enable LoRA for all supported modules; note this may introduce minor performance overhead. `None` q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj, qkv_proj, gate_up_proj, all - + `--lora-paths` The list of LoRA adapters to load. Each adapter must be specified in one of the following formats: <PATH> \| <NAME>=<PATH> \| JSON with schema {"lora_name": str, "lora_path": str, "pinned": bool}. `None` Type: List[str] / JSON objects - + `--max-loras-per-batch` Maximum number of adapters for a running batch, including base-only requests. `8` Type: int - + `--max-loaded-loras` If specified, limits the maximum number of LoRA adapters loaded in CPU memory at a time. Must be ≥ --max-loras-per-batch. `None` Type: int - + `--lora-eviction-policy` LoRA adapter eviction policy when the GPU memory pool is full. `lru` lru, fifo - + `--lora-backend` Choose the kernel backend for multi-LoRA serving. `csgmv` triton, csgmv, ascend, torch_native - + `--max-lora-chunk-size` Maximum chunk size for the ChunkedSGMV LoRA backend. Only used when --lora-backend is csgmv. Larger values may improve performance. `16` 16, 32, 64, 128 - + `--lora-drain-wait-threshold` When any LoRA adapter request waits longer than this threshold (in seconds), the scheduler will selectively drain one running adapter to make room. This mitigates extreme tail latency under high or skewed workloads by preventing a small set of adapters from monopolizing batch slots. Set to 0 to disable draining (default). - `0` + `0.0` Type: float - + + `--experts-shared-outer-loras` + Force shared outer LoRA mode for MoE models. When set, w1/w3 lora_A and w2 lora_B are shared across experts (expert_dim=1). Use --no-experts-shared-outer-loras to force disable. By default this is auto-detected from adapter weights. + `None` + Type: str + + + `--lora-use-virtual-experts` + Enable virtual expert computation for MoE models. When set, the model will use virtual expert computation. + `False` + bool flag (set to enable) + + + `--lora-strict-loading` + Enable strict loading for LoRA adapters. When set, mismatched or missing keys in the adapter weights will raise an error. + `False` + Type: str + + ## Kernel Backends (Attention, Sampling, Grammar, GEMM) @@ -1176,85 +1350,103 @@ Please consult the documentation below and [server_args.py](https://github.com/s - + `--attention-backend` Choose the kernels for attention layers. `None` - triton, torch_native, flex_attention, dsa, cutlass_mla, fa3, fa4, flashinfer, flashmla, trtllm_mla, trtllm_mha, dual_chunk_flash_attn, aiter, wave, intel_amx, ascend + triton, torch_native, flex_attention, dsa, nsa, dsv4, compressed, cutlass_mla, fa3, fa4, flashinfer, flashmla, trtllm_mla, cutedsl_mla, tokenspeed_mla, trtllm_mha, dual_chunk_flash_attn, aiter, wave, intel_amx, ascend, intel_xpu - + `--prefill-attention-backend` Choose the kernels for prefill attention layers (have priority over --attention-backend). `None` - triton, torch_native, flex_attention, dsa, cutlass_mla, fa3, fa4, flashinfer, flashmla, trtllm_mla, trtllm_mha, dual_chunk_flash_attn, aiter, wave, intel_amx, ascend + triton, torch_native, flex_attention, dsa, nsa, dsv4, compressed, cutlass_mla, fa3, fa4, flashinfer, flashmla, trtllm_mla, cutedsl_mla, tokenspeed_mla, trtllm_mha, dual_chunk_flash_attn, aiter, wave, intel_amx, ascend, intel_xpu - + `--decode-attention-backend` Choose the kernels for decode attention layers (have priority over --attention-backend). `None` - triton, torch_native, flex_attention, dsa, cutlass_mla, fa3, fa4, flashinfer, flashmla, trtllm_mla, trtllm_mha, dual_chunk_flash_attn, aiter, wave, intel_amx, ascend + triton, torch_native, flex_attention, dsa, nsa, dsv4, compressed, cutlass_mla, fa3, fa4, flashinfer, flashmla, trtllm_mla, cutedsl_mla, tokenspeed_mla, trtllm_mha, dual_chunk_flash_attn, aiter, wave, intel_amx, ascend, intel_xpu - + `--sampling-backend` Choose the kernels for sampling layers. `None` flashinfer, pytorch, ascend - + `--grammar-backend` Choose the backend for grammar-guided decoding. `None` xgrammar, outlines, llguidance, none - + `--mm-attention-backend` Set multimodal attention backend. `None` - sdpa, fa3, fa4, triton_attn, ascend_attn, aiter_attn + sdpa, fa3, fa4, triton_attn, ascend_attn, aiter_attn, flashinfer_cudnn, amx_attn, xpu_attn - + `--dsa-prefill-backend` DSA backend for the prefill stage (overrides `--attention-backend` when running DeepSeek DSA-style attention). - `flashmla_sparse` + Auto (hardware-dependent) flashmla_sparse, flashmla_kv, flashmla_auto, fa3, tilelang, aiter, trtllm - + `--dsa-decode-backend` DSA backend for the decode stage when running DeepSeek DSA-style attention. Overrides `--attention-backend` for decoding. - `fa3` - flashmla_sparse, flashmla_kv, fa3, tilelang, aiter, trtllm + Auto (hardware-dependent) + flashmla_sparse, flashmla_kv, flashmla_auto, fa3, tilelang, aiter, trtllm - + `--dsa-topk-backend` Choose the DSA indexer top-k backend. The `torch` backend currently requires `SGLANG_DSA_FUSE_TOPK=false`. `sgl-kernel` sgl-kernel, torch, flashinfer - + `--enable-deepseek-v4-fp4-indexer` Enable the experimental FP4 C4 indexer path for DeepSeek V4. When unset, SGLang keeps the default DeepSeek V4 indexer path. Requires SM100 GPUs with DeepGEMM FP4 indexer support. `False` bool flag (set to enable) - + `--fp8-gemm-backend` Choose the runner backend for Blockwise FP8 GEMM operations. Options: 'auto' (default, auto-selects based on hardware), 'deep_gemm' (JIT-compiled; enabled by default on NVIDIA Hopper (SM90) and Blackwell (SM100) when DeepGEMM is installed), 'flashinfer_trtllm' (FlashInfer TRTLLM backend; SM100/SM103 only), 'flashinfer_cutlass' (FlashInfer CUTLASS backend, SM120 only), 'flashinfer_deepgemm' (Hopper SM90 only, uses swapAB optimization for small M dimensions in decoding), 'cutlass' (optimal for Hopper/Blackwell GPUs and high-throughput), 'triton' (fallback, widely compatible), 'aiter' (ROCm only). `auto` auto, deep_gemm, flashinfer_trtllm, flashinfer_cutlass, flashinfer_deepgemm, cutlass, triton, aiter - + `--fp4-gemm-backend` Choose the runner backend for NVFP4 GEMM operations. Options: 'auto' (default; selects flashinfer_cutedsl on SM100, marlin on SM80-SM90, flashinfer_cutlass otherwise (including SM120)), 'cutlass' (SGLang CUTLASS kernel), 'flashinfer_cutlass' (FlashInfer CUTLASS backend), 'flashinfer_cudnn' (FlashInfer cuDNN backend, optimal on CUDA 13+ with cuDNN 9.15+), 'flashinfer_cutedsl' (FlashInfer CuTe DSL backend), 'flashinfer_trtllm' (FlashInfer TensorRT-LLM backend, requires different weight preparation with shuffling), 'marlin' (weight-only W4A16 fallback for SM80-SM90). All FlashInfer backends fall back to sgl-kernel CUTLASS when FlashInfer is unavailable. - auto + `auto` auto, cutlass, flashinfer_cudnn, flashinfer_cutedsl, flashinfer_cutlass, flashinfer_trtllm, marlin - + `--disable-flashinfer-autotune` Flashinfer autotune is enabled by default. Set this flag to disable the autotune. `False` bool flag (set to enable) - + + `--radix-cache-backend` + Name of a radix-cache backend previously registered via register_radix_cache_backend. Omit this flag to use the built-in default cache selection chain. + `None` + Type: str + + + `--nsa-prefill-backend` + [Deprecated] Use --dsa-prefill-backend instead. + Auto + flashmla_sparse, flashmla_kv, flashmla_auto, fa3, tilelang, aiter, trtllm + + + `--nsa-decode-backend` + [Deprecated] Use --dsa-decode-backend instead. + Auto + flashmla_sparse, flashmla_kv, flashmla_auto, fa3, tilelang, aiter, trtllm + + ## Speculative decoding @@ -1274,97 +1466,115 @@ Please consult the documentation below and [server_args.py](https://github.com/s - + `--speculative-algorithm` Speculative algorithm. `None` `EAGLE`, `EAGLE3`, `NEXTN`, `STANDALONE`, `NGRAM` - - `--speculative-draft-model-path` `--speculative-draft-model` + + `--speculative-draft-model-path`
`--speculative-draft-model` The path of the draft model weights. This can be a local folder or a Hugging Face repo ID. `None` Type: str - + `--speculative-draft-model-revision` The specific draft model version to use. It can be a branch name, a tag name, or a commit id. If unspecified, will use the default version. `None` Type: str - + `--speculative-draft-load-format` The format of the draft model weights to load. If not specified, will use the same format as `--load-format`. Use 'dummy' to initialize draft model weights with random values for profiling. `None` - Same as `--load-format` options + auto, pt, safetensors, npcache, dummy, sharded_state, gguf, bitsandbytes, mistral, layered, flash_rl, remote, remote_instance, fastsafetensors, private, runai_streamer - + `--speculative-num-steps` The number of steps sampled from draft model in Speculative Decoding. `None` Type: int - + `--speculative-eagle-topk` The number of tokens sampled from the draft model in eagle2 each step. `None` Type: int - + `--speculative-num-draft-tokens` The number of tokens sampled from the draft model in Speculative Decoding. `None` Type: int - + `--speculative-accept-threshold-single` Accept a draft token if its probability in the target model is greater than this threshold. `1.0` Type: float - + `--speculative-accept-threshold-acc` The accept probability of a draft token is raised from its target probability p to min(1, p / threshold_acc). `1.0` Type: float - + `--speculative-token-map` The path of the draft model's small vocab table. `None` Type: str - + `--speculative-attention-mode` Attention backend for speculative decoding operations (both target verify and draft extend). Can be one of 'prefill' (default) or 'decode'. `prefill` - prefill, decode + prefill, decode - + `--speculative-draft-attention-backend` Attention backend for speculative decoding drafting. `None` Same as attention backend options - + `--speculative-moe-runner-backend` MOE backend for EAGLE speculative decoding, see `--moe-runner-backend` for options. Same as moe runner backend if unset. `None` - Same as `--moe-runner-backend` options + auto, deep_gemm, triton, triton_kernel, flashinfer_trtllm, experimental_sgl_trtllm, flashinfer_trtllm_routed, flashinfer_cutlass, flashinfer_mxfp4, flashinfer_cutedsl, cutlass, aiter, marlin - + `--speculative-moe-a2a-backend` MOE A2A backend for EAGLE speculative decoding, see `--moe-a2a-backend` for options. Same as moe a2a backend if unset. `None` - Same as `--moe-a2a-backend` options + none, deepep, mooncake, nixl, mori, ascend_fuseep, flashinfer, megamoe - + `--speculative-draft-model-quantization` The quantization method for speculative model. `None` Same as `--quantization` options - + + `--speculative-dflash-block-size` + DFLASH only. Block size (verify window length). Alias of --speculative-num-draft-tokens for DFLASH. + `None` + Type: int + + + `--speculative-draft-window-size` + Sliding window size for the draft model. Honored by Llama EAGLE-3 (`LlamaForCausalLMEagle3`) and DFLASH only; other EAGLE-3 backends (e.g. MLA-based drafters) silently ignore it. For Llama EAGLE-3, the drafter only attends to the most recent N keys (verifier hidden states + its own outputs); the verifier is unaffected. For DFLASH, the draft worker keeps a recent target-token window in its local KV cache (paged backends may retain up to one extra page on the left for alignment). Default is full attention/context. + `None` + Type: int + + + `--speculative-dflash-draft-window-size` + — + — + Type: int + + ## Ngram speculative decoding @@ -1384,37 +1594,73 @@ Please consult the documentation below and [server_args.py](https://github.com/s - - --speculative-ngram-min-bfs-breadth + + `--speculative-ngram-min-bfs-breadth` The minimum breadth for BFS (Breadth-First Search) in ngram speculative decoding. `1` Type: int - - --speculative-ngram-max-bfs-breadth + + `--speculative-ngram-max-bfs-breadth` The maximum breadth for BFS (Breadth-First Search) in ngram speculative decoding. - 10 + `10` Type: int - - --speculative-ngram-match-type + + `--speculative-ngram-match-type` Ngram tree-building mode. BFS selects recency-based expansion and PROB selects frequency-based expansion. This setting is forwarded to the ngram cache implementation. - BFS + `BFS` BFS, PROB - - --speculative-ngram-max-trie-depth + + `--speculative-ngram-max-trie-depth` Maximum suffix length stored and matched by the ngram trie. - 18 + `18` Type: int - - --speculative-ngram-capacity + + `--speculative-ngram-capacity` The cache capacity for ngram speculative decoding. 10000000 Type: int - + + `--speculative-ngram-external-corpus-path` + Path to an external JSONL corpus to pre-load into SAM at startup. Additional corpora can be added at runtime via POST /add_external_corpus. + `None` + Type: str + + + `--speculative-ngram-external-sam-budget` + Number of draft nodes reserved for the external SAM subtree in ngram speculative decoding. + `0` + Type: int + + + `--speculative-ngram-external-corpus-max-tokens` + Fail startup if the tokenized external ngram corpus exceeds this many tokens. Tune this based on your CPU memory budget. + `10000000` + Type: int + + + `--speculative-adaptive` + Enable adaptive speculative decoding that dynamically adjusts num_steps based on acceptance rate. + `False` + bool flag (set to enable) + + + `--speculative-adaptive-config` + Path to a JSON config file for adaptive speculative decoding tuning knobs. + `None` + Type: str + + + `--speculative-skip-dp-mlp-sync` + Skip the extra MLP sync that the scheduler performs before merging a new batch when speculative decoding + DP attention are both enabled. + `False` + bool flag (set to enable) + + ## Multi-layer Eagle speculative decoding @@ -1434,7 +1680,7 @@ Please consult the documentation below and [server_args.py](https://github.com/s - + `--enable-multi-layer-eagle` Enable multi-layer Eagle speculative decoding. `False` @@ -1460,162 +1706,174 @@ Please consult the documentation below and [server_args.py](https://github.com/s - - ` --expert-parallel-size`
`--ep-size`
`--ep` + + `--expert-parallel-size`
`--ep-size`
`--ep` The expert parallelism size. - ` 1` + `1` Type: int - - ` --moe-a2a-backend` + + `--moe-a2a-backend` Select the backend for all-to-all communication for expert parallelism. - ` none` - none, deepep, mooncake, mori, nixl, ascend_fuseep + `none` + none, deepep, mooncake, nixl, mori, ascend_fuseep, flashinfer, megamoe - - ` --moe-runner-backend` + + `--moe-runner-backend` Choose the runner backend for MoE. - ` auto` - auto, deep_gemm, triton, triton_kernel, flashinfer_trtllm, flashinfer_trtllm_routed, flashinfer_cutlass, flashinfer_mxfp4, flashinfer_cutedsl, cutlass, marlin + `auto` + auto, deep_gemm, triton, triton_kernel, flashinfer_trtllm, experimental_sgl_trtllm, flashinfer_trtllm_routed, flashinfer_cutlass, flashinfer_mxfp4, flashinfer_cutedsl, cutlass, aiter, marlin - - ` --flashinfer-mxfp4-moe-precision` + + `--flashinfer-mxfp4-moe-precision` Choose the computation precision of flashinfer mxfp4 moe - ` default` + `default` default, bf16 - - ` --enable-flashinfer-allreduce-fusion` + + `--enable-flashinfer-allreduce-fusion` Enable FlashInfer allreduce fusion with Residual RMSNorm. - ` False` + `False` bool flag (set to enable) - - --enable-aiter-allreduce-fusion + + `--enable-aiter-allreduce-fusion` Enable aiter allreduce fusion with Residual RMSNorm. - False + `False` bool flag (set to enable) - - --deepep-mode + + `--deepep-mode` Select the mode when enable DeepEP MoE, could be normal, low_latency or auto. Default is auto, which means low_latency for decode batch and normal for prefill batch. - auto + `auto` normal, low_latency, auto - - --deepep-dispatcher-output-dtype + + `--deepep-dispatcher-output-dtype` Select DeepEP dispather output dtype, could be bf16, fp8, int8 (only Ascend A2/A3 NPU), nvfp4 or auto. Default is auto, which follows a priority order (server argument → deprecated env var → input_global_scale check → dispatcher_output_dtype from quant_config → flashinfer/cutlass backend → NPU BF16 default → GPU FP8 default). - auto - bf16, fp8, int8, nvfp4, auto + `auto` + auto, bf16, fp8, int8, nvfp4 - - --ep-num-redundant-experts + + `--ep-num-redundant-experts` Allocate this number of redundant experts in expert parallel. - 0 + `0` Type: int - - --ep-dispatch-algorithm + + `--ep-dispatch-algorithm` The algorithm to choose ranks for redundant experts in expert parallel. None Type: str - - --init-expert-location + + `--init-expert-location` Initial location of EP experts. - trivial + `trivial` Type: str - - --enable-eplb + + `--enable-eplb` Enable EPLB algorithm - False + `False` bool flag (set to enable) - - --eplb-algorithm + + `--eplb-algorithm` Chosen EPLB algorithm - auto + `auto` Type: str - - --eplb-rebalance-num-iterations + + `--eplb-rebalance-num-iterations` Number of iterations to automatically trigger a EPLB re-balance. - 1000 + `1000` Type: int - - --eplb-rebalance-layers-per-chunk + + `--eplb-rebalance-layers-per-chunk` Number of layers to rebalance per forward pass. None Type: int - - --eplb-min-rebalancing-utilization-threshold + + `--eplb-min-rebalancing-utilization-threshold` Minimum threshold for GPU average utilization to trigger EPLB rebalancing. Must be in the range [0.0, 1.0]. - 1.0 + `1.0` Type: float - - --expert-distribution-recorder-mode + + `--expert-distribution-recorder-mode` Mode of expert distribution recorder. ` None` Type: str - - --expert-distribution-recorder-buffer-size + + `--expert-distribution-recorder-buffer-size` Circular buffer size of expert distribution recorder. Set to -1 to denote infinite buffer. None Type: int - - --enable-expert-distribution-metrics + + `--enable-expert-distribution-metrics` Enable logging metrics for expert balancedness - False + `False` bool flag (set to enable) - - --deepep-config + + `--deepep-config` Tuned DeepEP config suitable for your own cluster. It can be either a string with JSON content or a file path. ` None` Type: str - - --moe-dense-tp-size + + `--moe-dense-tp-size` TP size for MoE dense MLP layers. This flag is useful when, with large TP size, there are errors caused by weights in MLP layers having dimension smaller than the min dimension GEMM supports. ` none` Type: int - - --elastic-ep-backend + + `--elastic-ep-backend` Specify the collective communication backend for elastic EP. Currently supports 'mooncake'. ` None` - none, mooncake + none, mooncake, nixl - - --enable-elastic-expert-backup + + `--enable-elastic-expert-backup` Enable elastic EP backend to backup expert weights in DRAM feature. Currently supports 'mooncake'. - False + `False` bool flag (set to enable) - - ` --mooncake-ib-device` + + `--mooncake-ib-device` The InfiniBand devices for Mooncake Backend transfer, accepts multiple comma-separated devices (e.g., --mooncake-ib-device mlx5_0,mlx5_1). Default is None, which triggers automatic device detection when Mooncake Backend is enabled. ` None` Type: str - + `--enable-deepep-waterfill` Enable DeepEP Waterfill: dispatch the shared expert as the 9th routed expert to the least-loaded EP rank. Automatically sets `--moe-a2a-backend deepep`, implicitly enables shared-expert fusion, and supports `--deepep-mode auto`, `normal`, or `low_latency`. Use `auto` or `low_latency` for production decode so CUDA graph remains enabled. Supported on DeepSeek-V3/R1 with EP >= 2. By default, Waterfill uses the static local-batch path; set `SGLANG_DISABLE_STATIC_WATERFILL=1` to force dynamic Waterfill with runtime EP all-reduce. `False` bool flag (set to enable) - + `--elastic-ep-rejoin` Indicates that this process is a relaunched elastic EP rank that should rejoin an existing process group during rank recovery. `False` bool flag (set to enable) + + `--flashinfer-allreduce-fusion-backend` + Enable FlashInfer allreduce fusion and choose backend. Defaults to auto. 'auto': choose mnnvl on SM90 single-node systems and SM100/SM103 single-node or multi-node systems; choose trtllm otherwise. 'trtllm': available on single-node systems only. 'mnnvl': available on SM90 single-node systems and SM100/SM103 single-node or multi-node systems via MNNVL fabric. Fuses allreduce with Residual + RMSNorm for supported MoE models. + `None` + auto, trtllm, mnnvl + + + `--enforce-disable-flashinfer-allreduce-fusion` + Enforce disable FlashInfer allreduce fusion. + `False` + bool flag (set to enable) + @@ -1636,37 +1894,73 @@ Please consult the documentation below and [server_args.py](https://github.com/s - + `--max-mamba-cache-size` The maximum size of the mamba cache. `None` Type: int - + `--mamba-ssm-dtype` - The data type of the SSM states in mamba cache. - `float32` + The data type of the SSM states in mamba cache. If not set, read from the model config. + Auto (from model config) float32, bfloat16, float16 - + `--mamba-full-memory-ratio` The ratio of mamba state memory to full kv cache memory. `0.9` Type: float - + `--mamba-scheduler-strategy` The strategy to use for mamba scheduler. auto currently defaults to no_buffer. 1. no_buffer does not support overlap scheduler due to not allocating extra mamba state buffers. Branching point caching support is feasible but not implemented. 2. extra_buffer supports overlap schedule by allocating extra mamba state buffers to track mamba state for caching (mamba state usage per running req becomes 2x for non-spec; 1+(1/(2+speculative_num_draft_tokens))x for spec dec (e.g. 1.16x if speculative_num_draft_tokens==4)). 2a. extra_buffer is strictly better for non-KV-cache-bound cases; for KV-cache-bound cases, the tradeoff depends on whether enabling overlap outweighs reduced max running requests. 2b. mamba caching at radix cache branching point is strictly better than non-branch but requires kernel support, currently only extra_buffer supports branching. `auto` - auto, no_buffer, extra_buffer + auto, no_buffer, extra_buffer, extra_buffer_lazy - + `--mamba-track-interval` The interval (in tokens) to track the mamba state during decode. Only used when --mamba-scheduler-strategy is extra_buffer. Must be divisible by page_size if set, and must be >= speculative_num_draft_tokens when using speculative decoding. `256` Type: int - + + `--enable-int8-mamba-checkpoint` + Store radix-cached linear-attn (mamba) states in int8 (separate checkpoint pool) for ~2x cached-prefix capacity at fixed memory. + `False` + bool flag (set to enable) + + + `--int8-mamba-ckpt-size` + Number of int8 mamba checkpoint slots (default: 2x the active mamba pool size). + `None` + Type: int + + + `--mamba-backend` + Choose the kernel backend for Mamba SSM operations. Default is 'triton'. Options: 'triton' (default), 'flashinfer' (requires FlashInfer with Mamba support). + `triton` + triton, flashinfer + + + `--linear-attn-backend` + The default kernel backend for linear attention (GDN/KDA). Can be overridden per-mode by --linear-attn-decode-backend and --linear-attn-prefill-backend. + `triton` + triton, cutedsl, flashinfer + + + `--linear-attn-decode-backend` + Override the kernel backend for linear attention decode. If not set, uses --linear-attn-backend. + `None` + triton, cutedsl, flashinfer + + + `--linear-attn-prefill-backend` + Override the kernel backend for linear attention prefill/extend. If not set, uses --linear-attn-backend. + `None` + triton, cutedsl, flashinfer + + ## Hierarchical cache @@ -1686,55 +1980,55 @@ Please consult the documentation below and [server_args.py](https://github.com/s - + `--enable-hierarchical-cache` Enable hierarchical cache `False` bool flag (set to enable) - + `--hicache-ratio` The ratio of the size of host KV cache memory pool to the size of device pool. `2.0` Type: float - + `--hicache-size` The size of host KV cache memory pool in gigabytes, which will override the hicache_ratio if set. `0` Type: int - + `--hicache-write-policy` The write policy of hierarchical cache. `write_through` - `write_back`, `write_through`, `write_through_selective` + write_back, write_through, write_through_selective - + `--hicache-io-backend` The IO backend for KV cache transfer between CPU and GPU `kernel` - `direct`, `kernel`, `kernel_ascend` + direct, kernel, kernel_ascend - + `--hicache-mem-layout` The layout of host memory pool for hierarchical cache. - `layer_first` - `layer_first`, `page_first`, `page_first_direct`, `page_first_kv_split`, `page_head` + `page_first` + layer_first, page_first, page_first_direct, page_first_kv_split, page_head - + `--hicache-storage-backend` The storage backend for hierarchical KV cache. Built-in backends: file, mooncake, hf3fs, nixl, aibrix. For dynamic backend, use --hicache-storage-backend-extra-config to specify: backend_name (custom name), module_path (Python module path), class_name (backend class name). `None` - `file`, `mooncake`, `hf3fs`, `nixl`, `aibrix`, `dynamic`, `eic` + file, mooncake, hf3fs, nixl, aibrix, dynamic, eic, simm - + `--hicache-storage-prefetch-policy` Control when prefetching from the storage backend should stop. `timeout` - `best_effort`, `wait_complete`, `timeout` + best_effort, wait_complete, timeout - + `--hicache-storage-backend-extra-config` A dictionary in JSON string format, or a string starting with a `@` followed by a config file in JSON/YAML/TOML format, containing extra configuration for the storage backend. `None` @@ -1760,13 +2054,19 @@ Please consult the documentation below and [server_args.py](https://github.com/s - - `--hierarchical-sparse-attention-extra-config` + + `--hisparse-config`
`--hierarchical-sparse-attention-extra-config` A dictionary in JSON string format for hierarchical sparse attention configuration. Required fields: `algorithm` (str), `backend` (str). All other fields are algorithm-specific and passed to the algorithm constructor. `None` Type: str - + + `--enable-hisparse` + Enable hierarchical sparse attention + `False` + bool flag (set to enable) + + ## LMCache @@ -1786,13 +2086,13 @@ Please consult the documentation below and [server_args.py](https://github.com/s - + `--enable-lmcache` Using LMCache as an alternative hierarchical cache solution `False` bool flag (set to enable) - + `--lmcache-config-file` Path to the LMCache YAML configuration file `None` @@ -1818,37 +2118,37 @@ Please consult the documentation below and [server_args.py](https://github.com/s - + `--kt-weight-path` [ktransformers parameter] The path of the quantized expert weights for amx kernel. A local folder. `None` Type: str - + `--kt-method` [ktransformers parameter] Quantization formats for CPU execution. `AMXINT4` Type: str - + `--kt-cpuinfer` [ktransformers parameter] The number of CPUInfer threads. `None` Type: int - + `--kt-threadpool-count` [ktransformers parameter] One-to-one with the number of NUMA nodes (one thread pool per NUMA). `2` Type: int - + `--kt-num-gpu-experts` [ktransformers parameter] The number of GPU experts. `None` Type: int - + `--kt-max-deferred-experts-per-token` [ktransformers parameter] Maximum number of experts deferred to CPU per token. All MoE layers except the final one use this value; the final layer always uses 0. `None` @@ -1875,13 +2175,13 @@ Please consult the documentation below and [server_args.py](https://github.com/s - + `--dllm-algorithm` The diffusion LLM algorithm, such as LowConfidence. `None` Type: str - + `--dllm-algorithm-config` The diffusion LLM algorithm configurations. Must be a YAML file. `None` @@ -1907,31 +2207,31 @@ Please consult the documentation below and [server_args.py](https://github.com/s - + `--cpu-offload-gb` How many GBs of RAM to reserve for CPU offloading. `0` Type: int - + `--offload-group-size` Number of layers per group in offloading. `-1` Type: int - + `--offload-num-in-group` Number of layers to be offloaded within a group. `1` Type: int - + `--offload-prefetch-step` Steps to prefetch in offloading. `1` Type: int - + `--offload-mode` Mode of offloading. `cpu` @@ -1957,13 +2257,14 @@ Please consult the documentation below and [server_args.py](https://github.com/s - - `--multi-item-scoring-delimiter` - Delimiter token ID for multi-item scoring. Used to combine Query and Items into a single sequence: Query<delimiter>Item1<delimiter>Item2<delimiter>... This enables efficient batch processing of multiple items against a single query. - `None` - Type: int + + + `--enable-mis` + Enable Multi-Item Scoring optimization. Combines query and multiple items into a single sequence for efficient batch processing. Requires --attention-backend flashinfer; auto-disables CUDA graph, radix cache, and chunked prefill. + `False` + bool flag (set to enable) - + ## Optimization/debug options @@ -1983,444 +2284,535 @@ Please consult the documentation below and [server_args.py](https://github.com/s - - --disable-radix-cache + + `--disable-radix-cache` Disable RadixAttention for prefix caching. - False + `False` bool flag (set to enable) - - --cuda-graph-config + + `--cuda-graph-config` Canonical per-phase CUDA graph settings as JSON, e.g. {`{"decode":{"backend":"full","max_bs":256},"prefill":{"backend":"tc_piecewise","tc_compiler":"eager"}}`}. JSON wins over the per-phase --cuda-graph-* convenience flags and over the legacy flags. Allowed backends: full, breakable, tc_piecewise, disabled (full is decode-only). `None` Type: JSON (dict-of-dicts) - - --cuda-graph-backend-decode + + `--cuda-graph-backend-decode` Backend for the decode phase. Folds into cuda_graph_config[decode].backend. `None` full, breakable, tc_piecewise, disabled - - --cuda-graph-backend-prefill + + `--cuda-graph-backend-prefill` Backend for the prefill phase. Folds into cuda_graph_config[prefill].backend. `None` breakable, tc_piecewise, disabled - - --cuda-graph-max-bs-decode + + `--cuda-graph-max-bs-decode` Maximum batch size captured for the decode CUDA graph. `None` Type: int - - --cuda-graph-max-bs-prefill + + `--cuda-graph-max-bs-prefill` Maximum batch size captured for the prefill CUDA graph. `None` Type: int - - --cuda-graph-bs-decode + + `--cuda-graph-bs-decode` Explicit list of batch sizes to capture for the decode CUDA graph. `None` List[int] - - --cuda-graph-bs-prefill + + `--cuda-graph-bs-prefill` Explicit list of batch sizes to capture for the prefill CUDA graph. `None` List[int] - - --cuda-graph-tc-compiler + + `--cuda-graph-tc-compiler` Compiler used by the tc_piecewise backend (only the prefill phase consumes it today). `None` eager, inductor - - --disable-cuda-graph-padding + + `--disable-cuda-graph-padding` Disable cuda graph when padding is needed. Still uses cuda graph when padding is not needed. - False + `False` bool flag (set to enable) - - --enable-profile-cuda-graph + + `--enable-profile-cuda-graph` Enable profiling of cuda graph capture. - False + `False` bool flag (set to enable) - - --debug-cuda-graph + + `--debug-cuda-graph` Eager-mode CUDA graph via the breakable backend: graph breaks let every op run eagerly while still going through the capture/replay path. Useful for debugging capture/replay issues. - False + `False` bool flag (set to enable) - - --enable-cudagraph-gc + + `--enable-cudagraph-gc` Enable garbage collection during CUDA graph capture. If disabled (default), GC is frozen during capture to speed up the process. - False + `False` bool flag (set to enable) - - --enable-layerwise-nvtx-marker + + `--enable-layerwise-nvtx-marker` Enable layerwise NVTX profiling annotations for the model. This adds NVTX markers to every layer for detailed per-layer performance analysis with Nsight Systems. - False + `False` bool flag (set to enable) - - --enable-nccl-nvls + + `--enable-nccl-nvls` Enable NCCL NVLS for prefill heavy requests when available. - False + `False` bool flag (set to enable) - - --enable-symm-mem + + `--enable-symm-mem` Enable NCCL symmetric memory for fast collectives. - False + `False` bool flag (set to enable) - - --disable-flashinfer-cutlass-moe-fp4-allgather + + `--disable-flashinfer-cutlass-moe-fp4-allgather` Disables quantize before all-gather for flashinfer cutlass moe. - False + `False` bool flag (set to enable) - - --enable-tokenizer-batch-encode + + `--enable-tokenizer-batch-encode` Enable batch tokenization for improved performance when processing multiple text inputs. Do not use with image inputs, pre-tokenized input_ids, or input_embeds. - False + `False` bool flag (set to enable) - - --disable-tokenizer-batch-decode + + `--disable-tokenizer-batch-decode` Disable batch decoding when decoding multiple completions. - False + `False` bool flag (set to enable) - - --disable-outlines-disk-cache + + `--disable-outlines-disk-cache` Disable disk cache of outlines to avoid possible crashes related to file system or high concurrency. - False + `False` bool flag (set to enable) - - --disable-custom-all-reduce + + `--disable-custom-all-reduce` Disable the custom all-reduce kernel and fall back to NCCL. - False + `False` bool flag (set to enable) - - --enable-mscclpp + + `--enable-mscclpp` Enable using mscclpp for small messages for all-reduce kernel and fall back to NCCL. - False + `False` bool flag (set to enable) - - --enable-torch-symm-mem + + `--enable-torch-symm-mem` Enable using torch symm mem for all-reduce kernel and fall back to NCCL. Only supports CUDA device SM90 and above. SM90 supports world size 4, 6, 8. SM10 supports world size 6, 8. - False + `False` bool flag (set to enable) - - --disable-overlap-schedule + + `--disable-overlap-schedule` Disable the overlap scheduler, which overlaps the CPU scheduler with GPU model worker. - False + `False` bool flag (set to enable) - - --enable-mixed-chunk + + `--enable-mixed-chunk` Enabling mixing prefill and decode in a batch when using chunked prefill. - False + `False` bool flag (set to enable) - - --enable-dp-attention + + `--enable-dp-attention` Enabling data parallelism for attention and tensor parallelism for FFN. The dp size should be equal to the tp size. Currently DeepSeek-V2 and Qwen 2/3 MoE models are supported. - False + `False` bool flag (set to enable) - - --enable-dp-lm-head + + `--enable-dp-lm-head` Enable vocabulary parallel across the attention TP group to avoid all-gather across DP groups, optimizing performance under DP attention. - False + `False` bool flag (set to enable) - - --enable-two-batch-overlap + + `--enable-two-batch-overlap` Enabling two micro batches to overlap. - False + `False` bool flag (set to enable) - - --enable-single-batch-overlap + + `--enable-single-batch-overlap` Let computation and communication overlap within one micro batch. - False + `False` bool flag (set to enable) - - --tbo-token-distribution-threshold + + `--tbo-token-distribution-threshold` The threshold of token distribution between two batches in micro-batch-overlap, determines whether to two-batch-overlap or two-chunk-overlap. Set to 0 denote disable two-chunk-overlap. - 0.48 + `0.48` Type: float - - --enable-torch-compile + + `--enable-torch-compile` Optimize the model with torch.compile. Experimental feature. - False + `False` bool flag (set to enable) - - --enable-torch-compile-debug-mode + + `--enable-torch-compile-debug-mode` Enable debug mode for torch compile. - False + `False` bool flag (set to enable) - - --torch-compile-max-bs + + `--torch-compile-max-bs` Set the maximum batch size when using torch compile. - 32 + `32` Type: int - - --cuda-graph-max-bs + + `--cuda-graph-max-bs` Deprecated alias for --cuda-graph-max-bs-decode. `None` Type: int - - --cuda-graph-bs + + `--cuda-graph-bs` Deprecated alias for --cuda-graph-bs-decode. `None` List[int] - - --disable-cuda-graph + + `--disable-cuda-graph` Deprecated. Use --cuda-graph-backend-decode=disabled and/or --cuda-graph-backend-prefill=disabled. False bool flag (set to enable) - - --enable-breakable-cuda-graph + + `--enable-breakable-cuda-graph` Deprecated alias for --cuda-graph-backend-prefill=breakable. False bool flag (set to enable) - - --disable-prefill-cuda-graph + + `--disable-prefill-cuda-graph` Disable the prefill-phase CUDA graph. Convenience for --cuda-graph-backend-prefill=disabled. - False + `False` bool flag (set to enable) - - --disable-decode-cuda-graph + + `--disable-decode-cuda-graph` Disable the decode-phase CUDA graph. Convenience for --cuda-graph-backend-decode=disabled. - False + `False` bool flag (set to enable) - - --disable-piecewise-cuda-graph + + `--disable-piecewise-cuda-graph` Deprecated alias for --cuda-graph-backend-prefill=disabled. False bool flag (set to enable) - - --enforce-piecewise-cuda-graph + + `--enforce-piecewise-cuda-graph` Deprecated alias for --cuda-graph-backend-prefill=tc_piecewise. Explicitly setting the prefill backend now skips the auto-disable cascade automatically. False bool flag (set to enable) - - --piecewise-cuda-graph-tokens + + `--piecewise-cuda-graph-tokens` Deprecated alias for --cuda-graph-bs-prefill. `None` List[int] - - --piecewise-cuda-graph-compiler + + `--piecewise-cuda-graph-compiler` Deprecated alias for --cuda-graph-tc-compiler. eager eager, inductor - - --piecewise-cuda-graph-max-tokens + + `--piecewise-cuda-graph-max-tokens` Deprecated alias for --cuda-graph-max-bs-prefill. 4096 Type: int - - --torchao-config + + `--torchao-config` Optimize the model with torchao. Experimental feature. Current choices are: int8dq, int8wo, int4wo-<group_size>, fp8wo, fp8dq-per_tensor, fp8dq-per_row - `` + `""` Type: str - - --enable-nan-detection - Enable the NaN detection for debugging purposes. - False - bool flag (set to enable) - - - --enable-p2p-check + + + `--enable-p2p-check` Enable P2P check for GPU access, otherwise the p2p access is allowed by default. - False + `False` bool flag (set to enable) - - --triton-attention-reduce-in-fp32 + + `--triton-attention-reduce-in-fp32` Cast the intermediate attention results to fp32 to avoid possible crashes related to fp16. This only affects Triton attention kernels. - False + `False` bool flag (set to enable) - - --triton-attention-num-kv-splits + + `--triton-attention-num-kv-splits` The number of KV splits in flash decoding Triton kernel. Larger value is better in longer context scenarios. The default value is 8. - 8 + `8` Type: int - - --triton-attention-split-tile-size + + `--triton-attention-split-tile-size` The size of split KV tile in flash decoding Triton kernel. Used for deterministic inference. `None` Type: int - - --num-continuous-decode-steps + + `--num-continuous-decode-steps` Run multiple continuous decoding steps to reduce scheduling overhead. This can potentially increase throughput but may also increase time-to-first-token latency. The default value is 1, meaning only run one decoding step at a time. - 1 + `1` Type: int - - --delete-ckpt-after-loading + + `--delete-ckpt-after-loading` Delete the model checkpoint after loading the model. - False + `False` bool flag (set to enable) - - --enable-memory-saver + + `--enable-memory-saver` Allow saving memory using release_memory_occupation and resume_memory_occupation - False + `False` bool flag (set to enable) - - --enable-weights-cpu-backup + + `--enable-weights-cpu-backup` Save model weights to CPU memory during release_weights_occupation and resume_weights_occupation - False + `False` bool flag (set to enable) - - --enable-draft-weights-cpu-backup + + `--enable-draft-weights-cpu-backup` Save draft model weights to CPU memory during release_weights_occupation and resume_weights_occupation - False + `False` bool flag (set to enable) - - --allow-auto-truncate + + `--allow-auto-truncate` Allow automatically truncating requests that exceed the maximum input length instead of returning an error. - False + `False` bool flag (set to enable) - - --enable-custom-logit-processor + + `--enable-custom-logit-processor` Enable users to pass custom logit processors to the server (disabled by default for security) - False + `False` bool flag (set to enable) - - --flashinfer-mla-disable-ragged + + `--flashinfer-mla-disable-ragged` Not using ragged prefill wrapper when running flashinfer mla - False + `False` bool flag (set to enable) - - --disable-shared-experts-fusion + + `--disable-shared-experts-fusion` Disable shared experts fusion optimization for deepseek v3/r1. - False + `False` bool flag (set to enable) - - --disable-chunked-prefix-cache + + `--disable-chunked-prefix-cache` Disable chunked prefix cache feature for deepseek, which should save overhead for short sequences. - False + `False` bool flag (set to enable) - - --disable-fast-image-processor + + `--disable-fast-image-processor` Adopt base image processor instead of fast image processor. - False + `False` bool flag (set to enable) - - --keep-mm-feature-on-device + + `--keep-mm-feature-on-device` Keep multimodal feature tensors on device after processing to save D2H copy. - False + `False` bool flag (set to enable) - - --enable-return-hidden-states + + `--enable-return-hidden-states` Enable returning hidden states with responses. - False + `False` bool flag (set to enable) - - --enable-return-routed-experts + + `--enable-return-routed-experts` Enable returning routed experts of each layer with responses. - False + `False` bool flag (set to enable) - - --scheduler-recv-interval + + `--scheduler-recv-interval` The interval to poll requests in scheduler. Can be set to >1 to reduce the overhead of this. - 1 + `1` Type: int - - --numa-node + + `--numa-node` Sets the numa node for the subprocesses. i-th element corresponds to i-th subprocess. `None` List[int] - - --enable-deterministic-inference + + `--enable-deterministic-inference` Enable deterministic inference mode with batch invariant ops. - False + `False` bool flag (set to enable) - - --rl-on-policy-target + + `--rl-on-policy-target` The training system that SGLang needs to match for true on-policy. `None` fsdp - - --enable-attn-tp-input-scattered + + `--enable-attn-tp-input-scattered` Allow input of attention to be scattered when only using tensor parallelism, to reduce the computational load of operations such as qkv latent. - False + `False` bool flag (set to enable) - - --enable-prefill-cp + + `--enable-prefill-cp` Enable context parallelism for the prefill phase. Select the layout with --cp-strategy. - False + `False` bool flag (set to enable) - - --cp-strategy + + `--cp-strategy` Sharding strategy for prefill CP. zigzag is the former in-seq-split mode; interleave is the former round-robin-split mode. None zigzag, interleave - - --enable-fused-qk-norm-rope + + `--enable-fused-qk-norm-rope` Enable fused qk normalization and rope rotary embedding. - False + `False` bool flag (set to enable) - - --enable-precise-embedding-interpolation + + `--enable-precise-embedding-interpolation` Enable corner alignment for resize of embeddings grid to ensure more accurate(but slower) evaluation of interpolated embedding values. - False + `False` bool flag (set to enable) + + `--kv-canary` + KV cache canary mode. 'none' disables the canary (default). 'log' prints them while the server keeps running (production-safe). 'raise' fails the server on the first detected mismatch (CI lane). + `none` + none, log, raise + + + `--kv-canary-real-data` + Check the real KV-cache in the canary. 'none' (default) disables the feature. 'partial' checks the first 16 bytes of each real-KV slot. 'all' checks the full real-KV slot. + `none` + Type: str + + + `--kv-canary-sweep-interval` + Every N forward steps, run a full-pool sweep. + `0` + Type: int + + + `--pre-warm-nccl` + Pre-warm NCCL/RCCL communicators during startup to reduce P99 TTFT cold-start latency. Default: enabled for AMD/HIP (RCCL), disabled for NVIDIA/CUDA (NCCL). + `False` + bool flag (set to enable) + + + `--enable-dp-attention-local-control-broadcast` + With DP-attention, send control messages to every DP group leader and broadcast within attn_tp_group instead of the full tp_group. Eliminates a costly all-ranks gloo sync on every scheduler iteration. + `False` + bool flag (set to enable) + + + `--enforce-shared-experts-fusion` + Enforce shared experts fusion even when it would normally be disabled (e.g. under DeepEP). Mutually exclusive with --disable-shared-experts-fusion. + `False` + bool flag (set to enable) + + + `--enable-return-indexer-topk` + Enable returning indexer topk indices of layers with indexer with responses. + `False` + bool flag (set to enable) + + + `--disable-attn-tp-gather` + Disable scheduler-side attn_tp_gather (the upstream SP path that pads num_tokens to attn_tp_size and pre-allocates a gathered buffer). Use for models that manage SP scatter/gather at the model level (e.g., perform their own all_gather/reduce_scatter inside attention) and do not consume the upstream gathered_buffer. Without this, the cuda graph runner pads num_tokens to attn_tp_size, which can cause kernel autotuners to select wrong-sized variants at small batches. + `False` + bool flag (set to enable) + + + `--enable-dsa-prefill-context-parallel` + [Deprecated] Use --enable-prefill-cp instead. + — + Type: str + + + `--enable-nsa-prefill-context-parallel` + [Deprecated] Use --enable-prefill-cp instead. + — + Type: str + + + `--enable-prefill-context-parallel` + [Deprecated] Use --enable-prefill-cp instead. + — + Type: str + + + `--dsa-prefill-cp-mode` + [Deprecated] Use --cp-strategy {zigzag,interleave} instead. 'in-seq-split' maps to 'zigzag'; 'round-robin-split' maps to 'interleave'. + `round-robin-split` + in-seq-split, round-robin-split + + + `--nsa-prefill-cp-mode` + [Deprecated] Use --cp-strategy instead. + Auto + in-seq-split, round-robin-split + + + `--prefill-cp-mode` + [Deprecated] Use --cp-strategy {zigzag,interleave} instead. 'in-seq-split' maps to 'zigzag'. + `in-seq-split` + in-seq-split + + + `--enable-fused-moe-sum-all-reduce` + Enable fused moe triton and sum all reduce. + `False` + bool flag (set to enable) + + + `--gc-threshold` + Set the garbage collection thresholds (the collection frequency). Accepts 1 to 3 integers. + — + Type: int (one or more) + @@ -2441,19 +2833,19 @@ Please consult the documentation below and [server_args.py](https://github.com/s - + `--enable-dynamic-batch-tokenizer` Enable async dynamic batch tokenizer for improved performance when multiple requests arrive concurrently. `False` bool flag (set to enable) - + `--dynamic-batch-tokenizer-batch-size` [Only used if --enable-dynamic-batch-tokenizer is set] Maximum batch size for dynamic batch tokenizer. `32` Type: int - + `--dynamic-batch-tokenizer-batch-timeout` [Only used if --enable-dynamic-batch-tokenizer is set] Timeout in seconds for batching tokenization requests. `0.002` @@ -2479,25 +2871,25 @@ Please consult the documentation below and [server_args.py](https://github.com/s - + `--debug-tensor-dump-output-folder` The output folder for dumping tensors. `None` Type: str - + `--debug-tensor-dump-layers` The layer ids to dump. Dump all layers if not specified. `None` Type: JSON list - + `--debug-tensor-dump-input-file` The input filename for dumping tensors `None` Type: str - + `--debug-tensor-dump-inject` Inject the outputs from jax as the input of every layer. `False` @@ -2523,46 +2915,58 @@ Please consult the documentation below and [server_args.py](https://github.com/s - - --disaggregation-mode + + `--disaggregation-mode` Only used for PD disaggregation. "prefill" for prefill-only server, and "decode" for decode-only server. If not specified, it is not PD disaggregated - null + `null` null, prefill, decode - - --disaggregation-transfer-backend + + `--disaggregation-transfer-backend` The backend for disaggregation transfer. Default is mooncake. - mooncake - mooncake, nixl, ascend, fake + `mooncake` + mooncake, nixl, ascend, fake, mori, mooncake_tcp - - --disaggregation-bootstrap-port + + `--disaggregation-bootstrap-port` Bootstrap server port on the prefill server. Default is 8998. - 8998 + `8998` Type: int - - --disaggregation-ib-device + + `--disaggregation-ib-device` The InfiniBand devices for disaggregation transfer, accepts single device (e.g., --disaggregation-ib-device mlx5_0) or multiple comma-separated devices (e.g., --disaggregation-ib-device mlx5_0,mlx5_1). Default is None, which triggers automatic device detection when mooncake backend is enabled. None Type: str - - --disaggregation-decode-enable-offload-kvcache + + `--disaggregation-decode-enable-offload-kvcache` Enable async KV cache offloading on decode server (PD mode). `False` bool flag (set to enable) - - --num-reserved-decode-tokens + + `--num-reserved-decode-tokens` Number of decode tokens that will have memory reserved when adding new request to the running batch. - 512 + `512` Type: int - - --disaggregation-decode-polling-interval + + `--disaggregation-decode-polling-interval` The interval to poll requests in decode server. Can be set to >1 to reduce the overhead of this. - 1 + `1` + Type: int + + + `--disaggregation-decode-enable-radix-cache` + Enable radix cache on decode server (PD mode). Caches KV prefixes to avoid redundant transfers. Incompatible with --enable-hisparse, speculative decoding, and --disaggregation-transfer-backend fake. + `False` + bool flag (set to enable) + + + `--optimistic-prefill-retries` + Number of optimistic prefill retries that will skip the bootstrap wait. + `0` Type: int @@ -2585,31 +2989,49 @@ Please consult the documentation below and [server_args.py](https://github.com/s - + `--encoder-only` For MLLM with an encoder, launch an encoder-only server `False` bool flag (set to enable) - + `--language-only` For VLM, load weights for the language model only. `False` bool flag (set to enable) - + `--encoder-transfer-backend` The backend for encoder disaggregation transfer. Default is zmq_to_scheduler. `zmq_to_scheduler` - `zmq_to_scheduler`, `zmq_to_tokenizer`, `mooncake` + zmq_to_scheduler, zmq_to_tokenizer, mooncake - + `--encoder-urls` List of encoder server urls. `[]` Type: JSON list - + + `--encoder-bootstrap-port` + Port for the EncoderBootstrapServer that runs in the language-only tokenizer manager process. Encoders register here, and language-only receivers fetch the current URL list from here. + `8997` + Type: int + + + `--encoder-register-urls` + One or more EncoderBootstrapServer URLs to register this encoder with on startup, for dynamic encoder discovery. Example: --encoder-register-urls http://prefill0:8997 http://prefill1:8997. Used with --encoder-only servers. + `[]` + Type: str (one or more) + + + `--enable-adaptive-dispatch-to-encoder` + When enabled, adaptively dispatch: multi-image requests go to encoder in language_only epd mode, single-image requests are processed locally. + `False` + bool flag (set to enable) + + ## Custom weight loader @@ -2629,60 +3051,78 @@ Please consult the documentation below and [server_args.py](https://github.com/s - - --custom-weight-loader + + `--custom-weight-loader` The custom dataloader which used to update the model. Should be set with a valid import path, such as my_package.weight_load_func None List[str] - - --weight-loader-disable-mmap + + `--weight-loader-disable-mmap` Disable mmap while loading weight using safetensors. `False` bool flag (set to enable) - - --weight-loader-prefetch-checkpoints + + `--weight-loader-prefetch-checkpoints` Prefetch checkpoint files into OS page cache before loading. Each rank prefetches a fraction of the shards in a background thread, reducing total network I/O on shared filesystems (NFS/Lustre) from N\*checkpoint to 1\*checkpoint. Recommended for models on network storage. - False + `False` bool flag (set to enable) - - --weight-loader-prefetch-num-threads + + `--weight-loader-prefetch-num-threads` Number of threads per rank for checkpoint prefetching. - 4 + `4` Type: int - - --remote-instance-weight-loader-seed-instance-ip + + `--remote-instance-weight-loader-seed-instance-ip` The ip of the seed instance for loading weights from remote instance. None Type: str - - --remote-instance-weight-loader-seed-instance-service-port + + `--remote-instance-weight-loader-seed-instance-service-port` The service port of the seed instance for loading weights from remote instance. None Type: int - - --remote-instance-weight-loader-send-weights-group-ports + + `--remote-instance-weight-loader-send-weights-group-ports` The communication group ports for loading weights from remote instance. None Type: JSON list - - --remote-instance-weight-loader-backend + + `--remote-instance-weight-loader-backend` The backend for loading weights from remote instance. Can be 'transfer_engine', 'nccl', or 'modelexpress'. Default is 'nccl'. - nccl + `nccl` transfer_engine, nccl, modelexpress - - --remote-instance-weight-loader-start-seed-via-transfer-engine + + `--remote-instance-weight-loader-start-seed-via-transfer-engine` Start seed server via transfer engine backend for remote instance weight loader. - False + `False` bool flag (set to enable) + + `--weight-loader-drop-cache-after-load` + Call posix_fadvise(DONTNEED) on each safetensors shard after loading it. + `False` + bool flag (set to enable) + + + `--engine-info-bootstrap-port` + Port for the engine info bootstrap server. Default is 6789. Must be set explicitly when running multiple instances on the same node. + `6789` + Type: int + + + `--modelexpress-config` + JSON config for ModelExpress P2P weight loading. Keys: "url" (optional gRPC host:port override), "transport" ("nixl" or "transfer_engine"). Example: '{"url": "localhost:8001", "transport": "nixl"}' + `None` + Type: str + @@ -2703,19 +3143,19 @@ Please consult the documentation below and [server_args.py](https://github.com/s - + `--enable-pdmux` Enable PD-Multiplexing, PD running on greenctx stream. `False` bool flag (set to enable) - + `--pdmux-config-path` The path of the PD-Multiplexing config file. `None` Type: str - + `--sm-group-num` Number of sm partition groups. `8` @@ -2741,7 +3181,7 @@ Please consult the documentation below and [server_args.py](https://github.com/s - + `--config` Read CLI options from a config file. Must be a YAML file with configuration options. `None` @@ -2767,46 +3207,35 @@ Please consult the documentation below and [server_args.py](https://github.com/s + - --mm-max-concurrent-calls - The max concurrent calls for async mm data processing. - 32 - Type: int - - - --mm-per-request-timeout - The timeout for each multi-modal request in seconds. - 10.0 - Type: int - - - --enable-broadcast-mm-inputs-process + `--enable-broadcast-mm-inputs-process` Enable broadcast mm-inputs process in scheduler. - False + `False` bool flag (set to enable) - - --mm-process-config + + `--mm-process-config` Multimodal preprocessing config, a json config contains keys: image, video, audio. {} Type: JSON / Dict - - --mm-enable-dp-encoder + + `--mm-enable-dp-encoder` Enabling data parallelism for mm encoder. The dp size will be set to the tp size automatically. - False + `False` bool flag (set to enable) - - --limit-mm-data-per-request + + `--limit-mm-data-per-request` Limit the number of multimodal inputs per request. e.g. '{"image": 1, "video": 1, "audio": 1}' `None` Type: JSON / Dict - - --enable-mm-global-cache + + `--enable-mm-global-cache` Enable Mooncake-backed global multimodal embedding cache on encoder servers so repeated images can reuse cached ViT embeddings instead of recomputing them. - False + `False` bool flag (set to enable) @@ -2829,19 +3258,19 @@ Please consult the documentation below and [server_args.py](https://github.com/s - + `--decrypted-config-file` The path of the decrypted config file. `None` Type: str - + `--decrypted-draft-config-file` The path of the decrypted draft config file. `None` Type: str - + `--enable-prefix-mm-cache` Enable prefix multimodal cache. Currently only supports mm-only. `False` @@ -2867,7 +3296,7 @@ Please consult the documentation below and [server_args.py](https://github.com/s - + `--forward-hooks` JSON-formatted list of forward hook specifications. Each element must include `target_modules` (list of glob patterns matched against `model.named_modules()` names) and `hook_factory` (Python import path to a factory, e.g. `my_package.hooks:make_hook`). An optional `name` field is used for logging, and an optional `config` object is passed as a `dict` to the factory. `None` @@ -2893,7 +3322,7 @@ Please consult the documentation below and [server_args.py](https://github.com/s - + `--msprobe-dump-config` The path of the JSON configuration file for msProbe. If specified, enables msProbe dump. `None` @@ -2919,83 +3348,24 @@ Please consult the documentation below and [server_args.py](https://github.com/s - - `--enable-ep-moe` - NOTE: --enable-ep-moe is deprecated. Please set `--ep-size` to the same value as `--tp-size` instead. - `None` - N/A - - - `--enable-deepep-moe` - NOTE: --enable-deepep-moe is deprecated. Please set `--moe-a2a-backend` to 'deepep' instead. - `None` - N/A - - + + `--prefill-round-robin-balance` Note: Note: --prefill-round-robin-balance is deprecated now. `None` N/A - - `--enable-flashinfer-cutlass-moe` - NOTE: --enable-flashinfer-cutlass-moe is deprecated. Please set `--moe-runner-backend` to 'flashinfer_cutlass' instead. - `None` - N/A - - - `--enable-flashinfer-cutedsl-moe` - NOTE: --enable-flashinfer-cutedsl-moe is deprecated. Please set `--moe-runner-backend` to 'flashinfer_cutedsl' instead. - `None` - N/A - - - `--enable-flashinfer-trtllm-moe` - NOTE: --enable-flashinfer-trtllm-moe is deprecated. Please set `--moe-runner-backend` to 'flashinfer_trtllm' instead. - `None` - N/A - - - `--enable-triton-kernel-moe` - NOTE: --enable-triton-kernel-moe is deprecated. Please set `--moe-runner-backend` to 'triton_kernel' instead. - `None` - N/A - - - `--enable-flashinfer-mxfp4-moe` - NOTE: --enable-flashinfer-mxfp4-moe is deprecated. Please set `--moe-runner-backend` to 'flashinfer_mxfp4' instead. - `None` - N/A - - - `--crash-on-nan` - Crash the server on nan logprobs. - `False` - Type: str - - + + + + + `--hybrid-kvcache-ratio` Mix ratio in [0,1] between uniform and hybrid kv buffers (0.0 = pure uniform: swa_size / full_size = 1)(1.0 = pure hybrid: swa_size / full_size = local_attention_size / context_length) `None` Optional[float] - - `--load-watch-interval` - The interval of load watching in seconds. - `0.1` - Type: float - - - `--nsa-prefill` - Deprecated alias for `--dsa-prefill-backend`. - `flashmla_sparse` - `flashmla_sparse`, `flashmla_decode`, `fa3`, `tilelang`, `aiter` - - - `--nsa-decode` - Deprecated alias for `--dsa-decode-backend`. - `flashmla_kv` - `flashmla_prefill`, `flashmla_kv`, `fa3`, `tilelang`, `aiter` - + + diff --git a/docs_new/docs/references/environment_variables.mdx b/docs_new/docs/references/environment_variables.mdx index c3d32088f..44bb8fb87 100644 --- a/docs_new/docs/references/environment_variables.mdx +++ b/docs_new/docs/references/environment_variables.mdx @@ -5,7 +5,7 @@ metatags: --- SGLang supports various environment variables that can be used to configure its runtime behavior. This document provides a comprehensive list and aims to stay updated over time. -*Note: SGLang uses two prefixes for environment variables: `SGL_` and `SGLANG_`. This is likely due to historical reasons. While both are currently supported for different settings, future versions might consolidate them.* +*Note: The canonical prefix for all SGLang environment variables is `SGLANG_`. The legacy `SGL_` prefix is deprecated: any `SGL_*` variable set in the environment is automatically rewritten to its `SGLANG_*` equivalent at import time with a deprecation warning, and the alias will be removed in a future release. A few variables keep an upstream/vendor prefix (e.g. `MOONCAKE_*`, `ASCEND_*`) because that is their canonical name.* ## General Configuration @@ -167,11 +167,6 @@ SGLang supports various environment variables that can be used to configure its Set the maximum number of requests per poll, with a negative value indicating no limit `-1` - - `SGLANG_DISABLE_FA4_WARMUP` - Disable Flash Attention 4 warmup passes (set to 1, true, yes, or on to disable) - `false` - `SGLANG_DATA_PARALLEL_BUDGET_INTERVAL` Interval for DPBudget updates @@ -217,11 +212,6 @@ SGLang supports various environment variables that can be used to configure its Size of preallocated GPU buffer (in GB) for NCCL symmetric memory pool to limit memory fragmentation. Only have an effect when server arg `--enable-symm-mem` is set. -1 - - SGLANG_CUSTOM_ALLREDUCE_ALGO - The algorithm of custom all-reduce. Set to oneshot or 1stage to force use one-shot. Set to twoshot or 2stage to force use two-shot. - `` - SGLANG_SKIP_SOFTMAX_PREFILL_THRESHOLD_SCALE_FACTOR Skip-softmax threshold scale factor for TRT-LLM prefill attention in flashinfer. None means standard attention. See https://arxiv.org/abs/2512.12087 @@ -336,6 +326,96 @@ SGLang supports various environment variables that can be used to configure its Run shared experts on an alternate stream when single batch overlap is enabled on GB200. When not setting this flag, shared experts and down gemm will be overlapped with DeepEP combine together. `"false"` + + SGLANG_DISABLED_MODEL_ARCHS + Comma-separated list of model architectures to disable from auto-registration. + Not set + + + SGLANG_SORT_WEIGHT_FILES + Controls weight-file ordering for load-time I/O optimization. -1 disables sorting/staggering (original order); 0 sorts files only; a value k greater than 0 sorts and staggers per-rank order with factor k for better multi-rank I/O concurrency. + 0 + + + SGLANG_RETURN_ORIGINAL_LOGPROB + Return the original (pre-temperature) logprobs instead of the post-sampling values. + false + + + SGLANG_ENABLE_COLOCATED_BATCH_GEN + Enable colocated batch generation. + false + + + SGLANG_ENABLE_MOE_DEFERRED_FINALIZE + Defer the MoE finalize step to overlap it with other work. + false + + + SGLANG_PATCH_TOKENIZER + Patch the tokenizer to cache all_special_tokens/all_special_ids (notably for Kimi tiktoken, where ITL can otherwise regress under high batch size). + true + + + SGLANG_ENABLE_LOGITS_PROCESSER_CHUNK + Process logits in chunks to reduce peak memory. + false + + + SGLANG_LOGITS_PROCESSER_CHUNK_SIZE + Chunk size (in tokens) used when logits-processor chunking is enabled. + 2048 + + + SGLANG_FLASHINFER_USE_PAGED + Use the paged FlashInfer attention path. + false + + + SGLANG_FLASHINFER_WORKSPACE_SIZE + FlashInfer workspace size in bytes (default ≈ 384 MiB). + 402653184 + + + SGLANG_PREP_IN_CUDA_GRAPH + Capture input preparation inside the CUDA graph. + true + + + SGLANG_EAGER_INPUT_NO_COPY + In eager forward, wrap the ForwardBatch's own tensors instead of copying them into the CUDA graph buffer registry (skips a per-iter device-to-device copy). + false + + + SGLANG_DEEPGEMM_SANITY_CHECK + Run extra sanity checks on DeepGEMM kernels. + false + + + SGLANG_DEEPGEMM_PDL + Enable Programmatic Dependent Launch (PDL) for DeepGEMM kernels. + true + + + SGLANG_PP_PARALLEL_DEEPGEMM_WARMUP + Run DeepGEMM warmup in parallel across pipeline-parallel ranks. + false + + + SGLANG_DISABLE_STATIC_WATERFILL + Force dynamic DeepEP waterfill with runtime EP all-reduce instead of the default static local-batch path. + false + + + SGLANG_NIXL_EP_BF16_DISPATCH + Use BF16 for NIXL-EP dispatch. + false + + + SGLANG_NIXL_EP_NUM_MAX_DISPATCH_TOKENS_PER_RANK + Maximum number of dispatched tokens per GPU for NIXL-EP. + 128 + @@ -461,6 +541,36 @@ SGLang supports various environment variables that can be used to configure its Experimental. When enabled, broadcast the finalized NSA/DSA indexer top-k result from attention TP rank 0 to the other attention TP ranks. This can mitigate top-k mismatches in TP attention runs at the cost of some speed. false + + SGLANG_MORI_SEND_AUX_RDMA + Send CPU-resident AUX data via RDMA instead of ZMQ TCP. + false + + + SGLANG_MORI_TRANSFER_SHARDS + Number of sharded synchronous worker threads draining KV transfers; also bounds outstanding transfers (primary RDMA send-queue throttle). + 8 + + + SGLANG_MORI_WAIT_POLL_MS + Poll cadence (ms) at which a transfer worker wakes to check the SLA while waiting for completion. + 1000 + + + SGLANG_MORI_TRANSFER_TIMEOUT_MS + Per-transfer SLA (ms) before a KV transfer is failed; 0 disables the SLA. + 0 + + + SGLANG_DSA_HIP_DISABLE_PRESHUFFLE + Disable weight pre-shuffle on the HIP DSA path. SGLANG_NSA_HIP_DISABLE_PRESHUFFLE is a deprecated alias. + false + + + SGLANG_DSA_MQA_LOGITS_FREE_MEM_FRACTION + Fraction of free memory the MQA-logits step may use on the DSA path. + 0.2 + @@ -550,6 +660,61 @@ SGLang supports various environment variables that can be used to configure its Use the fused parallel QK RMSNorm kernel for MiniMax-M2.x on CUDA when attention TP size > 1 `false` + + SGLANG_ENABLE_STRICT_MEM_CHECK_DURING_BUSY + Enable strict memory checks while the scheduler is busy. + 0 + + + SGLANG_ENABLE_STRICT_MEM_CHECK_DURING_IDLE + Enable strict memory checks while the scheduler is idle. + true + + + SGLANG_NATIVE_MOVE_KV_CACHE + Use the native implementation to move KV cache entries. + false + + + SGLANG_USE_BREAKABLE_CUDA_GRAPH + Use a breakable CUDA graph so it can be interrupted/rebuilt at runtime. + false + + + SGLANG_MEMORY_SAVER_CUDA_GRAPH + Allow CUDA graphs under the release/resume memory saver. + false + + + SGLANG_GEMMA_OUT_OF_PLACE_POSITION_MUTATION + Use out-of-place position mutation for Gemma models. + false + + + SGLANG_MAMBA_CONV_DTYPE + dtype for the Mamba convolution state. + bfloat16 + + + SGLANG_MAMBA_SSM_DTYPE + dtype for the Mamba SSM state (defaults to the model dtype when unset). + Not set + + + SGLANG_EMBEDDINGS_SPARSE_HEAD + Name of the sparse-embeddings head to expose for embedding models. + Not set + + + SGLANG_DSV4_FP4_EXPERTS + Whether DeepSeek V4 experts use FP4. Set to false when using an FP4-to-FP8 converted DeepSeek V4 checkpoint. + true + + + SGLANG_DSV4_REASONING_EFFORT + Default reasoning_effort for the DeepSeek V4 chat encoder when a request does not set it (accepts max, high; empty means unset). + "" + @@ -669,6 +834,41 @@ SGLang supports various environment variables that can be used to configure its Comma-separated bundle indices for Ray actor placement (e.g., "0,1,2,3"). Must match world_size. Enables fine-grained GPU assignment in custom placement groups. Not set + + SGLANG_CPU_QUANTIZATION + Enable CPU-side quantization. + false + + + SGLANG_USE_DYNAMIC_MXFP4_LINEAR + Use dynamic MXFP4 quantization for linear layers. + false + + + USE_TRITON_W8A8_FP8_KERNEL + Use the Triton W8A8 FP8 kernel. + false + + + SGLANG_USE_MESSAGE_QUEUE_BROADCASTER + Use the shared-memory message-queue broadcaster for inter-process tensor broadcast. + true + + + SGLANG_DISTRIBUTED_INIT_METHOD_OVERRIDE + Override the init method used by torch.distributed.init_process_group. Set to env:// to use an externally-created TCPStore via MASTER_ADDR/MASTER_PORT. + Not set + + + SGLANG_TCP_STORE_PORT + Port for the torch.distributed TCPStore. + 29600 + + + SGLANG_SYNC_TOKEN_IDS_ACROSS_TP + Synchronize sampled token ids across tensor-parallel ranks. + false + @@ -833,6 +1033,46 @@ SGLang supports various environment variables that can be used to configure its Config BatchSpanProcessor.max_export_batch_size if tracing is enabled `64` + + SGLANG_PROFILE_V2 + Use the v2 profiler implementation. + false + + + SGLANG_DETECT_SLOW_RANK + Detect and report ranks that fall behind during collective ops. + false + + + SGLANG_FORCE_SHUTDOWN + Force an immediate process-group shutdown on exit. + false + + + SGLANG_PYSPY_DUMP_BEFORE_CRASH + Capture a py-spy stack dump of all processes before crashing. + true + + + SGLANG_CUDA_COREDUMP + Enable CUDA coredump generation (auto-injects the required CUDA_* env vars). + false + + + SGLANG_CUDA_COREDUMP_DIR + Directory for CUDA coredumps. If unset, resolves to RUNNER_TEMP in CI, else /tmp. + Not set + + + SGLANG_CUDA_COREDUMP_BEFORE_CRASH + Trigger a CUDA coredump before crashing. + true + + + SGLANG_CUDA_COREDUMP_BEFORE_CRASH_WAIT_SECS + Seconds to wait for the CUDA coredump to finish before exiting. + 60.0 + @@ -882,6 +1122,36 @@ SGLang supports various environment variables that can be used to configure its Use huge pages for host KV cache allocations (HiCache / disaggregation offload). Valid values: 2MB (2 MiB pages via MAP_HUGE_2MB) or 1GB (1 GiB pages via MAP_HUGE_1GB). Requires huge pages to be pre-allocated on the host OS (/proc/sys/vm/nr_hugepages or /sys/kernel/mm/hugepages). If the allocation fails, the allocator logs a warning and falls back to regular page-size mmap automatically. Not set (uses OS default page size) + + SGLANG_HICACHE_HF3FS_CONFIG_PATH + Path to the HiCache HF3FS backend config file. + Not set + + + SGLANG_HICACHE_FILE_BACKEND_STORAGE_DIR + Storage directory for the HiCache file backend. + Not set + + + SGLANG_HICACHE_FILE_BACKEND_MAX_SIZE + Max size for HiCache file-backend LRU eviction (accepts SI/IEC suffixes; 0 disables eviction). + Not set (eviction off) + + + SGLANG_HICACHE_FILE_BACKEND_EVICTION_RATIO + Target fraction to evict down to when the file-backend max size is reached. + 0.9 + + + SGLANG_HICACHE_FILE_BACKEND_MIN_FREE_SPACE + Minimum free space to keep on the file-backend volume (accepts SI/IEC suffixes). + 0 + + + SGLANG_HICACHE_NIXL_BACKEND_STORAGE_DIR + Storage directory for the HiCache NIXL backend. + Not set + @@ -907,5 +1177,1066 @@ SGLang supports various environment variables that can be used to configure its Controls the strictness level of tool call parsing and validation. <br>Level 0: Off - No strict validation <br>Level 1: Function strict - Enables structural tag constraints for all tools (even if none have strict=True set) <br>Level 2: Parameter strict - Enforces strict parameter validation for all tools, treating them as if they all have strict=True set 0 + + SGLANG_DEFAULT_THINKING + Enable model thinking/reasoning output by default. + false + + + SGLANG_MAX_THINK_TOKENS + Cap on thinking tokens. Negative means unlimited; 0 or greater caps the count. + -1 + + + + +## Logging & Observability + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Environment VariableDescriptionDefault Value
SGLANG_LOG_GCLog Python garbage-collection pauses.false
SGLANG_LOG_FORWARD_ITERSLog each forward iteration.false
SGLANG_LOG_MSLog per-step timing in milliseconds.false
SGLANG_LOG_REQUEST_EXCEEDED_MSLog requests whose processing time exceeds this many milliseconds. -1 disables.-1
SGLANG_LOG_SCHEDULER_STATUS_TARGETTarget (e.g. a file path) for periodic scheduler-status logging.""
SGLANG_LOG_SCHEDULER_STATUS_INTERVALInterval (seconds) between scheduler-status log lines.60.0
+ +## Constrained Decoding (Grammar) + + + + + + + + + + + + + + + + + + + + + + + + + + +
Environment VariableDescriptionDefault Value
SGLANG_GRAMMAR_POLL_INTERVALPoll interval (seconds) for asynchronous grammar compilation.0.005
SGLANG_GRAMMAR_MAX_POLL_ITERATIONSMaximum poll iterations before grammar compilation is treated as stuck.10000
+ +## Scheduler & Batching + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Environment VariableDescriptionDefault Value
SGLANG_INIT_NEW_TOKEN_RATIOInitial new-token ratio used for memory planning.0.7
SGLANG_MIN_NEW_TOKEN_RATIO_FACTORFloor factor for the new-token ratio after decay.0.14
SGLANG_NEW_TOKEN_RATIO_DECAY_STEPSNumber of steps over which the new-token ratio decays.600
SGLANG_RETRACT_DECODE_STEPSNumber of decode steps to look ahead when deciding to retract.20
SGLANG_EMPTY_CACHE_INTERVALInterval (seconds) at which to empty the device cache; set this if memory accumulates over a long serving period. -1 disables.-1
SGLANG_FORCE_STREAM_INTERVALFor non-streaming requests, flush intermediate output batches to the tokenizer manager every N decoded tokens (lower to 1 for accurate TTFT benchmarking).50
SGLANG_DYNAMIC_CHUNKING_SMOOTH_FACTORSmoothing factor for dynamic prefill chunking.0.75
SGLANG_SWA_EVICTION_INTERVAL_MULTIPLIERMultiplier applied to the sliding-window-attention eviction interval.1.0
SGLANG_ENABLE_UNIFIED_RADIX_TREEUse the unified radix-tree cache implementation.false
SGLANG_EXPERIMENTAL_CPP_RADIX_TREEUse the experimental C++ radix-tree implementation.false
SGLANG_RADIX_FORCE_MISSForce radix-cache misses (debugging/benchmarking).false
SGLANG_SCHEDULER_SKIP_ALL_GATHERSkip the scheduler all-gather step.false
SGLANG_ENABLE_WAR_BARRIERForce-enable the write-after-read barrier for the overlap scheduler even when CUDA is not detected (e.g. AMD/ROCm). On CUDA the barrier is always enabled.false
SGLANG_PP_SKIP_PURE_CHUNKED_OUTPUT_COMMIn pipeline parallel, skip output send/recv when a batch is entirely non-final chunked-prefill requests.false
SGLANG_KILLPG_ON_SCHEDULER_EXCEPTIONKill the whole process group when the scheduler raises an exception.false
SGLANG_REQUEST_STATE_WAIT_TIMEOUTTokenizer-manager request-state wait timeout (seconds).4
+ +## PD Disaggregation (Runtime) + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Environment VariableDescriptionDefault Value
SGLANG_DISAGGREGATION_THREAD_POOL_SIZEThread-pool size for KV transfers. Defaults to a value computed from the CPU count at runtime.Not set (computed at runtime)
SGLANG_DISAGGREGATION_QUEUE_SIZEDisaggregation transfer queue size.4
SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUTTimeout (seconds) for the disaggregation bootstrap handshake.300
SGLANG_DISAGGREGATION_WAITING_TIMEOUTTimeout (seconds) for a request waiting on KV transfer.300
SGLANG_DISAGGREGATION_HEARTBEAT_INTERVALInterval (seconds) between disaggregation heartbeats.5.0
SGLANG_DISAGGREGATION_HEARTBEAT_MAX_FAILUREConsecutive heartbeat failures tolerated before a peer is considered dead.2
SGLANG_DISAGGREGATION_BOOTSTRAP_ENTRY_CLEANUP_INTERVALInterval (seconds) for cleaning up stale bootstrap entries.120
SGLANG_DISAGGREGATION_NIXL_BACKENDNIXL transport backend for disaggregation.UCX
SGLANG_DISAGGREGATION_NIXL_BACKEND_PARAMSJSON parameters passed to the NIXL backend.{'{}'}
SGLANG_DISAGGREGATION_ALL_CP_RANKS_TRANSFERHave all context-parallel ranks participate in KV transfer.false
SGLANG_DISAGGREGATION_FORCE_QUERY_PREFILL_DP_RANKForce querying the prefill DP rank for routing.false
SGLANG_DISAGGREGATION_NUM_PRE_ALLOCATE_REQSExtra slots in req_to_token_pool for decode workers (effective when max_num_reqs greater than 32), letting more KV transfers overlap decode.0
+ +## Mooncake KV Store & Transfer + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Environment VariableDescriptionDefault Value
SGLANG_HICACHE_MOONCAKE_CONFIG_PATHPath to the HiCache Mooncake store config file.Not set
SGLANG_HICACHE_MOONCAKE_REUSE_TEReuse the Mooncake transfer engine across HiCache operations.true
SGLANG_MOONCAKE_SEND_AUX_TCPSend Mooncake AUX data over TCP.false
SGLANG_ENABLE_FAILED_SESSION_PROBEProbe failed Mooncake sessions for recovery.false
SGLANG_FAILED_SESSION_PROBE_INTERVAL_SInterval (seconds) between failed-session probes.30.0
MOONCAKE_MASTERAddress of the Mooncake master.Not set
MOONCAKE_CLIENTMooncake client identifier.Not set
MOONCAKE_LOCAL_HOSTNAMELocal hostname advertised to Mooncake.localhost
MOONCAKE_TE_META_DATA_SERVERMooncake transfer-engine metadata server.P2PHANDSHAKE
MOONCAKE_GLOBAL_SEGMENT_SIZEMooncake global segment size.4gb
MOONCAKE_PROTOCOLMooncake transport protocol.rdma
MOONCAKE_DEVICEMooncake RDMA device(s).""
MOONCAKE_MASTER_METRICS_PORTPort for Mooncake master metrics.9003
MOONCAKE_CHECK_SERVERCheck connectivity to the Mooncake server on startup.false
MOONCAKE_STANDALONE_STORAGERun Mooncake in standalone storage mode.false
MOONCAKE_ENABLE_SSD_OFFLOADEnable SSD offload in Mooncake.false
MOONCAKE_OFFLOAD_FILE_STORAGE_PATHFile storage path for Mooncake SSD offload.Not set
ENABLE_ASCEND_TRANSFER_WITH_MOONCAKEEnable Ascend NPU transfers via Mooncake.false
ASCEND_NPU_PHY_IDPhysical Ascend NPU id used for Mooncake transfers. -1 auto-detects.-1
+ +## Attention & Kernels + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Environment VariableDescriptionDefault Value
SGLANG_TRITON_DECODE_ATTN_STATIC_KV_SPLITSUse static KV splits in the Triton decode-attention kernel.false
SGLANG_MUSA_FA3_FORCE_UPDATE_METADATAForce FA3 metadata updates on MThreads MUSA.false
SGLANG_SKIP_SGL_KERNEL_VERSION_CHECKSkip the sgl-kernel version compatibility check.false
+ +## Deterministic Inference + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Environment VariableDescriptionDefault Value
SGLANG_ENABLE_DETERMINISTIC_INFERENCEEnable deterministic inference (fixed reduction/accumulation order).false
SGLANG_USE_1STAGE_ALLREDUCEUse the 1-stage all-reduce kernel on AMD (deterministic, fixed accumulation order). If unset, it is auto-enabled when deterministic inference is on.false
SGLANG_FLASHINFER_PREFILL_SPLIT_TILE_SIZEFlashInfer prefill split-tile size for deterministic attention.4096
SGLANG_FLASHINFER_DECODE_SPLIT_TILE_SIZEFlashInfer decode split-tile size for deterministic attention.2048
SGLANG_TRITON_PREFILL_TRUNCATION_ALIGN_SIZETriton prefill truncation alignment size for deterministic attention.4096
SGLANG_TRITON_DECODE_SPLIT_TILE_SIZETriton decode split-tile size for deterministic attention.256
+ +## Speculative Decoding & Overlap + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Environment VariableDescriptionDefault Value
SGLANG_ENABLE_OVERLAP_PLAN_STREAMPlan the next step on a separate stream to overlap with the current step (Overlap Spec V2).false
SGLANG_SPEC_ENABLE_STRICT_FILTER_CHECKEnable strict filter checks in speculative decoding.true
SGLANG_SPEC_SKIP_ZERO_STEP_DRAFT_EXTENDSkip draft_extend while adaptive spec is at steps=0; saves a draft forward but the draft KV goes stale.false
SGLANG_NGRAM_FORCE_GREEDY_VERIFYForce greedy verification for the n-gram speculative path.false
SGLANG_SANITIZE_NAN_LOGITSSanitize NaN logits before sampling kernels and emit a throttled warning.true
+ +## EPLB (Expert Parallel Load Balancing) + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Environment VariableDescriptionDefault Value
SGLANG_EXPERT_DISTRIBUTION_RECORDER_DIROutput directory for the expert-distribution recorder./tmp
SGLANG_ENABLE_EPLB_BALANCEDNESS_METRICEmit an EPLB balancedness metric.false
SGLANG_LOG_EXPERT_LOCATION_METADATALog expert-location metadata.false
SGLANG_EXPERT_LOCATION_UPDATER_LOG_INPUTLog inputs to the expert-location updater.false
SGLANG_EXPERT_LOCATION_UPDATER_LOG_METRICSLog metrics from the expert-location updater.false
+ +## AMD & ROCm + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Environment VariableDescriptionDefault Value
SGLANG_USE_AITER_AGUse the AITER all-gather implementation.true
SGLANG_USE_AITER_UNIFIED_ATTNUse the AITER unified attention kernel.false
SGLANG_USE_AITER_FP8_PER_TOKENUse AITER FP8 per-token quantization.false
SGLANG_USE_AITER_MOE_GU_ITLVSelect the AITER MoE gate/up tile layout: true interleaves, false uses the separated layout.true
SGLANG_AITER_FUSE_RMSNORM_PADFuse the residual-add + RMSNorm + zero-pad triplet before the MoE block via the AITER Triton kernel (TP=1, post-attention layernorm path only).false
SGLANG_AITER_KV_CACHE_LAYOUTPhysical layout for the MHA KV cache on AITER: nhd or vectorized_5d (SHUFFLE layout enabling pa_decode_gluon).nhd
SGLANG_ROCM_FUSED_DECODE_MLAUse the fused decode MLA kernel on ROCm.false
SGLANG_ROCM_DISABLE_LINEARQUANTDisable linear-layer quantization on ROCm.false
+ +## NPU (Ascend) + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Environment VariableDescriptionDefault Value
SGLANG_NPU_DISABLE_ACL_FORMAT_WEIGHTDisable ACL-format weight conversion on NPU.false
SGLANG_NPU_USE_MULTI_STREAMUse multiple streams on NPU.false
SGLANG_NPU_USE_MLAPOUse the MLAPO path on NPU.false
SGLANG_NPU_FUSED_MOE_MODEFused MoE mode selector for NPU.1
SGLANG_NPU_FORWARD_NATIVE_GELUTANHUse the native gelu-tanh activation forward (for Skywork-Reward-Gemma-2-27B-v0.2).false
SGLANG_NPU_FORWARD_NATIVE_GEMMA_RMS_NORMUse the native Gemma RMSNorm forward (for Skywork-Reward-Gemma-2-27B-v0.2).false
SGLANG_USE_AG_AFTER_QLORADelay all-gather until after QLoRA for better DeepSeek V3.2 performance.false
SGLANG_EXPERIMENTAL_LORA_OPTIMaster switch for the experimental TRT-LLM LoRA fast path. When off, all fine-grained opt switches read false.false
SGLANG_ZBAL_LOCAL_MEM_SIZELocal memory size for the ZBAL (zero-buffer accelerate library) path (NPU only).0
SGLANG_ZBAL_BOOTSTRAP_URLBootstrap URL for the ZBAL path (NPU only).""
+ +## Apple Silicon (MLX / MPS) + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Environment VariableDescriptionDefault Value
SGLANG_USE_MLXUse the MLX backend on Apple Silicon.false
SGLANG_MLX_USE_CUSTOM_ROPEUse the custom RoPE kernel on MLX.false
SGLANG_MLX_FUSE_SWIGLUFuse the SwiGLU activation on MLX.false
SGLANG_MLX_CLEAR_CACHE_STEPSNumber of decode steps between mx.clear_cache() calls. 0 disables cache clearing.256
+ +## Multimodal (VLM) + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Environment VariableDescriptionDefault Value
SGLANG_VLM_CACHE_SIZE_MBSize (MB) of the VLM feature cache.100
SGLANG_IMAGE_MAX_PIXELSMaximum number of pixels per image before resizing.12845056
SGLANG_RESIZE_RESAMPLEResampling filter used when resizing images (e.g. bilinear, bicubic).""
SGLANG_MM_SKIP_COMPUTE_HASHSkip computing multimodal-item hashes.false
SGLANG_MM_AVOID_RETOKENIZEFor pre-tokenized (list[int]) multimodal prompts, preserve the user's original tokens to avoid retokenization drift.true
SGLANG_VIT_ENABLE_CUDA_GRAPHCapture the vision encoder (ViT) in a CUDA graph.false
SGLANG_USE_CUDA_IPC_TRANSPORTUse CUDA IPC transport for multimodal-item tensors.false
SGLANG_USE_IPC_POOL_HANDLE_CACHECache CUDA IPC pool handles.false
SGLANG_MM_FEATURE_CACHE_MBSize (MB) of the multimodal feature cache.1024
SGLANG_MM_ITEM_MEM_POOL_RECYCLE_INTERVAL_SECInterval (seconds) for recycling the multimodal-item memory pool.0.05
+ +## Encoder / EPD (Multimodal Disaggregation) + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Environment VariableDescriptionDefault Value
SGLANG_ENCODER_MM_RECEIVER_MODEEncoder receiver selection used by EPD paths: http or grpc.http
SGLANG_ENCODER_GRPC_TIMEOUT_SECSgRPC timeout (seconds) for encoder communication.60
SGLANG_ENCODER_RECV_TIMEOUTEncoder receive timeout (seconds).180.0
SGLANG_ENCODER_SEND_TIMEOUTEncoder send timeout (seconds).180.0
SGLANG_ENCODER_HTTP_TIMEOUTEncoder HTTP timeout (seconds).1800.0
SGLANG_ENCODER_REQ_TIMEOUTEncoder per-request timeout (seconds).180.0
SGLANG_ENCODER_DISPATCH_MIN_ITEMSMinimum items before the encoder dispatches a batch.2
SGLANG_ENCODER_MAX_BATCH_SIZEMaximum encoder batch size.8
SGLANG_ENCODER_PREPROC_WORKERSNumber of encoder preprocessing workers.8
SGLANG_ENCODER_IMAGE_PROCESSOR_USE_GPURun the image processor on the GPU.false
SGLANG_ENCODER_BOOTSTRAP_HEALTH_CHECK_INTERVALEncoderBootstrapServer health-check interval (seconds). 0 disables it.10.0
SGLANG_ENCODER_BOOTSTRAP_HEALTH_CHECK_TIMEOUTEncoderBootstrapServer health-check timeout (seconds).2.0
SGLANG_EMBEDDING_POOL_SIZE_MBPersistent receiver-side GPU embedding pool size (MB) for Mooncake EPD transport. 0 disables (per-request register/deregister).4096
SGLANG_ENCODER_DP_WORKER_MAX_INFLIGHTMaximum in-flight requests per encoder DP worker.64
SGLANG_BACKUP_PORT_BASEBase port for elastic-EP backup ports.10000
+ +## HTTP & gRPC Server + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Environment VariableDescriptionDefault Value
SGLANG_TIMEOUT_KEEP_ALIVEHTTP keep-alive timeout (seconds).5
SGLANG_UVICORN_WORKER_HEALTHCHECK_TIMEOUTUvicorn multiprocess supervisor per-worker health-check interval (seconds).10
SGLANG_ENABLE_HEALTH_ENDPOINT_GENERATIONHave the /health endpoint run a generation as part of the check.true
SGLANG_WARMUP_TIMEOUTIf a warmup forward batch takes longer than this many seconds, the server crashes to avoid hanging. -1 disables; increase (e.g. to 1800) to accommodate kernel JIT precompile.-1
SGLANG_ENABLE_GRPCEnable the native gRPC server (internal, not yet user-facing).false
SGLANG_GRPC_PORTPort for the native gRPC server.Not set
SGLANG_GRANIAN_PARENT_PIDParent PID for the Granian HTTP/2 worker supervisor.Not set
+ +## NUMA & CPU + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Environment VariableDescriptionDefault Value
SGLANG_NUMA_BIND_V2Use the v2 NUMA binding implementation.true
SGLANG_AUTO_NUMA_BINDAutomatically bind processes to NUMA nodes.false
SGLANG_CRASH_ON_NUMA_BIND_FAILURECrash if NUMA binding fails instead of warning.false
+ +## Metrics + + + + + + + + + + + + + + + + + + + + + + + + + + +
Environment VariableDescriptionDefault Value
SGLANG_ENABLE_METRICS_DEVICE_TIMEREnable device-timer-based metrics.false
SGLANG_ENABLE_METRICS_DP_ATTENTIONEnable data-parallel attention metrics.false
+ +## External Models + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Environment VariableDescriptionDefault Value
SGLANG_EXTERNAL_MODEL_PACKAGEPython package providing external model implementations.""
SGLANG_EXTERNAL_MM_MODEL_ARCHExternal multimodal model architecture name.""
SGLANG_EXTERNAL_MM_PROCESSOR_PACKAGEPython package providing the external multimodal processor.""
+ +## Plugin System + + + + + + + + + + + + + + + + + + + + + + + + +
Environment VariableDescriptionDefault Value
SGLANG_PLATFORMPlatform plugin name to load.""
SGLANG_PLUGINSComma-separated list of plugins to load.""