From ed82def55fac34f116ca94a0e1e5ae062cef73b6 Mon Sep 17 00:00:00 2001 From: Cheng Wan <54331508+ch-wan@users.noreply.github.com> Date: Sun, 6 Sep 2026 21:40:31 -0700 Subject: [PATCH] [Config] Round 6.2: the field declarations move to their namespaces, and the record is assembled from them (#38047) Second of four; stacked on #38046. Mechanical relocation plus one design change that the relocation makes possible. **Review by checking the identity proofs at the bottom** -- nothing here is meant to change behaviour. ## The declarations move `ServerArgs` carried all 487 declarations in one 4,462-line file, each tagged with an `NS("...")` marker naming the namespace it belongs to -- structure supplied by annotation, in a file a namespace away from the `arg_groups/*_hook.py` that resolves it. They move to `arg_groups/fields/`: one module per top-level namespace, one class per leaf namespace (21 of them, `exec.moe` becomes `exec_.py::ExecMoe`). The class carries the `_NS_PATH` it stands for, so the module a field is declared in *is* its namespace and the marker is redundant -- `namespace_of` reads the declaring class instead. `NS` stays for the one case a class cannot express: a single ad-hoc dataclass whose fields span namespaces, which is what the config-bag tests build. Two things travel with the declarations. The `*_CHOICES` lists and the `add_*_choices` adders that extend them move to `arg_groups/choices.py`, since the fields naming them can no longer import from `server_args` without a cycle; `server_args` re-exports all of them, because out-of-tree plugins have always reached them there. And five fields whose only annotation element was the namespace marker become plain annotations -- `A` is `Annotated`, which needs two arguments, so stripping the marker would have left them invalid. `server_args.py` goes from 4,458 lines to about 1,000. ## The record is assembled, not inherited Inheriting the namespace classes would make the record's contents a property of which classes happen to appear in a base list. That is correct only while every namespace declares nothing but operator input, and it stops being correct the moment a derived field is declared: `attn_tp_size` belongs in `parallel.py` next to the leaves it is derived from, and inheriting `Parallel` would put it on the record -- where it is neither input nor safe, since the record is what crosses a process boundary and a derived width pickled to a subprocess is a stamp that elastic scale-up will not refresh. `collect_input_fields` takes the classes that declare input and returns their annotations, defaults and namespaces. Each source's annotations are resolved in its own module and handed on as type objects; carried across as text they would be re-evaluated where they land, and the composing module deliberately imports none of the names the declarations use. A namespace can now declare both halves side by side, and which half reaches the record is one readable call rather than an invariant spread across a base-class list. Nothing is registered on the derived side yet -- this is what makes it possible. `ServerArgs` is still one flat dataclass with 494 attributes, so `server_args.tp_size`, `ServerArgs(model_path=..., tp_size=8)`, pickling to a subprocess and every existing call site are untouched. ### Field order is a contract, so it is written down A dataclass turns field order into a positional constructor signature, and collecting whole namespaces groups fields that used to be interleaved. Keeping `model_path` first is not enough: `ServerArgs("dummy", "/tmp/tokenizer")` would set `load_format="/tmp/tokenizer"` and leave `tokenizer_path=None`, which then selects an invalid model loader -- silently, at a call site that did not change. So `arg_groups/field_order.py` records the order the record had before the split, and `collect_input_fields` orders what it collects by it. A field the record declares that the frozen order does not name goes after it, in declaration order -- the only backward-compatible place for a new field anyway, so a new declaration needs no edit there. The list is a compatibility record and nothing else reads it; the namespace a field belongs to is still the module it is declared in. ## Verification Four ways, all against the base commit: | check | result | |---|---| | `namespace_of` map, field by field | 494 / 494, **0 differences** | | CLI surface (options, defaults, choices, actions) | 507 / 507, **0 differences** | | field order, name by name | 494 / 494, **identical to the base** | | resolution result, 24 launch shapes x 489 fields | **0 differences** | | names importable from `sglang.srt.server_args` | nothing lost | Plus a full registered-unit sweep (648 files) against the stack's merge-base: 19 failures on both sides, the same 19, none of them config. --- python/sglang/srt/arg_groups/arg_utils.py | 74 +- python/sglang/srt/arg_groups/choices.py | 289 ++ python/sglang/srt/arg_groups/field_order.py | 511 +++ .../sglang/srt/arg_groups/fields/__init__.py | 70 + python/sglang/srt/arg_groups/fields/device.py | 83 + python/sglang/srt/arg_groups/fields/disagg.py | 167 + python/sglang/srt/arg_groups/fields/exec_.py | 863 ++++ python/sglang/srt/arg_groups/fields/lora.py | 119 + python/sglang/srt/arg_groups/fields/memory.py | 242 ++ python/sglang/srt/arg_groups/fields/mm.py | 154 + python/sglang/srt/arg_groups/fields/model.py | 385 ++ .../srt/arg_groups/fields/observability.py | 243 ++ .../sglang/srt/arg_groups/fields/parallel.py | 277 ++ .../sglang/srt/arg_groups/fields/schedule.py | 264 ++ .../sglang/srt/arg_groups/fields/serving.py | 300 ++ python/sglang/srt/arg_groups/fields/spec.py | 260 ++ python/sglang/srt/runtime_context.py | 34 +- python/sglang/srt/server_args.py | 3731 +---------------- 18 files changed, 4442 insertions(+), 3624 deletions(-) create mode 100644 python/sglang/srt/arg_groups/choices.py create mode 100644 python/sglang/srt/arg_groups/field_order.py create mode 100644 python/sglang/srt/arg_groups/fields/__init__.py create mode 100644 python/sglang/srt/arg_groups/fields/device.py create mode 100644 python/sglang/srt/arg_groups/fields/disagg.py create mode 100644 python/sglang/srt/arg_groups/fields/exec_.py create mode 100644 python/sglang/srt/arg_groups/fields/lora.py create mode 100644 python/sglang/srt/arg_groups/fields/memory.py create mode 100644 python/sglang/srt/arg_groups/fields/mm.py create mode 100644 python/sglang/srt/arg_groups/fields/model.py create mode 100644 python/sglang/srt/arg_groups/fields/observability.py create mode 100644 python/sglang/srt/arg_groups/fields/parallel.py create mode 100644 python/sglang/srt/arg_groups/fields/schedule.py create mode 100644 python/sglang/srt/arg_groups/fields/serving.py create mode 100644 python/sglang/srt/arg_groups/fields/spec.py diff --git a/python/sglang/srt/arg_groups/arg_utils.py b/python/sglang/srt/arg_groups/arg_utils.py index 5c29992cd..93dde06e6 100644 --- a/python/sglang/srt/arg_groups/arg_utils.py +++ b/python/sglang/srt/arg_groups/arg_utils.py @@ -43,13 +43,11 @@ import copy import dataclasses import functools import types +from collections.abc import Callable from typing import ( Annotated, Any, - Callable, - List, Literal, - Optional, Union, get_args, get_origin, @@ -78,15 +76,15 @@ class Arg: """CLI argument metadata attached to a dataclass field via ``Annotated``.""" help: str = "" - choices: Optional[list] = None - aliases: Optional[List[str]] = None - cli_name: Optional[str] = None - type_parser: Optional[Callable] = None - nargs: Optional[str] = None - required: Optional[bool] = None - action: Optional[Any] = None - action_kwargs: Optional[dict] = None - const: Optional[Any] = None + choices: list | None = None + aliases: list[str] | None = None + cli_name: str | None = None + type_parser: Callable | None = None + nargs: str | None = None + required: bool | None = None + action: Any | None = None + action_kwargs: dict | None = None + const: Any | None = None # When True, this field is skipped by add_cli_args_from_dataclass. # Use for fields that have no CLI surface (e.g. injected via Python only). no_cli: bool = False @@ -113,26 +111,50 @@ class NS: field: A[int, "help", NS("parallel")] = 1 field: A[str, Arg(help="…"), NS("exec.moe")] = "auto" - Kept separate from ``Arg`` (CLI metadata) so the ~400 existing bare-string / - multiline field annotations gain a namespace by *appending* one element, - without rewriting each ``Arg(...)`` call. ``namespace_of`` reads it to build - the RuntimeContext config-bag tree.""" + ``ServerArgs`` no longer uses it: its fields are declared in the + ``arg_groups/fields/`` classes, each of which carries the ``_NS_PATH`` it + stands for, so the module a declaration lives in *is* its namespace. What + is left for this marker is the case a class cannot express -- one ad-hoc + dataclass whose fields span several namespaces, which is what the + config-bag tests build.""" path: str -@functools.lru_cache(maxsize=None) +@functools.cache def namespace_of(cls) -> dict: - """``{field_name: dotted namespace path}`` from the ``NS`` marker in each - field's ``Annotated`` metadata. + """``{field_name: dotted namespace path}``, read from the declaring class. - Fields without an ``NS`` marker are absent from the map (the coverage lint - flags them). Non-dataclass types yield an empty map.""" + A field's namespace is where it is declared: each class in + ``arg_groups/fields/`` carries the ``_NS_PATH`` it stands for, and + ``ServerArgs`` composes them. Walking the MRO therefore answers "which + namespace owns this field" without a per-field marker -- the file the + declaration sits in is the marker. + + A class that is not built that way -- an ad-hoc dataclass spanning several + namespaces, which is what the config-bag tests construct -- falls back to + the per-field ``NS`` marker. A field with neither is absent from the map + (the coverage lint flags them). Non-dataclass types yield an empty map. + """ if not dataclasses.is_dataclass(cls): return {} + # An assembled record: the collector recorded who declared each field, + # because there are no base classes left to ask. + out = dict(getattr(cls, "_NS_BY_FIELD", None) or {}) + # A class that still inherits its namespaces: nearest declaration wins, so + # walk the MRO front to back and keep the first answer. + for base in cls.__mro__: + path = base.__dict__.get("_NS_PATH") + if path is None: + continue + for name in getattr(base, "__annotations__", {}): + out.setdefault(name, path) + if len(out) == len(dataclasses.fields(cls)): + return out hints = get_type_hints(cls, include_extras=True) - out = {} for field in dataclasses.fields(cls): + if field.name in out: + continue tp = hints.get(field.name, field.type) if get_origin(tp) is Annotated: for a in get_args(tp)[1:]: @@ -142,7 +164,7 @@ def namespace_of(cls) -> dict: return out -@functools.lru_cache(maxsize=None) +@functools.cache def field_names(cls) -> frozenset: """Names of ``cls`` dataclass fields — what a declaration may name.""" if not dataclasses.is_dataclass(cls): @@ -150,7 +172,7 @@ def field_names(cls) -> frozenset: return frozenset(field.name for field in dataclasses.fields(cls)) -@functools.lru_cache(maxsize=None) +@functools.cache def resolvable_fields(cls) -> frozenset: """Names of ``cls`` dataclass fields whose ``Arg`` metadata declares ``resolvable=True`` — the whitelist for config resolution. @@ -287,7 +309,7 @@ def _field_to_cli_name(name: str) -> str: # --------------------------------------------------------------------------- -def add_cli_args_from_dataclass(parser, cls, *, fields: Optional[List[str]] = None): +def add_cli_args_from_dataclass(parser, cls, *, fields: list[str] | None = None): """Add argparse arguments for every ``A[T, "help"]`` or ``A[T, Arg(...)]`` field. Fields without an ``Arg`` or bare-string annotation are silently skipped — @@ -362,7 +384,7 @@ def add_cli_args_from_dataclass(parser, cls, *, fields: Optional[List[str]] = No # Check for List[X] — but skip if type_parser is set (the parser # handles the whole value as a single string, e.g. json_list_type). origin = get_origin(inner_type) - if (origin is list or origin is List) and arg_meta.type_parser is None: + if origin is list and arg_meta.type_parser is None: elem_args = get_args(inner_type) elem_type = elem_args[0] if elem_args else str type_func = _infer_type_func(elem_type) diff --git a/python/sglang/srt/arg_groups/choices.py b/python/sglang/srt/arg_groups/choices.py new file mode 100644 index 000000000..c735df017 --- /dev/null +++ b/python/sglang/srt/arg_groups/choices.py @@ -0,0 +1,289 @@ +"""Enumerated choices shared by the config field declarations. + +These lived in ``server_args.py`` beside the fields that name them. The fields +moved to ``arg_groups/fields/``, and ``server_args`` imports the field modules, +so the lists cannot stay there without a cycle. ``server_args`` re-exports them +for the handful of modules that import them from their old home. +""" + +LOAD_FORMAT_CHOICES = [ + "auto", + "pt", + "safetensors", + "npcache", + "dummy", + "sharded_state", + "presharded", + "gguf", + # Experimental and intentionally narrow: expert_pack is validated only for + # DeepSeek-V4-Flash-0731 MXFP4 GGUF (MXFP4 experts, FP8 dense weights) + # and KIMI-K3-MXP4-DERISKED-Q2_K-*.gguf (Q2_K gate/up, Q3_K down weights): + # https://huggingface.co/unsloth/DeepSeek-V4-Flash-0731-GGUF + # https://huggingface.co/Blackfrost-AI/KIMI-K3-Q2_K-GGUF-ABLITERATED + "expert_pack", + "bitsandbytes", + "mistral", + "layered", + "flash_rl", + "remote", + "remote_instance", + "fastsafetensors", + "private", + "runai_streamer", +] + +# TODO: this list should likely contain only methods that support online quantization, or that support using custom quantization classes compatible with a given `quant_method` in config.json. +# Some of the choices here do NOT support online quantization. +QUANTIZATION_CHOICES = [ + "awq", + "fp8", # MOE + linear online quantization. + "mxfp8", # MOE + linear online quantization. + "gptq", + "gptq_marlin", + "awq_marlin", + "bitsandbytes", + "gguf", + # Modelopt has some online quantization support through ModelOptModelLoader. + "modelopt", + "modelopt_fp8", + "modelopt_fp4", + "nvfp4_online", + "modelopt_mixed", + "petit_nvfp4", + "w8a8_int8", # mentioned in quantization.md documentation, supporting compressed-tensors quant_method. + "w8a8_fp8", # mentioned in quantization.md documentation, supporting compressed-tensors quant_method. + "moe_wna16", # custom loading logic for gptq/awq checkpoints (likely untested/unused) + "w4afp8", + "mxfp4", # MOE-only. + "auto-round", + "auto-round-int8", + "compressed-tensors", # for Ktransformers + "modelslim", # for NPU + "mxfp_w4a8", # for NPU W4A8 (MXFP4 weights + MXFP8 activations) + "quark", # AMD Quark quantizer (FP8 / MXFP4 / Int4FP8 etc.) + "quark_int4fp8_moe", + "quark_mxfp4", # Online MOE + linear quantization (incl. NVFP4 -> MXFP4 requantization). + # Apple Silicon MLX backend — on-the-fly quantization of fp16 weights at load + # time via mlx.nn.quantize. Only takes effect when SGLANG_USE_MLX=1. + "mlx_q4", # 4 bits, group_size=64 (mlx-community default) + "mlx_q8", # 8 bits, group_size=64 + "unquant", + "humming", +] + +ATTENTION_BACKEND_CHOICES = [ + # Common + "triton", + "torch_native", + "flex_attention", + "dsa", + "nsa", # Deprecated alias for "dsa" + "dsv4", + "compressed", # Deprecated alias for "dsv4" + # NVIDIA specific + "cutlass_mla", + "fa3", + "fa4", + "flashinfer", + "flashmla", + "trtllm_mla", + "cutedsl_mla", + "tokenspeed_mla", + "trtllm_mha", + "dual_chunk_flash_attn", + "hpc_ops", # HPC-Ops (https://github.com/Tencent/hpc-ops), Hopper (SM90) only, requires --page-size 64 + "minicpm_flashattn", + "minicpm_flashinfer", + # AMD specific + "aiter", + "wave", + # Other platforms + "intel_amx", + "ascend", + "intel_xpu", +] + +# trtllm_mha is valid for decode-only dense-MQA drafts. DFLASH rejects it +# earlier when its per-layer attention requirements are not met. +DRAFT_ATTENTION_BACKEND_CHOICES = [ + "flashinfer", + "fa3", + "fa4", + "triton", + "ascend", + "trtllm_mha", +] + +DETERMINISTIC_ATTENTION_BACKEND_CHOICES = [ + "ascend", + "fa3", + "fa4", + "flashinfer", + "intel_xpu", + "triton", +] + +DISAGG_TRANSFER_BACKEND_CHOICES = [ + "mooncake", + "nixl", + "ascend", + "fake", + "mori", + "mooncake_tcp", +] + +GRAMMAR_BACKEND_CHOICES = ["xgrammar", "outlines", "llguidance", "none"] + +SAMPLING_BACKEND_CHOICES = {"flashinfer", "pytorch", "ascend"} + +MOE_RUNNER_BACKEND_CHOICES = [ + "auto", + "deep_gemm", + "triton", + "triton_kernel", + "flashinfer_trtllm", + "experimental_sgl_trtllm", + "flashinfer_trtllm_routed", + "flashinfer_cutlass", + "flashinfer_mxfp4", + "flashinfer_cutedsl", + "cutlass", + "aiter", + "marlin", + "humming", + "experimental_sgl_marlin", + "hpc_ops", # HPC-Ops (https://github.com/Tencent/hpc-ops), FP8 MoE on Hopper (SM90) only + "megamoe", + "intel_xpu", +] + +MXFP8_MOE_RUNNER_BACKEND_CHOICES = [ + "cutlass", + "deep_gemm", + "flashinfer_trtllm", + "flashinfer_trtllm_routed", +] + +FP8_GEMM_RUNNER_BACKEND_CHOICES = [ + "auto", + "deep_gemm", + "flashinfer_trtllm", + "flashinfer_cutlass", + "flashinfer_deepgemm", + "flashinfer_cutedsl", + "cutlass", + "triton", + "aiter", +] + +FP4_GEMM_RUNNER_BACKEND_CHOICES = [ + "auto", + "flashinfer_cudnn", + "flashinfer_cutedsl", + "flashinfer_cutlass", + "flashinfer_trtllm", + "marlin", +] + +RADIX_EVICTION_POLICY_CHOICES = ["lru", "lfu", "slru", "priority"] + +RL_ON_POLICY_TARGET_CHOICES = ["fsdp"] + +LINEAR_ATTN_KERNEL_BACKEND_CHOICES = [ + "triton", + "cutedsl", + "flashinfer", + "flashkda", + "nvidia_kda", + "ptx_kda", + "helion", + "intel_xpu", +] + + +# -------------------------------------------------------------------------- +# Extension points: out-of-tree platforms and plugins extend these lists +# before ServerArgs is constructed. Each list owns its adder on the line +# below it. A list with no adder is not an extension point -- inline it into +# the field's Arg(choices=...) instead of hoisting it here. +# -------------------------------------------------------------------------- + +# --- Model loading and quantization --- + +add_load_format_choices = LOAD_FORMAT_CHOICES.extend +# NOTE: LoadFormat.IPC_CACHE intentionally has no public --load-format choice. +# It is an internal dispatch format set automatically by ModelRunner when the +# weight cache is enabled (weight_cache_mode != "off"). Exposing it as a CLI +# choice let users create contradictory combos (see _handle_load_format). + +add_quantization_method_choices = QUANTIZATION_CHOICES.extend + +# --- Attention backends --- + +add_attention_backend_choices = ATTENTION_BACKEND_CHOICES.extend + +add_draft_attention_backend_choices = DRAFT_ATTENTION_BACKEND_CHOICES.extend + +# Attention backends whose kernels read the chunked prefix-cache layout. +# Out-of-tree platforms may extend this list (via +# add_chunked_prefix_cache_attention_backend) before ServerArgs construction; +# the chunked-prefix gate is evaluated during resolution. +CHUNKED_PREFIX_CACHE_SUPPORTED_ATTENTION_BACKENDS = [ + "flashinfer", + "fa3", + "fa4", + "flashmla", + "cutedsl_mla", + "cutlass_mla", + "trtllm_mla", + "tokenspeed_mla", +] +add_chunked_prefix_cache_attention_backend = ( + CHUNKED_PREFIX_CACHE_SUPPORTED_ATTENTION_BACKENDS.append +) + +add_deterministic_attention_backend_choices = ( + DETERMINISTIC_ATTENTION_BACKEND_CHOICES.extend +) + +RADIX_SUPPORTED_DETERMINISTIC_ATTENTION_BACKEND = ["ascend", "fa3", "fa4", "triton"] +add_radix_supported_deterministic_attention_backend_choices = ( + RADIX_SUPPORTED_DETERMINISTIC_ATTENTION_BACKEND.extend +) + +# --- Transport --- + +add_disagg_transfer_backend_choices = DISAGG_TRANSFER_BACKEND_CHOICES.extend + +# --- Sampling and grammar --- + +add_grammar_backend_choices = GRAMMAR_BACKEND_CHOICES.extend + + +# --- MoE and GEMM runners --- + +add_moe_runner_backend_choices = MOE_RUNNER_BACKEND_CHOICES.extend + +add_mxfp8_moe_runner_backend_choices = MXFP8_MOE_RUNNER_BACKEND_CHOICES.extend + +add_fp8_gemm_runner_backend_choices = FP8_GEMM_RUNNER_BACKEND_CHOICES.extend + +add_fp4_gemm_runner_backend_choices = FP4_GEMM_RUNNER_BACKEND_CHOICES.extend + +# --- Cache and scheduling policy --- + +add_radix_eviction_policy_choices = RADIX_EVICTION_POLICY_CHOICES.extend + +# --- Reinforcement learning --- + +add_rl_on_policy_target_choices = RL_ON_POLICY_TARGET_CHOICES.extend + +# --- Linear attention --- + +add_linear_attn_kernel_backend_choices = LINEAR_ATTN_KERNEL_BACKEND_CHOICES.extend + +# -------------------------------------------------------------------------- +# Add new extension points at the end of the matching group above. A new +# choice list is inlined into its field by default; hoisting one here makes +# it public API for out-of-tree code and is a deliberate decision. +# -------------------------------------------------------------------------- diff --git a/python/sglang/srt/arg_groups/field_order.py b/python/sglang/srt/arg_groups/field_order.py new file mode 100644 index 000000000..c1a914ca8 --- /dev/null +++ b/python/sglang/srt/arg_groups/field_order.py @@ -0,0 +1,511 @@ +"""The order ``ServerArgs`` presents its fields in, frozen. + +A dataclass turns field order into a positional constructor signature, so +``ServerArgs(model_path, tokenizer_path)`` has to keep meaning what it means. +Grouping the declarations by namespace would move the second argument onto +another field, silently. + +A compatibility record and nothing else -- a field's namespace is the module it +is declared in, and only ``collect_input_fields`` reads this. A name that is not +here sorts after every name that is, which is the only backward-compatible +position for a new field anyway. +""" + +# fmt: off +POSITIONAL_FIELD_ORDER = ( + "model_path", + "tokenizer_path", + "tokenizer_mode", + "tokenizer_backend", + "tokenizer_worker_num", + "detokenizer_worker_num", + "skip_tokenizer_init", + "load_format", + "model_loader_extra_config", + "trust_remote_code", + "context_length", + "is_embedding", + "enable_multimodal", + "revision", + "model_impl", + "model_config_parser", + "json_model_override_args", + "dtype", + "quantization", + "quantization_param_path", + "kv_cache_dtype", + "enable_fp32_lm_head", + "modelopt_quant", + "modelopt_checkpoint_restore_path", + "modelopt_checkpoint_save_path", + "modelopt_export_path", + "quantize_and_serve", + "rl_quant_profile", + "enable_tf32_matmul", + "mem_fraction_static", + "max_running_requests", + "max_queued_requests", + "max_total_tokens", + "chunked_prefill_size", + "prefill_decode_interval", + "enable_dynamic_chunking", + "max_prefill_tokens", + "prefill_max_requests", + "schedule_policy", + "enable_priority_scheduling", + "disable_priority_preemption", + "default_priority_value", + "abort_on_priority_when_disabled", + "schedule_low_priority_values_first", + "priority_scheduling_preemption_threshold", + "retraction_policy", + "schedule_conservativeness", + "page_size", + "c128_page_size", + "swa_full_tokens_ratio", + "disable_hybrid_swa_memory", + "radix_eviction_policy", + "radix_eviction_policy_config", + "prefill_only_disable_kv_cache", + "disable_radix_cache", + "enable_page_major_kv_layout", + "enable_unified_memory", + "disable_chunked_prefix_cache", + "disable_overlap_schedule", + "num_continuous_decode_steps", + "scheduler_recv_interval", + "enable_mixed_chunk", + "nccl_port", + "dist_timeout", + "dist_init_addr", + "gated_launch_port", + "nnodes", + "node_rank", + "tp_size", + "dcp_size", + "pp_size", + "pp_max_micro_batch_size", + "pp_async_batch_depth", + "dp_size", + "load_balance_method", + "attn_cp_size", + "moe_dp_size", + "dwdp_size", + "dcp_comm_backend", + "dcp_replicate_q_proj", + "enable_prefill_cp", + "cp_strategy", + "enable_dsa_cache_layer_split", + "enable_dsa_prefill_context_parallel", + "dsa_prefill_cp_mode", + "enable_prefill_context_parallel", + "prefill_cp_mode", + "enable_cp_decode_attn_tp", + "enable_dp_attention", + "enable_dp_attention_local_control_broadcast", + "enable_dp_lm_head", + "enable_tp_lm_head_all_to_all", + "enable_attn_tp_input_scattered", + "enable_shared_experts_attn_tp", + "enable_dense_mlp_attn_tp", + "enable_layernorm_sp", + "disable_attn_tp_gather", + "enable_p2p_check", + "device", + "base_gpu_id", + "gpu_id_step", + "random_seed", + "mlx_enable_sampling", + "watchdog_timeout", + "soft_watchdog_timeout", + "sleep_on_idle", + "use_ray", + "custom_sigquit_handler", + "numa_node", + "gc_threshold", + "host", + "port", + "fastapi_root_path", + "smg_grpc_mode", + "grpc_mode", + "grpc_port", + "grpc_worker_threads", + "sidecar", + "sidecar_args", + "skip_server_warmup", + "warmups", + "enable_http2", + "http2_max_concurrent_streams", + "http2_initial_connection_window_size", + "ssl_keyfile", + "ssl_certfile", + "ssl_ca_certs", + "ssl_keyfile_password", + "enable_ssl_refresh", + "api_key", + "admin_api_key", + "served_model_name", + "weight_version", + "chat_template", + "hf_chat_template_name", + "completion_template", + "file_storage_path", + "enable_cache_report", + "return_input_ids", + "return_output_ids", + "reasoning_parser", + "default_chat_template_kwargs", + "strip_thinking_cache", + "enable_strict_thinking", + "tool_call_parser", + "tool_server", + "sampling_defaults", + "asr_max_buffer_seconds", + "asr_max_concurrent_sessions", + "preferred_sampling_params", + "allow_auto_truncate", + "stream_interval", + "batch_notify_size", + "stream_response_default_include_usage", + "incremental_streaming_output", + "enable_streaming_session", + "enable_session_radix_cache", + "log_level", + "log_level_http", + "log_requests", + "log_requests_level", + "log_requests_format", + "log_requests_target", + "uvicorn_access_log_exclude_prefixes", + "crash_dump_folder", + "show_time_cost", + "enable_metrics", + "smg_http_sidecar_port", + "enable_mfu_metrics", + "enable_metrics_for_all_schedulers", + "load_snapshot_publish_interval", + "tokenizer_metrics_custom_labels_header", + "tokenizer_metrics_allowed_custom_labels", + "extra_metric_labels", + "bucket_time_to_first_token", + "bucket_inter_token_latency", + "bucket_e2e_request_latency", + "prompt_tokens_buckets", + "generation_tokens_buckets", + "gc_warning_threshold_secs", + "decode_log_interval", + "enable_request_time_stats_logging", + "kv_events_config", + "load_publish_endpoint", + "enable_forward_pass_metrics", + "forward_pass_metrics_worker_id", + "forward_pass_metrics_ipc_name", + "enable_trace", + "trace_modules", + "otlp_traces_endpoint", + "export_metrics_to_file", + "export_metrics_to_file_dir", + "stat_loggers", + "constrained_json_whitespace_pattern", + "constrained_json_disable_any_whitespace", + "attention_backend", + "decode_attention_backend", + "enable_lean_attention", + "prefill_attention_backend", + "sampling_backend", + "grammar_backend", + "radix_cache_backend", + "mm_attention_backend", + "fp8_gemm_runner_backend", + "fp4_gemm_runner_backend", + "bf16_gemm_backend", + "dsa_prefill_backend", + "dsv4_prefill_backend", + "dsa_decode_backend", + "dsa_paged_mqa_logits_backend", + "dsa_topk_backend", + "disable_flashinfer_autotune", + "flashinfer_autotune_skip_ops", + "mamba_backend", + "cuda_graph_config", + "cuda_graph_backend_decode", + "cuda_graph_backend_prefill", + "cuda_graph_max_bs_decode", + "cuda_graph_max_bs_prefill", + "cuda_graph_bs_decode", + "cuda_graph_bs_prefill", + "cuda_graph_tc_compiler", + "disable_prefill_cuda_graph", + "disable_decode_cuda_graph", + "disable_cuda_graph", + "disable_cuda_graph_padding", + "enable_profile_cuda_graph", + "enable_cudagraph_gc", + "debug_cuda_graph", + "enable_layerwise_nvtx_marker", + "enable_nccl_nvls", + "enable_symm_mem", + "triton_attention_reduce_in_fp32", + "triton_attention_num_kv_splits", + "triton_attention_split_tile_size", + "flashinfer_mla_disable_ragged", + "enable_fused_qk_norm_rope", + "enable_precise_embedding_interpolation", + "enable_fused_moe_sum_all_reduce", + "enable_deepseek_v4_fp4_indexer", + "disable_custom_all_reduce", + "enable_mscclpp", + "enable_torch_symm_mem", + "enable_scattered_sconv", + "pre_warm_nccl", + "enable_quant_communications", + "enable_flashinfer_allreduce_fusion", + "enforce_disable_flashinfer_allreduce_fusion", + "flashinfer_allreduce_fusion_backend", + "enable_aiter_allreduce_fusion", + "enable_torch_compile", + "enable_torch_compile_debug_mode", + "torch_compile_max_bs", + "speculative_algorithm", + "uno_lora_path", + "speculative_draft_model_path", + "speculative_draft_model_revision", + "speculative_draft_load_format", + "speculative_num_steps", + "speculative_eagle_topk", + "speculative_num_draft_tokens", + "speculative_dflash_block_size", + "speculative_dspark_block_size", + "speculative_dspark_sps_table_path", + "speculative_dspark_confidence_sts_path", + "speculative_dspark_align_verify_tokens_to_graph_tier", + "speculative_accept_threshold_single", + "speculative_accept_threshold_acc", + "speculative_use_rejection_sampling", + "speculative_token_map", + "speculative_attention_mode", + "speculative_draft_attention_backend", + "speculative_dsa_topk_backend", + "speculative_draft_kv_cache_dtype", + "speculative_draft_window_size", + "speculative_moe_runner_backend", + "speculative_moe_a2a_backend", + "speculative_draft_model_quantization", + "_speculative_draft_quantization_explicitly_set", + "speculative_skip_dp_mlp_sync", + "enable_multi_layer_eagle", + "speculative_adaptive", + "speculative_adaptive_config", + "decoupled_spec_bind_endpoint", + "decoupled_spec_connect_endpoints", + "decoupled_spec_rank", + "decoupled_spec_role", + "spec_trace_dir", + "speculative_ngram_min_bfs_breadth", + "speculative_ngram_max_bfs_breadth", + "speculative_ngram_match_type", + "speculative_ngram_max_trie_depth", + "speculative_ngram_capacity", + "speculative_ngram_external_corpus_path", + "speculative_ngram_external_sam_budget", + "speculative_ngram_external_corpus_max_tokens", + "ep_size", + "moe_a2a_backend", + "enable_w4a4_mxfp4_megamoe", + "deepep_v2_mode", + "moe_runner_backend", + "flashinfer_mxfp4_moe_precision", + "deepep_mode", + "fuseep_mode", + "deepep_dispatcher_output_dtype", + "ep_num_redundant_experts", + "ep_dispatch_algorithm", + "init_expert_location", + "enable_eplb", + "eplb_algorithm", + "eplb_rebalance_num_iterations", + "eplb_rebalance_layers_per_chunk", + "eplb_min_rebalancing_utilization_threshold", + "expert_distribution_recorder_mode", + "expert_distribution_recorder_buffer_size", + "expert_balancedness_report_mode", + "deepep_config", + "moe_dense_tp_size", + "elastic_ep_backend", + "enable_elastic_expert_backup", + "mooncake_ib_device", + "enable_waterfill", + "ep_join_mode", + "ep_join_rank_offset", + "elastic_ep_initial_size", + "max_ep_size", + "elastic_ep_scale_timeout", + "elastic_ep_rejoin", + "disable_flashinfer_cutlass_moe_fp4_allgather", + "disable_shared_experts_fusion", + "enforce_shared_experts_fusion", + "max_mamba_cache_size", + "mamba_ssm_dtype", + "mamba_max_states_per_path", + "enable_mamba_cache_stochastic_rounding", + "mamba_cache_philox_rounds", + "mamba_full_memory_ratio", + "mamba_radix_cache_strategy", + "uses_mamba_radix_cache", + "mamba_track_interval", + "enable_int8_mamba_checkpoint", + "int8_mamba_ckpt_size", + "linear_attn_backend", + "linear_attn_decode_backend", + "linear_attn_prefill_backend", + "linear_attn_verify_backend", + "enable_linear_replayssm", + "linear_replayssm_cache_len", + "enable_linear_replayssm_spec", + "enable_hierarchical_cache", + "hicache_host_memory_mode", + "hicache_ratio", + "hicache_size", + "hicache_write_policy", + "hicache_io_backend", + "hicache_mem_layout", + "hicache_storage_backend", + "hicache_storage_prefetch_policy", + "hicache_storage_backend_extra_config", + "hicache_storage_prefetch_retry_poll_interval", + "hicache_storage_prefetch_retry_max_attempts", + "enable_unified_cache_external_linker", + "unified_cache_external_linker_backend", + "enable_hisparse", + "hisparse_config", + "enable_broadcast_mm_inputs_process", + "enable_prefix_mm_cache", + "mm_enable_dp_encoder", + "mm_process_config", + "mm_processor_worker_num", + "mm_io_worker_num", + "allowed_media_domains", + "media_url_max_file_size_mb", + "mm_preprocess_cache_size_mb", + "trust_mm_content_hashes", + "limit_mm_data_per_request", + "enable_mm_global_cache", + "image_processor_backend", + "mm_global_cache_backend", + "disable_fast_image_processor", + "mm_feature_transport", + "keep_mm_feature_on_device", + "enable_lora", + "enable_lora_overlap_loading", + "max_lora_rank", + "lora_target_modules", + "lora_paths", + "max_loaded_loras", + "max_loras_per_batch", + "lora_eviction_policy", + "lora_backend", + "max_lora_chunk_size", + "experts_shared_outer_loras", + "lora_use_virtual_experts", + "lora_strict_loading", + "lora_drain_wait_threshold", + "enable_two_batch_overlap", + "enable_single_batch_overlap", + "tbo_token_distribution_threshold", + "cpu_offload_gb", + "offload_group_size", + "offload_num_in_group", + "offload_prefetch_step", + "offload_mode", + "enable_lmcache", + "lmcache_config_file", + "enable_flexkv", + "flexkv_config_file", + "kt_weight_path", + "kt_method", + "kt_cpuinfer", + "kt_threadpool_count", + "kt_num_gpu_experts", + "kt_max_deferred_experts_per_token", + "dllm_algorithm", + "dllm_algorithm_config", + "dllm_fdfo", + "disaggregation_mode", + "disaggregation_transfer_backend", + "disaggregation_bootstrap_port", + "disaggregation_ib_device", + "disaggregation_decode_enable_radix_cache", + "disaggregation_decode_enable_offload_kvcache", + "disaggregation_decode_retraction_backup", + "num_reserved_decode_tokens", + "disaggregation_decode_extra_slots", + "disaggregation_decode_polling_interval", + "optimistic_prefill_attempts", + "encoder_only", + "language_only", + "language_model_only", + "encoder_transfer_backend", + "encoder_urls", + "encoder_bootstrap_port", + "encoder_register_urls", + "enable_adaptive_dispatch_to_encoder", + "enable_pdmux", + "pdmux_config_path", + "sm_group_num", + "startup_weight_load_mode", + "custom_weight_loader", + "weight_loader_disable_mmap", + "weight_loader_prefetch_checkpoints", + "weight_loader_prefetch_num_threads", + "weight_loader_drop_cache_after_load", + "remote_instance_weight_loader_seed_instance_ip", + "remote_instance_weight_loader_seed_instance_service_port", + "remote_instance_weight_loader_send_weights_group_ports", + "remote_instance_weight_loader_backend", + "remote_instance_weight_loader_start_seed_via_transfer_engine", + "engine_info_bootstrap_port", + "modelexpress_config", + "download_dir", + "model_checksum", + "delete_ckpt_after_loading", + "decrypted_config_file", + "decrypted_draft_config_file", + "checkpoint_engine_wait_weights_before_ready", + "enable_prefill_delayer", + "prefill_delayer_max_delay_passes", + "prefill_delayer_token_usage_low_watermark", + "prefill_delayer_forward_passes_buckets", + "prefill_delayer_wait_seconds_buckets", + "prefill_delayer_queue_min_ratio", + "prefill_delayer_max_delay_ms", + "min_free_slots_delay", + "enable_deterministic_inference", + "rl_on_policy_target", + "kv_canary", + "kv_canary_real_data", + "kv_canary_sweep_interval", + "enable_dynamic_batch_tokenizer", + "dynamic_batch_tokenizer_batch_size", + "dynamic_batch_tokenizer_batch_timeout", + "enable_tokenizer_batch_encode", + "disable_tokenizer_batch_decode", + "debug_tensor_dump_output_folder", + "debug_tensor_dump_layers", + "debug_tensor_dump_input_file", + "enable_memory_saver", + "enable_weights_cpu_backup", + "enable_draft_weights_cpu_backup", + "enable_custom_logit_processor", + "enable_return_hidden_states", + "return_hidden_states_mode", + "enable_return_routed_experts", + "enable_return_indexer_topk", + "disable_outlines_disk_cache", + "enable_mis", + "weight_cache_mode", + "weight_cache_socket", + "weight_cache_timeout", + "forward_hooks", + "msprobe_dump_config", +) +# fmt: on diff --git a/python/sglang/srt/arg_groups/fields/__init__.py b/python/sglang/srt/arg_groups/fields/__init__.py new file mode 100644 index 000000000..48ce80a05 --- /dev/null +++ b/python/sglang/srt/arg_groups/fields/__init__.py @@ -0,0 +1,70 @@ +"""Config field declarations, one module per top-level namespace. + +Each class carries the ``_NS_PATH`` it stands for, so the module a field is +declared in *is* its namespace. ``ServerArgs`` is assembled from the classes +that declare **operator input**; the derived fields, which nobody can type, +are declared beside them but are not collected into the record. +""" + +from __future__ import annotations + +import dataclasses +from typing import Any, Dict, List, Tuple, get_type_hints + +from sglang.srt.arg_groups.field_order import POSITIONAL_FIELD_ORDER + + +def collect_input_fields( + sources: List[type], +) -> Tuple[Dict[str, Any], Dict[str, Any], Dict[str, str]]: + """The annotations and defaults of every field these classes declare. + + Each source's annotations are resolved **in its own module** and handed on + as type objects. Strings never travel: an annotation carried across as text + would be re-evaluated where it lands, and the composing module does not + import the names the declarations use -- that is the whole point of them + living where they do. + + Returns the resolved annotations, the defaults, and ``{field: namespace}`` + -- the last because the assembled record has no base classes to read a + ``_NS_PATH`` off, and the namespace is still the class that declared it. + + Which classes are passed here is the record's contract: the record holds + what the operator asked for, so a class of derived fields is simply not in + the list, and the rule is one readable call rather than an invariant spread + across a base-class list. + + The result is ordered by ``POSITIONAL_FIELD_ORDER``, not by namespace: a + dataclass turns field order into a positional constructor signature, and + grouping fields that used to be interleaved would silently move + ``ServerArgs(model_path, tokenizer_path)``'s second argument onto another + field. Anything the record declares that the frozen order does not name + goes after it, in declaration order -- the only backward-compatible place + for a new field. + """ + annotations: Dict[str, Any] = {} + defaults: Dict[str, Any] = {} + for source in sources: + hints = get_type_hints(source, include_extras=True) + for field in dataclasses.fields(source): + if field.name in annotations: + raise ValueError( + f"{field.name!r} is declared by both " + f"{annotations[field.name][0].__name__} and {source.__name__}; " + "a field belongs to exactly one namespace" + ) + annotations[field.name] = (source, hints[field.name]) + if field.default is not dataclasses.MISSING: + defaults[field.name] = field.default + elif field.default_factory is not dataclasses.MISSING: + defaults[field.name] = dataclasses.field( + default_factory=field.default_factory + ) + known = [n for n in POSITIONAL_FIELD_ORDER if n in annotations] + rest = [n for n in annotations if n not in set(POSITIONAL_FIELD_ORDER)] + ordered = known + rest + return ( + {name: annotations[name][1] for name in ordered}, + {name: defaults[name] for name in ordered if name in defaults}, + {name: annotations[name][0]._NS_PATH for name in ordered}, + ) diff --git a/python/sglang/srt/arg_groups/fields/device.py b/python/sglang/srt/arg_groups/fields/device.py new file mode 100644 index 000000000..48ed843da --- /dev/null +++ b/python/sglang/srt/arg_groups/fields/device.py @@ -0,0 +1,83 @@ +"""Config fields of the ``device`` namespace. + +One class per namespace. The class *is* the namespace: a field declared here +lands in the ``device`` bag, which is what ``get_device()`` returns, so a reader +spells it exactly as before. ``ServerArgs`` composes these classes, so the +record stays one flat object -- the split moves where declarations live, not +how config is shaped at runtime. +""" + +from __future__ import annotations + +import dataclasses +from typing import ( + Callable, + List, + Optional, +) + +from sglang.srt.arg_groups.arg_utils import A + + +@dataclasses.dataclass +class Device: + """Namespace ``device``.""" + + _NS_PATH = "device" + + # ------------------------------------------------------------------------- + # Device info and server timeout + # ------------------------------------------------------------------------- + device: A[ + Optional[str], + "The device to use ('cuda', 'xpu', 'hpu', 'npu', 'cpu', 'musa'). Defaults to auto-detection if not specified.", + ] = None + base_gpu_id: A[ + int, + "The base GPU ID to start allocating GPUs from. Useful when running multiple instances on the same machine.", + ] = 0 + gpu_id_step: A[ + int, + "The delta between consecutive GPU IDs that are used. For example, setting it to 2 will use GPU 0,2,4,...", + ] = 1 + random_seed: A[Optional[int], "The random seed."] = None + mlx_enable_sampling: A[ + bool, + ( + "MLX backend only: sample decode tokens (temperature / top-k / " + "top-p / min-p) instead of greedy argmax. Sampling runs inside " + "the lazy MLX graph, so it works with the overlap scheduler; " + "first tokens from prefill/extend are sampled too. Greedy " + "requests keep exact argmax behavior. Also enables on the MLX " + "path: grammar vocab masks and custom logit processors (these " + "break decode chaining per step; custom processors run on " + "pure-decode steps only), logit_bias, output logprobs (sampled " + "token / top-k / token_ids; prompt input logprobs are not " + "computed), NaN sanitization (SGLANG_SANITIZE_NAN_LOGITS), and " + "per-request sampling_seed under " + "--enable-deterministic-inference (deterministic within MLX " + "only). Penalties are not applied." + ), + ] = False + watchdog_timeout: A[ + float, + "Set watchdog timeout in seconds. If a forward batch takes longer than this, the server will crash to prevent hanging.", + ] = 300 + soft_watchdog_timeout: A[ + Optional[float], + "Set soft watchdog timeout in seconds. If a forward batch takes longer than this, the server will dump information for debugging.", + ] = None + sleep_on_idle: A[bool, "Reduce CPU usage when sglang is idle."] = False + use_ray: A[ + bool, + "Use Ray actors for scheduler process management.", + ] = False + custom_sigquit_handler: Optional[Callable] = None + numa_node: A[ + Optional[List[int]], + "Sets the numa node for the subprocesses. i-th element corresponds to i-th subprocess. If unset, will be automatically detected on NUMA systems.", + ] = None + gc_threshold: A[ + Optional[List[int]], + "Set the garbage collection thresholds (the collection frequency). Accepts 1 to 3 integers.", + ] = None diff --git a/python/sglang/srt/arg_groups/fields/disagg.py b/python/sglang/srt/arg_groups/fields/disagg.py new file mode 100644 index 000000000..a7251916e --- /dev/null +++ b/python/sglang/srt/arg_groups/fields/disagg.py @@ -0,0 +1,167 @@ +"""Config fields of the ``disagg`` namespace. + +One class per namespace. The class *is* the namespace: a field declared here +lands in the ``disagg`` bag, which is what ``get_disagg()`` returns, so a reader +spells it exactly as before. ``ServerArgs`` composes these classes, so the +record stays one flat object -- the split moves where declarations live, not +how config is shaped at runtime. +""" + +from __future__ import annotations + +import dataclasses +from typing import ( + List, + Literal, + Optional, +) + +from sglang.srt.arg_groups.arg_utils import ( + A, + Arg, +) +from sglang.srt.arg_groups.choices import DISAGG_TRANSFER_BACKEND_CHOICES +from sglang.srt.utils.common import json_list_type + + +@dataclasses.dataclass +class Disagg: + """Namespace ``disagg``.""" + + _NS_PATH = "disagg" + + # Decoupled speculative decoding: draft and verify run as + # separate engines, currently connected by a ZMQ IPC mesh. + decoupled_spec_bind_endpoint: A[ + Optional[str], + "ZMQ endpoint this engine binds for its inbound channel in decoupled " + "speculative decoding (verifier: result PULL; drafter: control PULL).", + ] = None + decoupled_spec_connect_endpoints: A[ + Optional[List[str]], + Arg( + help="Peer inbound (bind) endpoints to connect to, ordered by peer " + "rank, for decoupled speculative decoding.", + type_parser=json_list_type, + ), + ] = None + decoupled_spec_rank: A[ + Optional[int], + "This engine's rank within its own role space (verifier-rank or " + "drafter-rank) for decoupled speculative decoding.", + ] = None + decoupled_spec_role: A[ + Literal["null", "verifier", "drafter"], + "Role in decoupled speculative decoding: 'null' disables it, 'verifier' " + "runs the target/verify half, 'drafter' runs the draft half.", + ] = "null" + + # ------------------------------------------------------------------------- + # PD disaggregation + # ------------------------------------------------------------------------- + disaggregation_mode: A[ + Literal["null", "prefill", "decode"], + 'Only used for PD disaggregation. "prefill" for prefill-only server, and "decode" for decode-only server. If not specified, it is not PD disaggregated', + ] = "null" + disaggregation_transfer_backend: A[ + str, + Arg( + help="The backend for disaggregation transfer. Default is mooncake.", + choices=DISAGG_TRANSFER_BACKEND_CHOICES, + ), + ] = "mooncake" + disaggregation_bootstrap_port: A[ + int, "Bootstrap server port on the prefill server. Default is 8998." + ] = 8998 + disaggregation_ib_device: A[ + Optional[str], + 'The InfiniBand devices for disaggregation transfer. Supports a single device (e.g., --disaggregation-ib-device mlx5_0), a shared comma-separated list (e.g., --disaggregation-ib-device mlx5_0,mlx5_1), a per-GPU JSON mapping (e.g., --disaggregation-ib-device \'{"0": "mlx5_0,mlx5_1", "1": "mlx5_2"}\'), or a path to a JSON file containing that mapping. Default is None, which triggers automatic device detection when mooncake backend is enabled.', + ] = None + disaggregation_decode_enable_radix_cache: A[ + bool, + "Enable radix cache on decode server (PD mode). Caches KV prefixes to avoid redundant transfers. Incompatible with --enable-hisparse, speculative decoding, and --disaggregation-transfer-backend fake.", + ] = False + disaggregation_decode_enable_offload_kvcache: A[ + bool, "Enable async KV cache offloading on decode server (PD mode)." + ] = False + disaggregation_decode_retraction_backup: A[ + Optional[str], + Arg( + help=( + "Storage backend for KV preserved across PD decode retraction. " + "'cpu_tensor' uses per-request CPU tensors. 'host_pool' uses " + "a reserved HiCache pool and does not fall back on exhaustion. " + "If omitted, the backend is inferred from the decode KV pool." + ), + choices=["cpu_tensor", "host_pool"], + ), + ] = None + num_reserved_decode_tokens: A[ + int, + "Number of decode tokens that will have memory reserved when adding new request to the running batch.", + ] = 512 + disaggregation_decode_extra_slots: A[ + Optional[int], + "Number of extra decode req_to_token slots pre-allocated for in-transfer requests (PD mode). If unset, defaults to 0 (or 2x the per-worker running batch for small batches).", + ] = None + disaggregation_decode_polling_interval: A[ + int, + "The interval to poll requests in decode server. Can be set to >1 to reduce the overhead of this.", + ] = 1 + optimistic_prefill_attempts: A[ + int, "Number of optimistic prefill forward passes that skip the bootstrap wait." + ] = 0 + + # ------------------------------------------------------------------------- + # Encode prefill disaggregation + # ------------------------------------------------------------------------- + encoder_only: A[ + bool, + "For MLLM with an encoder, launch an encoder-only server", + ] = False + language_only: A[ + bool, + "For VLM, load weights for the language model only.", + ] = False + language_model_only: A[ + bool, + "Skip the multimodal encoder entirely: its weights are never loaded and the " + "tower is never built, freeing that GPU memory for KV cache. Multimodal " + "requests are rejected. Unlike --language-only this is a standalone mode, " + "not part of encoder/decoder disaggregation.", + ] = False + encoder_transfer_backend: A[ + str, + Arg( + help="The backend for encoder disaggregation transfer. Auto selects a model- and TP-aware backend.", + choices=["auto", "zmq_to_scheduler", "zmq_to_tokenizer", "mooncake"], + ), + ] = "auto" + encoder_urls: A[List[str], "List of encoder server urls."] = dataclasses.field( + default_factory=list + ) + encoder_bootstrap_port: A[ + int, + "Port for the EncoderBootstrapServer that runs in the language-only tokenizer manager process. Encoders register here, and language-only receivers fetch the current URL list from here.", + ] = 8997 + encoder_register_urls: A[ + List[str], + "One or more EncoderBootstrapServer URLs to register this encoder with on startup, for dynamic encoder discovery. Example: --encoder-register-urls http://prefill0:8997 http://prefill1:8997. Used with --encoder-only servers.", + ] = dataclasses.field(default_factory=list) + enable_adaptive_dispatch_to_encoder: A[ + bool, + "When enabled, adaptively dispatch: multi-image requests go to encoder in language_only epd mode, single-image requests are processed locally.", + ] = False + + # ------------------------------------------------------------------------- + # PD-Multiplexing + # ------------------------------------------------------------------------- + enable_pdmux: A[ + bool, + "Enable PD-Multiplexing, PD running on greenctx stream.", + ] = False + pdmux_config_path: A[ + Optional[str], + "The path of the PD-Multiplexing config file.", + ] = None + sm_group_num: A[int, "Number of sm partition groups."] = 8 diff --git a/python/sglang/srt/arg_groups/fields/exec_.py b/python/sglang/srt/arg_groups/fields/exec_.py new file mode 100644 index 000000000..ab70e7d12 --- /dev/null +++ b/python/sglang/srt/arg_groups/fields/exec_.py @@ -0,0 +1,863 @@ +"""Config fields of the ``exec`` namespace. + +One class per namespace. The class *is* the namespace: a field declared here +lands in the ``exec`` bag, which is what ``get_exec()`` returns, so a reader +spells it exactly as before. ``ServerArgs`` composes these classes, so the +record stays one flat object -- the split moves where declarations live, not +how config is shaped at runtime. +""" + +from __future__ import annotations + +import argparse +import dataclasses +from typing import ( + List, + Literal, + Optional, +) + +from sglang.srt.arg_groups.arg_utils import ( + A, + Arg, +) +from sglang.srt.arg_groups.choices import ( + ATTENTION_BACKEND_CHOICES, + FP4_GEMM_RUNNER_BACKEND_CHOICES, + FP8_GEMM_RUNNER_BACKEND_CHOICES, + GRAMMAR_BACKEND_CHOICES, + LINEAR_ATTN_KERNEL_BACKEND_CHOICES, + MOE_RUNNER_BACKEND_CHOICES, + RL_ON_POLICY_TARGET_CHOICES, +) +from sglang.srt.model_executor.cuda_graph_config import ( + Backend, + CudaGraphConfig, + parse_cuda_graph_config_arg, +) + + +@dataclasses.dataclass +class ExecFeatures: + """Namespace ``exec.features``.""" + + _NS_PATH = "exec.features" + enable_fp32_lm_head: A[ + bool, + "If set, the LM head outputs (logits) are in FP32.", + ] = False + enable_tf32_matmul: A[ + bool, + Arg( + help="Enable float32 matmuls to use TensorFloat32 precision for better performance (via torch.set_float32_matmul_precision). CUDA only.", + resolvable=True, + ), + ] = False + + # ------------------------------------------------------------------------- + # Misc runtime features + # ------------------------------------------------------------------------- + enable_memory_saver: A[ + bool, + "Allow saving memory using release_memory_occupation and resume_memory_occupation", + ] = False + enable_weights_cpu_backup: A[ + bool, + "Save model weights (both main model and draft model, if any) to CPU memory during release_weights_occupation and resume_weights_occupation", + ] = False + enable_draft_weights_cpu_backup: A[ + bool, + "Save draft model weights to CPU memory during release_weights_occupation and resume_weights_occupation", + ] = False + enable_custom_logit_processor: A[ + bool, + "Enable users to pass custom logit processors to the server (disabled by default for security)", + ] = False + enable_return_hidden_states: A[ + bool, + "Enable returning full hidden states with responses. Equivalent to " + "`--return-hidden-states-mode full`.", + ] = False + return_hidden_states_mode: A[ + Optional[str], + Arg( + help="Set the maximum hidden-state return mode supported by the " + "server. `last` allows requests with return_hidden_states=False or " + "`last`; `full` also allows return_hidden_states=True.", + choices=["last", "full"], + ), + ] = None + enable_return_routed_experts: A[ + bool, "Enable returning routed experts of each layer with responses." + ] = False + enable_return_indexer_topk: A[ + bool, + "Enable returning indexer topk indices of layers with indexer with responses.", + ] = False + disable_outlines_disk_cache: A[ + bool, + "Disable disk cache of outlines to avoid possible crashes related to file system or high concurrency.", + ] = False + enable_mis: A[ + bool, + "Enable Multi-Item Scoring optimization. Combines query and multiple items into a single sequence for efficient batch processing. Requires --attention-backend flashinfer; auto-disables CUDA graph, radix cache, and chunked prefill.", + ] = False + + +@dataclasses.dataclass +class ExecKernel: + """Namespace ``exec.kernel``.""" + + _NS_PATH = "exec.kernel" + + # ------------------------------------------------------------------------- + # Kernel backend + # ------------------------------------------------------------------------- + attention_backend: A[ + Optional[str], + Arg( + help="Choose the kernels for attention layers.", + choices=ATTENTION_BACKEND_CHOICES, + resolvable=True, + ), + ] = None + decode_attention_backend: A[ + Optional[str], + Arg( + help="Choose the kernels for decode attention layers (have priority over --attention-backend).", + choices=ATTENTION_BACKEND_CHOICES, + resolvable=True, + ), + ] = None + enable_lean_attention: A[ + Optional[bool], + "Enable Lean (Work-Centric) Attention decode kernel for long-context serving. When None (default), uses auto-gate that activates Lean for long contexts and falls back to standard kernel for short contexts. Set to True to force enable, False to force disable.", + ] = None + prefill_attention_backend: A[ + Optional[str], + Arg( + help="Choose the kernels for prefill attention layers (have priority over --attention-backend).", + choices=ATTENTION_BACKEND_CHOICES, + resolvable=True, + ), + ] = None + sampling_backend: A[ + Optional[str], + Arg( + help="Choose the kernels for sampling layers.", + no_cli=True, + resolvable=True, + ), + ] = None + grammar_backend: A[ + Optional[str], + Arg( + help="Choose the backend for grammar-guided decoding.", + choices=GRAMMAR_BACKEND_CHOICES, + ), + ] = None + fp8_gemm_runner_backend: A[ + str, + Arg( + help="Choose the runner backend for Blockwise FP8 GEMM operations. Options: 'auto' (default, auto-selects based on hardware; MXFP8 dense picks flashinfer_cutedsl on SM100/SM103 and FlashInfer CUTLASS on other supported Blackwell GPUs), 'deep_gemm' (JIT-compiled; enabled by default on NVIDIA Hopper (SM90) and Blackwell (SM100) when DeepGEMM is installed), 'flashinfer_trtllm' (optimal for Blackwell and low-latency), 'flashinfer_cutlass' (FlashInfer CUTLASS groupwise FP8 GEMM), 'flashinfer_cutedsl' (FlashInfer CuTe DSL MXFP8 GEMM on SM100/SM103), 'flashinfer_deepgemm' (Hopper SM90 only; uses swapAB optimization for small M dimensions in decoding), 'cutlass' (optimal for SM120 GPUs), 'triton' (fallback, widely compatible), 'aiter' (ROCm only). ", + cli_name="--fp8-gemm-backend", + choices=FP8_GEMM_RUNNER_BACKEND_CHOICES, + resolvable=True, + ), + ] = "auto" + fp4_gemm_runner_backend: A[ + str, + Arg( + help="Choose the runner backend for NVFP4 GEMM operations. Options: 'auto' (default; selects flashinfer_cutedsl on SM100, marlin on SM80-SM90, flashinfer_cutlass otherwise (including SM120)), 'flashinfer_cutlass' (FlashInfer CUTLASS backend), 'flashinfer_cudnn' (FlashInfer cuDNN backend, optimal on CUDA 13+ with cuDNN 9.15+), 'flashinfer_cutedsl' (FlashInfer CuTe DSL backend), 'flashinfer_trtllm' (FlashInfer TensorRT-LLM backend, requires different weight preparation with shuffling), 'marlin' (weight-only W4A16 fallback for SM80+). ", + cli_name="--fp4-gemm-backend", + choices=FP4_GEMM_RUNNER_BACKEND_CHOICES, + resolvable=True, + ), + ] = "auto" + bf16_gemm_backend: A[ + str, + Arg( + help="Choose the backend for unquantized BF16 GEMM operations. Options: 'auto' (default; selects 'cutedsl' on SM10x GPUs, except deterministic inference selects 'torch'; otherwise uses cuBLAS via torch.nn.functional.linear), 'cutedsl' (SGLang JIT CuTe DSL TGV BF16 GEMM on SM10x; dispatches between the allowlisted low-M Split-K kernel, the CuTe DSL kernel, and cuBLAS; set SGLANG_ENABLE_BF16_SPLITK_GEMM=0 to disable Split-K), 'flashinfer_pr4266' (legacy compatibility alias for the optimized CuTe DSL path), 'gemv', 'torch' (always uses cuBLAS via torch.nn.functional.linear).", + cli_name="--bf16-gemm-backend", + choices=["auto", "cutedsl", "flashinfer_pr4266", "gemv", "torch"], + ), + ] = "auto" + dsa_prefill_backend: A[ + Optional[str], + Arg( + help="DSA (DeepSeek Sparse Attention) prefill backend. If not specified, auto-detects based on hardware and kv_cache_dtype.", + choices=[ + "flashmla_sparse", + "flashmla_sparse_q8", + "flashmla_kv", + "flashmla_auto", + "flashinfer_sparse_mla", + "fa3", + "tilelang", + "aiter", + "trtllm", + ], + resolvable=True, + ), + ] = None + dsv4_prefill_backend: A[ + str, + Arg( + help=( + "DeepSeek-V4 sparse prefill backend. 'auto' and " + "'flashmla_sparse' use the existing BF16 sparse prefill path; " + "'flashmla_sparse_q8' enables the Q8KV8 sparse prefill path." + ), + choices=["auto", "flashmla_sparse", "flashmla_sparse_q8"], + ), + ] = "auto" + dsa_decode_backend: A[ + Optional[str], + Arg( + help="DSA (DeepSeek Sparse Attention) decode backend. If not specified, auto-detects based on hardware and kv_cache_dtype.", + choices=[ + "flashmla_sparse", + "flashmla_sparse_q8", + "flashmla_kv", + "flashmla_auto", + "flashinfer_sparse_mla", + "fa3", + "tilelang", + "aiter", + "trtllm", + ], + resolvable=True, + ), + ] = None + dsa_paged_mqa_logits_backend: A[ + str, + Arg( + help="DSA indexer paged MQA logits kernel backend. Options: 'auto' (default; DeepGEMM on CUDA, aiter on ROCm), 'deepgemm', 'cutedsl' (CuTe DSL kernel, SM 100 (Blackwell) only; wins at low batch size and long context), 'aiter' (ROCm only).", + choices=["auto", "deepgemm", "cutedsl", "aiter"], + ), + ] = "auto" + dsa_topk_backend: A[ + str, + Arg( + help="DSA indexer top-k backend for the target model. Options: 'sgl-kernel', 'torch', 'flashinfer'. The 'torch' backend currently requires SGLANG_DSA_FUSE_TOPK=false.", + choices=["sgl-kernel", "torch", "flashinfer"], + resolvable=True, + ), + ] = "sgl-kernel" + disable_flashinfer_autotune: A[ + bool, + "Disable FlashInfer autotuning.", + ] = False + flashinfer_autotune_skip_ops: A[ + Optional[List[str]], + Arg( + help=( + "FlashInfer custom-op identifiers to skip during autotuning. " + "Skipped ops use FlashInfer's heuristic fallback. SGLang " + "temporarily skips mxfp8_gemm by default due to an IMA." + ), + nargs="+", + ), + ] = None + triton_attention_reduce_in_fp32: A[ + bool, + "Cast the intermediate attention results to fp32 to avoid possible crashes related to fp16." + "This only affects Triton attention kernels.", + ] = False + triton_attention_num_kv_splits: A[ + int, + "The number of KV splits in flash decoding Triton kernel. Larger value is better in longer context scenarios. The default value is 8.", + ] = 8 + triton_attention_split_tile_size: A[ + Optional[int], + "The size of split KV tile in flash decoding Triton kernel. Used for deterministic inference.", + ] = None + flashinfer_mla_disable_ragged: A[ + bool, "Not using ragged prefill wrapper when running flashinfer mla" + ] = False + enable_fused_qk_norm_rope: A[ + bool, "Enable fused qk normalization and rope rotary embedding." + ] = False + enable_precise_embedding_interpolation: A[ + bool, + "Enable corner alignment for resize of embeddings grid to ensure more accurate(but slower) evaluation of interpolated embedding values.", + ] = False + enable_deepseek_v4_fp4_indexer: A[ + bool, + "Enable the experimental FP4 C4 indexer path for DeepSeek V4. Default keeps the existing indexer implementation.", + ] = False + + +@dataclasses.dataclass +class ExecMamba: + """Namespace ``exec.mamba``.""" + + _NS_PATH = "exec.mamba" + mamba_backend: A[ + str, + Arg( + help="Choose the kernel backend for Mamba SSM operations. Default is 'triton'. Options: 'triton' (default), 'flashinfer' (requires FlashInfer with Mamba support).", + choices=["triton", "flashinfer"], + ), + ] = "triton" + mamba_ssm_dtype: A[ + Optional[str], + Arg( + help="The data type of the SSM states in mamba cache. If not set, will be read from model config (mamba_ssm_dtype).", + choices=["float32", "bfloat16", "float16"], + ), + ] = None + mamba_max_states_per_path: A[ + int, + "Maximum number of cached Mamba states retained per root-to-tail path " + "(-1 means unlimited). When enabled, after each insert the shallowest eligible " + "interior states beyond the cap are removed while their full KV remains. " + "Tail, fork, and locked nodes are preserved. Must be -1 or a positive integer.", + ] = -1 + enable_mamba_cache_stochastic_rounding: A[ + bool, + "Enable stochastic rounding when writing FP16 Mamba SSM cache states. Requires --mamba-ssm-dtype float16 and CUDA. With --mamba-backend triton, requires SM100.", + ] = False + mamba_cache_philox_rounds: A[ + int, + "Number of Philox rounds to use for stochastic rounding of FP16 Mamba SSM cache writes. Triton uses the Triton default when set to 0; FlashInfer uses 10 rounds when set to 0.", + ] = 0 + mamba_radix_cache_strategy: A[ + str, + Arg( + help="The strategy to use for mamba radix cache.", + choices=["auto", "no_buffer", "extra_buffer", "extra_buffer_lazy"], + resolvable=True, + ), + ] = "auto" + uses_mamba_radix_cache: A[ + bool, + Arg( + help="(Derived) whether the model routes through the hybrid-mamba " + "radix cache handling; resolved from the model architecture, no " + "CLI surface.", + no_cli=True, + resolvable=True, + ), + ] = False + mamba_track_interval: A[ + int, + "The interval to track the mamba state during decode.", + ] = 256 + enable_int8_mamba_checkpoint: A[ + bool, + "Store radix-cached linear-attn (mamba) states in int8 (separate checkpoint pool) for ~2x cached-prefix capacity at fixed memory.", + ] = False + int8_mamba_ckpt_size: A[ + Optional[int], + "Number of int8 mamba checkpoint slots (default: 2x the active mamba pool size).", + ] = None + linear_attn_backend: A[ + str, + Arg( + help="The default kernel backend for linear attention (GDN/KDA). Can be overridden per-mode by --linear-attn-decode-backend and --linear-attn-prefill-backend. The Helion backend is KDA-only.", + choices=LINEAR_ATTN_KERNEL_BACKEND_CHOICES, + ), + ] = "triton" + linear_attn_decode_backend: A[ + Optional[str], + Arg( + help="Override the kernel backend for linear attention decode. If not set, uses --linear-attn-backend.", + choices=LINEAR_ATTN_KERNEL_BACKEND_CHOICES, + ), + ] = None + linear_attn_prefill_backend: A[ + Optional[str], + Arg( + help="Override the kernel backend for linear attention prefill/extend. If not set, uses --linear-attn-backend; compatible SM100 GDN models may automatically select FlashInfer.", + choices=LINEAR_ATTN_KERNEL_BACKEND_CHOICES, + ), + ] = None + linear_attn_verify_backend: A[ + Optional[str], + Arg( + help="Override the kernel backend for linear attention speculative target-verify. If not set, follows the decode backend (flashinfer decode -> flashinfer verify, otherwise triton). KDA supports triton, nv_cutedsl, and flashinfer verify backends.", + choices=LINEAR_ATTN_KERNEL_BACKEND_CHOICES + ["nv_cutedsl"], + ), + ] = None + # ReplaySSM buffered output-only linear-attn decode (GDN + KDA): per-slot + # ring + periodic flush to cut per-step HBM state traffic. + enable_linear_replayssm: A[ + bool, + "Enable the ReplaySSM buffered output-only linear-attn decode kernel. " + "Primarily a GDN (scalar-gate) decode-bandwidth optimization (~1.2-1.5x " + "at batch >= 64). KDA uses its selected Triton or Helion implementation, " + "but its per-K gate ring is larger and ReplaySSM is typically slower " + "than packed KDA decode; benchmark before enabling it. Requires the " + "Triton linear-attn decode backend, or Helion for KDA, and " + "--mamba-radix-cache-strategy no_buffer (the default).", + ] = False + linear_replayssm_cache_len: A[ + int, + "Ring-buffer length L for ReplaySSM linear-attn decode. The full recurrent state is flushed to HBM every L decode steps.", + ] = 16 + # ReplaySSM spec-verify (Part B of RFC #28511): linear-attn target-verify via + # fold-every-commit instead of per-draft full-state snapshots -- the verify + # stores each draft step's raw inputs into a per-slot window and the commit + # replays the accepted prefix into the fp32 checkpoint. GDN sizes the window + # to the draft maximum; KDA folds a (raw v, pre-norm k, gate, beta) ring of + # length --linear-replayssm-cache-len. Linear-chain (topk <= 1) only. + enable_linear_replayssm_spec: A[ + bool, + "Enable the ReplaySSM spec-verify: fold-every-commit -- a per-slot raw-input window replaces the recurrent verify's per-draft full-state snapshots. GDN or KDA hybrid linear-attn models, linear-chain (--speculative-eagle-topk in {None, 1}) only.", + ] = False + + +@dataclasses.dataclass +class ExecGraph: + """Namespace ``exec.graph``.""" + + _NS_PATH = "exec.graph" + + # ------------------------------------------------------------------------- + # Cuda graphs + # ------------------------------------------------------------------------- + cuda_graph_config: A[ + Optional[CudaGraphConfig], + Arg( + help='Per-phase CUDA graph settings as JSON, e.g. \'{"decode":{"backend":"full","max_bs":256},"prefill":{"backend":"tc_piecewise","tc_compiler":"eager"}}\'. Allowed backends per phase: full, breakable, tc_piecewise, disabled (full is decode-only). JSON wins over the per-phase --cuda-graph-* convenience flags and over legacy flags.', + type_parser=parse_cuda_graph_config_arg, + ), + ] = None + cuda_graph_backend_decode: A[ + Optional[Literal["full", "breakable", "tc_piecewise", "disabled"]], + Arg( + help="Backend for the decode phase. Folds into cuda_graph_config[decode].backend.", + choices=Backend.ALL, + ), + ] = None + cuda_graph_backend_prefill: A[ + Optional[Literal["full", "breakable", "tc_piecewise", "disabled"]], + Arg( + help="Backend for the prefill phase. Folds into cuda_graph_config[prefill].backend.", + choices=Backend.ALL, + ), + ] = None + cuda_graph_max_bs_decode: A[ + Optional[int], "Maximum batch size captured for the decode cuda graph." + ] = None + cuda_graph_max_bs_prefill: A[ + Optional[int], "Maximum batch size captured for the prefill cuda graph." + ] = None + cuda_graph_bs_decode: A[ + Optional[List[int]], + "Explicit list of batch sizes to capture for the decode cuda graph.", + ] = None + cuda_graph_bs_prefill: A[ + Optional[List[int]], + "Explicit list of batch sizes to capture for the prefill cuda graph.", + ] = None + cuda_graph_tc_compiler: A[ + Optional[Literal["eager", "inductor"]], + "Compiler used by the tc_piecewise backend (currently only the prefill phase consumes it).", + ] = None + disable_prefill_cuda_graph: A[ + bool, + "Disable the prefill-phase CUDA graph. Convenience for --cuda-graph-backend-prefill=disabled.", + ] = False + disable_decode_cuda_graph: A[ + bool, + "Disable the decode-phase CUDA graph. Convenience for --cuda-graph-backend-decode=disabled.", + ] = False + disable_cuda_graph: A[bool, Arg(no_cli=True)] = False + disable_cuda_graph_padding: A[ + bool, + "Disable cuda graph when padding is needed. Still uses cuda graph when padding is not needed.", + ] = False + enable_profile_cuda_graph: A[ + bool, + "Enable profiling of cuda graph capture.", + ] = False + enable_cudagraph_gc: A[ + bool, + "Enable garbage collection during CUDA graph capture. If disabled (default), GC is frozen during capture to speed up the process.", + ] = False + debug_cuda_graph: A[ + bool, + "Enable debug/eager mode for CUDA graph using breakable CUDA graph. When enabled, graph breaks are inserted so every operation runs eagerly while still going through the CUDA graph capture / replay path. Useful for debugging CUDA graph capture / replay issues.", + ] = False + + # ------------------------------------------------------------------------- + # Torch compile + # ------------------------------------------------------------------------- + enable_torch_compile: A[ + bool, "Optimize the model with torch.compile. Experimental feature." + ] = False + enable_torch_compile_debug_mode: A[ + bool, + "Enable debug mode for torch compile", + ] = False + torch_compile_max_bs: A[ + int, + "Set the maximum batch size when using torch compile.", + ] = 32 + + +@dataclasses.dataclass +class ExecComm: + """Namespace ``exec.comm``.""" + + _NS_PATH = "exec.comm" + + # ------------------------------------------------------------------------- + # Communication and kernels + # ------------------------------------------------------------------------- + enable_layerwise_nvtx_marker: A[ + bool, "Enable layerwise NVTX profiling annotations for the model." + ] = False + enable_nccl_nvls: A[ + bool, "Enable NCCL NVLS for prefill heavy requests when available." + ] = False + enable_symm_mem: A[ + bool, + Arg( + help="Enable NCCL symmetric memory for fast collectives.", + resolvable=True, + ), + ] = False + disable_custom_all_reduce: A[ + bool, + Arg( + help="Disable the custom all-reduce kernel and fall back to NCCL.", + resolvable=True, + ), + ] = False + enable_mscclpp: A[ + bool, + "Enable using mscclpp for small messages for all-reduce kernel and fall back to NCCL.", + ] = False + enable_torch_symm_mem: A[ + bool, + "Enable using torch symm mem for all-reduce kernel and fall back to NCCL. Only supports CUDA device SM90 and above. SM90 supports world size 4, 6, 8. SM100 supports world size 6, 8.", + ] = False + enable_scattered_sconv: A[ + bool, + "Inkling: replace the attention/MLP output all-reduce with a hidden-dimension reduce-scatter, run the channelwise output short convolution on the [T, H/P] shard, then all-gather before the residual add. This shards the convolution cache across tensor-parallel ranks without changing communication volume.", + ] = False + pre_warm_nccl: A[ + bool, + "Pre-warm NCCL/RCCL communicators during startup to reduce P99 TTFT cold-start latency. Default: enabled for AMD/HIP (RCCL), disabled for NVIDIA/CUDA (NCCL).", + ] = False + enable_quant_communications: A[ + Optional[bool], + "Enable INT8 quantization of TP communications (limited support).", + ] = False + enable_flashinfer_allreduce_fusion: A[bool, Arg(no_cli=True)] = False + enforce_disable_flashinfer_allreduce_fusion: A[ + bool, + "Enforce disable FlashInfer allreduce fusion.", + ] = False + flashinfer_allreduce_fusion_backend: A[ + Optional[Literal["auto", "trtllm", "mnnvl"]], + Arg( + help=( + "Enable FlashInfer allreduce fusion and choose backend. " + "Requires SM90 or SM10X NVIDIA GPUs. " + "Defaults to auto. " + "'auto': choose mnnvl on Blackwell (SM100/SM103) systems " + "(single- and multi-node) and trtllm on SM90 single-node systems. " + "'trtllm': available on single-node systems only. " + "'mnnvl': available on SM90 single-node systems and SM100/SM103 " + "single-node or multi-node systems via MNNVL fabric. " + "Fuses allreduce with Residual + RMSNorm for supported MoE models." + ), + resolvable=True, + ), + ] = None + enable_aiter_allreduce_fusion: A[ + bool, Arg(help="Enable Aiter AllReduce Fusion.", resolvable=True) + ] = False + + +@dataclasses.dataclass +class ExecMoe: + """Namespace ``exec.moe``.""" + + _NS_PATH = "exec.moe" + enable_fused_moe_sum_all_reduce: A[ + bool, + "Enable fused moe triton and sum all reduce.", + ] = False + moe_a2a_backend: A[ + Literal[ + "none", + "deepep", + "mooncake", + "nixl", + "mori", + "ascend_fuseep", + "flashinfer", + "megamoe", + "deepep_v2", + "ascend_tp", + "pplx", + ], + Arg( + help="Choose the backend for MoE A2A.", + choices=[ + "none", + "deepep", + "mooncake", + "nixl", + "mori", + "ascend_fuseep", + "flashinfer", + "megamoe", + "deepep_v2", + "pplx", + "ascend_tp", + ], + resolvable=True, + ), + ] = "none" + enable_w4a4_mxfp4_megamoe: A[ + bool, + "Enable the W4A4 MXFP4 MegaMoE path with DeepGEMM's " + "mxf4xmxf4 MMA type. Use with " + "--moe-a2a-backend megamoe.", + ] = False + deepep_v2_mode: A[ + Literal["direct", "hybrid"], + "DeepEP v2 ElasticBuffer communication topology, fixed at server init: " + "`direct` (single-node NVLink) or `hybrid` (multi-node scale-out). " + "Layout/grouped-GEMM and the decode CUDA graph are chosen per batch by " + "inference phase, independent of this knob; not equivalent to DeepEP v1 " + "normal/low_latency.", + ] = "direct" + moe_runner_backend: A[ + str, + Arg( + help="Choose the runner backend for MoE.", + choices=MOE_RUNNER_BACKEND_CHOICES, + resolvable=True, + ), + ] = "auto" + flashinfer_mxfp4_moe_precision: A[ + Literal["default", "bf16", "fp8"], + "Choose the computation precision of flashinfer mxfp4 moe. " + "On SM90, `fp8` selects the Humming-style MXFP4-weight x FP8-activation " + "path introduced by FlashInfer #3738 and requires FlashInfer >= 0.6.18.", + ] = "default" + deepep_mode: A[ + Literal["auto", "normal", "low_latency"], + "Select the mode when enable DeepEP or MoriEP MoE, could be `normal`, `low_latency` or `auto`. Default is `auto`, which means `low_latency` for decode batch and `normal` for prefill batch.", + ] = "auto" + fuseep_mode: A[ + Literal[1, 2], + "Select the mode when enable Ascend FuseEP MoE, 1 -> dispatch_gmm_combine_decode is executed；2 -> dispatch_ffn_combine is executed (support hybrid deployment when 2).", + ] = 2 + deepep_dispatcher_output_dtype: A[ + Literal["auto", "bf16", "fp8", "int8", "nvfp4"], + "Select DeepEP dispatcher output dtype", + ] = "auto" + ep_num_redundant_experts: A[ + int, "Allocate this number of redundant experts in expert parallel." + ] = 0 + ep_dispatch_algorithm: A[ + Optional[Literal["static", "dynamic", "fake", "lp"]], + "The algorithm to choose ranks for redundant experts in expert parallel.", + ] = None + init_expert_location: A[str, "Initial location of EP experts."] = "trivial" + enable_eplb: A[bool, "Enable EPLB algorithm"] = False + eplb_algorithm: A[str, "Chosen EPLB algorithm"] = "auto" + eplb_rebalance_num_iterations: A[ + int, "Number of iterations to automatically trigger a EPLB re-balance." + ] = 1000 + eplb_rebalance_layers_per_chunk: A[ + Optional[int], + "Number of layers to rebalance per forward pass.", + ] = None + eplb_min_rebalancing_utilization_threshold: A[ + float, + "Minimum threshold for GPU average utilization to trigger EPLB rebalancing. Must be in the range [0.0, 1.0].", + ] = 1.0 + expert_distribution_recorder_mode: A[ + Optional[Literal["stat", "stat_approx", "per_pass", "per_token"]], + "Mode of expert distribution recorder.", + ] = None + expert_distribution_recorder_buffer_size: A[ + Optional[int], + "Circular buffer size of expert distribution recorder. Set to -1 to denote infinite buffer.", + ] = None + expert_balancedness_report_mode: A[ + Literal["off", "server_log", "prometheus", "both"], + "Where to report expert balancedness. Options: off, server_log, prometheus, both.", + ] = "off" + deepep_config: A[ + Optional[str], + "Tuned DeepEP config suitable for your own cluster. It can be either a string with JSON content or a file path.", + ] = None + elastic_ep_backend: A[ + Literal[None, "mooncake", "nixl"], + Arg( + help="Specify the collective communication backend for elastic EP. Supports 'mooncake' and 'nixl'.", + choices=["none", "mooncake", "nixl"], + ), + ] = None + enable_elastic_expert_backup: A[ + bool, + "Enable elastic expert backup feature.", + ] = False + mooncake_ib_device: A[ + Optional[str], + "The InfiniBand devices for Mooncake Backend transfer, accepts multiple comma-separated devices (e.g., --mooncake-ib-device mlx5_0,mlx5_1). Default is None, which triggers automatic device detection when Mooncake Backend is enabled.", + ] = None + enable_waterfill: A[ + bool, + "Enable Waterfill: dispatch the fused shared expert as an extra routed expert slot to the least-loaded EP rank. Supports DeepEP and MegaMOE MoE A2A backends, implicitly enables shared-expert fusion, and supports --deepep-mode auto, normal, or low_latency when used with DeepEP. Use auto or low_latency for production DeepEP decode so CUDA graph remains enabled. Supported on DeepSeek-V3/R1 with EP >= 2.", + ] = False + ep_join_mode: A[ + Optional[Literal["scale", "recover"]], + Arg( + help="Join mode for elastic EP. 'recover' rejoins an existing slot after a fault. 'scale' joins as a new rank beyond the original group size and requires --node-rank 1.", + cli_name="--elastic-ep-join-mode", + choices=["scale", "recover"], + ), + ] = None + elastic_ep_scale_timeout: A[ + float, "Timeout in seconds for a pending elastic EP scale operation." + ] = 600 + elastic_ep_rejoin: A[ + bool, + "[Deprecated] Alias for --elastic-ep-join-mode recover.", + ] = False + disable_flashinfer_cutlass_moe_fp4_allgather: A[ + bool, "Disables quantize before all-gather for flashinfer cutlass moe." + ] = False + disable_shared_experts_fusion: A[ + bool, + Arg( + help="Disable the built-in shared experts fusion optimization for DeepSeek V3/R1. Note: Waterfill (--enable-waterfill) routes the shared expert as an extra MoE slot, so the shared expert is not separated from the MoE path when Waterfill is enabled.", + resolvable=True, + ), + ] = False + enforce_shared_experts_fusion: A[ + bool, + "Enforce shared experts fusion even when it would normally be disabled (e.g. under DeepEP). Mutually exclusive with --disable-shared-experts-fusion.", + ] = False + + # ------------------------------------------------------------------------- + # Ktransformers/AMX expert parallelism + # ------------------------------------------------------------------------- + kt_weight_path: A[ + Optional[str], + "[ktransformers parameter] The path of the quantized expert weights for amx kernel. A local folder.", + ] = None + kt_method: A[ + str, "[ktransformers parameter] Quantization formats for CPU execution." + ] = "AMXINT4" + kt_cpuinfer: A[ + Optional[int], "[ktransformers parameter] The number of CPUInfer threads." + ] = None + kt_threadpool_count: A[ + int, + "[ktransformers parameter] One-to-one with the number of NUMA nodes (one thread pool per NUMA).", + ] = 2 + kt_num_gpu_experts: A[ + Optional[int], "[ktransformers parameter] The number of GPU experts." + ] = None + kt_max_deferred_experts_per_token: A[ + Optional[int], + "[ktransformers parameter] Maximum number of experts deferred to CPU per token. All MoE layers except the final one use this value; the final layer always uses 0.", + ] = None + + +@dataclasses.dataclass +class ExecOverlap: + """Namespace ``exec.overlap``.""" + + _NS_PATH = "exec.overlap" + + # ------------------------------------------------------------------------- + # Two batch overlap + # ------------------------------------------------------------------------- + enable_two_batch_overlap: A[ + bool, + "Enabling two micro batches to overlap.", + ] = False + enable_single_batch_overlap: A[ + bool, "Let computation and communication overlap within one micro batch." + ] = False + tbo_token_distribution_threshold: A[ + float, + "The threshold of token distribution between two batches in micro-batch-overlap, determines whether to two-batch-overlap or two-chunk-overlap. Set to 0 denote disable two-chunk-overlap.", + ] = 0.48 + + +@dataclasses.dataclass +class ExecOffload: + """Namespace ``exec.offload``.""" + + _NS_PATH = "exec.offload" + + # ------------------------------------------------------------------------- + # Offloading + # ------------------------------------------------------------------------- + cpu_offload_gb: A[ + int, + "How many GBs of RAM to reserve for CPU offloading.", + ] = 0 + offload_group_size: A[ + int, + "Number of layers per group in offloading.", + ] = -1 + offload_num_in_group: A[ + int, + "Number of layers to be offloaded within a group.", + ] = 1 + offload_prefetch_step: A[ + int, + "Steps to prefetch in offloading.", + ] = 1 + offload_mode: A[str, "Mode of offloading."] = "cpu" + + +@dataclasses.dataclass +class ExecDllm: + """Namespace ``exec.dllm``.""" + + _NS_PATH = "exec.dllm" + + # ------------------------------------------------------------------------- + # Diffusion LLM + # ------------------------------------------------------------------------- + dllm_algorithm: A[ + Optional[str], "The diffusion LLM algorithm, such as LowConfidence." + ] = None + dllm_algorithm_config: A[ + Optional[str], + "The diffusion LLM algorithm configurations. Must be a YAML file.", + ] = None + dllm_fdfo: A[ + bool, + Arg( + help="Enable First-Done-First-Out (FDFO) scheduling for diffusion LLM inference. Enabled by default; use --no-dllm-fdfo to fall back to synchronous block scheduling.", + action=argparse.BooleanOptionalAction, + ), + ] = True + + +@dataclasses.dataclass +class ExecDeterministic: + """Namespace ``exec.deterministic``.""" + + _NS_PATH = "exec.deterministic" + + # ------------------------------------------------------------------------- + # Deterministic inference + # ------------------------------------------------------------------------- + enable_deterministic_inference: A[ + bool, "Enable deterministic inference mode with batch invariant ops." + ] = False + rl_on_policy_target: A[ + Optional[str], + Arg( + help="The training system that SGLang needs to match for true on-policy.", + choices=RL_ON_POLICY_TARGET_CHOICES, + ), + ] = None diff --git a/python/sglang/srt/arg_groups/fields/lora.py b/python/sglang/srt/arg_groups/fields/lora.py new file mode 100644 index 000000000..57262eb8b --- /dev/null +++ b/python/sglang/srt/arg_groups/fields/lora.py @@ -0,0 +1,119 @@ +"""Config fields of the ``lora`` namespace. + +One class per namespace. The class *is* the namespace: a field declared here +lands in the ``lora`` bag, which is what ``get_lora()`` returns, so a reader +spells it exactly as before. ``ServerArgs`` composes these classes, so the +record stays one flat object -- the split moves where declarations live, not +how config is shaped at runtime. +""" + +from __future__ import annotations + +import argparse +import dataclasses +from typing import ( + List, + Optional, + Union, +) + +from sglang.srt.arg_groups.arg_utils import ( + A, + Arg, +) +from sglang.srt.arg_groups.argparse_actions import LoRAPathAction +from sglang.srt.lora.lora_registry import LoRARef +from sglang.srt.utils.common import ( + LORA_TARGET_ALL_MODULES, + SUPPORTED_LORA_TARGET_MODULES, +) + + +@dataclasses.dataclass +class Lora: + """Namespace ``lora``.""" + + _NS_PATH = "lora" + + # ------------------------------------------------------------------------- + # LoRA + # ------------------------------------------------------------------------- + enable_lora: A[ + Optional[bool], + "Enable LoRA support for the model. This argument is automatically set to True if `--lora-paths` is provided for backward compatibility.", + ] = None + enable_lora_overlap_loading: A[ + Optional[bool], + "Enable asynchronous LoRA weight loading in order to overlap H2D transfers with GPU compute. This should be enabled if you find that your LoRA workloads are bottlenecked by adapter weight loading, for example when frequently loading large LoRA adapters.", + ] = None + max_lora_rank: A[ + Optional[int], + "The maximum rank of LoRA adapters. If not specified, it will be automatically inferred from the adapters provided in --lora-paths.", + ] = None + lora_target_modules: A[ + Optional[Union[set[str], List[str]]], + Arg( + help="The union set of all target modules where LoRA should be applied. If not specified, it will be automatically inferred from the adapters provided in --lora-paths. If 'all' is specified, all supported modules will be targeted.", + nargs="*", + choices=SUPPORTED_LORA_TARGET_MODULES + [LORA_TARGET_ALL_MODULES], + ), + ] = None + lora_paths: A[ + Optional[Union[dict[str, str], List[dict[str, str]], List[str], List[LoRARef]]], + Arg( + help='The list of LoRA adapters to load. Each adapter must be specified in one of the following formats: | = | JSON with schema {"lora_name":str,"lora_path":str,"pinned":bool}', + action=LoRAPathAction, + action_kwargs={"type": str, "nargs": "*"}, + ), + ] = None + max_loaded_loras: A[ + Optional[int], + "If specified, it limits the maximum number of LoRA adapters loaded in CPU memory at a time. The value must be greater than or equal to `--max-loras-per-batch`.", + ] = None + max_loras_per_batch: A[ + int, + "Maximum number of adapters for a running batch, include base-only request.", + ] = 8 + lora_eviction_policy: A[ + str, + Arg( + help="LoRA adapter eviction policy when memory pool is full. 'lru': Least Recently Used (default, better cache efficiency). 'fifo': First-In-First-Out.", + choices=["lru", "fifo"], + ), + ] = "lru" + lora_backend: A[ + str, + Arg( + help="Choose the kernel backend for multi-LoRA serving.", + choices=["triton", "csgmv", "ascend", "torch_native"], + ), + ] = "csgmv" + max_lora_chunk_size: A[ + Optional[int], + Arg( + help="Maximum chunk size for the ChunkedSGMV LoRA backend. Only used when --lora-backend is 'csgmv'. Choosing a larger value might improve performance.", + choices=[16, 32, 64, 128], + ), + ] = 16 + experts_shared_outer_loras: A[ + Optional[bool], + Arg( + help="Force shared outer LoRA mode for MoE models. When set, w1/w3 lora_A and w2 lora_B are shared across experts (expert_dim=1). Use --no-experts-shared-outer-loras to force disable. By default this is auto-detected from adapter weights.", + action=argparse.BooleanOptionalAction, + ), + ] = None + lora_use_virtual_experts: A[ + bool, + "Enable virtual expert computation for MoE models. When set, the model will use virtual expert computation.", + ] = False + lora_strict_loading: A[ + bool, + Arg( + help="Enable strict loading for LoRA adapters. When set, mismatched or missing keys in the adapter weights will raise an error.", + action=argparse.BooleanOptionalAction, + ), + ] = False + lora_drain_wait_threshold: A[ + float, + "When any LoRA adapter request waits longer than this threshold (in seconds), the scheduler will selectively drain one running adapter to make room. This mitigates extreme tail latency under high or skewed workloads by preventing a small set of adapters from monopolizing batch slots. Set to 0 to disable draining (default).", + ] = 0.0 diff --git a/python/sglang/srt/arg_groups/fields/memory.py b/python/sglang/srt/arg_groups/fields/memory.py new file mode 100644 index 000000000..e184b0530 --- /dev/null +++ b/python/sglang/srt/arg_groups/fields/memory.py @@ -0,0 +1,242 @@ +"""Config fields of the ``memory`` namespace. + +One class per namespace. The class *is* the namespace: a field declared here +lands in the ``memory`` bag, which is what ``get_memory()`` returns, so a reader +spells it exactly as before. ``ServerArgs`` composes these classes, so the +record stays one flat object -- the split moves where declarations live, not +how config is shaped at runtime. +""" + +from __future__ import annotations + +import dataclasses +import json +from typing import ( + Any, + Dict, + Optional, +) + +from sglang.srt.arg_groups.arg_utils import ( + A, + Arg, +) +from sglang.srt.arg_groups.choices import RADIX_EVICTION_POLICY_CHOICES + + +@dataclasses.dataclass +class Memory: + """Namespace ``memory``.""" + + _NS_PATH = "memory" + radix_eviction_policy: A[ + str, + Arg( + help=( + "The eviction policy of radix trees. 'lru' stands for Least " + "Recently Used, 'lfu' stands for Least Frequently Used, 'slru' " + "stands for Segmented Least Recently Used, and 'priority' evicts " + "lower-priority requests first. See " + "https://docs.sglang.io/docs/advanced_features/radix_eviction_policy " + "for what each policy optimizes for." + ), + choices=RADIX_EVICTION_POLICY_CHOICES, + ), + ] = "lru" + radix_eviction_policy_config: A[ + Optional[Dict[str, Any]], + Arg( + help=( + "Tuning parameters for --radix-eviction-policy, as a json object " + "passed to the policy as keyword arguments. Only 'slru' takes any " + "today: protected_threshold (int, default 2), e.g. " + "'{\"protected_threshold\": 4}'. An unrecognized key fails at " + "startup, naming the key and the policy. See " + "https://docs.sglang.io/docs/advanced_features/radix_eviction_policy#policy-parameters " + "for the full parameter list." + ), + type_parser=json.loads, + ), + ] = None + disable_radix_cache: A[ + bool, + Arg( + help="Disable RadixAttention for prefix caching.", + resolvable=True, + ), + ] = False + enable_page_major_kv_layout: A[ + bool, + "Enable the page-major KV layout: lay out the Mamba state and full/SWA " + "KV caches in a page-granularity envelope (page is the outermost axis, " + "layer-major within a page) instead of the default per-layer " + "(layer-major) layout. Requires the Triton attention / linear-attn / " + "Mamba backends.", + ] = False + enable_unified_memory: A[ + bool, + "Replace the statically-partitioned hybrid-model pools (full-attn KV + " + "SWA/Mamba state) with one byte buffer split dynamically between " + "sub-pools. Requires the Triton attention / linear-attn / Mamba " + "backends; not yet compatible with PD disaggregation or speculative " + "decoding.", + ] = False + enable_session_radix_cache: A[ + bool, + "Track per-session references on UnifiedRadixCache KV: eviction consumes unreferenced entries before referenced ones, and closing a session only dereferences its KV.", + ] = False + radix_cache_backend: A[ + Optional[str], + "Name of a radix-cache backend previously registered via register_radix_cache_backend. Omit this flag to use the built-in default cache selection chain.", + ] = None + + # ------------------------------------------------------------------------- + # Hierarchical cache + # ------------------------------------------------------------------------- + enable_hierarchical_cache: A[bool, "Enable hierarchical cache"] = False + hicache_host_memory_mode: A[ + str, + Arg( + help="Whether host memory is a persistent HiCache tier (cache) or a transient staging buffer between GPU and the storage backend (buffer_only). buffer_only requires --hicache-storage-backend.", + choices=["cache", "buffer_only"], + ), + ] = "cache" + hicache_ratio: A[ + Optional[float], + "The ratio of the size of host KV cache memory pool to the size of device pool. Defaults to 2.0 in cache mode, 1.2 in buffer_only mode, or 0.2 for backup-only host-pool decode retraction.", + ] = None + hicache_size: A[ + int, + "The size of host KV cache memory pool in gigabytes. Overrides --hicache-ratio in either host memory mode.", + ] = 0 + hicache_write_policy: A[ + str, + Arg( + help="The write policy of hierarchical cache.", + choices=["write_back", "write_through", "write_through_selective"], + ), + ] = "write_through" + hicache_io_backend: A[ + str, + Arg( + help="The IO backend for KV cache transfer between CPU and GPU", + choices=["direct", "kernel", "kernel_ascend"], + ), + ] = "kernel" + hicache_mem_layout: A[ + str, + Arg( + help="The layout of host memory pool for hierarchical cache.", + choices=[ + "layer_first", + "page_first", + "page_first_direct", + "page_first_kv_split", + "page_head", + ], + ), + ] = "page_first" + hicache_storage_backend: A[ + Optional[str], + Arg( + help="The storage backend for hierarchical KV cache. Built-in backends: file, mooncake, hf3fs, nixl, aibrix. For dynamic backend, use --hicache-storage-backend-extra-config to specify: backend_name (custom name), module_path (Python module path), class_name (backend class name).", + choices=[ + "file", + "sim", + "mooncake", + "hf3fs", + "nixl", + "aibrix", + "dynamic", + "eic", + "simm", + "mori", + "shm", + ], + ), + ] = None + hicache_storage_prefetch_policy: A[ + str, + Arg( + help="Control when prefetching from the storage backend should stop.", + choices=["best_effort", "wait_complete", "timeout"], + ), + ] = "timeout" + hicache_storage_backend_extra_config: A[ + Optional[str], + "A dictionary in JSON string format, or a string starting with a leading '@' and a config file in JSON/YAML/TOML format, containing extra configuration for the storage backend.", + ] = None + hicache_storage_prefetch_retry_poll_interval: A[ + int, + Arg( + help=( + "Scheduling passes a queued request waits after a storage " + "prefetch miss before the availability check is retried " + "(under load the first check can run before the needed " + "backup commits). 0 disables retries." + ), + ), + ] = 0 + hicache_storage_prefetch_retry_max_attempts: A[ + int, + "Maximum storage prefetch retries per request when --hicache-storage-prefetch-retry-poll-interval is set.", + ] = 4 + + # ------------------------------------------------------------------------- + # Unified Radix Cache + # ------------------------------------------------------------------------- + enable_unified_cache_external_linker: A[ + bool, + "Link UnifiedRadixCache directly to an external KV store (direct L3), with no host cache tier.", + ] = False + unified_cache_external_linker_backend: A[ + str, + Arg( + help="Storage backend for --enable-unified-cache-external-linker.", + choices=["mooncake", "mori"], + ), + ] = "mooncake" + + # ------------------------------------------------------------------------- + # Hierarchical sparse attention + # ------------------------------------------------------------------------- + enable_hisparse: A[bool, "Enable hierarchical sparse attention"] = False + hisparse_config: A[ + Optional[str], + Arg( + help='A dictionary in JSON string format for hierarchical sparse attention configuration. Example: \'{"top_k": 2048, "device_buffer_size": 4096, "host_to_device_ratio": 2}\'', + aliases=["--hierarchical-sparse-attention-extra-config"], + ), + ] = None + + # ------------------------------------------------------------------------- + # LMCache + # ------------------------------------------------------------------------- + enable_lmcache: A[ + bool, "Using LMCache as an alternative hierarchical cache solution" + ] = False + lmcache_config_file: A[ + Optional[str], + "Path to the LMCache YAML configuration file", + ] = None + + # ------------------------------------------------------------------------- + # FlexKV + # ------------------------------------------------------------------------- + enable_flexkv: A[ + bool, + ( + "Route the default RadixCache through FlexKV's KVManager for " + "host-tier (CPU / SSD / Remote) KV cache offload. Equivalent " + "to --radix-cache-backend=flexkv but also participates in the " + "auto-selection chain alongside --enable-lmcache." + ), + ] = False + flexkv_config_file: A[ + Optional[str], + ( + "Path to the FlexKV YAML / JSON configuration file. " + "Equivalent to setting the FLEXKV_CONFIG_PATH environment " + "variable." + ), + ] = None diff --git a/python/sglang/srt/arg_groups/fields/mm.py b/python/sglang/srt/arg_groups/fields/mm.py new file mode 100644 index 000000000..c42832bef --- /dev/null +++ b/python/sglang/srt/arg_groups/fields/mm.py @@ -0,0 +1,154 @@ +"""Config fields of the ``mm`` namespace. + +One class per namespace. The class *is* the namespace: a field declared here +lands in the ``mm`` bag, which is what ``get_mm()`` returns, so a reader +spells it exactly as before. ``ServerArgs`` composes these classes, so the +record stays one flat object -- the split moves where declarations live, not +how config is shaped at runtime. +""" + +from __future__ import annotations + +import dataclasses +import json +from typing import ( + Any, + Dict, + List, + Literal, + Optional, + Union, +) + +from sglang.srt.arg_groups.arg_utils import ( + A, + Arg, +) + + +@dataclasses.dataclass +class Mm: + """Namespace ``mm``.""" + + _NS_PATH = "mm" + enable_multimodal: A[ + Optional[bool], + "Enable the multimodal functionality for the served model. If the model being served is not multimodal, nothing will happen", + ] = None + mm_attention_backend: A[ + Optional[str], + Arg( + help="Set multimodal attention backend.", + choices=[ + "sdpa", + "fa3", + "fa4", + "triton_attn", + "ascend_attn", + "aiter_attn", + "flashinfer_cudnn", + "amx_attn", + "xpu_attn", + ], + ), + ] = None + + # ------------------------------------------------------------------------- + # Multi-modal optimization configs + # ------------------------------------------------------------------------- + enable_broadcast_mm_inputs_process: A[ + bool, + "Enable broadcast mm-inputs process in scheduler.", + ] = False + enable_prefix_mm_cache: A[ + bool, "Enable prefix multimodal cache. Currently only supports mm-only." + ] = False + mm_enable_dp_encoder: A[ + bool, + "Enabling data parallelism for mm encoder. The dp size will be set to the tp size automatically.", + ] = False + mm_process_config: A[ + Optional[Dict[str, Any]], + Arg( + help="Multimodal preprocessing config, a json config contains keys: `image`, `video`, `audio`", + type_parser=json.loads, + ), + ] = None + mm_processor_worker_num: A[ + int, + "Number of threads for multimodal processor calls. 0 selects the " + "model-specific default. Only processors with isolated-worker support " + "can use more than one thread.", + ] = 0 + mm_io_worker_num: A[ + int, + "Number of threads for multimodal data loading and decoding. 0 selects " + "the model-specific default. SGLANG_IO_WORKERS remains supported as an " + "environment override when this argument is 0.", + ] = 0 + allowed_media_domains: A[ + List[str], + "Restrict client-supplied HTTP(S) image, video, and audio URLs to these " + "exact hostnames. Redirect destinations are checked against the same " + "allowlist. When unset, remote media from any domain is allowed.", + ] = dataclasses.field(default_factory=list) + media_url_max_file_size_mb: A[ + int, + "Maximum size in MiB for one client-supplied remote media download. " + "The limit is enforced while streaming; set to 0 to disable it.", + ] = 64 + mm_preprocess_cache_size_mb: A[ + Optional[int], + "CPU memory budget for content-addressed multimodal preprocessing " + "artifacts. Unset selects a model-specific default (256 MiB for " + "Kimi-K3); 0 disables the cache. The budget is divided across " + "tokenizer workers and does not reserve GPU memory.", + ] = None + trust_mm_content_hashes: A[ + bool, + "Trust caller-provided multimodal SHA-256 content hashes. This can " + "skip reading media on a hot metadata-cache hit; only enable it when " + "the caller guarantees that hashes identify immutable media bytes.", + ] = False + limit_mm_data_per_request: A[ + Optional[Union[str, Dict[str, int]]], + Arg( + help='Limit the number of multimodal inputs per request. e.g. \'{"image": 1, "video": 1, "audio": 1}\'', + type_parser=json.loads, + ), + ] = None + enable_mm_global_cache: A[ + bool, + "Enable global multimodal embedding cache to skip redundant ViT inference.", + ] = False + image_processor_backend: A[ + Literal["auto", "torchvision", "pil"], + "Image processor backend. 'auto' lets Transformers select the best " + "available backend.", + ] = "auto" + mm_global_cache_backend: A[ + str, + Arg( + help="Storage backend for the multimodal global embedding cache. " + "Used when --enable-mm-global-cache is set.", + choices=["mooncake"], + ), + ] = "mooncake" + disable_fast_image_processor: A[ + bool, "Deprecated. Use --image-processor-backend=pil instead." + ] = False + mm_feature_transport: A[ + Optional[Literal["cpu", "cuda_ipc", "cuda_vmm"]], + "Transport multimodal features through CPU memory, a bounded CUDA IPC " + "pool, or a bounded CUDA VMM pool. " + "Unset uses cpu except for validated multi-node GB200/GB300 MNNVL models, " + "which use cuda_vmm when an IMEX channel is available. Select cuda_ipc " + "explicitly for single-node GPU transport. GPU transports reserve " + "SGLANG_MM_FEATURE_CACHE_MB (default 1024 MiB) on the base GPU and fall " + "back to CPU transport when the pool is full.", + ] = None + keep_mm_feature_on_device: A[ + bool, + "Deprecated. Use --mm-feature-transport=cuda_ipc for bounded GPU-resident " + "multimodal feature transport.", + ] = False diff --git a/python/sglang/srt/arg_groups/fields/model.py b/python/sglang/srt/arg_groups/fields/model.py new file mode 100644 index 000000000..62f925433 --- /dev/null +++ b/python/sglang/srt/arg_groups/fields/model.py @@ -0,0 +1,385 @@ +"""Config fields of the ``model`` namespace. + +One class per namespace. The class *is* the namespace: a field declared here +lands in the ``model`` bag, which is what ``get_model()`` returns, so a reader +spells it exactly as before. ``ServerArgs`` composes these classes, so the +record stays one flat object -- the split moves where declarations live, not +how config is shaped at runtime. +""" + +from __future__ import annotations + +import dataclasses +from typing import ( + Dict, + List, + Literal, + Optional, + Union, +) + +from sglang.srt.arg_groups.arg_utils import ( + A, + Arg, +) +from sglang.srt.arg_groups.choices import ( + LOAD_FORMAT_CHOICES, + QUANTIZATION_CHOICES, +) +from sglang.srt.utils.common import ( + human_readable_int, + json_list_type, + nullable_str, +) + + +@dataclasses.dataclass +class Model: + """Namespace ``model``.""" + + _NS_PATH = "model" + + # ------------------------------------------------------------------------- + # Model and tokenizer + # ------------------------------------------------------------------------- + model_path: A[ + str, + Arg( + help="The path of the model weights. This can be a local folder or a Hugging Face repo ID.", + aliases=["--model"], + ), + ] + load_format: A[ + str, + Arg( + help="The format of the model weights to load. " + '"auto" will try to load the weights in the safetensors format ' + "and fall back to the pytorch bin format if safetensors format " + "is not available. " + '"pt" will load the weights in the pytorch bin format. ' + '"safetensors" will load the weights in the safetensors format. ' + '"npcache" will load the weights in pytorch format and store ' + "a numpy cache to speed up the loading. " + '"dummy" will initialize the weights with random values, ' + "which is mainly for profiling." + '"gguf" will load the weights in the gguf format. ' + '"expert_pack" is experimental and loads only the validated ' + "DeepSeek-V4-Flash-0731 MXFP4 or text-only Kimi-K3 Q2_K GGUF " + "model with routed experts stored in an SSD expert pack. " + '"bitsandbytes" will load the weights using bitsandbytes ' + "quantization." + '"layered" loads weights layer by layer so that one can quantize a ' + "layer before loading another to make the peak memory envelope " + "smaller." + '"presharded" performs a normal first-time load (with quantization), ' + "then dumps a per-rank/per-tensor sharded checkpoint with content " + "deduplication into " + "/presharded//. " + "Subsequent runs with the same parallelism+quantization config " + "load directly from this presharded checkpoint and skip " + "re-quantization. " + "The dump directory must be on a shared filesystem across all " + "ranks/nodes. Optional model_loader_extra_config roots: " + "presharded_path (target) and draft_presharded_path (speculative " + "draft); each replaces /presharded and still gets a " + "config subfolder appended. Use a writable path when model_path " + "is read-only (e.g. HF cache mounts).", + choices=LOAD_FORMAT_CHOICES, + ), + ] = "auto" + model_loader_extra_config: A[ + str, + "Extra config for model loader. This will be passed to the model loader " + "corresponding to the chosen load_format. For load_format=presharded, " + "JSON may include presharded_path (target cache root), " + "draft_presharded_path (draft cache root), max_file_bytes, " + "hash_num_threads, and verify_on_load.", + ] = "{}" + trust_remote_code: A[ + bool, + "Whether or not to allow for custom models defined on the Hub in their own modeling files.", + ] = False + context_length: A[ + Optional[int], + Arg( + help="The model's maximum context length. Defaults to None (will use the value from the model's config.json instead)." + f"\n\n{human_readable_int.__doc__}", + type_parser=human_readable_int, + ), + ] = None + is_embedding: A[ + bool, + "Whether to use a CausalLM as an embedding model.", + ] = False + revision: A[ + Optional[str], + "The specific model version to use. It can be a branch name, a tag name, or a commit id. If unspecified, will use the default version.", + ] = None + model_impl: A[ + str, + Arg( + help=( + "Which implementation of the model to use.\n\n" + '* "auto" will try to use the SGLang implementation if it exists ' + "and fall back to the Transformers implementation if no SGLang " + "implementation is available.\n" + '* "sglang" will use the SGLang model implementation.\n' + '* "transformers" will use the Transformers model ' + '* "mindspore" will use the MindSpore model ' + "implementation.\n" + ) + ), + ] = "auto" + model_config_parser: A[ + str, + Arg( + help=( + 'Which model-config parser to use. "auto" picks "mistral" ' + 'via the is_mistral_model name heuristic, else "hf" ' + "(AutoConfig over config.json). Plugins can register additional " + "parsers via @register_model_config_parser." + ) + ), + ] = "auto" + json_model_override_args: A[ + str, + "A dictionary in JSON string format used to override default model configurations.", + ] = "{}" + + # ------------------------------------------------------------------------- + # Quantization and data type + # ------------------------------------------------------------------------- + dtype: A[ + str, + Arg( + help=( + "Data type for model weights and activations.\n\n" + '* "auto" will use FP16 precision for FP32 and FP16 models, and ' + "BF16 precision for BF16 models.\n" + '* "half" for FP16. Recommended for AWQ quantization.\n' + '* "float16" is the same as "half".\n' + '* "bfloat16" for a balance between precision and range.\n' + '* "float" is shorthand for FP32 precision.\n' + '* "float32" for FP32 precision.' + ), + choices=["auto", "half", "float16", "bfloat16", "float", "float32"], + resolvable=True, + ), + ] = "auto" + quantization: A[ + Optional[str], + Arg( + help="The quantization method.", + choices=QUANTIZATION_CHOICES, + resolvable=True, + ), + ] = None + quantization_param_path: A[ + Optional[str], + Arg( + help=( + "Path to the JSON file containing the KV cache scaling factors. " + "This should generally be supplied, when KV cache dtype is FP8. " + "Otherwise, KV cache scaling factors default to 1.0, which may " + "cause accuracy issues. " + ), + type_parser=nullable_str, + ), + ] = None + kv_cache_dtype: A[ + str, + Arg( + help=( + 'Data type for kv cache storage. "auto" will use model data type. ' + '"bf16" or "bfloat16" for BF16 KV cache. "fp8_e5m2" and ' + '"fp8_e4m3" are supported for CUDA 11.8+. "mxfp8" is supported ' + 'by the FA4 backend. "nvfp4" selects ' + 'the NVFP4 FP4 E2M1 KV cache recipe; "fp4_mx_block16" ' + "selects the MX-style block-size-16 FP4 E2M1 KV cache " + "recipe. Both require CUDA 12.8+ and PyTorch 2.8.0+" + ), + choices=[ + "auto", + "fp8_e5m2", + "fp8_e4m3", + "mxfp8", + "bf16", + "bfloat16", + "nvfp4", + "fp4_mx_block16", + "fp4_e2m1", + ], + resolvable=True, + ), + ] = "auto" + modelopt_quant: A[ + Optional[Union[str, Dict]], + ( + "The ModelOpt quantization configuration. Supported values: 'fp8', " + "'int4_awq', 'w4a8_awq', 'nvfp4', 'nvfp4_awq'. This requires the " + "NVIDIA Model Optimizer library to be installed: pip install " + "nvidia-modelopt" + ), + ] = None + modelopt_checkpoint_restore_path: A[ + Optional[str], + ( + "Path to restore a previously saved ModelOpt quantized checkpoint. " + "If provided, the quantization process will be skipped and the model " + "will be loaded from this checkpoint." + ), + ] = None + modelopt_checkpoint_save_path: A[ + Optional[str], + ( + "Path to save the ModelOpt quantized checkpoint after quantization. " + "This allows reusing the quantized model in future runs." + ), + ] = None + modelopt_export_path: A[ + Optional[str], + ( + "Path to export the quantized model in HuggingFace format after " + "ModelOpt quantization. The exported model can then be used directly " + "with SGLang for inference. If not provided, the model will not be " + "exported." + ), + ] = None + quantize_and_serve: A[ + bool, + ( + "Quantize the model with ModelOpt and immediately serve it without " + "exporting. This is useful for development and prototyping. For " + "production, it's recommended to use separate quantization and " + "deployment steps." + ), + ] = False + rl_quant_profile: A[ + Optional[str], + "Path to the FlashRL quantization profile. Required when using --load-format flash_rl.", + ] = None # For flash_rl load format + + # ------------------------------------------------------------------------- + # Model weight update and weight loading + # ------------------------------------------------------------------------- + startup_weight_load_mode: A[ + Literal["serial", "overlap"], + ( + "Control startup weight loading relative to CUDA graph capture. " + "'serial' preserves the existing startup order; 'overlap' stages " + "checkpoint files while CUDA graphs are captured and commits the " + "real weights afterward." + ), + ] = "serial" + custom_weight_loader: A[ + Optional[List[str]], + Arg( + help="The custom dataloader which used to update the model. Should be set with a valid import path, such as my_package.weight_load_func", + nargs="*", + ), + ] = None + weight_loader_disable_mmap: A[ + bool, + "Disable mmap while loading weight using safetensors.", + ] = False + weight_loader_prefetch_checkpoints: A[ + bool, + "Prefetch checkpoint files into OS page cache before loading. Each rank prefetches a fraction of the shards, reducing total network I/O on shared filesystems (NFS/Lustre) from N*checkpoint to 1*checkpoint. Recommended for models on network storage. When enabled, multi-threaded safetensors loading is disabled by default to avoid I/O oversubscription with the prefetch threads; set enable_multithread_load=true in --model-loader-extra-config to keep multi-threaded loading (e.g. on local NVMe where prefetch is a no-op).", + ] = False + weight_loader_prefetch_num_threads: A[ + int, "Number of threads per rank for checkpoint prefetching (default: 4)." + ] = 4 + weight_loader_drop_cache_after_load: A[ + bool, "Call posix_fadvise(DONTNEED) on each safetensors shard after loading it." + ] = False + remote_instance_weight_loader_seed_instance_ip: A[ + Optional[str], + "The ip of the seed instance for loading weights from remote instance.", + ] = None + remote_instance_weight_loader_seed_instance_service_port: A[ + Optional[int], + "The service port of the seed instance for loading weights from remote instance.", + ] = None + remote_instance_weight_loader_send_weights_group_ports: A[ + Optional[List[int]], + Arg( + help="The communication group ports for loading weights from remote instance.", + type_parser=json_list_type, + ), + ] = None + remote_instance_weight_loader_backend: A[ + Literal["transfer_engine", "nccl", "modelexpress"], + "The backend for loading weights from remote instance. Can be 'transfer_engine', 'nccl', or 'modelexpress'. Default is 'nccl'.", + ] = "nccl" + remote_instance_weight_loader_start_seed_via_transfer_engine: A[ + bool, + "Start seed server via transfer engine backend for remote instance weight loader.", + ] = False + engine_info_bootstrap_port: A[ + int, + "Port for the engine info bootstrap server. Default is 6789. Must be set explicitly when running multiple instances on the same node.", + ] = 6789 + modelexpress_config: A[ + Optional[str], + 'JSON config for ModelExpress P2P weight loading. Keys: "url" (optional gRPC host:port override), "transport" ("nixl" or "transfer_engine"). Example: \'{"url": "localhost:8001", "transport": "nixl"}\'', + ] = None + download_dir: A[ + Optional[str], + "Model download directory for huggingface.", + ] = None + model_checksum: A[ + Optional[str], + Arg( + help="Model file integrity verification. If provided without value, uses model-path as HF repo ID. Otherwise, provide checksums JSON file path or HuggingFace repo ID.", + nargs="?", + const="", + ), + ] = None + delete_ckpt_after_loading: A[ + bool, + "Delete the model checkpoint after loading the model.", + ] = False + # Checkpoint decryption + decrypted_config_file: A[ + Optional[str], + "The path of the decrypted config file.", + ] = None + decrypted_draft_config_file: A[ + Optional[str], + "The path of the decrypted draft config file.", + ] = None + checkpoint_engine_wait_weights_before_ready: A[ + bool, + "If set, the server will wait for initial weights to be loaded via checkpoint-engine or other update methods before serving inference requests.", + ] = False + + # ------------------------------------------------------------------------- + # Weight cache + # ------------------------------------------------------------------------- + weight_cache_mode: A[ + str, + Arg( + help="Weight cache mode. 'off': normal disk loading. " + "'daemon': launch weight cache daemon (holds weights in GPU memory). " + "Engine-spawned daemons are co-terminal with the engine and do NOT " + "persist across restarts, so this alone does not speed up restart " + "(the first start is slower). For fast recovery, run the standalone " + "daemon (python -m sglang.srt.weight_cache.daemon) and connect with " + "'client'. 'client': connect to existing daemon and load via IPC.", + choices=["off", "daemon", "client"], + ), + ] = "off" + weight_cache_socket: A[ + Optional[str], + Arg( + help="Unix socket path for weight cache daemon (client mode)." + "If not set, derives the path from SGLANG_WEIGHT_CACHE_SOCKET_TEMPLATE " + "using the caller's physical GPU UUID.", + ), + ] = None + weight_cache_timeout: A[ + int, + Arg( + help="Timeout in seconds for weight cache daemon readiness (default: 1800).", + ), + ] = 1800 diff --git a/python/sglang/srt/arg_groups/fields/observability.py b/python/sglang/srt/arg_groups/fields/observability.py new file mode 100644 index 000000000..98f8f51e0 --- /dev/null +++ b/python/sglang/srt/arg_groups/fields/observability.py @@ -0,0 +1,243 @@ +"""Config fields of the ``observability`` namespace. + +One class per namespace. The class *is* the namespace: a field declared here +lands in the ``observability`` bag, which is what ``get_observability()`` returns, so a reader +spells it exactly as before. ``ServerArgs`` composes these classes, so the +record stays one flat object -- the split moves where declarations live, not +how config is shaped at runtime. +""" + +from __future__ import annotations + +import argparse +import dataclasses +import json +from typing import ( + Any, + Dict, + List, + Optional, +) + +from sglang.srt.arg_groups.arg_utils import ( + A, + Arg, +) +from sglang.srt.utils.common import json_list_type + + +@dataclasses.dataclass +class Observability: + """Namespace ``observability``.""" + + _NS_PATH = "observability" + + # ------------------------------------------------------------------------- + # Logging, metrics, and tracing + # ------------------------------------------------------------------------- + log_level: A[str, "The logging level of all loggers."] = "info" + log_level_http: A[ + Optional[str], + "The logging level of HTTP server. If not set, reuse --log-level by default.", + ] = None + log_requests: A[ + bool, + "Log metadata, inputs, outputs of all requests. The verbosity is decided by --log-requests-level", + ] = False + log_requests_level: A[ + int, + Arg( + help="0: Log metadata (no sampling parameters). 1: Log metadata and sampling parameters. 2: Log metadata, sampling parameters and partial input/output. 3: Log every input/output.", + choices=[0, 1, 2, 3], + ), + ] = 2 + log_requests_format: A[ + str, + Arg( + help="Format for request logging: 'text' (human-readable) or 'json' (structured)", + choices=["text", "json"], + ), + ] = "text" + log_requests_target: A[ + Optional[List[str]], + "Target(s) for request logging: 'stdout' and/or directory path(s) for file output. Can specify multiple targets, e.g., '--log-requests-target stdout /my/path'. ", + ] = None + uvicorn_access_log_exclude_prefixes: A[ + List[str], + Arg( + help="Exclude uvicorn access logs whose request path starts with any of these prefixes. Defaults to empty (disabled). Example: --uvicorn-access-log-exclude-prefixes /metrics /health", + nargs="*", + ), + ] = dataclasses.field(default_factory=list) + crash_dump_folder: A[ + Optional[str], + "Folder path to dump requests from the last 5 min before a crash (if any). If not specified, crash dumping is disabled.", + ] = None + show_time_cost: A[bool, "Show time cost of custom marks."] = False + enable_metrics: A[bool, "Enable log prometheus metrics."] = False + smg_http_sidecar_port: A[ + Optional[int], + Arg( + help="Port for the HTTP sidecar server in legacy SMG gRPC mode (--smg-grpc-mode). Serves Prometheus metrics and profiling endpoints. Defaults to --port + 1. Not used in HTTP mode.", + aliases=["--grpc-http-sidecar-port"], + ), + ] = None + enable_mfu_metrics: A[ + bool, + "Enable estimated MFU-related prometheus metrics.", + ] = False + enable_metrics_for_all_schedulers: A[ + bool, + "Enable --enable-metrics-for-all-schedulers when you want schedulers on all TP ranks (not just TP 0) to record request metrics separately. This is especially useful when dp_attention is enabled, as otherwise all metrics appear to come from TP 0.", + ] = False + load_snapshot_publish_interval: A[ + int, + "Publish load snapshot to shared memory every N decode iterations. Prefill and idle always publish immediately.", + ] = 15 + tokenizer_metrics_custom_labels_header: A[ + str, "Specify the HTTP header for passing custom labels for tokenizer metrics." + ] = "x-custom-labels" + tokenizer_metrics_allowed_custom_labels: A[ + Optional[List[str]], + "The custom labels allowed for tokenizer metrics. The labels are specified via a dict in '--tokenizer-metrics-custom-labels-header' field in HTTP requests, e.g., {'label1': 'value1', 'label2': 'value2'} is allowed if '--tokenizer-metrics-allowed-custom-labels label1 label2' is set.", + ] = None + extra_metric_labels: A[ + Optional[Dict[str, str]], + Arg( + help='The custom labels for metrics. e.g. \'{"label1": "value1", "label2": "value2"}\'', + type_parser=json.loads, + ), + ] = None + bucket_time_to_first_token: A[ + Optional[List[float]], + "The buckets of time to first token, specified as a list of floats.", + ] = None + bucket_inter_token_latency: A[ + Optional[List[float]], + "The buckets of inter-token latency, specified as a list of floats.", + ] = None + bucket_e2e_request_latency: A[ + Optional[List[float]], + "The buckets of end-to-end request latency, specified as a list of floats.", + ] = None + prompt_tokens_buckets: A[ + Optional[List[str]], + "The buckets rule of prompt tokens. " + "Supports 3 rule types: 'default' uses predefined buckets; 'tse ' " + "generates two sides exponential distributed buckets (e.g., 'tse 1000 2 8' generates buckets " + "[984.0, 992.0, 996.0, 998.0, 1000.0, 1002.0, 1004.0, 1008.0, 1016.0]).); 'custom " + " ...' uses custom bucket values (e.g., 'custom 10 50 100 500').", + ] = None + generation_tokens_buckets: A[ + Optional[List[str]], + "The buckets rule for generation tokens histogram. " + "Supports 3 rule types: 'default' uses predefined buckets; 'tse ' " + "generates two sides exponential distributed buckets (e.g., 'tse 1000 2 8' generates buckets " + "[984.0, 992.0, 996.0, 998.0, 1000.0, 1002.0, 1004.0, 1008.0, 1016.0]).); 'custom " + " ...' uses custom bucket values (e.g., 'custom 10 50 100 500').", + ] = None + gc_warning_threshold_secs: A[ + float, + "The threshold for long GC warning. If a GC takes longer than this, a warning will be logged. Set to 0 to disable.", + ] = 0.0 + decode_log_interval: A[ + int, + "The log and metrics reporting interval (in decode iterations) for decode batches.", + ] = 40 + enable_request_time_stats_logging: A[ + bool, + "Enable per request time stats logging", + ] = False + kv_events_config: A[ + Optional[str], + "Config in json format for NVIDIA dynamo KV event publishing. Publishing will be enabled if this flag is used. Runtime-load publishing for load-aware routers is a separate opt-in; see --load-publish-endpoint.", + ] = None + load_publish_endpoint: A[ + Optional[str], + "Opt in to the runtime-load PUB socket that load-aware routers subscribe to. Off by default (unset or 'off'). Use 'auto' to reserve the dp_size ports packed after the --kv-events-config range, or a wildcard-host TCP address (e.g. tcp://*:6000) to place it explicitly; rank r binds port+r and /server_info advertises the base under the kv_events block. Requires --kv-events-config to describe a publisher (routers discover the base through /server_info); startup fails if this is set without one, is not bindable, or overlaps the KV range. Note: 'auto' reserves 2*dp_size ports from the KV base — space co-hosted engines accordingly. The router-facing update cadence follows --load-snapshot-publish-interval (shared to avoid double-collecting the snapshot), so a large value there also staleness-caps this feed.", + ] = None + enable_forward_pass_metrics: A[ + bool, + "Enable per-iteration forward pass metrics via ZMQ IPC. External consumers (e.g. Dynamo planner) subscribe to the IPC endpoint exposed in server_args.forward_pass_metrics_ipc_name.", + ] = False + forward_pass_metrics_worker_id: A[ + str, + Arg(help=argparse.SUPPRESS), + ] = "" + forward_pass_metrics_ipc_name: A[ + Optional[str], + Arg(help=argparse.SUPPRESS), + ] = None + enable_trace: A[bool, "Enable opentelemetry trace"] = False + trace_modules: A[ + str, + "Select the components to trace. Available options are 'request' and 'mooncake'. Format: ,,...", + ] = "request" + otlp_traces_endpoint: A[ + str, + "Config opentelemetry collector endpoint if --enable-trace is set. format: :", + ] = "localhost:4317" + # RequestMetricsExporter configuration + export_metrics_to_file: A[ + bool, + "Export performance metrics for each request to local file (e.g. for forwarding to external systems).", + ] = False + export_metrics_to_file_dir: A[ + Optional[str], + "Directory path for writing performance metrics files (required when --export-metrics-to-file is enabled).", + ] = None + # Class-level DI for the five *MetricsCollector classes. Maps collector role + # (one of: "scheduler", "tokenizer", "storage", "radix_cache", "expert_dispatch") + # to a subclass of the matching base collector. The five instantiation sites + # read from this map and fall back to the base class. Class-object only (no + # CLI surface) since this exists for embedded use cases that pass a Python + # class directly. Default None preserves existing behavior. + stat_loggers: Optional[Dict[str, type]] = None + + # ------------------------------------------------------------------------- + # KV canary + # ------------------------------------------------------------------------- + kv_canary: A[ + str, + Arg( + help="KV cache canary mode. 'none' disables the canary (default). 'log' prints them while the server keeps running (production-safe). 'raise' fails the server on the first detected mismatch (CI lane).", + choices=["none", "log", "raise"], + ), + ] = "none" + kv_canary_real_data: str = "none" + kv_canary_sweep_interval: A[ + int, + "Every N forward steps, run a full-pool sweep.", + ] = 0 + + # ------------------------------------------------------------------------- + # Debug tensor dumps + # ------------------------------------------------------------------------- + debug_tensor_dump_output_folder: A[ + Optional[str], + "The output folder for dumping tensors. In Eagle mode, tensor outputs from draft and target models are stored in separate subdirectories ('draft' and 'target').", + ] = None + # None means dump all layers. + debug_tensor_dump_layers: A[ + Optional[List[int]], "The layer ids to dump. Dump all layers if not specified." + ] = None + # TODO(guoyuhong): clean the old dumper code. + debug_tensor_dump_input_file: A[ + Optional[str], + "The input filename for dumping tensors", + ] = None + + # ------------------------------------------------------------------------- + # Custom hooks, probe, and plugins + # ------------------------------------------------------------------------- + forward_hooks: A[ + Optional[List[dict[str, Any]]], + Arg( + help="JSON-formatted forward hook specifications to attach to the model.", + type_parser=json_list_type, + ), + ] = None + msprobe_dump_config: A[ + Optional[str], + "The path of the JSON configuration file for msProbe. If specified, enables msProbe dump.", + ] = None diff --git a/python/sglang/srt/arg_groups/fields/parallel.py b/python/sglang/srt/arg_groups/fields/parallel.py new file mode 100644 index 000000000..bb3457f6b --- /dev/null +++ b/python/sglang/srt/arg_groups/fields/parallel.py @@ -0,0 +1,277 @@ +"""Config fields of the ``parallel`` namespace. + +One class per namespace. The class *is* the namespace: a field declared here +lands in the ``parallel`` bag, which is what ``get_parallel()`` returns, so a reader +spells it exactly as before. ``ServerArgs`` composes these classes, so the +record stays one flat object -- the split moves where declarations live, not +how config is shaped at runtime. +""" + +from __future__ import annotations + +import argparse +import dataclasses +from typing import Optional + +from sglang.srt.arg_groups.arg_utils import ( + A, + Arg, +) + + +@dataclasses.dataclass +class Parallel: + """Namespace ``parallel``.""" + + _NS_PATH = "parallel" + + # ------------------------------------------------------------------------- + # Distributed topology and parallelism (TP, PP, DP, CP) + # ------------------------------------------------------------------------- + nccl_port: A[ + Optional[int], + "The port for NCCL distributed environment setup. Defaults to a random port.", + ] = None + dist_timeout: A[ + Optional[int], "Set timeout for torch.distributed initialization." + ] = None + dist_init_addr: A[ + Optional[str], + Arg( + help="The host address for initializing distributed backend (e.g., `192.168.0.2:25000`).", + aliases=["--nccl-init-addr"], + ), + ] = None + gated_launch_port: A[ + Optional[int], + "The port of the gated launch control server. When set, every rank blocks right after the distributed environment is initialized, before any sizable GPU allocation, until `POST /gate/activate` is sent to this port on the host of the first rank. This lets an external orchestrator defer the memory hungry part of startup to a safe window. Defaults to None, which disables the gate.", + ] = None + nnodes: A[int, "The number of nodes."] = 1 + node_rank: A[int, "The node rank."] = 0 + tp_size: A[ + int, + Arg( + help="The tensor parallelism size.", + aliases=["--tensor-parallel-size"], + ), + ] = 1 + dcp_size: A[ + int, + Arg( + help="The decode context parallelism size.", + aliases=["--decode-context-parallel-size"], + ), + ] = 1 + pp_size: A[ + int, + Arg( + help="The pipeline parallelism size.", + aliases=["--pipeline-parallel-size"], + ), + ] = 1 + pp_max_micro_batch_size: A[ + Optional[int], "The maximum micro batch size in pipeline parallelism." + ] = None + pp_async_batch_depth: A[ + int, + "The async batch depth of pipeline parallelism.", + ] = 0 + dp_size: A[ + int, + Arg( + help="The data parallelism size.", + aliases=["--data-parallel-size"], + ), + ] = 1 + load_balance_method: A[ + str, + Arg( + help="The load balancing strategy for data parallelism.", + choices=[ + "auto", + "round_robin", + "follow_bootstrap_room", + "total_requests", + "total_tokens", + ], + ), + ] = "auto" + attn_cp_size: A[ + int, + Arg( + help="The attention context parallelism size.", + aliases=["--attention-context-parallel-size"], + resolvable=True, + ), + ] = 1 + moe_dp_size: A[ + int, + Arg( + help="The moe data parallelism size.", + aliases=["--moe-data-parallel-size"], + ), + ] = 1 + dwdp_size: A[ + int, + Arg( + help="DWDP (Distributed Weight Data Parallelism) group size. " + "When > 1, MoE prefill uses weight prefetch instead of token all-to-all. " + "Must equal tp_size. Only supported with --disaggregation-mode null or prefill.", + ), + ] = 1 + dcp_comm_backend: A[ + str, + Arg( + help="Communication backend for the decode context-parallel (DCP) " + "attention reduction: 'ag_rs' (AllGather + ReduceScatter), 'a2a' " + "(fused NCCL All-to-All exchange of output+LSE + local Triton LSE " + "combine), or 'fi_a2a' (FlashInfer MNNVL All-to-All kernel; requires " + "SM90+ and MNNVL fabric memory, e.g. GB200 NVL72).", + choices=["ag_rs", "a2a", "fi_a2a"], + resolvable=True, + ), + ] = "ag_rs" + dcp_replicate_q_proj: A[ + Optional[bool], + Arg( + help="For MLA decode context parallelism with the a2a/fi_a2a " + "backend: replicate the Q projection so each DCP rank computes the " + "full-head query locally (redundant projection compute), eliminating " + "the per-layer head-dim all-gather of Q. Trades a small amount of " + "extra GEMM for one fewer collective per layer. Use " + "--no-dcp-replicate-q-proj to disable the model-specific default.", + action=argparse.BooleanOptionalAction, + resolvable=True, + ), + ] = None + enable_prefill_cp: A[ + bool, + "Enable context parallelism for the prefill phase. Select the layout with --cp-strategy.", + ] = False + cp_strategy: A[ + Optional[str], + Arg( + help="Sharding strategy for prefill CP. 'zigzag' is the former in-seq-split mode; 'interleave' is the former round-robin-split mode.", + choices=("zigzag", "interleave"), + ), + ] = None + # Split DSA GPU KV/indexer cache layers across CP ranks. + enable_dsa_cache_layer_split: A[ + bool, + "Split DSA (DeepSeek Sparse Attention) GPU KV/indexer cache layers across context-parallel ranks to reduce per-rank KV memory. Currently only supported with the mooncake transfer backend (mooncake / mooncake_tcp); mori/nixl support will be added later by the community.", + ] = False + enable_dsa_prefill_context_parallel: A[bool, Arg(no_cli=True)] = False + dsa_prefill_cp_mode: A[str, Arg(no_cli=True)] = "round-robin-split" + enable_prefill_context_parallel: A[bool, Arg(no_cli=True)] = False + prefill_cp_mode: A[str, Arg(no_cli=True)] = "in-seq-split" + enable_cp_decode_attn_tp: A[ + bool, + "Enable attention tensor-parallel weight slicing during decode under context parallel (cp_size>1). Slices the replicated attention linears to the local CP partition, eliminating redundant decode GEMMs.", + ] = False + # DP attention + enable_dp_attention: A[ + bool, + Arg( + help="Enabling data parallelism for attention and tensor parallelism for FFN. The dp size should be equal to the tp size. Currently DeepSeek-V2 and Qwen 2/3 MoE models are supported.", + resolvable=True, + ), + ] = False + enable_dp_attention_local_control_broadcast: A[ + bool, + "With DP-attention, send control messages to every DP group leader and broadcast within attn_tp_group instead of the full tp_group. Eliminates a costly all-ranks gloo sync on every scheduler iteration.", + ] = False + enable_dp_lm_head: A[ + bool, + Arg( + help="Enable vocabulary parallel across the attention TP group to avoid all-gather across DP groups, optimizing performance under DP attention.", + resolvable=True, + ), + ] = False + enable_tp_lm_head_all_to_all: A[ + Optional[bool], + Arg( + help="Use all-to-all instead of TP all-gather followed by DP scatter " + "for the TP-sharded LM head under DP attention. By default this is " + "enabled only on decode-only PD nodes with pure DP attention " + "(tp_size == dp_size > 1 and attn_cp_size == 1), and disabled on " + "prefill-only and colocated nodes. Pass " + "--no-enable-tp-lm-head-all-to-all to opt out. The path is " + "incompatible with --enable-dp-lm-head; batches without an equal " + "padded row count fall back to the existing all-gather path.", + action=argparse.BooleanOptionalAction, + resolvable=True, + ), + ] = None + enable_attn_tp_input_scattered: A[ + bool, + "Allow input of attention to be scattered when only using tensor parallelism, to reduce the computational load of operations such as qkv latent.", + ] = False + enable_shared_experts_attn_tp: A[ + bool, + "Shard shared expert weights across the attention TP group when using an expert-parallel all-to-all backend.", + ] = False + enable_dense_mlp_attn_tp: A[ + bool, + "Shard dense MLP weights across the attention TP group under DP attention.", + ] = False + enable_layernorm_sp: A[ + bool, + "Enable Megatron-style sequence parallelism (arXiv:2205.05198) for the " + "LayerNorm/residual regions under pure tensor parallelism: the row-parallel " + "all-reduce becomes reduce-scatter + all-gather, so LayerNorm runs on " + "sequence-sharded activations with no extra communication volume. " + "Prefill only; Qwen3 dense; requires tp_size > 1 and NVLink/NVSwitch.", + ] = False + disable_attn_tp_gather: A[ + bool, + "Disable scheduler-side attn_tp_gather (the upstream SP path " + "that pads num_tokens to attn_tp_size and pre-allocates a gathered " + "buffer). Use for models that manage SP scatter/gather at the " + "model level (e.g., perform their own all_gather/reduce_scatter " + "inside attention) and do not consume the upstream gathered_buffer. " + "Without this, the cuda graph runner pads num_tokens to attn_tp_size, " + "which can cause kernel autotuners to select wrong-sized variants " + "at small batches.", + ] = False + enable_p2p_check: A[ + bool, + "Enable P2P check for GPU access, otherwise the p2p access is allowed by default.", + ] = False + + # ------------------------------------------------------------------------- + # Expert parallelism + # ------------------------------------------------------------------------- + ep_size: A[ + int, + Arg( + help="The expert parallelism size.", + aliases=["--expert-parallel-size", "--ep"], + resolvable=True, + ), + ] = 1 + moe_dense_tp_size: A[ + Optional[int], + Arg( + help="TP size for MoE dense MLP layers. This flag is useful when, with large TP size, there are errors caused by weights in MLP layers having dimension smaller than the min dimension GEMM supports.", + resolvable=True, + ), + ] = None + ep_join_rank_offset: A[ + int, + Arg( + help=( + "Global rank offset of an elastic EP joining group. Scale " + "joiners must set this to the current effective EP size." + ), + cli_name="--elastic-ep-join-rank-offset", + ), + ] = 0 + elastic_ep_initial_size: A[ + Optional[int], + "EP size used to define the immutable per-rank expert storage layout. " + "Scale joiners must use the primary deployment's launch-time EP size.", + ] = None + max_ep_size: A[ + Optional[int], + "Maximum EP size the server can scale to at runtime. Pre-allocates active-rank state and backend buffers to this size. Defaults to the launch-time world size.", + ] = None diff --git a/python/sglang/srt/arg_groups/fields/schedule.py b/python/sglang/srt/arg_groups/fields/schedule.py new file mode 100644 index 000000000..7ee9aca2e --- /dev/null +++ b/python/sglang/srt/arg_groups/fields/schedule.py @@ -0,0 +1,264 @@ +"""Config fields of the ``schedule`` namespace. + +One class per namespace. The class *is* the namespace: a field declared here +lands in the ``schedule`` bag, which is what ``get_schedule()`` returns, so a reader +spells it exactly as before. ``ServerArgs`` composes these classes, so the +record stays one flat object -- the split moves where declarations live, not +how config is shaped at runtime. +""" + +from __future__ import annotations + +import dataclasses +from typing import ( + List, + Optional, +) + +from sglang.srt.arg_groups.arg_utils import ( + A, + Arg, +) +from sglang.srt.utils.common import human_readable_int + + +@dataclasses.dataclass +class Schedule: + """Namespace ``schedule``.""" + + _NS_PATH = "schedule" + + # ------------------------------------------------------------------------- + # Memory and scheduling + # ------------------------------------------------------------------------- + mem_fraction_static: A[ + Optional[float], + "The fraction of the memory used for static allocation (model weights and KV cache memory pool). Use a smaller value if you see out-of-memory errors.", + ] = None + max_running_requests: A[ + Optional[int], + "The maximum number of running requests.", + ] = None + max_queued_requests: A[ + Optional[int], + "The maximum number of queued requests. This option is ignored when using disaggregation-mode.", + ] = None + max_total_tokens: A[ + Optional[int], + Arg( + help=( + "The maximum number of tokens in the memory pool. If not " + "specified, it will be automatically calculated based on the " + "memory usage fraction. This option is typically used for " + "development and debugging purposes." + + f"\n\n{human_readable_int.__doc__}" + ), + type_parser=human_readable_int, + ), + ] = None + chunked_prefill_size: A[ + Optional[int], + "The maximum number of tokens in a chunk for the chunked prefill. Setting this to -1 means disabling chunked prefill.", + ] = None + prefill_decode_interval: A[ + int, + "The number of decode rounds to run after a prefill batch before scheduling the next prefill. In data-parallel attention mode, the interval is synchronized across all DP ranks. Set to 0 to disable.", + ] = 0 + enable_dynamic_chunking: A[ + bool, + "Enable dynamic chunk size adjustment for pipeline parallelism. When enabled, chunk sizes are dynamically calculated based on fitted function to maintain consistent execution time across chunks.", + ] = False + max_prefill_tokens: A[ + int, + Arg( + help=( + "The maximum number of tokens in a prefill batch. The real bound " + "will be the maximum of this value and the model's maximum " + "context length." + f"\n\n{human_readable_int.__doc__}" + ), + type_parser=human_readable_int, + ), + ] = 16384 + prefill_max_requests: A[ + Optional[int], + "The maximum number of requests in a prefill batch. If not specified, there is no limit.", + ] = None + schedule_policy: A[ + str, + Arg( + help="The scheduling policy of the requests.", + choices=[ + "lpm", + "random", + "fcfs", + "dfs-weight", + "lof", + "priority", + "routing-key", + ], + ), + ] = "fcfs" + enable_priority_scheduling: A[ + bool, + "Enable priority scheduling. Requests with higher priority integer values will be scheduled first by default.", + ] = False + disable_priority_preemption: A[ + bool, + "Disable priority scheduling preemption.", + ] = False + default_priority_value: A[ + Optional[int], "Default priority for requests without explicit priority." + ] = None + abort_on_priority_when_disabled: A[ + bool, + "If set, abort requests that specify a priority when priority scheduling is disabled.", + ] = False + schedule_low_priority_values_first: A[ + bool, + "If specified with --enable-priority-scheduling, the scheduler will schedule requests with lower priority integer values first.", + ] = False + priority_scheduling_preemption_threshold: A[ + int, + "Minimum difference in priorities for an incoming request to have to preempt running request(s).", + ] = 10 + retraction_policy: A[ + str, + Arg( + help=( + "The decode retraction policy to use when the KV cache is full. " + "'length' preserves the existing behavior and retracts short-output, " + "long-input requests first. 'priority' retracts lower-priority " + "requests first, using the same priority direction as priority " + "scheduling." + ), + choices=["length", "priority"], + ), + ] = "length" + schedule_conservativeness: A[ + float, + "How conservative the schedule policy is. A larger value means more conservative scheduling. Use a larger value if you see requests being retracted frequently.", + ] = 1.0 + page_size: A[ + Optional[int], Arg(help="The number of tokens in a page.", resolvable=True) + ] = None + c128_page_size: A[ + int, + "The physical page size of the NPU DSV4 C128 KV cache. Must be a positive multiple of 16.", + ] = 16 + swa_full_tokens_ratio: A[ + Optional[float], + Arg( + help=( + "The ratio of SWA layer KV tokens / full layer KV tokens, regardless " + "of the number of swa:full layers. It should be between 0 and 1. " + "E.g. 0.5 means if each swa layer has 50 tokens, then each full " + "layer has 100 tokens." + ), + resolvable=True, + fallback=0.8, + ), + ] = None + disable_hybrid_swa_memory: A[ + bool, Arg(help="Disable the hybrid SWA memory pool.", resolvable=True) + ] = False + prefill_only_disable_kv_cache: A[ + bool, + "Skip the physical KV cache allocation for embedding-mode prefill-only workloads. Currently only valid with --is-embedding, --chunked-prefill-size=-1, --disable-radix-cache, an FA prefill backend, and non-FP4 KV cache so the fa_skip_kv_cache path is active (no layer reads or writes the cache). Other prefill-only workloads such as scoring/MIS may benefit from this later once their attention paths stop using paged KV. Scheduler admission accounting is unchanged; per-layer K/V tensors are sized to (page_size, head_num, head_dim) placeholders so GPU memory is not wasted.", + ] = False + disable_chunked_prefix_cache: A[ + bool, + "Disable chunked prefix cache feature for deepseek, which should save overhead for short sequences.", + ] = False + disable_overlap_schedule: A[ + bool, + Arg( + help="Disable the overlap scheduler, which overlaps the CPU scheduler with GPU model worker.", + resolvable=True, + ), + ] = False + num_continuous_decode_steps: A[ + int, + "Run multiple continuous decoding steps to reduce scheduling overhead. This can potentially increase throughput but may also increase time-to-first-token latency. The default value is 1, meaning only run one decoding step at a time.", + ] = 1 + scheduler_recv_interval: A[ + int, + "The interval to poll requests in scheduler. Can be set to >1 to reduce the overhead of this.", + ] = 1 + enable_mixed_chunk: A[ + bool, + "Enabling mixing prefill and decode in a batch when using chunked prefill.", + ] = False + + # ------------------------------------------------------------------------- + # Mamba cache and linear attn + # ------------------------------------------------------------------------- + max_mamba_cache_size: A[ + Optional[int], + "The maximum size of the mamba cache.", + ] = None + mamba_full_memory_ratio: A[ + Optional[float], + Arg( + help="The ratio of mamba state memory to full kv cache memory.", + resolvable=True, + fallback=0.9, + ), + ] = None + + # ------------------------------------------------------------------------- + # Prefill delayer + # ------------------------------------------------------------------------- + enable_prefill_delayer: A[ + bool, "Enable prefill delayer for DP attention to reduce idle time." + ] = False + prefill_delayer_max_delay_passes: A[ + int, + "Maximum forward passes to delay prefill.", + ] = 30 + prefill_delayer_token_usage_low_watermark: A[ + Optional[float], "Token usage low watermark for prefill delayer." + ] = None + prefill_delayer_forward_passes_buckets: A[ + Optional[List[float]], + "Custom buckets for prefill delayer forward passes histogram. 0 and max_delay_passes-1 will be auto-added.", + ] = None + prefill_delayer_wait_seconds_buckets: A[ + Optional[List[float]], + "Custom buckets for prefill delayer wait seconds histogram. 0 will be auto-added.", + ] = None + prefill_delayer_queue_min_ratio: A[ + Optional[float], + ( + "Opt-in to the adaptive queue-based delay trigger (independent of the " + "slot-based one). Delays prefill until the waiting queue reaches " + "min(running_req * ratio, prefill_max_requests), falling back to the " + "observed max_prefill_bs when no request limit is set. Unset (default) " + "keeps the original slot-only behavior. Typical: 0.1 ~ 0.5." + ), + ] = None + prefill_delayer_max_delay_ms: A[ + Optional[float], + ( + "Wall-clock cap (ms) on a single queue-trigger delay; once exceeded, " + "prefill is force-released to bound worst-case TTFT. Only consulted " + "when --prefill-delayer-queue-min-ratio is set. Typical: 1000 ~ " + "5000; defaults to 5000 if unset." + ), + ] = None + + # ------------------------------------------------------------------------- + # Min free slots delay (prefill refill batching) + # ------------------------------------------------------------------------- + min_free_slots_delay: A[ + Optional[int], + ( + "Hold new prefills until at least N running-request slots have freed " + "up, so they are admitted in one batch instead of one at a time. " + "Useful when each admission is disproportionately expensive, e.g. " + "speculative decoding with a separate draft prefill pass. An " + "explicit value always wins, capped by max-running-requests " + "(1 disables). When unset, DFlash workloads auto-enable the " + "formula; other workloads stay disabled. Not supported with " + "pipeline parallelism." + ), + ] = None diff --git a/python/sglang/srt/arg_groups/fields/serving.py b/python/sglang/srt/arg_groups/fields/serving.py new file mode 100644 index 000000000..7877140a7 --- /dev/null +++ b/python/sglang/srt/arg_groups/fields/serving.py @@ -0,0 +1,300 @@ +"""Config fields of the ``serving`` namespace. + +One class per namespace. The class *is* the namespace: a field declared here +lands in the ``serving`` bag, which is what ``get_serving()`` returns, so a reader +spells it exactly as before. ``ServerArgs`` composes these classes, so the +record stays one flat object -- the split moves where declarations live, not +how config is shaped at runtime. +""" + +from __future__ import annotations + +import dataclasses +import json +from typing import ( + Any, + Dict, + List, + Optional, +) + +from sglang.srt.arg_groups.arg_utils import ( + A, + Arg, +) +from sglang.srt.utils.common import json_list_type + + +@dataclasses.dataclass +class Serving: + """Namespace ``serving``.""" + + _NS_PATH = "serving" + tokenizer_path: A[Optional[str], "The path of the tokenizer."] = None + tokenizer_mode: A[ + str, + Arg( + help="Tokenizer mode. 'auto' will use the fast tokenizer if available, " + "and 'slow' will always use the slow tokenizer.", + choices=["auto", "slow"], + ), + ] = "auto" + tokenizer_backend: A[ + str, + Arg( + help="Tokenizer backend. 'huggingface' uses the default HuggingFace " + "tokenizers library, and 'fastokens' uses the fastokens library " + "for faster tokenization. Requires the fastokens package to be installed.", + choices=["huggingface", "fastokens"], + ), + ] = "huggingface" + tokenizer_worker_num: A[ + int, + "The worker num of the tokenizer manager.", + ] = 1 + detokenizer_worker_num: A[ + int, + "The worker num of the detokenizer manager.", + ] = 1 + skip_tokenizer_init: A[ + bool, "If set, skip init tokenizer and pass input_ids in generate request." + ] = False + + # ------------------------------------------------------------------------- + # HTTP server + # ------------------------------------------------------------------------- + host: A[str, "The host of the HTTP server."] = "127.0.0.1" + port: A[int, "The port of the HTTP server."] = 30000 + fastapi_root_path: A[ + str, + "App is behind a path based routing proxy.", + ] = "" + smg_grpc_mode: A[ + bool, + "Use the legacy SMG gRPC server (smg-grpc-servicer) instead of the HTTP " + "server. Replaces the deprecated --grpc-mode.", + ] = False + grpc_mode: A[ + bool, "(Deprecated, use --smg-grpc-mode) Legacy SMG gRPC server selector." + ] = False + grpc_port: A[ + Optional[int], + "Port for the native gRPC server, started alongside HTTP. Setting this " + "(or SGLANG_GRPC_PORT) enables the native gRPC server; it is off by " + "default. In legacy --smg-grpc-mode this is the SMG server port and " + "defaults to --port + 10000.", + ] = None + # Env-only (SGLANG_GRPC_WORKER_THREADS); a field so the projection sees it. + grpc_worker_threads: A[Optional[int], Arg(no_cli=True)] = None + sidecar: A[ + Optional[str], + "Start a locally managed sidecar against the native gRPC server. " + "The selected module must expose main(argv) and read the resolved " + "native gRPC endpoint from SGLANG_GRPC_ENDPOINT. Requires --grpc-port " + "or SGLANG_GRPC_PORT.", + ] = None + sidecar_args: A[ + Optional[List[str]], + Arg( + help="JSON array passed to the selected sidecar module's " + "main(argv) function. --sidecar-shutdown-timeout SECONDS is " + "consumed by SGLang.", + type_parser=json_list_type, + ), + ] = None + skip_server_warmup: A[bool, "If set, skip warmup."] = False + warmups: A[ + Optional[str], + "Specify custom warmup functions (csv) to run before server starts eg. --warmups=warmup_name1,warmup_name2 will run the functions `warmup_name1` and `warmup_name2` specified in warmup.py before the server starts listening for requests", + ] = None + enable_http2: A[ + bool, + "Use Granian instead of Uvicorn as the ASGI server, enabling HTTP/1.1 and HTTP/2 auto-negotiation. Clients may use h2c (cleartext HTTP/2) or plain HTTP/1.1. Requires 'pip install sglang[http2]'.", + ] = False + http2_max_concurrent_streams: A[ + int, + "Maximum number of concurrent streams advertised on each HTTP/2 " + "connection (1 to 2^32 - 1). Only applies with --enable-http2.", + ] = 200 + http2_initial_connection_window_size: A[ + int, + "Initial connection-level HTTP/2 receive window in bytes (1024 to " + "2^31 - 1). Only applies with --enable-http2.", + ] = 1024 * 1024 + + # ------------------------------------------------------------------------- + # SSL/TLS + # ------------------------------------------------------------------------- + ssl_keyfile: A[ + Optional[str], + "The file path to the SSL key file.", + ] = None + ssl_certfile: A[ + Optional[str], + "The file path to the SSL certificate file.", + ] = None + ssl_ca_certs: A[Optional[str], "The CA certificates file."] = None + ssl_keyfile_password: A[ + Optional[str], + "The password to decrypt the SSL keyfile.", + ] = None + enable_ssl_refresh: A[ + bool, + "Enable automatic SSL certificate hot-reloading when cert/key files change on disk. Requires --ssl-certfile and --ssl-keyfile.", + ] = False + + # ------------------------------------------------------------------------- + # API related + # ------------------------------------------------------------------------- + api_key: A[ + Optional[str], + "Set API key of the server. It is also used in the OpenAI API compatible server.", + ] = None + admin_api_key: A[ + Optional[str], + "Set admin API key for sensitive management endpoints (e.g. /clear_hicache_storage_backend). When set, admin endpoints require this key and do NOT accept --api-key.", + ] = None + served_model_name: A[ + Optional[str], + "Override the model name returned by the v1/models endpoint in OpenAI API server.", + ] = None + weight_version: A[ + str, + "Version identifier for the model weights. Defaults to 'default' if not specified.", + ] = "default" + chat_template: A[ + Optional[str], + "The buliltin chat template name or the path of the chat template file. This is only used for OpenAI-compatible API server.", + ] = None + hf_chat_template_name: A[ + Optional[str], + "When the HuggingFace tokenizer has multiple chat templates (e.g., 'default', 'tool_use', 'rag'), specify which named template to use. If not set, the first available template is used.", + ] = None + completion_template: A[ + Optional[str], + "The buliltin completion template name or the path of the completion template file. This is only used for OpenAI-compatible API server. only for code completion currently.", + ] = None + file_storage_path: A[ + str, + "The path of the file storage in backend.", + ] = "sglang_storage" + enable_cache_report: A[ + bool, + "Return number of cached tokens in usage.prompt_tokens_details for each openai request.", + ] = False + return_input_ids: A[ + bool, + "Return prompt (input) token ids on the response-level sglext extension for every chat completion request, as if return_input_ids_in_sglext were set on the request.", + ] = False + return_output_ids: A[ + bool, + "Return sampled output token ids on the response-level sglext extension for every chat completion request, as if return_output_ids_in_sglext were set on the request.", + ] = False + reasoning_parser: Optional[str] = None + default_chat_template_kwargs: A[ + Optional[Dict[str, Any]], + Arg( + help="Default chat template kwargs applied to every request when not " + "overridden per-request. Keys must match what the model's chat template " + "expects (e.g. enable_thinking, thinking, reasoning_effort). Per-request " + "chat_template_kwargs takes precedence.", + type_parser=json.loads, + ), + ] = None + strip_thinking_cache: A[ + bool, + "Skip caching reasoning-model output (thinking + answer) in the radix tree on finish; keep only the prompt prefix. Opt-in: changes cache contents.", + ] = False + enable_strict_thinking: A[ + bool, + "Enable strict token filtering during the thinking phase. Blocks model-specific excluded tokens (e.g., tool call markers) during reasoning. Requires a grammar backend that supports token filtering.", + ] = False + tool_call_parser: Optional[str] = None + tool_server: A[ + Optional[str], + "Either 'demo' or a comma-separated list of tool server urls to use for the model. If not specified, no tool server will be used.", + ] = None + sampling_defaults: A[ + str, + Arg( + help="Where to get default sampling parameters. 'openai' uses SGLang/OpenAI defaults (temperature=1.0, top_p=1.0, etc.). 'model' uses the model's generation_config.json to get the recommended sampling parameters if available. Default is 'model'.", + choices=["openai", "model"], + ), + ] = "model" + asr_max_buffer_seconds: A[ + int, + "Maximum seconds of PCM audio the streaming ASR WebSocket handler will accumulate before closing the session with a buffer_overflow error. Guards against OOM when a client streams audio faster than inference can consume it. Default 60s.", + ] = 60 + asr_max_concurrent_sessions: A[ + int, + "Maximum number of concurrent realtime ASR WebSocket sessions served by /v1/realtime. New connections beyond this cap are accepted, sent an error{code:too_many_sessions} frame, and closed. Default 32.", + ] = 32 + preferred_sampling_params: A[ + Optional[str], + Arg( + help="json-formatted sampling settings that will be returned in /get_model_info", + type_parser=json.loads, + ), + ] = None + allow_auto_truncate: A[ + bool, + "Allow automatically truncating requests that exceed the maximum input length instead of returning an error.", + ] = False + + # ------------------------------------------------------------------------- + # Streaming + # ------------------------------------------------------------------------- + stream_interval: A[ + int, + "The interval (or buffer size) for streaming in terms of the token length. A smaller value makes streaming smoother, while a larger value makes the throughput higher", + ] = 1 + batch_notify_size: A[ + int, + "Number of streaming notifications to batch before yielding to the event loop. Reduces asyncio wakeup overhead under high concurrency.", + ] = 16 + stream_response_default_include_usage: A[ + bool, + "Include usage in every streaming response (even when stream_options is not specified).", + ] = False + incremental_streaming_output: A[ + bool, + "Whether to output as a sequence of disjoint segments.", + ] = False + enable_streaming_session: A[ + bool, "Enable streaming session mode and StreamingSession wrapper." + ] = False + + # ------------------------------------------------------------------------- + # Constrained decoding + # ------------------------------------------------------------------------- + constrained_json_whitespace_pattern: A[ + Optional[str], + "(outlines and llguidance backends only) Regex pattern for syntactic whitespaces allowed in JSON constrained output. For example, to allow the model generate consecutive whitespaces, set the pattern to [\n\t ]*", + ] = None + constrained_json_disable_any_whitespace: A[ + bool, + "(xgrammar and llguidance backends only) Enforce compact representation in JSON constrained output.", + ] = False + + # ------------------------------------------------------------------------- + # Dynamic batch tokenizer + # ------------------------------------------------------------------------- + enable_dynamic_batch_tokenizer: A[ + bool, + "Enable async dynamic batch tokenizer for improved performance when multiple requests arrive concurrently.", + ] = False + dynamic_batch_tokenizer_batch_size: A[ + int, + "[Only used if --enable-dynamic-batch-tokenizer is set] Maximum batch size for dynamic batch tokenizer.", + ] = 32 + dynamic_batch_tokenizer_batch_timeout: A[ + float, + "[Only used if --enable-dynamic-batch-tokenizer is set] Timeout in seconds for batching tokenization requests.", + ] = 0.002 + enable_tokenizer_batch_encode: A[ + bool, + "Enable batch tokenization for improved performance when processing multiple text inputs. Do not use with image inputs, pre-tokenized input_ids, or input_embeds.", + ] = False + disable_tokenizer_batch_decode: A[ + bool, "Disable batch decoding when decoding multiple completions." + ] = False diff --git a/python/sglang/srt/arg_groups/fields/spec.py b/python/sglang/srt/arg_groups/fields/spec.py new file mode 100644 index 000000000..f71db3296 --- /dev/null +++ b/python/sglang/srt/arg_groups/fields/spec.py @@ -0,0 +1,260 @@ +"""Config fields of the ``spec`` namespace. + +One class per namespace. The class *is* the namespace: a field declared here +lands in the ``spec`` bag, which is what ``get_spec()`` returns, so a reader +spells it exactly as before. ``ServerArgs`` composes these classes, so the +record stays one flat object -- the split moves where declarations live, not +how config is shaped at runtime. +""" + +from __future__ import annotations + +import dataclasses +from typing import ( + Literal, + Optional, +) + +from sglang.srt.arg_groups.arg_utils import ( + A, + Arg, +) +from sglang.srt.arg_groups.choices import ( + LOAD_FORMAT_CHOICES, + MOE_RUNNER_BACKEND_CHOICES, + QUANTIZATION_CHOICES, +) + + +@dataclasses.dataclass +class Spec: + """Namespace ``spec``.""" + + _NS_PATH = "spec" + # ------------------------------------------------------------------------- + # Speculative decoding + # ------------------------------------------------------------------------- + speculative_algorithm: A[ + Optional[str], + "Speculative algorithm. Builtins: EAGLE, EAGLE3, NEXTN, STANDALONE, NGRAM, DFLASH, DSPARK, UNO. Or any name registered via `SpeculativeAlgorithm.register`.", + ] = None + uno_lora_path: A[Optional[str], "Path to the UNO draft LoRA checkpoint."] = None + speculative_draft_model_path: A[ + Optional[str], + Arg( + help="The path of the draft model weights. This can be a local folder or a Hugging Face repo ID.", + aliases=["--speculative-draft-model"], + ), + ] = None + speculative_draft_model_revision: A[ + Optional[str], + "The specific draft model version to use. It can be a branch name, a tag name, or a commit id. If unspecified, will use the default version.", + ] = None + speculative_draft_load_format: A[ + Optional[str], + Arg( + help="The format of the draft model weights to load. If not specified, will use the same format as --load-format. Use 'dummy' to initialize draft model weights with random values for profiling.", + choices=LOAD_FORMAT_CHOICES, + ), + ] = None + speculative_num_steps: A[ + Optional[int], + "The number of steps sampled from draft model in Speculative Decoding.", + ] = None + speculative_eagle_topk: A[ + Optional[int], + "The number of tokens sampled from the draft model in eagle2 each step.", + ] = None + speculative_num_draft_tokens: A[ + Optional[int], + "The number of tokens sampled from the draft model in Speculative Decoding.", + ] = None + speculative_dflash_block_size: A[ + Optional[int], + "DFLASH only. Block size (verify window length). Alias of --speculative-num-draft-tokens for DFLASH.", + ] = None + speculative_dspark_block_size: A[ + Optional[int], + "DSPARK only. Draft block size gamma (number of proposed draft tokens). The verify window is gamma + 1, so this sets --speculative-num-draft-tokens = gamma + 1. Omit to auto-infer gamma from the draft checkpoint block_size.", + ] = None + speculative_dspark_sps_table_path: A[ + Optional[str], + "DSPARK only. Path to a pre-profiled SPS cost table (JSON) built offline with " + "sglang.benchmark.dspark_sps_profiler, consumed by the ragged-verify " + "scheduler (cap-accept / compact). Omit for an uninitialized flat " + "constant-SPS table: the budget degenerates to verify-all (zero throughput " + "gain by itself).", + ] = None + speculative_dspark_confidence_sts_path: A[ + Optional[str], + "DSPARK only. Optional path to a per-position STS (sequential temperature " + "scaling) calibration JSON, fit offline with sglang.benchmark.dspark_sts_fit. " + "Calibrates the confidence-head survival probabilities the ragged-verify " + "scheduler consumes. Omit to use identity (no calibration); losslessness is " + "unaffected either way.", + ] = None + speculative_dspark_align_verify_tokens_to_graph_tier: A[ + bool, + "DSPARK compact ragged-verify only. Fill the per-request verify lengths so " + "the total verify-token count reaches the cuda-graph tier the forward is " + "already padded to: round the dp-max scheduled total up to the captured " + "token bucket and let the top-k allocator admit that many real draft tokens " + "(confidence-ordered). This recovers the padding the forward pays for anyway " + "-- both the cuda-graph bucket round-up and the dp cross-rank max -- turning " + "it into extra real verification at the same step time. Off by default; when " + "off the schedule is byte-for-byte unchanged.", + ] = False + speculative_accept_threshold_single: A[ + float, + "Accept a draft token if its probability in the target model is greater than this threshold.", + ] = 1.0 + speculative_accept_threshold_acc: A[ + float, + "The accept probability of a draft token is raised from its target probability p to min(1, p / threshold_acc).", + ] = 1.0 + speculative_use_rejection_sampling: A[ + bool, "Use rejection sampling for speculative decoding (requires topk=1)." + ] = False + speculative_token_map: A[ + Optional[str], + "The path of the draft model's small vocab table.", + ] = None + speculative_attention_mode: A[ + str, + Arg( + help="Attention backend for speculative decoding operations (both target verify and draft extend). Can be one of 'prefill' (default) or 'decode'.", + choices=["prefill", "decode"], + resolvable=True, + ), + ] = "prefill" + speculative_draft_attention_backend: A[ + Optional[str], + Arg( + help="Attention backend for speculative decoding drafting.", + resolvable=True, + ), + ] = None + speculative_dsa_topk_backend: A[ + str, + Arg( + help="DSA indexer top-k backend for speculative draft workers. Options: 'sgl-kernel', 'torch', 'flashinfer'. The 'torch' backend currently requires SGLANG_DSA_FUSE_TOPK=false.", + choices=["sgl-kernel", "torch", "flashinfer"], + ), + ] = "sgl-kernel" + speculative_draft_kv_cache_dtype: A[ + Optional[str], + Arg( + help="KV cache dtype for the speculative draft model only. The draft pool is " + "allocated with one slot per target token (draft and target share a slot index " + "space), so for a small draft it can still rival the target pool: a 5-layer " + "DFLASH draft costs 10240 bytes/token in bf16. Setting fp8_e4m3 halves the draft " + "pool; the saving shows up as free device memory, so raise " + "--mem-fraction-static to convert it into KV capacity. Default follows " + "--kv-cache-dtype.", + choices=["auto", "fp8_e5m2", "fp8_e4m3", "bf16", "bfloat16"], + ), + ] = None + speculative_draft_window_size: A[ + Optional[int], + "Sliding window size for the draft model. Honored by Llama EAGLE-3 (`LlamaForCausalLMEagle3`) and DFLASH only; other EAGLE-3 backends (e.g. MLA-based drafters) silently ignore it. For Llama EAGLE-3, the drafter only attends to the most recent N keys (verifier hidden states + its own outputs); the verifier is unaffected. For DFLASH, the draft worker keeps a recent target-token window in its local KV cache (paged backends may retain up to one extra page on the left for alignment). Default is full attention/context.", + ] = None + speculative_moe_runner_backend: A[ + Optional[str], + Arg( + help="Choose the runner backend for MoE in speculative decoding.", + choices=MOE_RUNNER_BACKEND_CHOICES, + resolvable=True, + ), + ] = None + speculative_moe_a2a_backend: A[ + Optional[str], + Arg( + help="Choose the backend for MoE A2A in speculative decoding", + choices=[ + "none", + "deepep", + "mooncake", + "nixl", + "mori", + "ascend_fuseep", + "flashinfer", + "megamoe", + "deepep_v2", + "pplx", + "ascend_tp", + ], + resolvable=True, + ), + ] = None + speculative_draft_model_quantization: A[ + Optional[str], + Arg( + help="The quantization method for speculative model.", + choices=QUANTIZATION_CHOICES, + ), + ] = None + # Internal provenance used after the public draft quantization inherits the + # target value. It is a dataclass field so ServerArgs round-trips preserve + # whether the user explicitly set the draft option; it has no CLI surface. + _speculative_draft_quantization_explicitly_set: A[ + Optional[bool], + Arg(no_cli=True), + ] = None + speculative_skip_dp_mlp_sync: A[ + bool, + "Skip the extra MLP sync that the scheduler performs before merging a new batch when speculative decoding + DP attention are both enabled.", + ] = False + enable_multi_layer_eagle: A[ + bool, + Arg( + help="Enable multi-layer Eagle speculative decoding.", + resolvable=True, + ), + ] = False + speculative_adaptive: A[ + bool, + "Enable adaptive speculative decoding that dynamically adjusts num_steps based on acceptance rate.", + ] = False + speculative_adaptive_config: A[ + Optional[str], + "Path to a JSON config file for adaptive speculative decoding tuning knobs.", + ] = None + spec_trace_dir: A[ + Optional[str], "Directory to write decoupled speculative decoding trace files." + ] = None + + # ------------------------------------------------------------------------- + # Speculative decoding (ngram) + # ------------------------------------------------------------------------- + speculative_ngram_min_bfs_breadth: A[ + int, + "The minimum breadth for BFS (Breadth-First Search) in ngram speculative decoding.", + ] = 1 + speculative_ngram_max_bfs_breadth: A[ + int, + "The maximum breadth for BFS (Breadth-First Search) in ngram speculative decoding.", + ] = 10 + speculative_ngram_match_type: A[ + Literal["BFS", "PROB"], + "The match type for cache tree.", + ] = "BFS" + speculative_ngram_max_trie_depth: A[ + int, + "The max trie depth for ngram speculative decoding.", + ] = 18 + speculative_ngram_capacity: A[ + int, + "The cache capacity for ngram speculative decoding.", + ] = 10 * 1000 * 1000 + speculative_ngram_external_corpus_path: A[ + Optional[str], + "Path to an external JSONL corpus to pre-load into SAM at startup. Additional corpora can be added at runtime via POST /add_external_corpus.", + ] = None + speculative_ngram_external_sam_budget: A[ + int, + "Number of draft nodes reserved for the external SAM subtree in ngram speculative decoding.", + ] = 0 + speculative_ngram_external_corpus_max_tokens: A[ + int, + "Fail startup if the tokenized external ngram corpus exceeds this many tokens. Tune this based on your CPU memory budget.", + ] = 10000000 diff --git a/python/sglang/srt/runtime_context.py b/python/sglang/srt/runtime_context.py index c9cd6e120..2c5f1e494 100644 --- a/python/sglang/srt/runtime_context.py +++ b/python/sglang/srt/runtime_context.py @@ -33,8 +33,8 @@ shims over this slot). ``get_model()`` / ``get_spec()`` / ``get_lora()`` / ``get_mm()`` / ``get_disagg()`` / ``get_serving()`` / ``get_observability()`` return the resolved **config namespace bags** — the single source of truth for config, -snapshotted from ``server_args`` at publish and driven by the ``NS(...)`` -metadata on each field (multi-level under ``exec.*``). Reads are attribute +snapshotted from ``server_args`` at publish, one bag per namespace class in +``arg_groups/fields/`` (multi-level under ``exec.*``). Reads are attribute chains (``get_exec().moe.moe_runner_backend``); bags are read-only by bare assignment (written via ``override``). @@ -796,14 +796,16 @@ class _ConfigBag: def _build_config_bags(server_args: Any) -> dict: - """Snapshot the resolution result into the namespace bag tree, driven by - the ``NS(...)`` metadata on the dataclass fields. Each leaf comes from + """Snapshot the resolution result into the namespace bag tree. + + The tree is ``namespace_of``: each field is placed by the namespace class + that declares it (``arg_groups/fields/``). Each leaf comes from ``resolution_result`` -- the declaration if resolution made one, else what - the caller supplied. Returns - ``{top_level_name: _ConfigBag}``, arbitrarily nested (``exec.moe.eplb.…``). - Only dataclass fields carry ``NS`` markers, so derived properties/methods are - naturally excluded (they stay on the bag). A name used as both a leaf and a - subgroup at the same level is a hard error — no silent shadowing.""" + the caller supplied. Returns ``{top_level_name: _ConfigBag}``, arbitrarily + nested (``exec.moe.eplb.…``). Only dataclass fields are placed, so derived + properties and methods are naturally excluded (they stay on the bag). A + name used as both a leaf and a subgroup at the same level is a hard error + — no silent shadowing.""" from sglang.srt.arg_groups.arg_utils import namespace_of from sglang.srt.arg_groups.overrides import resolution_result @@ -812,12 +814,12 @@ def _build_config_bags(server_args: Any) -> dict: for field, path in namespace_of(type(server_args)).items(): value = resolution_result(server_args, field, _MISSING) if value is _MISSING: - # Every NS-declared field is a dataclass field, so a resolved config + # Every placed field is a dataclass field, so a resolved config # always carries it; a miss means a malformed/partial config object # was published. Fail loud here rather than silently omitting the # leaf (which surfaces later as a confusing "not a published leaf"). raise AttributeError( - f"config field {field!r} is declared NS({path!r}) but absent from " + f"config field {field!r} belongs to namespace {path!r} but is absent from " f"the published {type(server_args).__name__}; cannot project its bag leaf" ) parts = path.split(".") @@ -953,8 +955,8 @@ class RuntimeContext: ) self._server_args = server_args # Snapshot resolved config into the namespace bags (the single source of - # truth for config reads). Driven by NS(...) metadata; a mock/partial - # config with no NS markers yields an empty tree (no bags projected). + # truth for config reads). Placed by `namespace_of`; a mock/partial + # config that declares no namespace yields an empty tree (no bags). self._config_bags = _build_config_bags(server_args) spec = self._config_bags.get("spec") if spec is not None: @@ -1024,7 +1026,7 @@ class RuntimeContext: no write-through, so the old "wrote one store, read another" desync class cannot occur. - Each flat field name is routed to its bag by the ``NS`` metadata (flat + Each flat field name is routed to its bag by ``namespace_of`` (flat names are unique across namespaces). Validation is all-or-nothing: an unknown / unprojected field aborts before any write. ``source`` is recorded for provenance / reproduction. @@ -1042,7 +1044,7 @@ class RuntimeContext: path = nsmap.get(name) if path is None: raise ValueError( - f"override: unknown config field {name!r} (no NS namespace) — " + f"override: unknown config field {name!r} (no namespace) — " "not a resolved config leaf" ) parts = path.split(".") @@ -1076,7 +1078,7 @@ class RuntimeContext: path = namespace_of(type(self._server_args)).get(name) if path is None: - raise ValueError(f"{name!r} is not a config leaf (no NS namespace)") + raise ValueError(f"{name!r} is not a config leaf (no namespace)") parts = path.split(".") bag = self.config_bag(parts[0]) for seg in parts[1:]: diff --git a/python/sglang/srt/server_args.py b/python/sglang/srt/server_args.py index b433effcf..4bb8d857c 100644 --- a/python/sglang/srt/server_args.py +++ b/python/sglang/srt/server_args.py @@ -37,11 +37,10 @@ import argparse import copy import dataclasses import functools -import json import logging import tempfile import uuid -from typing import Any, Callable, Dict, List, Literal, Optional, Union +from typing import Any, Dict, List, Optional from sglang.kernels.ops.kv_canary.consts import RealKvHashMode from sglang.srt.arg_groups.arg_utils import NS, A, Arg, add_cli_args_from_dataclass @@ -85,274 +84,111 @@ from sglang.srt.utils.network import NetworkAddress, get_free_port, wait_port_av logger = logging.getLogger(__name__) -# -------------------------------------------------------------------------- -# Extension points: out-of-tree platforms and plugins extend these lists -# before ServerArgs is constructed. Each list owns its adder on the line -# below it. A list with no adder is not an extension point -- inline it into -# the field's Arg(choices=...) instead of hoisting it here. -# -------------------------------------------------------------------------- +# Re-exported. These were importable from this module while the field +# declarations that used them lived here; the declarations moved to +# `arg_groups/fields/` but out-of-tree code -- and `tokenizer_control_mixin` +# for `LoRARef` -- still reaches them through `sglang.srt.server_args`. +from sglang.srt.arg_groups.arg_utils import NS, A, Arg # noqa: F401 +from sglang.srt.arg_groups.argparse_actions import LoRAPathAction # noqa: F401 -# --- Model loading and quantization --- - -LOAD_FORMAT_CHOICES = [ - "auto", - "pt", - "safetensors", - "npcache", - "dummy", - "sharded_state", - "presharded", - "gguf", - # Experimental and intentionally narrow: expert_pack is validated only for - # DeepSeek-V4-Flash-0731 MXFP4 GGUF (MXFP4 experts, FP8 dense weights) - # and KIMI-K3-MXP4-DERISKED-Q2_K-*.gguf (Q2_K gate/up, Q3_K down weights): - # https://huggingface.co/unsloth/DeepSeek-V4-Flash-0731-GGUF - # https://huggingface.co/Blackfrost-AI/KIMI-K3-Q2_K-GGUF-ABLITERATED - "expert_pack", - "bitsandbytes", - "mistral", - "layered", - "flash_rl", - "remote", - "remote_instance", - "fastsafetensors", - "private", - "runai_streamer", -] -add_load_format_choices = LOAD_FORMAT_CHOICES.extend -# NOTE: LoadFormat.IPC_CACHE intentionally has no public --load-format choice. -# It is an internal dispatch format set automatically by ModelRunner when the -# weight cache is enabled (weight_cache_mode != "off"). Exposing it as a CLI -# choice let users create contradictory combos (see _handle_load_format). - -# TODO: this list should likely contain only methods that support online quantization, or that support using custom quantization classes compatible with a given `quant_method` in config.json. -# Some of the choices here do NOT support online quantization. -QUANTIZATION_CHOICES = [ - "awq", - "fp8", # MOE + linear online quantization. - "mxfp8", # MOE + linear online quantization. - "gptq", - "gptq_marlin", - "awq_marlin", - "bitsandbytes", - "gguf", - # Modelopt has some online quantization support through ModelOptModelLoader. - "modelopt", - "modelopt_fp8", - "modelopt_fp4", - "nvfp4_online", - "modelopt_mixed", - "petit_nvfp4", - "w8a8_int8", # mentioned in quantization.md documentation, supporting compressed-tensors quant_method. - "w8a8_fp8", # mentioned in quantization.md documentation, supporting compressed-tensors quant_method. - "moe_wna16", # custom loading logic for gptq/awq checkpoints (likely untested/unused) - "w4afp8", - "mxfp4", # MOE-only. - "auto-round", - "auto-round-int8", - "compressed-tensors", # for Ktransformers - "modelslim", # for NPU - "mxfp_w4a8", # for NPU W4A8 (MXFP4 weights + MXFP8 activations) - "quark", # AMD Quark quantizer (FP8 / MXFP4 / Int4FP8 etc.) - "quark_int4fp8_moe", - "quark_mxfp4", # Online MOE + linear quantization (incl. NVFP4 -> MXFP4 requantization). - # Apple Silicon MLX backend — on-the-fly quantization of fp16 weights at load - # time via mlx.nn.quantize. Only takes effect when SGLANG_USE_MLX=1. - "mlx_q4", # 4 bits, group_size=64 (mlx-community default) - "mlx_q8", # 8 bits, group_size=64 - "unquant", - "humming", -] -add_quantization_method_choices = QUANTIZATION_CHOICES.extend - -# --- Attention backends --- - -ATTENTION_BACKEND_CHOICES = [ - # Common - "triton", - "torch_native", - "flex_attention", - "dsa", - "nsa", # Deprecated alias for "dsa" - "dsv4", - "compressed", # Deprecated alias for "dsv4" - # NVIDIA specific - "cutlass_mla", - "fa3", - "fa4", - "flashinfer", - "flashmla", - "trtllm_mla", - "cutedsl_mla", - "tokenspeed_mla", - "trtllm_mha", - "dual_chunk_flash_attn", - "hpc_ops", # HPC-Ops (https://github.com/Tencent/hpc-ops), Hopper (SM90) only, requires --page-size 64 - "minicpm_flashattn", - "minicpm_flashinfer", - # AMD specific - "aiter", - "wave", - # Other platforms - "intel_amx", - "ascend", - "intel_xpu", -] -add_attention_backend_choices = ATTENTION_BACKEND_CHOICES.extend - -# trtllm_mha is valid for decode-only dense-MQA drafts. DFLASH rejects it -# earlier when its per-layer attention requirements are not met. -DRAFT_ATTENTION_BACKEND_CHOICES = [ - "flashinfer", - "fa3", - "fa4", - "triton", - "ascend", - "trtllm_mha", -] -add_draft_attention_backend_choices = DRAFT_ATTENTION_BACKEND_CHOICES.extend - -# Attention backends whose kernels read the chunked prefix-cache layout. -# Out-of-tree platforms may extend this list (via -# add_chunked_prefix_cache_attention_backend) before ServerArgs construction; -# the chunked-prefix gate is evaluated during resolution. -CHUNKED_PREFIX_CACHE_SUPPORTED_ATTENTION_BACKENDS = [ - "flashinfer", - "fa3", - "fa4", - "flashmla", - "cutedsl_mla", - "cutlass_mla", - "trtllm_mla", - "tokenspeed_mla", -] -add_chunked_prefix_cache_attention_backend = ( - CHUNKED_PREFIX_CACHE_SUPPORTED_ATTENTION_BACKENDS.append +# Re-exported for out-of-tree plugins, which have always reached these +# through `sglang.srt.server_args`. The lists and their adders moved to +# `arg_groups/choices.py` with the field declarations that name them. +from sglang.srt.arg_groups.choices import ( # noqa: F401 + ATTENTION_BACKEND_CHOICES, + CHUNKED_PREFIX_CACHE_SUPPORTED_ATTENTION_BACKENDS, + DETERMINISTIC_ATTENTION_BACKEND_CHOICES, + DISAGG_TRANSFER_BACKEND_CHOICES, + DRAFT_ATTENTION_BACKEND_CHOICES, + FP4_GEMM_RUNNER_BACKEND_CHOICES, + FP8_GEMM_RUNNER_BACKEND_CHOICES, + GRAMMAR_BACKEND_CHOICES, + LINEAR_ATTN_KERNEL_BACKEND_CHOICES, + LOAD_FORMAT_CHOICES, + MOE_RUNNER_BACKEND_CHOICES, + MXFP8_MOE_RUNNER_BACKEND_CHOICES, + QUANTIZATION_CHOICES, + RADIX_EVICTION_POLICY_CHOICES, + RADIX_SUPPORTED_DETERMINISTIC_ATTENTION_BACKEND, + RL_ON_POLICY_TARGET_CHOICES, + SAMPLING_BACKEND_CHOICES, + add_attention_backend_choices, + add_chunked_prefix_cache_attention_backend, + add_deterministic_attention_backend_choices, + add_disagg_transfer_backend_choices, + add_draft_attention_backend_choices, + add_fp4_gemm_runner_backend_choices, + add_fp8_gemm_runner_backend_choices, + add_grammar_backend_choices, + add_linear_attn_kernel_backend_choices, + add_load_format_choices, + add_moe_runner_backend_choices, + add_mxfp8_moe_runner_backend_choices, + add_quantization_method_choices, + add_radix_eviction_policy_choices, + add_radix_supported_deterministic_attention_backend_choices, + add_rl_on_policy_target_choices, +) +from sglang.srt.arg_groups.fields import collect_input_fields +from sglang.srt.arg_groups.fields.device import ( + Device, +) +from sglang.srt.arg_groups.fields.disagg import ( + Disagg, +) +from sglang.srt.arg_groups.fields.exec_ import ( + ExecComm, + ExecDeterministic, + ExecDllm, + ExecFeatures, + ExecGraph, + ExecKernel, + ExecMamba, + ExecMoe, + ExecOffload, + ExecOverlap, +) +from sglang.srt.arg_groups.fields.lora import ( + Lora, +) +from sglang.srt.arg_groups.fields.memory import ( + Memory, +) +from sglang.srt.arg_groups.fields.mm import ( + Mm, +) +from sglang.srt.arg_groups.fields.model import ( + Model, +) +from sglang.srt.arg_groups.fields.observability import ( + Observability, +) +from sglang.srt.arg_groups.fields.parallel import ( + Parallel, +) +from sglang.srt.arg_groups.fields.schedule import ( + Schedule, +) +from sglang.srt.arg_groups.fields.serving import ( + Serving, +) +from sglang.srt.arg_groups.fields.spec import ( + Spec, +) +from sglang.srt.lora.lora_registry import LoRARef # noqa: F401 +from sglang.srt.model_executor.cuda_graph_config import ( # noqa: F401 + CudaGraphConfig, + parse_cuda_graph_config_arg, +) +from sglang.srt.utils.common import ( # noqa: F401 + LORA_TARGET_ALL_MODULES, + SUPPORTED_LORA_TARGET_MODULES, + human_readable_int, + json_list_type, + nullable_str, ) -DETERMINISTIC_ATTENTION_BACKEND_CHOICES = [ - "ascend", - "fa3", - "fa4", - "flashinfer", - "intel_xpu", - "triton", -] -add_deterministic_attention_backend_choices = ( - DETERMINISTIC_ATTENTION_BACKEND_CHOICES.extend -) -RADIX_SUPPORTED_DETERMINISTIC_ATTENTION_BACKEND = ["ascend", "fa3", "fa4", "triton"] -add_radix_supported_deterministic_attention_backend_choices = ( - RADIX_SUPPORTED_DETERMINISTIC_ATTENTION_BACKEND.extend -) - -# --- Transport --- - -DISAGG_TRANSFER_BACKEND_CHOICES = [ - "mooncake", - "nixl", - "ascend", - "fake", - "mori", - "mooncake_tcp", -] -add_disagg_transfer_backend_choices = DISAGG_TRANSFER_BACKEND_CHOICES.extend - -# --- Sampling and grammar --- - -GRAMMAR_BACKEND_CHOICES = ["xgrammar", "outlines", "llguidance", "none"] -add_grammar_backend_choices = GRAMMAR_BACKEND_CHOICES.extend - -SAMPLING_BACKEND_CHOICES = {"flashinfer", "pytorch", "ascend"} - -# --- MoE and GEMM runners --- - -MOE_RUNNER_BACKEND_CHOICES = [ - "auto", - "deep_gemm", - "triton", - "triton_kernel", - "flashinfer_trtllm", - "experimental_sgl_trtllm", - "flashinfer_trtllm_routed", - "flashinfer_cutlass", - "flashinfer_mxfp4", - "flashinfer_cutedsl", - "cutlass", - "aiter", - "marlin", - "humming", - "experimental_sgl_marlin", - "hpc_ops", # HPC-Ops (https://github.com/Tencent/hpc-ops), FP8 MoE on Hopper (SM90) only - "megamoe", - "intel_xpu", -] -add_moe_runner_backend_choices = MOE_RUNNER_BACKEND_CHOICES.extend - -MXFP8_MOE_RUNNER_BACKEND_CHOICES = [ - "cutlass", - "deep_gemm", - "flashinfer_trtllm", - "flashinfer_trtllm_routed", -] -add_mxfp8_moe_runner_backend_choices = MXFP8_MOE_RUNNER_BACKEND_CHOICES.extend - -FP8_GEMM_RUNNER_BACKEND_CHOICES = [ - "auto", - "deep_gemm", - "flashinfer_trtllm", - "flashinfer_cutlass", - "flashinfer_deepgemm", - "flashinfer_cutedsl", - "cutlass", - "triton", - "aiter", -] -add_fp8_gemm_runner_backend_choices = FP8_GEMM_RUNNER_BACKEND_CHOICES.extend - -FP4_GEMM_RUNNER_BACKEND_CHOICES = [ - "auto", - "flashinfer_cudnn", - "flashinfer_cutedsl", - "flashinfer_cutlass", - "flashinfer_trtllm", - "marlin", -] -add_fp4_gemm_runner_backend_choices = FP4_GEMM_RUNNER_BACKEND_CHOICES.extend - -# --- Cache and scheduling policy --- - -RADIX_EVICTION_POLICY_CHOICES = ["lru", "lfu", "slru", "priority"] -add_radix_eviction_policy_choices = RADIX_EVICTION_POLICY_CHOICES.extend - -# --- Reinforcement learning --- - -RL_ON_POLICY_TARGET_CHOICES = ["fsdp"] -add_rl_on_policy_target_choices = RL_ON_POLICY_TARGET_CHOICES.extend - -# --- Linear attention --- - -LINEAR_ATTN_KERNEL_BACKEND_CHOICES = [ - "triton", - "cutedsl", - "flashinfer", - "flashkda", - "nvidia_kda", - "ptx_kda", - "helion", - "intel_xpu", -] -add_linear_attn_kernel_backend_choices = LINEAR_ATTN_KERNEL_BACKEND_CHOICES.extend - -# -------------------------------------------------------------------------- -# Add new extension points at the end of the matching group above. A new -# choice list is inlined into its field by default; hoisting one here makes -# it public API for out-of-tree code and is a deliberate decision. -# -------------------------------------------------------------------------- - - -@dataclasses.dataclass class ServerArgs: """Server-wide configuration for SGLang. @@ -392,3324 +228,14 @@ class ServerArgs: - The ``--config`` meta-argument (not a dataclass field). Everything else should use the ``A[T, ...]`` annotation. + + The fields live in ``arg_groups/fields/`` -- one class per config + namespace, assembled here rather than inherited, so what the record holds + is one readable call and not a property of a base-class list. The order + they appear in is ``POSITIONAL_FIELD_ORDER``, not the order the namespaces + are listed in: field order *is* the positional constructor signature. """ - # ------------------------------------------------------------------------- - # Model and tokenizer - # ------------------------------------------------------------------------- - model_path: A[ - str, - Arg( - help="The path of the model weights. This can be a local folder or a Hugging Face repo ID.", - aliases=["--model"], - ), - NS("model"), - ] - tokenizer_path: A[Optional[str], "The path of the tokenizer.", NS("serving")] = None - tokenizer_mode: A[ - str, - Arg( - help="Tokenizer mode. 'auto' will use the fast tokenizer if available, " - "and 'slow' will always use the slow tokenizer.", - choices=["auto", "slow"], - ), - NS("serving"), - ] = "auto" - tokenizer_backend: A[ - str, - Arg( - help="Tokenizer backend. 'huggingface' uses the default HuggingFace " - "tokenizers library, and 'fastokens' uses the fastokens library " - "for faster tokenization. Requires the fastokens package to be installed.", - choices=["huggingface", "fastokens"], - ), - NS("serving"), - ] = "huggingface" - tokenizer_worker_num: A[ - int, "The worker num of the tokenizer manager.", NS("serving") - ] = 1 - detokenizer_worker_num: A[ - int, "The worker num of the detokenizer manager.", NS("serving") - ] = 1 - skip_tokenizer_init: A[ - bool, - "If set, skip init tokenizer and pass input_ids in generate request.", - NS("serving"), - ] = False - load_format: A[ - str, - Arg( - help="The format of the model weights to load. " - '"auto" will try to load the weights in the safetensors format ' - "and fall back to the pytorch bin format if safetensors format " - "is not available. " - '"pt" will load the weights in the pytorch bin format. ' - '"safetensors" will load the weights in the safetensors format. ' - '"npcache" will load the weights in pytorch format and store ' - "a numpy cache to speed up the loading. " - '"dummy" will initialize the weights with random values, ' - "which is mainly for profiling." - '"gguf" will load the weights in the gguf format. ' - '"expert_pack" is experimental and loads only the validated ' - "DeepSeek-V4-Flash-0731 MXFP4 or text-only Kimi-K3 Q2_K GGUF " - "model with routed experts stored in an SSD expert pack. " - '"bitsandbytes" will load the weights using bitsandbytes ' - "quantization." - '"layered" loads weights layer by layer so that one can quantize a ' - "layer before loading another to make the peak memory envelope " - "smaller." - '"presharded" performs a normal first-time load (with quantization), ' - "then dumps a per-rank/per-tensor sharded checkpoint with content " - "deduplication into " - "/presharded//. " - "Subsequent runs with the same parallelism+quantization config " - "load directly from this presharded checkpoint and skip " - "re-quantization. " - "The dump directory must be on a shared filesystem across all " - "ranks/nodes. Optional model_loader_extra_config roots: " - "presharded_path (target) and draft_presharded_path (speculative " - "draft); each replaces /presharded and still gets a " - "config subfolder appended. Use a writable path when model_path " - "is read-only (e.g. HF cache mounts).", - choices=LOAD_FORMAT_CHOICES, - ), - NS("model"), - ] = "auto" - model_loader_extra_config: A[ - str, - "Extra config for model loader. This will be passed to the model loader " - "corresponding to the chosen load_format. For load_format=presharded, " - "JSON may include presharded_path (target cache root), " - "draft_presharded_path (draft cache root), max_file_bytes, " - "hash_num_threads, and verify_on_load.", - NS("model"), - ] = "{}" - trust_remote_code: A[ - bool, - "Whether or not to allow for custom models defined on the Hub in their own modeling files.", - NS("model"), - ] = False - context_length: A[ - Optional[int], - Arg( - help="The model's maximum context length. Defaults to None (will use the value from the model's config.json instead)." - f"\n\n{human_readable_int.__doc__}", - type_parser=human_readable_int, - ), - NS("model"), - ] = None - is_embedding: A[ - bool, "Whether to use a CausalLM as an embedding model.", NS("model") - ] = False - enable_multimodal: A[ - Optional[bool], - "Enable the multimodal functionality for the served model. If the model being served is not multimodal, nothing will happen", - NS("mm"), - ] = None - revision: A[ - Optional[str], - "The specific model version to use. It can be a branch name, a tag name, or a commit id. If unspecified, will use the default version.", - NS("model"), - ] = None - model_impl: A[ - str, - Arg( - help=( - "Which implementation of the model to use.\n\n" - '* "auto" will try to use the SGLang implementation if it exists ' - "and fall back to the Transformers implementation if no SGLang " - "implementation is available.\n" - '* "sglang" will use the SGLang model implementation.\n' - '* "transformers" will use the Transformers model ' - '* "mindspore" will use the MindSpore model ' - "implementation.\n" - ) - ), - NS("model"), - ] = "auto" - model_config_parser: A[ - str, - Arg( - help=( - 'Which model-config parser to use. "auto" picks "mistral" ' - 'via the is_mistral_model name heuristic, else "hf" ' - "(AutoConfig over config.json). Plugins can register additional " - "parsers via @register_model_config_parser." - ) - ), - NS("model"), - ] = "auto" - json_model_override_args: A[ - str, - "A dictionary in JSON string format used to override default model configurations.", - NS("model"), - ] = "{}" - - # ------------------------------------------------------------------------- - # Quantization and data type - # ------------------------------------------------------------------------- - dtype: A[ - str, - Arg( - help=( - "Data type for model weights and activations.\n\n" - '* "auto" will use FP16 precision for FP32 and FP16 models, and ' - "BF16 precision for BF16 models.\n" - '* "half" for FP16. Recommended for AWQ quantization.\n' - '* "float16" is the same as "half".\n' - '* "bfloat16" for a balance between precision and range.\n' - '* "float" is shorthand for FP32 precision.\n' - '* "float32" for FP32 precision.' - ), - choices=["auto", "half", "float16", "bfloat16", "float", "float32"], - resolvable=True, - ), - NS("model"), - ] = "auto" - quantization: A[ - Optional[str], - Arg( - help="The quantization method.", - choices=QUANTIZATION_CHOICES, - resolvable=True, - ), - NS("model"), - ] = None - quantization_param_path: A[ - Optional[str], - Arg( - help=( - "Path to the JSON file containing the KV cache scaling factors. " - "This should generally be supplied, when KV cache dtype is FP8. " - "Otherwise, KV cache scaling factors default to 1.0, which may " - "cause accuracy issues. " - ), - type_parser=nullable_str, - ), - NS("model"), - ] = None - kv_cache_dtype: A[ - str, - Arg( - help=( - 'Data type for kv cache storage. "auto" will use model data type. ' - '"bf16" or "bfloat16" for BF16 KV cache. "fp8_e5m2" and ' - '"fp8_e4m3" are supported for CUDA 11.8+. "mxfp8" is supported ' - 'by the FA4 backend. "nvfp4" selects ' - 'the NVFP4 FP4 E2M1 KV cache recipe; "fp4_mx_block16" ' - "selects the MX-style block-size-16 FP4 E2M1 KV cache " - "recipe. Both require CUDA 12.8+ and PyTorch 2.8.0+" - ), - choices=[ - "auto", - "fp8_e5m2", - "fp8_e4m3", - "mxfp8", - "bf16", - "bfloat16", - "nvfp4", - "fp4_mx_block16", - "fp4_e2m1", - ], - resolvable=True, - ), - NS("model"), - ] = "auto" - enable_fp32_lm_head: A[ - bool, "If set, the LM head outputs (logits) are in FP32.", NS("exec.features") - ] = False - modelopt_quant: A[ - Optional[Union[str, Dict]], - ( - "The ModelOpt quantization configuration. Supported values: 'fp8', " - "'int4_awq', 'w4a8_awq', 'nvfp4', 'nvfp4_awq'. This requires the " - "NVIDIA Model Optimizer library to be installed: pip install " - "nvidia-modelopt" - ), - NS("model"), - ] = None - modelopt_checkpoint_restore_path: A[ - Optional[str], - ( - "Path to restore a previously saved ModelOpt quantized checkpoint. " - "If provided, the quantization process will be skipped and the model " - "will be loaded from this checkpoint." - ), - NS("model"), - ] = None - modelopt_checkpoint_save_path: A[ - Optional[str], - ( - "Path to save the ModelOpt quantized checkpoint after quantization. " - "This allows reusing the quantized model in future runs." - ), - NS("model"), - ] = None - modelopt_export_path: A[ - Optional[str], - ( - "Path to export the quantized model in HuggingFace format after " - "ModelOpt quantization. The exported model can then be used directly " - "with SGLang for inference. If not provided, the model will not be " - "exported." - ), - NS("model"), - ] = None - quantize_and_serve: A[ - bool, - ( - "Quantize the model with ModelOpt and immediately serve it without " - "exporting. This is useful for development and prototyping. For " - "production, it's recommended to use separate quantization and " - "deployment steps." - ), - NS("model"), - ] = False - rl_quant_profile: A[ - Optional[str], - "Path to the FlashRL quantization profile. Required when using --load-format flash_rl.", - NS("model"), - ] = None # For flash_rl load format - enable_tf32_matmul: A[ - bool, - Arg( - help="Enable float32 matmuls to use TensorFloat32 precision for better performance (via torch.set_float32_matmul_precision). CUDA only.", - resolvable=True, - ), - NS("exec.features"), - ] = False - - # ------------------------------------------------------------------------- - # Memory and scheduling - # ------------------------------------------------------------------------- - mem_fraction_static: A[ - Optional[float], - "The fraction of the memory used for static allocation (model weights and KV cache memory pool). Use a smaller value if you see out-of-memory errors.", - NS("schedule"), - ] = None - max_running_requests: A[ - Optional[int], "The maximum number of running requests.", NS("schedule") - ] = None - max_queued_requests: A[ - Optional[int], - "The maximum number of queued requests. This option is ignored when using disaggregation-mode.", - NS("schedule"), - ] = None - max_total_tokens: A[ - Optional[int], - Arg( - help=( - "The maximum number of tokens in the memory pool. If not " - "specified, it will be automatically calculated based on the " - "memory usage fraction. This option is typically used for " - "development and debugging purposes." - + f"\n\n{human_readable_int.__doc__}" - ), - type_parser=human_readable_int, - ), - NS("schedule"), - ] = None - chunked_prefill_size: A[ - Optional[int], - "The maximum number of tokens in a chunk for the chunked prefill. Setting this to -1 means disabling chunked prefill.", - NS("schedule"), - ] = None - prefill_decode_interval: A[ - int, - "The number of decode rounds to run after a prefill batch before scheduling the next prefill. In data-parallel attention mode, the interval is synchronized across all DP ranks. Set to 0 to disable.", - NS("schedule"), - ] = 0 - enable_dynamic_chunking: A[ - bool, - "Enable dynamic chunk size adjustment for pipeline parallelism. When enabled, chunk sizes are dynamically calculated based on fitted function to maintain consistent execution time across chunks.", - NS("schedule"), - ] = False - max_prefill_tokens: A[ - int, - Arg( - help=( - "The maximum number of tokens in a prefill batch. The real bound " - "will be the maximum of this value and the model's maximum " - "context length." + f"\n\n{human_readable_int.__doc__}" - ), - type_parser=human_readable_int, - ), - NS("schedule"), - ] = 16384 - prefill_max_requests: A[ - Optional[int], - "The maximum number of requests in a prefill batch. If not specified, there is no limit.", - NS("schedule"), - ] = None - schedule_policy: A[ - str, - Arg( - help="The scheduling policy of the requests.", - choices=[ - "lpm", - "random", - "fcfs", - "dfs-weight", - "lof", - "priority", - "routing-key", - ], - ), - NS("schedule"), - ] = "fcfs" - enable_priority_scheduling: A[ - bool, - "Enable priority scheduling. Requests with higher priority integer values will be scheduled first by default.", - NS("schedule"), - ] = False - disable_priority_preemption: A[ - bool, "Disable priority scheduling preemption.", NS("schedule") - ] = False - default_priority_value: A[ - Optional[int], - "Default priority for requests without explicit priority.", - NS("schedule"), - ] = None - abort_on_priority_when_disabled: A[ - bool, - "If set, abort requests that specify a priority when priority scheduling is disabled.", - NS("schedule"), - ] = False - schedule_low_priority_values_first: A[ - bool, - "If specified with --enable-priority-scheduling, the scheduler will schedule requests with lower priority integer values first.", - NS("schedule"), - ] = False - priority_scheduling_preemption_threshold: A[ - int, - "Minimum difference in priorities for an incoming request to have to preempt running request(s).", - NS("schedule"), - ] = 10 - retraction_policy: A[ - str, - Arg( - help=( - "The decode retraction policy to use when the KV cache is full. " - "'length' preserves the existing behavior and retracts short-output, " - "long-input requests first. 'priority' retracts lower-priority " - "requests first, using the same priority direction as priority " - "scheduling." - ), - choices=["length", "priority"], - ), - NS("schedule"), - ] = "length" - schedule_conservativeness: A[ - float, - "How conservative the schedule policy is. A larger value means more conservative scheduling. Use a larger value if you see requests being retracted frequently.", - NS("schedule"), - ] = 1.0 - page_size: A[ - Optional[int], - Arg(help="The number of tokens in a page.", resolvable=True), - NS("schedule"), - ] = None - c128_page_size: A[ - int, - "The physical page size of the NPU DSV4 C128 KV cache. Must be a positive multiple of 16.", - NS("schedule"), - ] = 16 - swa_full_tokens_ratio: A[ - Optional[float], - Arg( - help=( - "The ratio of SWA layer KV tokens / full layer KV tokens, regardless " - "of the number of swa:full layers. It should be between 0 and 1. " - "E.g. 0.5 means if each swa layer has 50 tokens, then each full " - "layer has 100 tokens." - ), - resolvable=True, - fallback=0.8, - ), - NS("schedule"), - ] = None - disable_hybrid_swa_memory: A[ - bool, - Arg(help="Disable the hybrid SWA memory pool.", resolvable=True), - NS("schedule"), - ] = False - radix_eviction_policy: A[ - str, - Arg( - help=( - "The eviction policy of radix trees. 'lru' stands for Least " - "Recently Used, 'lfu' stands for Least Frequently Used, 'slru' " - "stands for Segmented Least Recently Used, and 'priority' evicts " - "lower-priority requests first. See " - "https://docs.sglang.io/docs/advanced_features/radix_eviction_policy " - "for what each policy optimizes for." - ), - choices=RADIX_EVICTION_POLICY_CHOICES, - ), - NS("memory"), - ] = "lru" - radix_eviction_policy_config: A[ - Optional[Dict[str, Any]], - Arg( - help=( - "Tuning parameters for --radix-eviction-policy, as a json object " - "passed to the policy as keyword arguments. Only 'slru' takes any " - "today: protected_threshold (int, default 2), e.g. " - "'{\"protected_threshold\": 4}'. An unrecognized key fails at " - "startup, naming the key and the policy. See " - "https://docs.sglang.io/docs/advanced_features/radix_eviction_policy#policy-parameters " - "for the full parameter list." - ), - type_parser=json.loads, - ), - NS("memory"), - ] = None - prefill_only_disable_kv_cache: A[ - bool, - "Skip the physical KV cache allocation for embedding-mode prefill-only workloads. Currently only valid with --is-embedding, --chunked-prefill-size=-1, --disable-radix-cache, an FA prefill backend, and non-FP4 KV cache so the fa_skip_kv_cache path is active (no layer reads or writes the cache). Other prefill-only workloads such as scoring/MIS may benefit from this later once their attention paths stop using paged KV. Scheduler admission accounting is unchanged; per-layer K/V tensors are sized to (page_size, head_num, head_dim) placeholders so GPU memory is not wasted.", - NS("schedule"), - ] = False - disable_radix_cache: A[ - bool, - Arg( - help="Disable RadixAttention for prefix caching.", - resolvable=True, - ), - NS("memory"), - ] = False - enable_page_major_kv_layout: A[ - bool, - "Enable the page-major KV layout: lay out the Mamba state and full/SWA " - "KV caches in a page-granularity envelope (page is the outermost axis, " - "layer-major within a page) instead of the default per-layer " - "(layer-major) layout. Requires the Triton attention / linear-attn / " - "Mamba backends.", - NS("memory"), - ] = False - enable_unified_memory: A[ - bool, - "Replace the statically-partitioned hybrid-model pools (full-attn KV + " - "SWA/Mamba state) with one byte buffer split dynamically between " - "sub-pools. Requires the Triton attention / linear-attn / Mamba " - "backends; not yet compatible with PD disaggregation or speculative " - "decoding.", - NS("memory"), - ] = False - disable_chunked_prefix_cache: A[ - bool, - "Disable chunked prefix cache feature for deepseek, which should save overhead for short sequences.", - NS("schedule"), - ] = False - disable_overlap_schedule: A[ - bool, - Arg( - help="Disable the overlap scheduler, which overlaps the CPU scheduler with GPU model worker.", - resolvable=True, - ), - NS("schedule"), - ] = False - num_continuous_decode_steps: A[ - int, - "Run multiple continuous decoding steps to reduce scheduling overhead. This can potentially increase throughput but may also increase time-to-first-token latency. The default value is 1, meaning only run one decoding step at a time.", - NS("schedule"), - ] = 1 - scheduler_recv_interval: A[ - int, - "The interval to poll requests in scheduler. Can be set to >1 to reduce the overhead of this.", - NS("schedule"), - ] = 1 - enable_mixed_chunk: A[ - bool, - "Enabling mixing prefill and decode in a batch when using chunked prefill.", - NS("schedule"), - ] = False - - # ------------------------------------------------------------------------- - # Distributed topology and parallelism (TP, PP, DP, CP) - # ------------------------------------------------------------------------- - nccl_port: A[ - Optional[int], - "The port for NCCL distributed environment setup. Defaults to a random port.", - NS("parallel"), - ] = None - dist_timeout: A[ - Optional[int], - "Set timeout for torch.distributed initialization.", - NS("parallel"), - ] = None - dist_init_addr: A[ - Optional[str], - Arg( - help="The host address for initializing distributed backend (e.g., `192.168.0.2:25000`).", - aliases=["--nccl-init-addr"], - ), - NS("parallel"), - ] = None - gated_launch_port: A[ - Optional[int], - "The port of the gated launch control server. When set, every rank blocks right after the distributed environment is initialized, before any sizable GPU allocation, until `POST /gate/activate` is sent to this port on the host of the first rank. This lets an external orchestrator defer the memory hungry part of startup to a safe window. Defaults to None, which disables the gate.", - NS("parallel"), - ] = None - nnodes: A[int, "The number of nodes.", NS("parallel")] = 1 - node_rank: A[int, "The node rank.", NS("parallel")] = 0 - tp_size: A[ - int, - Arg( - help="The tensor parallelism size.", - aliases=["--tensor-parallel-size"], - ), - NS("parallel"), - ] = 1 - dcp_size: A[ - int, - Arg( - help="The decode context parallelism size.", - aliases=["--decode-context-parallel-size"], - ), - NS("parallel"), - ] = 1 - pp_size: A[ - int, - Arg( - help="The pipeline parallelism size.", - aliases=["--pipeline-parallel-size"], - ), - NS("parallel"), - ] = 1 - pp_max_micro_batch_size: A[ - Optional[int], - "The maximum micro batch size in pipeline parallelism.", - NS("parallel"), - ] = None - pp_async_batch_depth: A[ - int, "The async batch depth of pipeline parallelism.", NS("parallel") - ] = 0 - dp_size: A[ - int, - Arg( - help="The data parallelism size.", - aliases=["--data-parallel-size"], - ), - NS("parallel"), - ] = 1 - load_balance_method: A[ - str, - Arg( - help="The load balancing strategy for data parallelism.", - choices=[ - "auto", - "round_robin", - "follow_bootstrap_room", - "total_requests", - "total_tokens", - ], - ), - NS("parallel"), - ] = "auto" - attn_cp_size: A[ - int, - Arg( - help="The attention context parallelism size.", - aliases=["--attention-context-parallel-size"], - resolvable=True, - ), - NS("parallel"), - ] = 1 - moe_dp_size: A[ - int, - Arg( - help="The moe data parallelism size.", - aliases=["--moe-data-parallel-size"], - ), - NS("parallel"), - ] = 1 - dwdp_size: A[ - int, - Arg( - help="DWDP (Distributed Weight Data Parallelism) group size. " - "When > 1, MoE prefill uses weight prefetch instead of token all-to-all. " - "Must equal tp_size. Only supported with --disaggregation-mode null or prefill.", - ), - NS("parallel"), - ] = 1 - dcp_comm_backend: A[ - str, - Arg( - help="Communication backend for the decode context-parallel (DCP) " - "attention reduction: 'ag_rs' (AllGather + ReduceScatter), 'a2a' " - "(fused NCCL All-to-All exchange of output+LSE + local Triton LSE " - "combine), or 'fi_a2a' (FlashInfer MNNVL All-to-All kernel; requires " - "SM90+ and MNNVL fabric memory, e.g. GB200 NVL72).", - choices=["ag_rs", "a2a", "fi_a2a"], - resolvable=True, - ), - NS("parallel"), - ] = "ag_rs" - dcp_replicate_q_proj: A[ - Optional[bool], - Arg( - help="For MLA decode context parallelism with the a2a/fi_a2a " - "backend: replicate the Q projection so each DCP rank computes the " - "full-head query locally (redundant projection compute), eliminating " - "the per-layer head-dim all-gather of Q. Trades a small amount of " - "extra GEMM for one fewer collective per layer. Use " - "--no-dcp-replicate-q-proj to disable the model-specific default.", - action=argparse.BooleanOptionalAction, - resolvable=True, - ), - NS("parallel"), - ] = None - enable_prefill_cp: A[ - bool, - "Enable context parallelism for the prefill phase. Select the layout with --cp-strategy.", - NS("parallel"), - ] = False - cp_strategy: A[ - Optional[str], - Arg( - help="Sharding strategy for prefill CP. 'zigzag' is the former in-seq-split mode; 'interleave' is the former round-robin-split mode.", - choices=("zigzag", "interleave"), - ), - NS("parallel"), - ] = None - # Split DSA GPU KV/indexer cache layers across CP ranks. - enable_dsa_cache_layer_split: A[ - bool, - "Split DSA (DeepSeek Sparse Attention) GPU KV/indexer cache layers across context-parallel ranks to reduce per-rank KV memory. Currently only supported with the mooncake transfer backend (mooncake / mooncake_tcp); mori/nixl support will be added later by the community.", - NS("parallel"), - ] = False - enable_dsa_prefill_context_parallel: A[bool, Arg(no_cli=True), NS("parallel")] = ( - False - ) - dsa_prefill_cp_mode: A[str, Arg(no_cli=True), NS("parallel")] = "round-robin-split" - enable_prefill_context_parallel: A[bool, Arg(no_cli=True), NS("parallel")] = False - prefill_cp_mode: A[str, Arg(no_cli=True), NS("parallel")] = "in-seq-split" - enable_cp_decode_attn_tp: A[ - bool, - "Enable attention tensor-parallel weight slicing during decode under context parallel (cp_size>1). Slices the replicated attention linears to the local CP partition, eliminating redundant decode GEMMs.", - NS("parallel"), - ] = False - # DP attention - enable_dp_attention: A[ - bool, - Arg( - help="Enabling data parallelism for attention and tensor parallelism for FFN. The dp size should be equal to the tp size. Currently DeepSeek-V2 and Qwen 2/3 MoE models are supported.", - resolvable=True, - ), - NS("parallel"), - ] = False - enable_dp_attention_local_control_broadcast: A[ - bool, - "With DP-attention, send control messages to every DP group leader and broadcast within attn_tp_group instead of the full tp_group. Eliminates a costly all-ranks gloo sync on every scheduler iteration.", - NS("parallel"), - ] = False - enable_dp_lm_head: A[ - bool, - Arg( - help="Enable vocabulary parallel across the attention TP group to avoid all-gather across DP groups, optimizing performance under DP attention.", - resolvable=True, - ), - NS("parallel"), - ] = False - enable_tp_lm_head_all_to_all: A[ - Optional[bool], - Arg( - help="Use all-to-all instead of TP all-gather followed by DP scatter " - "for the TP-sharded LM head under DP attention. By default this is " - "enabled only on decode-only PD nodes with pure DP attention " - "(tp_size == dp_size > 1 and attn_cp_size == 1), and disabled on " - "prefill-only and colocated nodes. Pass " - "--no-enable-tp-lm-head-all-to-all to opt out. The path is " - "incompatible with --enable-dp-lm-head; batches without an equal " - "padded row count fall back to the existing all-gather path.", - action=argparse.BooleanOptionalAction, - resolvable=True, - ), - NS("parallel"), - ] = None - enable_attn_tp_input_scattered: A[ - bool, - "Allow input of attention to be scattered when only using tensor parallelism, to reduce the computational load of operations such as qkv latent.", - NS("parallel"), - ] = False - enable_shared_experts_attn_tp: A[ - bool, - "Shard shared expert weights across the attention TP group when using an expert-parallel all-to-all backend.", - NS("parallel"), - ] = False - enable_dense_mlp_attn_tp: A[ - bool, - "Shard dense MLP weights across the attention TP group under DP attention.", - NS("parallel"), - ] = False - enable_layernorm_sp: A[ - bool, - "Enable Megatron-style sequence parallelism (arXiv:2205.05198) for the " - "LayerNorm/residual regions under pure tensor parallelism: the row-parallel " - "all-reduce becomes reduce-scatter + all-gather, so LayerNorm runs on " - "sequence-sharded activations with no extra communication volume. " - "Prefill only; Qwen3 dense; requires tp_size > 1 and NVLink/NVSwitch.", - NS("parallel"), - ] = False - disable_attn_tp_gather: A[ - bool, - "Disable scheduler-side attn_tp_gather (the upstream SP path " - "that pads num_tokens to attn_tp_size and pre-allocates a gathered " - "buffer). Use for models that manage SP scatter/gather at the " - "model level (e.g., perform their own all_gather/reduce_scatter " - "inside attention) and do not consume the upstream gathered_buffer. " - "Without this, the cuda graph runner pads num_tokens to attn_tp_size, " - "which can cause kernel autotuners to select wrong-sized variants " - "at small batches.", - NS("parallel"), - ] = False - enable_p2p_check: A[ - bool, - "Enable P2P check for GPU access, otherwise the p2p access is allowed by default.", - NS("parallel"), - ] = False - - # ------------------------------------------------------------------------- - # Device info and server timeout - # ------------------------------------------------------------------------- - device: A[ - Optional[str], - "The device to use ('cuda', 'xpu', 'hpu', 'npu', 'cpu', 'musa'). Defaults to auto-detection if not specified.", - NS("device"), - ] = None - base_gpu_id: A[ - int, - "The base GPU ID to start allocating GPUs from. Useful when running multiple instances on the same machine.", - NS("device"), - ] = 0 - gpu_id_step: A[ - int, - "The delta between consecutive GPU IDs that are used. For example, setting it to 2 will use GPU 0,2,4,...", - NS("device"), - ] = 1 - random_seed: A[Optional[int], "The random seed.", NS("device")] = None - mlx_enable_sampling: A[ - bool, - ( - "MLX backend only: sample decode tokens (temperature / top-k / " - "top-p / min-p) instead of greedy argmax. Sampling runs inside " - "the lazy MLX graph, so it works with the overlap scheduler; " - "first tokens from prefill/extend are sampled too. Greedy " - "requests keep exact argmax behavior. Also enables on the MLX " - "path: grammar vocab masks and custom logit processors (these " - "break decode chaining per step; custom processors run on " - "pure-decode steps only), logit_bias, output logprobs (sampled " - "token / top-k / token_ids; prompt input logprobs are not " - "computed), NaN sanitization (SGLANG_SANITIZE_NAN_LOGITS), and " - "per-request sampling_seed under " - "--enable-deterministic-inference (deterministic within MLX " - "only). Penalties are not applied." - ), - NS("device"), - ] = False - watchdog_timeout: A[ - float, - "Set watchdog timeout in seconds. If a forward batch takes longer than this, the server will crash to prevent hanging.", - NS("device"), - ] = 300 - soft_watchdog_timeout: A[ - Optional[float], - "Set soft watchdog timeout in seconds. If a forward batch takes longer than this, the server will dump information for debugging.", - NS("device"), - ] = None - sleep_on_idle: A[bool, "Reduce CPU usage when sglang is idle.", NS("device")] = ( - False - ) - use_ray: A[ - bool, "Use Ray actors for scheduler process management.", NS("device") - ] = False - custom_sigquit_handler: A[Optional[Callable], NS("device")] = None - numa_node: A[ - Optional[List[int]], - "Sets the numa node for the subprocesses. i-th element corresponds to i-th subprocess. If unset, will be automatically detected on NUMA systems.", - NS("device"), - ] = None - gc_threshold: A[ - Optional[List[int]], - "Set the garbage collection thresholds (the collection frequency). Accepts 1 to 3 integers.", - NS("device"), - ] = None - - # ------------------------------------------------------------------------- - # HTTP server - # ------------------------------------------------------------------------- - host: A[str, "The host of the HTTP server.", NS("serving")] = "127.0.0.1" - port: A[int, "The port of the HTTP server.", NS("serving")] = 30000 - fastapi_root_path: A[ - str, "App is behind a path based routing proxy.", NS("serving") - ] = "" - smg_grpc_mode: A[ - bool, - "Use the legacy SMG gRPC server (smg-grpc-servicer) instead of the HTTP " - "server. Replaces the deprecated --grpc-mode.", - NS("serving"), - ] = False - grpc_mode: A[ - bool, - "(Deprecated, use --smg-grpc-mode) Legacy SMG gRPC server selector.", - NS("serving"), - ] = False - grpc_port: A[ - Optional[int], - "Port for the native gRPC server, started alongside HTTP. Setting this " - "(or SGLANG_GRPC_PORT) enables the native gRPC server; it is off by " - "default. In legacy --smg-grpc-mode this is the SMG server port and " - "defaults to --port + 10000.", - NS("serving"), - ] = None - # Env-only (SGLANG_GRPC_WORKER_THREADS); a field so the projection sees it. - grpc_worker_threads: A[Optional[int], Arg(no_cli=True), NS("serving")] = None - sidecar: A[ - Optional[str], - "Start a locally managed sidecar against the native gRPC server. " - "The selected module must expose main(argv) and read the resolved " - "native gRPC endpoint from SGLANG_GRPC_ENDPOINT. Requires --grpc-port " - "or SGLANG_GRPC_PORT.", - NS("serving"), - ] = None - sidecar_args: A[ - Optional[List[str]], - Arg( - help="JSON array passed to the selected sidecar module's " - "main(argv) function. --sidecar-shutdown-timeout SECONDS is " - "consumed by SGLang.", - type_parser=json_list_type, - ), - NS("serving"), - ] = None - skip_server_warmup: A[bool, "If set, skip warmup.", NS("serving")] = False - warmups: A[ - Optional[str], - "Specify custom warmup functions (csv) to run before server starts eg. --warmups=warmup_name1,warmup_name2 will run the functions `warmup_name1` and `warmup_name2` specified in warmup.py before the server starts listening for requests", - NS("serving"), - ] = None - enable_http2: A[ - bool, - "Use Granian instead of Uvicorn as the ASGI server, enabling HTTP/1.1 and HTTP/2 auto-negotiation. Clients may use h2c (cleartext HTTP/2) or plain HTTP/1.1. Requires 'pip install sglang[http2]'.", - NS("serving"), - ] = False - http2_max_concurrent_streams: A[ - int, - "Maximum number of concurrent streams advertised on each HTTP/2 " - "connection (1 to 2^32 - 1). Only applies with --enable-http2.", - NS("serving"), - ] = 200 - http2_initial_connection_window_size: A[ - int, - "Initial connection-level HTTP/2 receive window in bytes (1024 to " - "2^31 - 1). Only applies with --enable-http2.", - NS("serving"), - ] = 1024 * 1024 - - # ------------------------------------------------------------------------- - # SSL/TLS - # ------------------------------------------------------------------------- - ssl_keyfile: A[ - Optional[str], "The file path to the SSL key file.", NS("serving") - ] = None - ssl_certfile: A[ - Optional[str], "The file path to the SSL certificate file.", NS("serving") - ] = None - ssl_ca_certs: A[Optional[str], "The CA certificates file.", NS("serving")] = None - ssl_keyfile_password: A[ - Optional[str], "The password to decrypt the SSL keyfile.", NS("serving") - ] = None - enable_ssl_refresh: A[ - bool, - "Enable automatic SSL certificate hot-reloading when cert/key files change on disk. Requires --ssl-certfile and --ssl-keyfile.", - NS("serving"), - ] = False - - # ------------------------------------------------------------------------- - # API related - # ------------------------------------------------------------------------- - api_key: A[ - Optional[str], - "Set API key of the server. It is also used in the OpenAI API compatible server.", - NS("serving"), - ] = None - admin_api_key: A[ - Optional[str], - "Set admin API key for sensitive management endpoints (e.g. /clear_hicache_storage_backend). When set, admin endpoints require this key and do NOT accept --api-key.", - NS("serving"), - ] = None - served_model_name: A[ - Optional[str], - "Override the model name returned by the v1/models endpoint in OpenAI API server.", - NS("serving"), - ] = None - weight_version: A[ - str, - "Version identifier for the model weights. Defaults to 'default' if not specified.", - NS("serving"), - ] = "default" - chat_template: A[ - Optional[str], - "The buliltin chat template name or the path of the chat template file. This is only used for OpenAI-compatible API server.", - NS("serving"), - ] = None - hf_chat_template_name: A[ - Optional[str], - "When the HuggingFace tokenizer has multiple chat templates (e.g., 'default', 'tool_use', 'rag'), specify which named template to use. If not set, the first available template is used.", - NS("serving"), - ] = None - completion_template: A[ - Optional[str], - "The buliltin completion template name or the path of the completion template file. This is only used for OpenAI-compatible API server. only for code completion currently.", - NS("serving"), - ] = None - file_storage_path: A[ - str, "The path of the file storage in backend.", NS("serving") - ] = "sglang_storage" - enable_cache_report: A[ - bool, - "Return number of cached tokens in usage.prompt_tokens_details for each openai request.", - NS("serving"), - ] = False - return_input_ids: A[ - bool, - "Return prompt (input) token ids on the response-level sglext extension for every chat completion request, as if return_input_ids_in_sglext were set on the request.", - NS("serving"), - ] = False - return_output_ids: A[ - bool, - "Return sampled output token ids on the response-level sglext extension for every chat completion request, as if return_output_ids_in_sglext were set on the request.", - NS("serving"), - ] = False - reasoning_parser: A[Optional[str], NS("serving")] = None - default_chat_template_kwargs: A[ - Optional[Dict[str, Any]], - Arg( - help="Default chat template kwargs applied to every request when not " - "overridden per-request. Keys must match what the model's chat template " - "expects (e.g. enable_thinking, thinking, reasoning_effort). Per-request " - "chat_template_kwargs takes precedence.", - type_parser=json.loads, - ), - NS("serving"), - ] = None - strip_thinking_cache: A[ - bool, - "Skip caching reasoning-model output (thinking + answer) in the radix tree on finish; keep only the prompt prefix. Opt-in: changes cache contents.", - NS("serving"), - ] = False - enable_strict_thinking: A[ - bool, - "Enable strict token filtering during the thinking phase. Blocks model-specific excluded tokens (e.g., tool call markers) during reasoning. Requires a grammar backend that supports token filtering.", - NS("serving"), - ] = False - tool_call_parser: A[Optional[str], NS("serving")] = None - tool_server: A[ - Optional[str], - "Either 'demo' or a comma-separated list of tool server urls to use for the model. If not specified, no tool server will be used.", - NS("serving"), - ] = None - sampling_defaults: A[ - str, - Arg( - help="Where to get default sampling parameters. 'openai' uses SGLang/OpenAI defaults (temperature=1.0, top_p=1.0, etc.). 'model' uses the model's generation_config.json to get the recommended sampling parameters if available. Default is 'model'.", - choices=["openai", "model"], - ), - NS("serving"), - ] = "model" - asr_max_buffer_seconds: A[ - int, - "Maximum seconds of PCM audio the streaming ASR WebSocket handler will accumulate before closing the session with a buffer_overflow error. Guards against OOM when a client streams audio faster than inference can consume it. Default 60s.", - NS("serving"), - ] = 60 - asr_max_concurrent_sessions: A[ - int, - "Maximum number of concurrent realtime ASR WebSocket sessions served by /v1/realtime. New connections beyond this cap are accepted, sent an error{code:too_many_sessions} frame, and closed. Default 32.", - NS("serving"), - ] = 32 - preferred_sampling_params: A[ - Optional[str], - Arg( - help="json-formatted sampling settings that will be returned in /get_model_info", - type_parser=json.loads, - ), - NS("serving"), - ] = None - allow_auto_truncate: A[ - bool, - "Allow automatically truncating requests that exceed the maximum input length instead of returning an error.", - NS("serving"), - ] = False - - # ------------------------------------------------------------------------- - # Streaming - # ------------------------------------------------------------------------- - stream_interval: A[ - int, - "The interval (or buffer size) for streaming in terms of the token length. A smaller value makes streaming smoother, while a larger value makes the throughput higher", - NS("serving"), - ] = 1 - batch_notify_size: A[ - int, - "Number of streaming notifications to batch before yielding to the event loop. Reduces asyncio wakeup overhead under high concurrency.", - NS("serving"), - ] = 16 - stream_response_default_include_usage: A[ - bool, - "Include usage in every streaming response (even when stream_options is not specified).", - NS("serving"), - ] = False - incremental_streaming_output: A[ - bool, "Whether to output as a sequence of disjoint segments.", NS("serving") - ] = False - enable_streaming_session: A[ - bool, - "Enable streaming session mode and StreamingSession wrapper.", - NS("serving"), - ] = False - enable_session_radix_cache: A[ - bool, - "Track per-session references on UnifiedRadixCache KV: eviction consumes unreferenced entries before referenced ones, and closing a session only dereferences its KV.", - NS("memory"), - ] = False - - # ------------------------------------------------------------------------- - # Logging, metrics, and tracing - # ------------------------------------------------------------------------- - log_level: A[str, "The logging level of all loggers.", NS("observability")] = "info" - log_level_http: A[ - Optional[str], - "The logging level of HTTP server. If not set, reuse --log-level by default.", - NS("observability"), - ] = None - log_requests: A[ - bool, - "Log metadata, inputs, outputs of all requests. The verbosity is decided by --log-requests-level", - NS("observability"), - ] = False - log_requests_level: A[ - int, - Arg( - help="0: Log metadata (no sampling parameters). 1: Log metadata and sampling parameters. 2: Log metadata, sampling parameters and partial input/output. 3: Log every input/output.", - choices=[0, 1, 2, 3], - ), - NS("observability"), - ] = 2 - log_requests_format: A[ - str, - Arg( - help="Format for request logging: 'text' (human-readable) or 'json' (structured)", - choices=["text", "json"], - ), - NS("observability"), - ] = "text" - log_requests_target: A[ - Optional[List[str]], - "Target(s) for request logging: 'stdout' and/or directory path(s) for file output. Can specify multiple targets, e.g., '--log-requests-target stdout /my/path'. ", - NS("observability"), - ] = None - uvicorn_access_log_exclude_prefixes: A[ - List[str], - Arg( - help="Exclude uvicorn access logs whose request path starts with any of these prefixes. Defaults to empty (disabled). Example: --uvicorn-access-log-exclude-prefixes /metrics /health", - nargs="*", - ), - NS("observability"), - ] = dataclasses.field(default_factory=list) - crash_dump_folder: A[ - Optional[str], - "Folder path to dump requests from the last 5 min before a crash (if any). If not specified, crash dumping is disabled.", - NS("observability"), - ] = None - show_time_cost: A[bool, "Show time cost of custom marks.", NS("observability")] = ( - False - ) - enable_metrics: A[bool, "Enable log prometheus metrics.", NS("observability")] = ( - False - ) - smg_http_sidecar_port: A[ - Optional[int], - Arg( - help="Port for the HTTP sidecar server in legacy SMG gRPC mode (--smg-grpc-mode). Serves Prometheus metrics and profiling endpoints. Defaults to --port + 1. Not used in HTTP mode.", - aliases=["--grpc-http-sidecar-port"], - ), - NS("observability"), - ] = None - enable_mfu_metrics: A[ - bool, "Enable estimated MFU-related prometheus metrics.", NS("observability") - ] = False - enable_metrics_for_all_schedulers: A[ - bool, - "Enable --enable-metrics-for-all-schedulers when you want schedulers on all TP ranks (not just TP 0) to record request metrics separately. This is especially useful when dp_attention is enabled, as otherwise all metrics appear to come from TP 0.", - NS("observability"), - ] = False - load_snapshot_publish_interval: A[ - int, - "Publish load snapshot to shared memory every N decode iterations. Prefill and idle always publish immediately.", - NS("observability"), - ] = 15 - tokenizer_metrics_custom_labels_header: A[ - str, - "Specify the HTTP header for passing custom labels for tokenizer metrics.", - NS("observability"), - ] = "x-custom-labels" - tokenizer_metrics_allowed_custom_labels: A[ - Optional[List[str]], - "The custom labels allowed for tokenizer metrics. The labels are specified via a dict in '--tokenizer-metrics-custom-labels-header' field in HTTP requests, e.g., {'label1': 'value1', 'label2': 'value2'} is allowed if '--tokenizer-metrics-allowed-custom-labels label1 label2' is set.", - NS("observability"), - ] = None - extra_metric_labels: A[ - Optional[Dict[str, str]], - Arg( - help='The custom labels for metrics. e.g. \'{"label1": "value1", "label2": "value2"}\'', - type_parser=json.loads, - ), - NS("observability"), - ] = None - bucket_time_to_first_token: A[ - Optional[List[float]], - "The buckets of time to first token, specified as a list of floats.", - NS("observability"), - ] = None - bucket_inter_token_latency: A[ - Optional[List[float]], - "The buckets of inter-token latency, specified as a list of floats.", - NS("observability"), - ] = None - bucket_e2e_request_latency: A[ - Optional[List[float]], - "The buckets of end-to-end request latency, specified as a list of floats.", - NS("observability"), - ] = None - prompt_tokens_buckets: A[ - Optional[List[str]], - "The buckets rule of prompt tokens. " - "Supports 3 rule types: 'default' uses predefined buckets; 'tse ' " - "generates two sides exponential distributed buckets (e.g., 'tse 1000 2 8' generates buckets " - "[984.0, 992.0, 996.0, 998.0, 1000.0, 1002.0, 1004.0, 1008.0, 1016.0]).); 'custom " - " ...' uses custom bucket values (e.g., 'custom 10 50 100 500').", - NS("observability"), - ] = None - generation_tokens_buckets: A[ - Optional[List[str]], - "The buckets rule for generation tokens histogram. " - "Supports 3 rule types: 'default' uses predefined buckets; 'tse ' " - "generates two sides exponential distributed buckets (e.g., 'tse 1000 2 8' generates buckets " - "[984.0, 992.0, 996.0, 998.0, 1000.0, 1002.0, 1004.0, 1008.0, 1016.0]).); 'custom " - " ...' uses custom bucket values (e.g., 'custom 10 50 100 500').", - NS("observability"), - ] = None - gc_warning_threshold_secs: A[ - float, - "The threshold for long GC warning. If a GC takes longer than this, a warning will be logged. Set to 0 to disable.", - NS("observability"), - ] = 0.0 - decode_log_interval: A[ - int, - "The log and metrics reporting interval (in decode iterations) for decode batches.", - NS("observability"), - ] = 40 - enable_request_time_stats_logging: A[ - bool, "Enable per request time stats logging", NS("observability") - ] = False - kv_events_config: A[ - Optional[str], - "Config in json format for NVIDIA dynamo KV event publishing. Publishing will be enabled if this flag is used. Runtime-load publishing for load-aware routers is a separate opt-in; see --load-publish-endpoint.", - NS("observability"), - ] = None - load_publish_endpoint: A[ - Optional[str], - "Opt in to the runtime-load PUB socket that load-aware routers subscribe to. Off by default (unset or 'off'). Use 'auto' to reserve the dp_size ports packed after the --kv-events-config range, or a wildcard-host TCP address (e.g. tcp://*:6000) to place it explicitly; rank r binds port+r and /server_info advertises the base under the kv_events block. Requires --kv-events-config to describe a publisher (routers discover the base through /server_info); startup fails if this is set without one, is not bindable, or overlaps the KV range. Note: 'auto' reserves 2*dp_size ports from the KV base — space co-hosted engines accordingly. The router-facing update cadence follows --load-snapshot-publish-interval (shared to avoid double-collecting the snapshot), so a large value there also staleness-caps this feed.", - NS("observability"), - ] = None - enable_forward_pass_metrics: A[ - bool, - "Enable per-iteration forward pass metrics via ZMQ IPC. External consumers (e.g. Dynamo planner) subscribe to the IPC endpoint exposed in server_args.forward_pass_metrics_ipc_name.", - NS("observability"), - ] = False - forward_pass_metrics_worker_id: A[ - str, Arg(help=argparse.SUPPRESS), NS("observability") - ] = "" - forward_pass_metrics_ipc_name: A[ - Optional[str], Arg(help=argparse.SUPPRESS), NS("observability") - ] = None - enable_trace: A[bool, "Enable opentelemetry trace", NS("observability")] = False - trace_modules: A[ - str, - "Select the components to trace. Available options are 'request' and 'mooncake'. Format: ,,...", - NS("observability"), - ] = "request" - otlp_traces_endpoint: A[ - str, - "Config opentelemetry collector endpoint if --enable-trace is set. format: :", - NS("observability"), - ] = "localhost:4317" - # RequestMetricsExporter configuration - export_metrics_to_file: A[ - bool, - "Export performance metrics for each request to local file (e.g. for forwarding to external systems).", - NS("observability"), - ] = False - export_metrics_to_file_dir: A[ - Optional[str], - "Directory path for writing performance metrics files (required when --export-metrics-to-file is enabled).", - NS("observability"), - ] = None - # Class-level DI for the five *MetricsCollector classes. Maps collector role - # (one of: "scheduler", "tokenizer", "storage", "radix_cache", "expert_dispatch") - # to a subclass of the matching base collector. The five instantiation sites - # read from this map and fall back to the base class. Class-object only (no - # CLI surface) since this exists for embedded use cases that pass a Python - # class directly. Default None preserves existing behavior. - stat_loggers: A[Optional[Dict[str, type]], NS("observability")] = None - - # ------------------------------------------------------------------------- - # Constrained decoding - # ------------------------------------------------------------------------- - constrained_json_whitespace_pattern: A[ - Optional[str], - "(outlines and llguidance backends only) Regex pattern for syntactic whitespaces allowed in JSON constrained output. For example, to allow the model generate consecutive whitespaces, set the pattern to [\n\t ]*", - NS("serving"), - ] = None - constrained_json_disable_any_whitespace: A[ - bool, - "(xgrammar and llguidance backends only) Enforce compact representation in JSON constrained output.", - NS("serving"), - ] = False - - # ------------------------------------------------------------------------- - # Kernel backend - # ------------------------------------------------------------------------- - attention_backend: A[ - Optional[str], - Arg( - help="Choose the kernels for attention layers.", - choices=ATTENTION_BACKEND_CHOICES, - resolvable=True, - ), - NS("exec.kernel"), - ] = None - decode_attention_backend: A[ - Optional[str], - Arg( - help="Choose the kernels for decode attention layers (have priority over --attention-backend).", - choices=ATTENTION_BACKEND_CHOICES, - resolvable=True, - ), - NS("exec.kernel"), - ] = None - enable_lean_attention: A[ - Optional[bool], - "Enable Lean (Work-Centric) Attention decode kernel for long-context serving. When None (default), uses auto-gate that activates Lean for long contexts and falls back to standard kernel for short contexts. Set to True to force enable, False to force disable.", - NS("exec.kernel"), - ] = None - prefill_attention_backend: A[ - Optional[str], - Arg( - help="Choose the kernels for prefill attention layers (have priority over --attention-backend).", - choices=ATTENTION_BACKEND_CHOICES, - resolvable=True, - ), - NS("exec.kernel"), - ] = None - sampling_backend: A[ - Optional[str], - Arg( - help="Choose the kernels for sampling layers.", - no_cli=True, - resolvable=True, - ), - NS("exec.kernel"), - ] = None - grammar_backend: A[ - Optional[str], - Arg( - help="Choose the backend for grammar-guided decoding.", - choices=GRAMMAR_BACKEND_CHOICES, - ), - NS("exec.kernel"), - ] = None - radix_cache_backend: A[ - Optional[str], - "Name of a radix-cache backend previously registered via register_radix_cache_backend. Omit this flag to use the built-in default cache selection chain.", - NS("memory"), - ] = None - mm_attention_backend: A[ - Optional[str], - Arg( - help="Set multimodal attention backend.", - choices=[ - "sdpa", - "fa3", - "fa4", - "triton_attn", - "ascend_attn", - "aiter_attn", - "flashinfer_cudnn", - "amx_attn", - "xpu_attn", - ], - ), - NS("mm"), - ] = None - fp8_gemm_runner_backend: A[ - str, - Arg( - help="Choose the runner backend for Blockwise FP8 GEMM operations. Options: 'auto' (default, auto-selects based on hardware; MXFP8 dense picks flashinfer_cutedsl on SM100/SM103 and FlashInfer CUTLASS on other supported Blackwell GPUs), 'deep_gemm' (JIT-compiled; enabled by default on NVIDIA Hopper (SM90) and Blackwell (SM100) when DeepGEMM is installed), 'flashinfer_trtllm' (optimal for Blackwell and low-latency), 'flashinfer_cutlass' (FlashInfer CUTLASS groupwise FP8 GEMM), 'flashinfer_cutedsl' (FlashInfer CuTe DSL MXFP8 GEMM on SM100/SM103), 'flashinfer_deepgemm' (Hopper SM90 only; uses swapAB optimization for small M dimensions in decoding), 'cutlass' (optimal for SM120 GPUs), 'triton' (fallback, widely compatible), 'aiter' (ROCm only). ", - cli_name="--fp8-gemm-backend", - choices=FP8_GEMM_RUNNER_BACKEND_CHOICES, - resolvable=True, - ), - NS("exec.kernel"), - ] = "auto" - fp4_gemm_runner_backend: A[ - str, - Arg( - help="Choose the runner backend for NVFP4 GEMM operations. Options: 'auto' (default; selects flashinfer_cutedsl on SM100, marlin on SM80-SM90, flashinfer_cutlass otherwise (including SM120)), 'flashinfer_cutlass' (FlashInfer CUTLASS backend), 'flashinfer_cudnn' (FlashInfer cuDNN backend, optimal on CUDA 13+ with cuDNN 9.15+), 'flashinfer_cutedsl' (FlashInfer CuTe DSL backend), 'flashinfer_trtllm' (FlashInfer TensorRT-LLM backend, requires different weight preparation with shuffling), 'marlin' (weight-only W4A16 fallback for SM80+). ", - cli_name="--fp4-gemm-backend", - choices=FP4_GEMM_RUNNER_BACKEND_CHOICES, - resolvable=True, - ), - NS("exec.kernel"), - ] = "auto" - bf16_gemm_backend: A[ - str, - Arg( - help="Choose the backend for unquantized BF16 GEMM operations. Options: 'auto' (default; selects 'cutedsl' on SM10x GPUs, except deterministic inference selects 'torch'; otherwise uses cuBLAS via torch.nn.functional.linear), 'cutedsl' (SGLang JIT CuTe DSL TGV BF16 GEMM on SM10x; dispatches between the allowlisted low-M Split-K kernel, the CuTe DSL kernel, and cuBLAS; set SGLANG_ENABLE_BF16_SPLITK_GEMM=0 to disable Split-K), 'flashinfer_pr4266' (legacy compatibility alias for the optimized CuTe DSL path), 'gemv', 'torch' (always uses cuBLAS via torch.nn.functional.linear).", - cli_name="--bf16-gemm-backend", - choices=["auto", "cutedsl", "flashinfer_pr4266", "gemv", "torch"], - ), - NS("exec.kernel"), - ] = "auto" - dsa_prefill_backend: A[ - Optional[str], - Arg( - help="DSA (DeepSeek Sparse Attention) prefill backend. If not specified, auto-detects based on hardware and kv_cache_dtype.", - choices=[ - "flashmla_sparse", - "flashmla_sparse_q8", - "flashmla_kv", - "flashmla_auto", - "flashinfer_sparse_mla", - "fa3", - "tilelang", - "aiter", - "trtllm", - ], - resolvable=True, - ), - NS("exec.kernel"), - ] = None - dsv4_prefill_backend: A[ - str, - Arg( - help=( - "DeepSeek-V4 sparse prefill backend. 'auto' and " - "'flashmla_sparse' use the existing BF16 sparse prefill path; " - "'flashmla_sparse_q8' enables the Q8KV8 sparse prefill path." - ), - choices=["auto", "flashmla_sparse", "flashmla_sparse_q8"], - ), - NS("exec.kernel"), - ] = "auto" - dsa_decode_backend: A[ - Optional[str], - Arg( - help="DSA (DeepSeek Sparse Attention) decode backend. If not specified, auto-detects based on hardware and kv_cache_dtype.", - choices=[ - "flashmla_sparse", - "flashmla_sparse_q8", - "flashmla_kv", - "flashmla_auto", - "flashinfer_sparse_mla", - "fa3", - "tilelang", - "aiter", - "trtllm", - ], - resolvable=True, - ), - NS("exec.kernel"), - ] = None - dsa_paged_mqa_logits_backend: A[ - str, - Arg( - help="DSA indexer paged MQA logits kernel backend. Options: 'auto' (default; DeepGEMM on CUDA, aiter on ROCm), 'deepgemm', 'cutedsl' (CuTe DSL kernel, SM 100 (Blackwell) only; wins at low batch size and long context), 'aiter' (ROCm only).", - choices=["auto", "deepgemm", "cutedsl", "aiter"], - ), - NS("exec.kernel"), - ] = "auto" - dsa_topk_backend: A[ - str, - Arg( - help="DSA indexer top-k backend for the target model. Options: 'sgl-kernel', 'torch', 'flashinfer'. The 'torch' backend currently requires SGLANG_DSA_FUSE_TOPK=false.", - choices=["sgl-kernel", "torch", "flashinfer"], - resolvable=True, - ), - NS("exec.kernel"), - ] = "sgl-kernel" - disable_flashinfer_autotune: A[ - bool, "Disable FlashInfer autotuning.", NS("exec.kernel") - ] = False - flashinfer_autotune_skip_ops: A[ - Optional[List[str]], - Arg( - help=( - "FlashInfer custom-op identifiers to skip during autotuning. " - "Skipped ops use FlashInfer's heuristic fallback. SGLang " - "temporarily skips mxfp8_gemm by default due to an IMA." - ), - nargs="+", - ), - NS("exec.kernel"), - ] = None - mamba_backend: A[ - str, - Arg( - help="Choose the kernel backend for Mamba SSM operations. Default is 'triton'. Options: 'triton' (default), 'flashinfer' (requires FlashInfer with Mamba support).", - choices=["triton", "flashinfer"], - ), - NS("exec.mamba"), - ] = "triton" - - # ------------------------------------------------------------------------- - # Cuda graphs - # ------------------------------------------------------------------------- - cuda_graph_config: A[ - Optional[CudaGraphConfig], - Arg( - help='Per-phase CUDA graph settings as JSON, e.g. \'{"decode":{"backend":"full","max_bs":256},"prefill":{"backend":"tc_piecewise","tc_compiler":"eager"}}\'. Allowed backends per phase: full, breakable, tc_piecewise, disabled (full is decode-only). JSON wins over the per-phase --cuda-graph-* convenience flags and over legacy flags.', - type_parser=parse_cuda_graph_config_arg, - ), - NS("exec.graph"), - ] = None - cuda_graph_backend_decode: A[ - Optional[Literal["full", "breakable", "tc_piecewise", "disabled"]], - Arg( - help="Backend for the decode phase. Folds into cuda_graph_config[decode].backend.", - choices=Backend.ALL, - ), - NS("exec.graph"), - ] = None - cuda_graph_backend_prefill: A[ - Optional[Literal["full", "breakable", "tc_piecewise", "disabled"]], - Arg( - help="Backend for the prefill phase. Folds into cuda_graph_config[prefill].backend.", - choices=Backend.ALL, - ), - NS("exec.graph"), - ] = None - cuda_graph_max_bs_decode: A[ - Optional[int], - "Maximum batch size captured for the decode cuda graph.", - NS("exec.graph"), - ] = None - cuda_graph_max_bs_prefill: A[ - Optional[int], - "Maximum batch size captured for the prefill cuda graph.", - NS("exec.graph"), - ] = None - cuda_graph_bs_decode: A[ - Optional[List[int]], - "Explicit list of batch sizes to capture for the decode cuda graph.", - NS("exec.graph"), - ] = None - cuda_graph_bs_prefill: A[ - Optional[List[int]], - "Explicit list of batch sizes to capture for the prefill cuda graph.", - NS("exec.graph"), - ] = None - cuda_graph_tc_compiler: A[ - Optional[Literal["eager", "inductor"]], - "Compiler used by the tc_piecewise backend (currently only the prefill phase consumes it).", - NS("exec.graph"), - ] = None - disable_prefill_cuda_graph: A[ - bool, - "Disable the prefill-phase CUDA graph. Convenience for --cuda-graph-backend-prefill=disabled.", - NS("exec.graph"), - ] = False - disable_decode_cuda_graph: A[ - bool, - "Disable the decode-phase CUDA graph. Convenience for --cuda-graph-backend-decode=disabled.", - NS("exec.graph"), - ] = False - disable_cuda_graph: A[bool, Arg(no_cli=True), NS("exec.graph")] = False - disable_cuda_graph_padding: A[ - bool, - "Disable cuda graph when padding is needed. Still uses cuda graph when padding is not needed.", - NS("exec.graph"), - ] = False - enable_profile_cuda_graph: A[ - bool, "Enable profiling of cuda graph capture.", NS("exec.graph") - ] = False - enable_cudagraph_gc: A[ - bool, - "Enable garbage collection during CUDA graph capture. If disabled (default), GC is frozen during capture to speed up the process.", - NS("exec.graph"), - ] = False - debug_cuda_graph: A[ - bool, - "Enable debug/eager mode for CUDA graph using breakable CUDA graph. When enabled, graph breaks are inserted so every operation runs eagerly while still going through the CUDA graph capture / replay path. Useful for debugging CUDA graph capture / replay issues.", - NS("exec.graph"), - ] = False - - # ------------------------------------------------------------------------- - # Communication and kernels - # ------------------------------------------------------------------------- - enable_layerwise_nvtx_marker: A[ - bool, - "Enable layerwise NVTX profiling annotations for the model.", - NS("exec.comm"), - ] = False - enable_nccl_nvls: A[ - bool, - "Enable NCCL NVLS for prefill heavy requests when available.", - NS("exec.comm"), - ] = False - enable_symm_mem: A[ - bool, - Arg( - help="Enable NCCL symmetric memory for fast collectives.", - resolvable=True, - ), - NS("exec.comm"), - ] = False - triton_attention_reduce_in_fp32: A[ - bool, - "Cast the intermediate attention results to fp32 to avoid possible crashes related to fp16." - "This only affects Triton attention kernels.", - NS("exec.kernel"), - ] = False - triton_attention_num_kv_splits: A[ - int, - "The number of KV splits in flash decoding Triton kernel. Larger value is better in longer context scenarios. The default value is 8.", - NS("exec.kernel"), - ] = 8 - triton_attention_split_tile_size: A[ - Optional[int], - "The size of split KV tile in flash decoding Triton kernel. Used for deterministic inference.", - NS("exec.kernel"), - ] = None - flashinfer_mla_disable_ragged: A[ - bool, - "Not using ragged prefill wrapper when running flashinfer mla", - NS("exec.kernel"), - ] = False - enable_fused_qk_norm_rope: A[ - bool, - "Enable fused qk normalization and rope rotary embedding.", - NS("exec.kernel"), - ] = False - enable_precise_embedding_interpolation: A[ - bool, - "Enable corner alignment for resize of embeddings grid to ensure more accurate(but slower) evaluation of interpolated embedding values.", - NS("exec.kernel"), - ] = False - enable_fused_moe_sum_all_reduce: A[ - bool, "Enable fused moe triton and sum all reduce.", NS("exec.moe") - ] = False - enable_deepseek_v4_fp4_indexer: A[ - bool, - "Enable the experimental FP4 C4 indexer path for DeepSeek V4. Default keeps the existing indexer implementation.", - NS("exec.kernel"), - ] = False - disable_custom_all_reduce: A[ - bool, - Arg( - help="Disable the custom all-reduce kernel and fall back to NCCL.", - resolvable=True, - ), - NS("exec.comm"), - ] = False - enable_mscclpp: A[ - bool, - "Enable using mscclpp for small messages for all-reduce kernel and fall back to NCCL.", - NS("exec.comm"), - ] = False - enable_torch_symm_mem: A[ - bool, - "Enable using torch symm mem for all-reduce kernel and fall back to NCCL. Only supports CUDA device SM90 and above. SM90 supports world size 4, 6, 8. SM100 supports world size 6, 8.", - NS("exec.comm"), - ] = False - enable_scattered_sconv: A[ - bool, - "Inkling: replace the attention/MLP output all-reduce with a hidden-dimension reduce-scatter, run the channelwise output short convolution on the [T, H/P] shard, then all-gather before the residual add. This shards the convolution cache across tensor-parallel ranks without changing communication volume.", - NS("exec.comm"), - ] = False - pre_warm_nccl: A[ - bool, - "Pre-warm NCCL/RCCL communicators during startup to reduce P99 TTFT cold-start latency. Default: enabled for AMD/HIP (RCCL), disabled for NVIDIA/CUDA (NCCL).", - NS("exec.comm"), - ] = False - enable_quant_communications: A[ - Optional[bool], - "Enable INT8 quantization of TP communications (limited support).", - NS("exec.comm"), - ] = False - enable_flashinfer_allreduce_fusion: A[bool, Arg(no_cli=True), NS("exec.comm")] = ( - False - ) - enforce_disable_flashinfer_allreduce_fusion: A[ - bool, "Enforce disable FlashInfer allreduce fusion.", NS("exec.comm") - ] = False - flashinfer_allreduce_fusion_backend: A[ - Optional[Literal["auto", "trtllm", "mnnvl"]], - Arg( - help=( - "Enable FlashInfer allreduce fusion and choose backend. " - "Requires SM90 or SM10X NVIDIA GPUs. " - "Defaults to auto. " - "'auto': choose mnnvl on Blackwell (SM100/SM103) systems " - "(single- and multi-node) and trtllm on SM90 single-node systems. " - "'trtllm': available on single-node systems only. " - "'mnnvl': available on SM90 single-node systems and SM100/SM103 " - "single-node or multi-node systems via MNNVL fabric. " - "Fuses allreduce with Residual + RMSNorm for supported MoE models." - ), - resolvable=True, - ), - NS("exec.comm"), - ] = None - enable_aiter_allreduce_fusion: A[ - bool, - Arg(help="Enable Aiter AllReduce Fusion.", resolvable=True), - NS("exec.comm"), - ] = False - - # ------------------------------------------------------------------------- - # Torch compile - # ------------------------------------------------------------------------- - enable_torch_compile: A[ - bool, - "Optimize the model with torch.compile. Experimental feature.", - NS("exec.graph"), - ] = False - enable_torch_compile_debug_mode: A[ - bool, "Enable debug mode for torch compile", NS("exec.graph") - ] = False - torch_compile_max_bs: A[ - int, "Set the maximum batch size when using torch compile.", NS("exec.graph") - ] = 32 - # ------------------------------------------------------------------------- - # Speculative decoding - # ------------------------------------------------------------------------- - speculative_algorithm: A[ - Optional[str], - "Speculative algorithm. Builtins: EAGLE, EAGLE3, NEXTN, STANDALONE, NGRAM, DFLASH, DSPARK, UNO. Or any name registered via `SpeculativeAlgorithm.register`.", - NS("spec"), - ] = None - uno_lora_path: A[ - Optional[str], - "Path to the UNO draft LoRA checkpoint.", - NS("spec"), - ] = None - speculative_draft_model_path: A[ - Optional[str], - Arg( - help="The path of the draft model weights. This can be a local folder or a Hugging Face repo ID.", - aliases=["--speculative-draft-model"], - ), - NS("spec"), - ] = None - speculative_draft_model_revision: A[ - Optional[str], - "The specific draft model version to use. It can be a branch name, a tag name, or a commit id. If unspecified, will use the default version.", - NS("spec"), - ] = None - speculative_draft_load_format: A[ - Optional[str], - Arg( - help="The format of the draft model weights to load. If not specified, will use the same format as --load-format. Use 'dummy' to initialize draft model weights with random values for profiling.", - choices=LOAD_FORMAT_CHOICES, - ), - NS("spec"), - ] = None - speculative_num_steps: A[ - Optional[int], - "The number of steps sampled from draft model in Speculative Decoding.", - NS("spec"), - ] = None - speculative_eagle_topk: A[ - Optional[int], - "The number of tokens sampled from the draft model in eagle2 each step.", - NS("spec"), - ] = None - speculative_num_draft_tokens: A[ - Optional[int], - "The number of tokens sampled from the draft model in Speculative Decoding.", - NS("spec"), - ] = None - speculative_dflash_block_size: A[ - Optional[int], - "DFLASH only. Block size (verify window length). Alias of --speculative-num-draft-tokens for DFLASH.", - NS("spec"), - ] = None - speculative_dspark_block_size: A[ - Optional[int], - "DSPARK only. Draft block size gamma (number of proposed draft tokens). The verify window is gamma + 1, so this sets --speculative-num-draft-tokens = gamma + 1. Omit to auto-infer gamma from the draft checkpoint block_size.", - NS("spec"), - ] = None - speculative_dspark_sps_table_path: A[ - Optional[str], - "DSPARK only. Path to a pre-profiled SPS cost table (JSON) built offline with " - "sglang.benchmark.dspark_sps_profiler, consumed by the ragged-verify " - "scheduler (cap-accept / compact). Omit for an uninitialized flat " - "constant-SPS table: the budget degenerates to verify-all (zero throughput " - "gain by itself).", - NS("spec"), - ] = None - speculative_dspark_confidence_sts_path: A[ - Optional[str], - "DSPARK only. Optional path to a per-position STS (sequential temperature " - "scaling) calibration JSON, fit offline with sglang.benchmark.dspark_sts_fit. " - "Calibrates the confidence-head survival probabilities the ragged-verify " - "scheduler consumes. Omit to use identity (no calibration); losslessness is " - "unaffected either way.", - NS("spec"), - ] = None - speculative_dspark_align_verify_tokens_to_graph_tier: A[ - bool, - "DSPARK compact ragged-verify only. Fill the per-request verify lengths so " - "the total verify-token count reaches the cuda-graph tier the forward is " - "already padded to: round the dp-max scheduled total up to the captured " - "token bucket and let the top-k allocator admit that many real draft tokens " - "(confidence-ordered). This recovers the padding the forward pays for anyway " - "-- both the cuda-graph bucket round-up and the dp cross-rank max -- turning " - "it into extra real verification at the same step time. Off by default; when " - "off the schedule is byte-for-byte unchanged.", - NS("spec"), - ] = False - speculative_accept_threshold_single: A[ - float, - "Accept a draft token if its probability in the target model is greater than this threshold.", - NS("spec"), - ] = 1.0 - speculative_accept_threshold_acc: A[ - float, - "The accept probability of a draft token is raised from its target probability p to min(1, p / threshold_acc).", - NS("spec"), - ] = 1.0 - speculative_use_rejection_sampling: A[ - bool, - "Use rejection sampling for speculative decoding (requires topk=1).", - NS("spec"), - ] = False - speculative_token_map: A[ - Optional[str], "The path of the draft model's small vocab table.", NS("spec") - ] = None - speculative_attention_mode: A[ - str, - Arg( - help="Attention backend for speculative decoding operations (both target verify and draft extend). Can be one of 'prefill' (default) or 'decode'.", - choices=["prefill", "decode"], - resolvable=True, - ), - NS("spec"), - ] = "prefill" - speculative_draft_attention_backend: A[ - Optional[str], - Arg( - help="Attention backend for speculative decoding drafting.", - resolvable=True, - ), - NS("spec"), - ] = None - speculative_dsa_topk_backend: A[ - str, - Arg( - help="DSA indexer top-k backend for speculative draft workers. Options: 'sgl-kernel', 'torch', 'flashinfer'. The 'torch' backend currently requires SGLANG_DSA_FUSE_TOPK=false.", - choices=["sgl-kernel", "torch", "flashinfer"], - ), - NS("spec"), - ] = "sgl-kernel" - speculative_draft_kv_cache_dtype: A[ - Optional[str], - Arg( - help="KV cache dtype for the speculative draft model only. The draft pool is " - "allocated with one slot per target token (draft and target share a slot index " - "space), so for a small draft it can still rival the target pool: a 5-layer " - "DFLASH draft costs 10240 bytes/token in bf16. Setting fp8_e4m3 halves the draft " - "pool; the saving shows up as free device memory, so raise " - "--mem-fraction-static to convert it into KV capacity. Default follows " - "--kv-cache-dtype.", - choices=["auto", "fp8_e5m2", "fp8_e4m3", "bf16", "bfloat16"], - ), - NS("spec"), - ] = None - speculative_draft_window_size: A[ - Optional[int], - "Sliding window size for the draft model. Honored by Llama EAGLE-3 (`LlamaForCausalLMEagle3`) and DFLASH only; other EAGLE-3 backends (e.g. MLA-based drafters) silently ignore it. For Llama EAGLE-3, the drafter only attends to the most recent N keys (verifier hidden states + its own outputs); the verifier is unaffected. For DFLASH, the draft worker keeps a recent target-token window in its local KV cache (paged backends may retain up to one extra page on the left for alignment). Default is full attention/context.", - NS("spec"), - ] = None - speculative_moe_runner_backend: A[ - Optional[str], - Arg( - help="Choose the runner backend for MoE in speculative decoding.", - choices=MOE_RUNNER_BACKEND_CHOICES, - resolvable=True, - ), - NS("spec"), - ] = None - speculative_moe_a2a_backend: A[ - Optional[str], - Arg( - help="Choose the backend for MoE A2A in speculative decoding", - choices=[ - "none", - "deepep", - "mooncake", - "nixl", - "mori", - "ascend_fuseep", - "flashinfer", - "megamoe", - "deepep_v2", - "pplx", - "ascend_tp", - ], - resolvable=True, - ), - NS("spec"), - ] = None - speculative_draft_model_quantization: A[ - Optional[str], - Arg( - help="The quantization method for speculative model.", - choices=QUANTIZATION_CHOICES, - ), - NS("spec"), - ] = None - # Internal provenance used after the public draft quantization inherits the - # target value. It is a dataclass field so ServerArgs round-trips preserve - # whether the user explicitly set the draft option; it has no CLI surface. - _speculative_draft_quantization_explicitly_set: A[ - Optional[bool], Arg(no_cli=True), NS("spec") - ] = None - speculative_skip_dp_mlp_sync: A[ - bool, - "Skip the extra MLP sync that the scheduler performs before merging a new batch when speculative decoding + DP attention are both enabled.", - NS("spec"), - ] = False - enable_multi_layer_eagle: A[ - bool, - Arg( - help="Enable multi-layer Eagle speculative decoding.", - resolvable=True, - ), - NS("spec"), - ] = False - speculative_adaptive: A[ - bool, - "Enable adaptive speculative decoding that dynamically adjusts num_steps based on acceptance rate.", - NS("spec"), - ] = False - speculative_adaptive_config: A[ - Optional[str], - "Path to a JSON config file for adaptive speculative decoding tuning knobs.", - NS("spec"), - ] = None - - # Decoupled speculative decoding: draft and verify run as - # separate engines, currently connected by a ZMQ IPC mesh. - decoupled_spec_bind_endpoint: A[ - Optional[str], - "ZMQ endpoint this engine binds for its inbound channel in decoupled " - "speculative decoding (verifier: result PULL; drafter: control PULL).", - NS("disagg"), - ] = None - decoupled_spec_connect_endpoints: A[ - Optional[List[str]], - Arg( - help="Peer inbound (bind) endpoints to connect to, ordered by peer " - "rank, for decoupled speculative decoding.", - type_parser=json_list_type, - ), - NS("disagg"), - ] = None - decoupled_spec_rank: A[ - Optional[int], - "This engine's rank within its own role space (verifier-rank or " - "drafter-rank) for decoupled speculative decoding.", - NS("disagg"), - ] = None - decoupled_spec_role: A[ - Literal["null", "verifier", "drafter"], - "Role in decoupled speculative decoding: 'null' disables it, 'verifier' " - "runs the target/verify half, 'drafter' runs the draft half.", - NS("disagg"), - ] = "null" - spec_trace_dir: A[ - Optional[str], - "Directory to write decoupled speculative decoding trace files.", - NS("spec"), - ] = None - - # ------------------------------------------------------------------------- - # Speculative decoding (ngram) - # ------------------------------------------------------------------------- - speculative_ngram_min_bfs_breadth: A[ - int, - "The minimum breadth for BFS (Breadth-First Search) in ngram speculative decoding.", - NS("spec"), - ] = 1 - speculative_ngram_max_bfs_breadth: A[ - int, - "The maximum breadth for BFS (Breadth-First Search) in ngram speculative decoding.", - NS("spec"), - ] = 10 - speculative_ngram_match_type: A[ - Literal["BFS", "PROB"], "The match type for cache tree.", NS("spec") - ] = "BFS" - speculative_ngram_max_trie_depth: A[ - int, "The max trie depth for ngram speculative decoding.", NS("spec") - ] = 18 - speculative_ngram_capacity: A[ - int, "The cache capacity for ngram speculative decoding.", NS("spec") - ] = 10 * 1000 * 1000 - speculative_ngram_external_corpus_path: A[ - Optional[str], - "Path to an external JSONL corpus to pre-load into SAM at startup. Additional corpora can be added at runtime via POST /add_external_corpus.", - NS("spec"), - ] = None - speculative_ngram_external_sam_budget: A[ - int, - "Number of draft nodes reserved for the external SAM subtree in ngram speculative decoding.", - NS("spec"), - ] = 0 - speculative_ngram_external_corpus_max_tokens: A[ - int, - "Fail startup if the tokenized external ngram corpus exceeds this many tokens. Tune this based on your CPU memory budget.", - NS("spec"), - ] = 10000000 - - # ------------------------------------------------------------------------- - # Expert parallelism - # ------------------------------------------------------------------------- - ep_size: A[ - int, - Arg( - help="The expert parallelism size.", - aliases=["--expert-parallel-size", "--ep"], - resolvable=True, - ), - NS("parallel"), - ] = 1 - moe_a2a_backend: A[ - Literal[ - "none", - "deepep", - "mooncake", - "nixl", - "mori", - "ascend_fuseep", - "flashinfer", - "megamoe", - "deepep_v2", - "ascend_tp", - "pplx", - ], - Arg( - help="Choose the backend for MoE A2A.", - choices=[ - "none", - "deepep", - "mooncake", - "nixl", - "mori", - "ascend_fuseep", - "flashinfer", - "megamoe", - "deepep_v2", - "pplx", - "ascend_tp", - ], - resolvable=True, - ), - NS("exec.moe"), - ] = "none" - enable_w4a4_mxfp4_megamoe: A[ - bool, - "Enable the W4A4 MXFP4 MegaMoE path with DeepGEMM's " - "mxf4xmxf4 MMA type. Use with " - "--moe-a2a-backend megamoe.", - NS("exec.moe"), - ] = False - deepep_v2_mode: A[ - Literal["direct", "hybrid"], - "DeepEP v2 ElasticBuffer communication topology, fixed at server init: " - "`direct` (single-node NVLink) or `hybrid` (multi-node scale-out). " - "Layout/grouped-GEMM and the decode CUDA graph are chosen per batch by " - "inference phase, independent of this knob; not equivalent to DeepEP v1 " - "normal/low_latency.", - NS("exec.moe"), - ] = "direct" - moe_runner_backend: A[ - str, - Arg( - help="Choose the runner backend for MoE.", - choices=MOE_RUNNER_BACKEND_CHOICES, - resolvable=True, - ), - NS("exec.moe"), - ] = "auto" - flashinfer_mxfp4_moe_precision: A[ - Literal["default", "bf16", "fp8"], - "Choose the computation precision of flashinfer mxfp4 moe. " - "On SM90, `fp8` selects the Humming-style MXFP4-weight x FP8-activation " - "path introduced by FlashInfer #3738 and requires FlashInfer >= 0.6.18.", - NS("exec.moe"), - ] = "default" - deepep_mode: A[ - Literal["auto", "normal", "low_latency"], - "Select the mode when enable DeepEP or MoriEP MoE, could be `normal`, `low_latency` or `auto`. Default is `auto`, which means `low_latency` for decode batch and `normal` for prefill batch.", - NS("exec.moe"), - ] = "auto" - fuseep_mode: A[ - Literal[1, 2], - "Select the mode when enable Ascend FuseEP MoE, 1 -> dispatch_gmm_combine_decode is executed；2 -> dispatch_ffn_combine is executed (support hybrid deployment when 2).", - NS("exec.moe"), - ] = 2 - deepep_dispatcher_output_dtype: A[ - Literal["auto", "bf16", "fp8", "int8", "nvfp4"], - "Select DeepEP dispatcher output dtype", - NS("exec.moe"), - ] = "auto" - ep_num_redundant_experts: A[ - int, - "Allocate this number of redundant experts in expert parallel.", - NS("exec.moe"), - ] = 0 - ep_dispatch_algorithm: A[ - Optional[Literal["static", "dynamic", "fake", "lp"]], - "The algorithm to choose ranks for redundant experts in expert parallel.", - NS("exec.moe"), - ] = None - init_expert_location: A[str, "Initial location of EP experts.", NS("exec.moe")] = ( - "trivial" - ) - enable_eplb: A[bool, "Enable EPLB algorithm", NS("exec.moe")] = False - eplb_algorithm: A[str, "Chosen EPLB algorithm", NS("exec.moe")] = "auto" - eplb_rebalance_num_iterations: A[ - int, - "Number of iterations to automatically trigger a EPLB re-balance.", - NS("exec.moe"), - ] = 1000 - eplb_rebalance_layers_per_chunk: A[ - Optional[int], "Number of layers to rebalance per forward pass.", NS("exec.moe") - ] = None - eplb_min_rebalancing_utilization_threshold: A[ - float, - "Minimum threshold for GPU average utilization to trigger EPLB rebalancing. Must be in the range [0.0, 1.0].", - NS("exec.moe"), - ] = 1.0 - expert_distribution_recorder_mode: A[ - Optional[Literal["stat", "stat_approx", "per_pass", "per_token"]], - "Mode of expert distribution recorder.", - NS("exec.moe"), - ] = None - expert_distribution_recorder_buffer_size: A[ - Optional[int], - "Circular buffer size of expert distribution recorder. Set to -1 to denote infinite buffer.", - NS("exec.moe"), - ] = None - expert_balancedness_report_mode: A[ - Literal["off", "server_log", "prometheus", "both"], - "Where to report expert balancedness. Options: off, server_log, prometheus, both.", - NS("exec.moe"), - ] = "off" - deepep_config: A[ - Optional[str], - "Tuned DeepEP config suitable for your own cluster. It can be either a string with JSON content or a file path.", - NS("exec.moe"), - ] = None - moe_dense_tp_size: A[ - Optional[int], - Arg( - help="TP size for MoE dense MLP layers. This flag is useful when, with large TP size, there are errors caused by weights in MLP layers having dimension smaller than the min dimension GEMM supports.", - resolvable=True, - ), - NS("parallel"), - ] = None - elastic_ep_backend: A[ - Literal[None, "mooncake", "nixl"], - Arg( - help="Specify the collective communication backend for elastic EP. Supports 'mooncake' and 'nixl'.", - choices=["none", "mooncake", "nixl"], - ), - NS("exec.moe"), - ] = None - enable_elastic_expert_backup: A[ - bool, "Enable elastic expert backup feature.", NS("exec.moe") - ] = False - mooncake_ib_device: A[ - Optional[str], - "The InfiniBand devices for Mooncake Backend transfer, accepts multiple comma-separated devices (e.g., --mooncake-ib-device mlx5_0,mlx5_1). Default is None, which triggers automatic device detection when Mooncake Backend is enabled.", - NS("exec.moe"), - ] = None - enable_waterfill: A[ - bool, - "Enable Waterfill: dispatch the fused shared expert as an extra routed expert slot to the least-loaded EP rank. Supports DeepEP and MegaMOE MoE A2A backends, implicitly enables shared-expert fusion, and supports --deepep-mode auto, normal, or low_latency when used with DeepEP. Use auto or low_latency for production DeepEP decode so CUDA graph remains enabled. Supported on DeepSeek-V3/R1 with EP >= 2.", - NS("exec.moe"), - ] = False - ep_join_mode: A[ - Optional[Literal["scale", "recover"]], - Arg( - help="Join mode for elastic EP. 'recover' rejoins an existing slot after a fault. 'scale' joins as a new rank beyond the original group size and requires --node-rank 1.", - cli_name="--elastic-ep-join-mode", - choices=["scale", "recover"], - ), - NS("exec.moe"), - ] = None - ep_join_rank_offset: A[ - int, - Arg( - help=( - "Global rank offset of an elastic EP joining group. Scale " - "joiners must set this to the current effective EP size." - ), - cli_name="--elastic-ep-join-rank-offset", - ), - NS("parallel"), - ] = 0 - elastic_ep_initial_size: A[ - Optional[int], - "EP size used to define the immutable per-rank expert storage layout. " - "Scale joiners must use the primary deployment's launch-time EP size.", - NS("parallel"), - ] = None - max_ep_size: A[ - Optional[int], - "Maximum EP size the server can scale to at runtime. Pre-allocates active-rank state and backend buffers to this size. Defaults to the launch-time world size.", - NS("parallel"), - ] = None - elastic_ep_scale_timeout: A[ - float, - "Timeout in seconds for a pending elastic EP scale operation.", - NS("exec.moe"), - ] = 600 - elastic_ep_rejoin: A[ - bool, "[Deprecated] Alias for --elastic-ep-join-mode recover.", NS("exec.moe") - ] = False - disable_flashinfer_cutlass_moe_fp4_allgather: A[ - bool, - "Disables quantize before all-gather for flashinfer cutlass moe.", - NS("exec.moe"), - ] = False - disable_shared_experts_fusion: A[ - bool, - Arg( - help="Disable the built-in shared experts fusion optimization for DeepSeek V3/R1. Note: Waterfill (--enable-waterfill) routes the shared expert as an extra MoE slot, so the shared expert is not separated from the MoE path when Waterfill is enabled.", - resolvable=True, - ), - NS("exec.moe"), - ] = False - enforce_shared_experts_fusion: A[ - bool, - "Enforce shared experts fusion even when it would normally be disabled (e.g. under DeepEP). Mutually exclusive with --disable-shared-experts-fusion.", - NS("exec.moe"), - ] = False - - # ------------------------------------------------------------------------- - # Mamba cache and linear attn - # ------------------------------------------------------------------------- - max_mamba_cache_size: A[ - Optional[int], "The maximum size of the mamba cache.", NS("schedule") - ] = None - mamba_ssm_dtype: A[ - Optional[str], - Arg( - help="The data type of the SSM states in mamba cache. If not set, will be read from model config (mamba_ssm_dtype).", - choices=["float32", "bfloat16", "float16"], - ), - NS("exec.mamba"), - ] = None - mamba_max_states_per_path: A[ - int, - "Maximum number of cached Mamba states retained per root-to-tail path " - "(-1 means unlimited). When enabled, after each insert the shallowest eligible " - "interior states beyond the cap are removed while their full KV remains. " - "Tail, fork, and locked nodes are preserved. Must be -1 or a positive integer.", - NS("exec.mamba"), - ] = -1 - enable_mamba_cache_stochastic_rounding: A[ - bool, - "Enable stochastic rounding when writing FP16 Mamba SSM cache states. Requires --mamba-ssm-dtype float16 and CUDA. With --mamba-backend triton, requires SM100.", - NS("exec.mamba"), - ] = False - mamba_cache_philox_rounds: A[ - int, - "Number of Philox rounds to use for stochastic rounding of FP16 Mamba SSM cache writes. Triton uses the Triton default when set to 0; FlashInfer uses 10 rounds when set to 0.", - NS("exec.mamba"), - ] = 0 - mamba_full_memory_ratio: A[ - Optional[float], - Arg( - help="The ratio of mamba state memory to full kv cache memory.", - resolvable=True, - fallback=0.9, - ), - NS("schedule"), - ] = None - mamba_radix_cache_strategy: A[ - str, - Arg( - help="The strategy to use for mamba radix cache.", - choices=["auto", "no_buffer", "extra_buffer", "extra_buffer_lazy"], - resolvable=True, - ), - NS("exec.mamba"), - ] = "auto" - uses_mamba_radix_cache: A[ - bool, - Arg( - help="(Derived) whether the model routes through the hybrid-mamba " - "radix cache handling; resolved from the model architecture, no " - "CLI surface.", - no_cli=True, - resolvable=True, - ), - NS("exec.mamba"), - ] = False - mamba_track_interval: A[ - int, "The interval to track the mamba state during decode.", NS("exec.mamba") - ] = 256 - enable_int8_mamba_checkpoint: A[ - bool, - "Store radix-cached linear-attn (mamba) states in int8 (separate checkpoint pool) for ~2x cached-prefix capacity at fixed memory.", - NS("exec.mamba"), - ] = False - int8_mamba_ckpt_size: A[ - Optional[int], - "Number of int8 mamba checkpoint slots (default: 2x the active mamba pool size).", - NS("exec.mamba"), - ] = None - linear_attn_backend: A[ - str, - Arg( - help="The default kernel backend for linear attention (GDN/KDA). Can be overridden per-mode by --linear-attn-decode-backend and --linear-attn-prefill-backend. The Helion backend is KDA-only.", - choices=LINEAR_ATTN_KERNEL_BACKEND_CHOICES, - ), - NS("exec.mamba"), - ] = "triton" - linear_attn_decode_backend: A[ - Optional[str], - Arg( - help="Override the kernel backend for linear attention decode. If not set, uses --linear-attn-backend.", - choices=LINEAR_ATTN_KERNEL_BACKEND_CHOICES, - ), - NS("exec.mamba"), - ] = None - linear_attn_prefill_backend: A[ - Optional[str], - Arg( - help="Override the kernel backend for linear attention prefill/extend. If not set, uses --linear-attn-backend; compatible SM100 GDN models may automatically select FlashInfer.", - choices=LINEAR_ATTN_KERNEL_BACKEND_CHOICES, - ), - NS("exec.mamba"), - ] = None - linear_attn_verify_backend: A[ - Optional[str], - Arg( - help="Override the kernel backend for linear attention speculative target-verify. If not set, follows the decode backend (flashinfer decode -> flashinfer verify, otherwise triton). KDA supports triton, nv_cutedsl, and flashinfer verify backends.", - choices=LINEAR_ATTN_KERNEL_BACKEND_CHOICES + ["nv_cutedsl"], - ), - NS("exec.mamba"), - ] = None - # ReplaySSM buffered output-only linear-attn decode (GDN + KDA): per-slot - # ring + periodic flush to cut per-step HBM state traffic. - enable_linear_replayssm: A[ - bool, - "Enable the ReplaySSM buffered output-only linear-attn decode kernel. " - "Primarily a GDN (scalar-gate) decode-bandwidth optimization (~1.2-1.5x " - "at batch >= 64). KDA uses its selected Triton or Helion implementation, " - "but its per-K gate ring is larger and ReplaySSM is typically slower " - "than packed KDA decode; benchmark before enabling it. Requires the " - "Triton linear-attn decode backend, or Helion for KDA, and " - "--mamba-radix-cache-strategy no_buffer (the default).", - NS("exec.mamba"), - ] = False - linear_replayssm_cache_len: A[ - int, - "Ring-buffer length L for ReplaySSM linear-attn decode. The full recurrent state is flushed to HBM every L decode steps.", - NS("exec.mamba"), - ] = 16 - # ReplaySSM spec-verify (Part B of RFC #28511): linear-attn target-verify via - # fold-every-commit instead of per-draft full-state snapshots -- the verify - # stores each draft step's raw inputs into a per-slot window and the commit - # replays the accepted prefix into the fp32 checkpoint. GDN sizes the window - # to the draft maximum; KDA folds a (raw v, pre-norm k, gate, beta) ring of - # length --linear-replayssm-cache-len. Linear-chain (topk <= 1) only. - enable_linear_replayssm_spec: A[ - bool, - "Enable the ReplaySSM spec-verify: fold-every-commit -- a per-slot raw-input window replaces the recurrent verify's per-draft full-state snapshots. GDN or KDA hybrid linear-attn models, linear-chain (--speculative-eagle-topk in {None, 1}) only.", - NS("exec.mamba"), - ] = False - - # ------------------------------------------------------------------------- - # Hierarchical cache - # ------------------------------------------------------------------------- - enable_hierarchical_cache: A[bool, "Enable hierarchical cache", NS("memory")] = ( - False - ) - hicache_host_memory_mode: A[ - str, - Arg( - help="Whether host memory is a persistent HiCache tier (cache) or a transient staging buffer between GPU and the storage backend (buffer_only). buffer_only requires --hicache-storage-backend.", - choices=["cache", "buffer_only"], - ), - NS("memory"), - ] = "cache" - hicache_ratio: A[ - Optional[float], - "The ratio of the size of host KV cache memory pool to the size of device pool. Defaults to 2.0 in cache mode, 1.2 in buffer_only mode, or 0.2 for backup-only host-pool decode retraction.", - NS("memory"), - ] = None - hicache_size: A[ - int, - "The size of host KV cache memory pool in gigabytes. Overrides --hicache-ratio in either host memory mode.", - NS("memory"), - ] = 0 - hicache_write_policy: A[ - str, - Arg( - help="The write policy of hierarchical cache.", - choices=["write_back", "write_through", "write_through_selective"], - ), - NS("memory"), - ] = "write_through" - hicache_io_backend: A[ - str, - Arg( - help="The IO backend for KV cache transfer between CPU and GPU", - choices=["direct", "kernel", "kernel_ascend"], - ), - NS("memory"), - ] = "kernel" - hicache_mem_layout: A[ - str, - Arg( - help="The layout of host memory pool for hierarchical cache.", - choices=[ - "layer_first", - "page_first", - "page_first_direct", - "page_first_kv_split", - "page_head", - ], - ), - NS("memory"), - ] = "page_first" - hicache_storage_backend: A[ - Optional[str], - Arg( - help="The storage backend for hierarchical KV cache. Built-in backends: file, mooncake, hf3fs, nixl, aibrix. For dynamic backend, use --hicache-storage-backend-extra-config to specify: backend_name (custom name), module_path (Python module path), class_name (backend class name).", - choices=[ - "file", - "sim", - "mooncake", - "hf3fs", - "nixl", - "aibrix", - "dynamic", - "eic", - "simm", - "mori", - "shm", - ], - ), - NS("memory"), - ] = None - hicache_storage_prefetch_policy: A[ - str, - Arg( - help="Control when prefetching from the storage backend should stop.", - choices=["best_effort", "wait_complete", "timeout"], - ), - NS("memory"), - ] = "timeout" - hicache_storage_backend_extra_config: A[ - Optional[str], - "A dictionary in JSON string format, or a string starting with a leading '@' and a config file in JSON/YAML/TOML format, containing extra configuration for the storage backend.", - NS("memory"), - ] = None - hicache_storage_prefetch_retry_poll_interval: A[ - int, - Arg( - help=( - "Scheduling passes a queued request waits after a storage " - "prefetch miss before the availability check is retried " - "(under load the first check can run before the needed " - "backup commits). 0 disables retries." - ), - ), - NS("memory"), - ] = 0 - hicache_storage_prefetch_retry_max_attempts: A[ - int, - "Maximum storage prefetch retries per request when --hicache-storage-prefetch-retry-poll-interval is set.", - NS("memory"), - ] = 4 - - # ------------------------------------------------------------------------- - # Unified Radix Cache - # ------------------------------------------------------------------------- - enable_unified_cache_external_linker: A[ - bool, - "Link UnifiedRadixCache directly to an external KV store (direct L3), with no host cache tier.", - NS("memory"), - ] = False - unified_cache_external_linker_backend: A[ - str, - Arg( - help="Storage backend for --enable-unified-cache-external-linker.", - choices=["mooncake", "mori"], - ), - NS("memory"), - ] = "mooncake" - - # ------------------------------------------------------------------------- - # Hierarchical sparse attention - # ------------------------------------------------------------------------- - enable_hisparse: A[bool, "Enable hierarchical sparse attention", NS("memory")] = ( - False - ) - hisparse_config: A[ - Optional[str], - Arg( - help='A dictionary in JSON string format for hierarchical sparse attention configuration. Example: \'{"top_k": 2048, "device_buffer_size": 4096, "host_to_device_ratio": 2}\'', - aliases=["--hierarchical-sparse-attention-extra-config"], - ), - NS("memory"), - ] = None - - # ------------------------------------------------------------------------- - # Multi-modal optimization configs - # ------------------------------------------------------------------------- - enable_broadcast_mm_inputs_process: A[ - bool, "Enable broadcast mm-inputs process in scheduler.", NS("mm") - ] = False - enable_prefix_mm_cache: A[ - bool, - "Enable prefix multimodal cache. Currently only supports mm-only.", - NS("mm"), - ] = False - mm_enable_dp_encoder: A[ - bool, - "Enabling data parallelism for mm encoder. The dp size will be set to the tp size automatically.", - NS("mm"), - ] = False - mm_process_config: A[ - Optional[Dict[str, Any]], - Arg( - help="Multimodal preprocessing config, a json config contains keys: `image`, `video`, `audio`", - type_parser=json.loads, - ), - NS("mm"), - ] = None - mm_processor_worker_num: A[ - int, - "Number of threads for multimodal processor calls. 0 selects the " - "model-specific default. Only processors with isolated-worker support " - "can use more than one thread.", - NS("mm"), - ] = 0 - mm_io_worker_num: A[ - int, - "Number of threads for multimodal data loading and decoding. 0 selects " - "the model-specific default. SGLANG_IO_WORKERS remains supported as an " - "environment override when this argument is 0.", - NS("mm"), - ] = 0 - allowed_media_domains: A[ - List[str], - "Restrict client-supplied HTTP(S) image, video, and audio URLs to these " - "exact hostnames. Redirect destinations are checked against the same " - "allowlist. When unset, remote media from any domain is allowed.", - NS("mm"), - ] = dataclasses.field(default_factory=list) - media_url_max_file_size_mb: A[ - int, - "Maximum size in MiB for one client-supplied remote media download. " - "The limit is enforced while streaming; set to 0 to disable it.", - NS("mm"), - ] = 64 - mm_preprocess_cache_size_mb: A[ - Optional[int], - "CPU memory budget for content-addressed multimodal preprocessing " - "artifacts. Unset selects a model-specific default (256 MiB for " - "Kimi-K3); 0 disables the cache. The budget is divided across " - "tokenizer workers and does not reserve GPU memory.", - NS("mm"), - ] = None - trust_mm_content_hashes: A[ - bool, - "Trust caller-provided multimodal SHA-256 content hashes. This can " - "skip reading media on a hot metadata-cache hit; only enable it when " - "the caller guarantees that hashes identify immutable media bytes.", - NS("mm"), - ] = False - limit_mm_data_per_request: A[ - Optional[Union[str, Dict[str, int]]], - Arg( - help='Limit the number of multimodal inputs per request. e.g. \'{"image": 1, "video": 1, "audio": 1}\'', - type_parser=json.loads, - ), - NS("mm"), - ] = None - enable_mm_global_cache: A[ - bool, - "Enable global multimodal embedding cache to skip redundant ViT inference.", - NS("mm"), - ] = False - image_processor_backend: A[ - Literal["auto", "torchvision", "pil"], - "Image processor backend. 'auto' lets Transformers select the best " - "available backend.", - NS("mm"), - ] = "auto" - mm_global_cache_backend: A[ - str, - Arg( - help="Storage backend for the multimodal global embedding cache. " - "Used when --enable-mm-global-cache is set.", - choices=["mooncake"], - ), - NS("mm"), - ] = "mooncake" - disable_fast_image_processor: A[ - bool, - "Deprecated. Use --image-processor-backend=pil instead.", - NS("mm"), - ] = False - mm_feature_transport: A[ - Optional[Literal["cpu", "cuda_ipc", "cuda_vmm"]], - "Transport multimodal features through CPU memory, a bounded CUDA IPC " - "pool, or a bounded CUDA VMM pool. " - "Unset uses cpu except for validated multi-node GB200/GB300 MNNVL models, " - "which use cuda_vmm when an IMEX channel is available. Select cuda_ipc " - "explicitly for single-node GPU transport. GPU transports reserve " - "SGLANG_MM_FEATURE_CACHE_MB (default 1024 MiB) on the base GPU and fall " - "back to CPU transport when the pool is full.", - NS("mm"), - ] = None - keep_mm_feature_on_device: A[ - bool, - "Deprecated. Use --mm-feature-transport=cuda_ipc for bounded GPU-resident " - "multimodal feature transport.", - NS("mm"), - ] = False - - # ------------------------------------------------------------------------- - # LoRA - # ------------------------------------------------------------------------- - enable_lora: A[ - Optional[bool], - "Enable LoRA support for the model. This argument is automatically set to True if `--lora-paths` is provided for backward compatibility.", - NS("lora"), - ] = None - enable_lora_overlap_loading: A[ - Optional[bool], - "Enable asynchronous LoRA weight loading in order to overlap H2D transfers with GPU compute. This should be enabled if you find that your LoRA workloads are bottlenecked by adapter weight loading, for example when frequently loading large LoRA adapters.", - NS("lora"), - ] = None - max_lora_rank: A[ - Optional[int], - "The maximum rank of LoRA adapters. If not specified, it will be automatically inferred from the adapters provided in --lora-paths.", - NS("lora"), - ] = None - lora_target_modules: A[ - Optional[Union[set[str], List[str]]], - Arg( - help="The union set of all target modules where LoRA should be applied. If not specified, it will be automatically inferred from the adapters provided in --lora-paths. If 'all' is specified, all supported modules will be targeted.", - nargs="*", - choices=SUPPORTED_LORA_TARGET_MODULES + [LORA_TARGET_ALL_MODULES], - ), - NS("lora"), - ] = None - lora_paths: A[ - Optional[Union[dict[str, str], List[dict[str, str]], List[str], List[LoRARef]]], - Arg( - help='The list of LoRA adapters to load. Each adapter must be specified in one of the following formats: | = | JSON with schema {"lora_name":str,"lora_path":str,"pinned":bool}', - action=LoRAPathAction, - action_kwargs={"type": str, "nargs": "*"}, - ), - NS("lora"), - ] = None - max_loaded_loras: A[ - Optional[int], - "If specified, it limits the maximum number of LoRA adapters loaded in CPU memory at a time. The value must be greater than or equal to `--max-loras-per-batch`.", - NS("lora"), - ] = None - max_loras_per_batch: A[ - int, - "Maximum number of adapters for a running batch, include base-only request.", - NS("lora"), - ] = 8 - lora_eviction_policy: A[ - str, - Arg( - help="LoRA adapter eviction policy when memory pool is full. 'lru': Least Recently Used (default, better cache efficiency). 'fifo': First-In-First-Out.", - choices=["lru", "fifo"], - ), - NS("lora"), - ] = "lru" - lora_backend: A[ - str, - Arg( - help="Choose the kernel backend for multi-LoRA serving.", - choices=["triton", "csgmv", "ascend", "torch_native"], - ), - NS("lora"), - ] = "csgmv" - max_lora_chunk_size: A[ - Optional[int], - Arg( - help="Maximum chunk size for the ChunkedSGMV LoRA backend. Only used when --lora-backend is 'csgmv'. Choosing a larger value might improve performance.", - choices=[16, 32, 64, 128], - ), - NS("lora"), - ] = 16 - experts_shared_outer_loras: A[ - Optional[bool], - Arg( - help="Force shared outer LoRA mode for MoE models. When set, w1/w3 lora_A and w2 lora_B are shared across experts (expert_dim=1). Use --no-experts-shared-outer-loras to force disable. By default this is auto-detected from adapter weights.", - action=argparse.BooleanOptionalAction, - ), - NS("lora"), - ] = None - lora_use_virtual_experts: A[ - bool, - "Enable virtual expert computation for MoE models. When set, the model will use virtual expert computation.", - NS("lora"), - ] = False - lora_strict_loading: A[ - bool, - Arg( - help="Enable strict loading for LoRA adapters. When set, mismatched or missing keys in the adapter weights will raise an error.", - action=argparse.BooleanOptionalAction, - ), - NS("lora"), - ] = False - lora_drain_wait_threshold: A[ - float, - "When any LoRA adapter request waits longer than this threshold (in seconds), the scheduler will selectively drain one running adapter to make room. This mitigates extreme tail latency under high or skewed workloads by preventing a small set of adapters from monopolizing batch slots. Set to 0 to disable draining (default).", - NS("lora"), - ] = 0.0 - - # ------------------------------------------------------------------------- - # Two batch overlap - # ------------------------------------------------------------------------- - enable_two_batch_overlap: A[ - bool, "Enabling two micro batches to overlap.", NS("exec.overlap") - ] = False - enable_single_batch_overlap: A[ - bool, - "Let computation and communication overlap within one micro batch.", - NS("exec.overlap"), - ] = False - tbo_token_distribution_threshold: A[ - float, - "The threshold of token distribution between two batches in micro-batch-overlap, determines whether to two-batch-overlap or two-chunk-overlap. Set to 0 denote disable two-chunk-overlap.", - NS("exec.overlap"), - ] = 0.48 - - # ------------------------------------------------------------------------- - # Offloading - # ------------------------------------------------------------------------- - cpu_offload_gb: A[ - int, "How many GBs of RAM to reserve for CPU offloading.", NS("exec.offload") - ] = 0 - offload_group_size: A[ - int, "Number of layers per group in offloading.", NS("exec.offload") - ] = -1 - offload_num_in_group: A[ - int, "Number of layers to be offloaded within a group.", NS("exec.offload") - ] = 1 - offload_prefetch_step: A[ - int, "Steps to prefetch in offloading.", NS("exec.offload") - ] = 1 - offload_mode: A[str, "Mode of offloading.", NS("exec.offload")] = "cpu" - - # ------------------------------------------------------------------------- - # LMCache - # ------------------------------------------------------------------------- - enable_lmcache: A[ - bool, - "Using LMCache as an alternative hierarchical cache solution", - NS("memory"), - ] = False - lmcache_config_file: A[ - Optional[str], "Path to the LMCache YAML configuration file", NS("memory") - ] = None - - # ------------------------------------------------------------------------- - # FlexKV - # ------------------------------------------------------------------------- - enable_flexkv: A[ - bool, - ( - "Route the default RadixCache through FlexKV's KVManager for " - "host-tier (CPU / SSD / Remote) KV cache offload. Equivalent " - "to --radix-cache-backend=flexkv but also participates in the " - "auto-selection chain alongside --enable-lmcache." - ), - NS("memory"), - ] = False - flexkv_config_file: A[ - Optional[str], - ( - "Path to the FlexKV YAML / JSON configuration file. " - "Equivalent to setting the FLEXKV_CONFIG_PATH environment " - "variable." - ), - NS("memory"), - ] = None - - # ------------------------------------------------------------------------- - # Ktransformers/AMX expert parallelism - # ------------------------------------------------------------------------- - kt_weight_path: A[ - Optional[str], - "[ktransformers parameter] The path of the quantized expert weights for amx kernel. A local folder.", - NS("exec.moe"), - ] = None - kt_method: A[ - str, - "[ktransformers parameter] Quantization formats for CPU execution.", - NS("exec.moe"), - ] = "AMXINT4" - kt_cpuinfer: A[ - Optional[int], - "[ktransformers parameter] The number of CPUInfer threads.", - NS("exec.moe"), - ] = None - kt_threadpool_count: A[ - int, - "[ktransformers parameter] One-to-one with the number of NUMA nodes (one thread pool per NUMA).", - NS("exec.moe"), - ] = 2 - kt_num_gpu_experts: A[ - Optional[int], - "[ktransformers parameter] The number of GPU experts.", - NS("exec.moe"), - ] = None - kt_max_deferred_experts_per_token: A[ - Optional[int], - "[ktransformers parameter] Maximum number of experts deferred to CPU per token. All MoE layers except the final one use this value; the final layer always uses 0.", - NS("exec.moe"), - ] = None - - # ------------------------------------------------------------------------- - # Diffusion LLM - # ------------------------------------------------------------------------- - dllm_algorithm: A[ - Optional[str], - "The diffusion LLM algorithm, such as LowConfidence.", - NS("exec.dllm"), - ] = None - dllm_algorithm_config: A[ - Optional[str], - "The diffusion LLM algorithm configurations. Must be a YAML file.", - NS("exec.dllm"), - ] = None - dllm_fdfo: A[ - bool, - Arg( - help="Enable First-Done-First-Out (FDFO) scheduling for diffusion LLM inference. Enabled by default; use --no-dllm-fdfo to fall back to synchronous block scheduling.", - action=argparse.BooleanOptionalAction, - ), - NS("exec.dllm"), - ] = True - - # ------------------------------------------------------------------------- - # PD disaggregation - # ------------------------------------------------------------------------- - disaggregation_mode: A[ - Literal["null", "prefill", "decode"], - 'Only used for PD disaggregation. "prefill" for prefill-only server, and "decode" for decode-only server. If not specified, it is not PD disaggregated', - NS("disagg"), - ] = "null" - disaggregation_transfer_backend: A[ - str, - Arg( - help="The backend for disaggregation transfer. Default is mooncake.", - choices=DISAGG_TRANSFER_BACKEND_CHOICES, - ), - NS("disagg"), - ] = "mooncake" - disaggregation_bootstrap_port: A[ - int, - "Bootstrap server port on the prefill server. Default is 8998.", - NS("disagg"), - ] = 8998 - disaggregation_ib_device: A[ - Optional[str], - 'The InfiniBand devices for disaggregation transfer. Supports a single device (e.g., --disaggregation-ib-device mlx5_0), a shared comma-separated list (e.g., --disaggregation-ib-device mlx5_0,mlx5_1), a per-GPU JSON mapping (e.g., --disaggregation-ib-device \'{"0": "mlx5_0,mlx5_1", "1": "mlx5_2"}\'), or a path to a JSON file containing that mapping. Default is None, which triggers automatic device detection when mooncake backend is enabled.', - NS("disagg"), - ] = None - disaggregation_decode_enable_radix_cache: A[ - bool, - "Enable radix cache on decode server (PD mode). Caches KV prefixes to avoid redundant transfers. Incompatible with --enable-hisparse, speculative decoding, and --disaggregation-transfer-backend fake.", - NS("disagg"), - ] = False - disaggregation_decode_enable_offload_kvcache: A[ - bool, - "Enable async KV cache offloading on decode server (PD mode).", - NS("disagg"), - ] = False - disaggregation_decode_retraction_backup: A[ - Optional[str], - Arg( - help=( - "Storage backend for KV preserved across PD decode retraction. " - "'cpu_tensor' uses per-request CPU tensors. 'host_pool' uses " - "a reserved HiCache pool and does not fall back on exhaustion. " - "If omitted, the backend is inferred from the decode KV pool." - ), - choices=["cpu_tensor", "host_pool"], - ), - NS("disagg"), - ] = None - num_reserved_decode_tokens: A[ - int, - "Number of decode tokens that will have memory reserved when adding new request to the running batch.", - NS("disagg"), - ] = 512 - disaggregation_decode_extra_slots: A[ - Optional[int], - "Number of extra decode req_to_token slots pre-allocated for in-transfer requests (PD mode). If unset, defaults to 0 (or 2x the per-worker running batch for small batches).", - NS("disagg"), - ] = None - disaggregation_decode_polling_interval: A[ - int, - "The interval to poll requests in decode server. Can be set to >1 to reduce the overhead of this.", - NS("disagg"), - ] = 1 - optimistic_prefill_attempts: A[ - int, - "Number of optimistic prefill forward passes that skip the bootstrap wait.", - NS("disagg"), - ] = 0 - - # ------------------------------------------------------------------------- - # Encode prefill disaggregation - # ------------------------------------------------------------------------- - encoder_only: A[ - bool, "For MLLM with an encoder, launch an encoder-only server", NS("disagg") - ] = False - language_only: A[ - bool, "For VLM, load weights for the language model only.", NS("disagg") - ] = False - language_model_only: A[ - bool, - "Skip the multimodal encoder entirely: its weights are never loaded and the " - "tower is never built, freeing that GPU memory for KV cache. Multimodal " - "requests are rejected. Unlike --language-only this is a standalone mode, " - "not part of encoder/decoder disaggregation.", - NS("disagg"), - ] = False - encoder_transfer_backend: A[ - str, - Arg( - help="The backend for encoder disaggregation transfer. Auto selects a model- and TP-aware backend.", - choices=["auto", "zmq_to_scheduler", "zmq_to_tokenizer", "mooncake"], - ), - NS("disagg"), - ] = "auto" - encoder_urls: A[List[str], "List of encoder server urls.", NS("disagg")] = ( - dataclasses.field(default_factory=list) - ) - encoder_bootstrap_port: A[ - int, - "Port for the EncoderBootstrapServer that runs in the language-only tokenizer manager process. Encoders register here, and language-only receivers fetch the current URL list from here.", - NS("disagg"), - ] = 8997 - encoder_register_urls: A[ - List[str], - "One or more EncoderBootstrapServer URLs to register this encoder with on startup, for dynamic encoder discovery. Example: --encoder-register-urls http://prefill0:8997 http://prefill1:8997. Used with --encoder-only servers.", - NS("disagg"), - ] = dataclasses.field(default_factory=list) - enable_adaptive_dispatch_to_encoder: A[ - bool, - "When enabled, adaptively dispatch: multi-image requests go to encoder in language_only epd mode, single-image requests are processed locally.", - NS("disagg"), - ] = False - - # ------------------------------------------------------------------------- - # PD-Multiplexing - # ------------------------------------------------------------------------- - enable_pdmux: A[ - bool, "Enable PD-Multiplexing, PD running on greenctx stream.", NS("disagg") - ] = False - pdmux_config_path: A[ - Optional[str], "The path of the PD-Multiplexing config file.", NS("disagg") - ] = None - sm_group_num: A[int, "Number of sm partition groups.", NS("disagg")] = 8 - - # ------------------------------------------------------------------------- - # Model weight update and weight loading - # ------------------------------------------------------------------------- - startup_weight_load_mode: A[ - Literal["serial", "overlap"], - ( - "Control startup weight loading relative to CUDA graph capture. " - "'serial' preserves the existing startup order; 'overlap' stages " - "checkpoint files while CUDA graphs are captured and commits the " - "real weights afterward." - ), - NS("model"), - ] = "serial" - custom_weight_loader: A[ - Optional[List[str]], - Arg( - help="The custom dataloader which used to update the model. Should be set with a valid import path, such as my_package.weight_load_func", - nargs="*", - ), - NS("model"), - ] = None - weight_loader_disable_mmap: A[ - bool, "Disable mmap while loading weight using safetensors.", NS("model") - ] = False - weight_loader_prefetch_checkpoints: A[ - bool, - "Prefetch checkpoint files into OS page cache before loading. Each rank prefetches a fraction of the shards, reducing total network I/O on shared filesystems (NFS/Lustre) from N*checkpoint to 1*checkpoint. Recommended for models on network storage. When enabled, multi-threaded safetensors loading is disabled by default to avoid I/O oversubscription with the prefetch threads; set enable_multithread_load=true in --model-loader-extra-config to keep multi-threaded loading (e.g. on local NVMe where prefetch is a no-op).", - NS("model"), - ] = False - weight_loader_prefetch_num_threads: A[ - int, - "Number of threads per rank for checkpoint prefetching (default: 4).", - NS("model"), - ] = 4 - weight_loader_drop_cache_after_load: A[ - bool, - "Call posix_fadvise(DONTNEED) on each safetensors shard after loading it.", - NS("model"), - ] = False - remote_instance_weight_loader_seed_instance_ip: A[ - Optional[str], - "The ip of the seed instance for loading weights from remote instance.", - NS("model"), - ] = None - remote_instance_weight_loader_seed_instance_service_port: A[ - Optional[int], - "The service port of the seed instance for loading weights from remote instance.", - NS("model"), - ] = None - remote_instance_weight_loader_send_weights_group_ports: A[ - Optional[List[int]], - Arg( - help="The communication group ports for loading weights from remote instance.", - type_parser=json_list_type, - ), - NS("model"), - ] = None - remote_instance_weight_loader_backend: A[ - Literal["transfer_engine", "nccl", "modelexpress"], - "The backend for loading weights from remote instance. Can be 'transfer_engine', 'nccl', or 'modelexpress'. Default is 'nccl'.", - NS("model"), - ] = "nccl" - remote_instance_weight_loader_start_seed_via_transfer_engine: A[ - bool, - "Start seed server via transfer engine backend for remote instance weight loader.", - NS("model"), - ] = False - engine_info_bootstrap_port: A[ - int, - "Port for the engine info bootstrap server. Default is 6789. Must be set explicitly when running multiple instances on the same node.", - NS("model"), - ] = 6789 - modelexpress_config: A[ - Optional[str], - 'JSON config for ModelExpress P2P weight loading. Keys: "url" (optional gRPC host:port override), "transport" ("nixl" or "transfer_engine"). Example: \'{"url": "localhost:8001", "transport": "nixl"}\'', - NS("model"), - ] = None - download_dir: A[ - Optional[str], "Model download directory for huggingface.", NS("model") - ] = None - model_checksum: A[ - Optional[str], - Arg( - help="Model file integrity verification. If provided without value, uses model-path as HF repo ID. Otherwise, provide checksums JSON file path or HuggingFace repo ID.", - nargs="?", - const="", - ), - NS("model"), - ] = None - delete_ckpt_after_loading: A[ - bool, "Delete the model checkpoint after loading the model.", NS("model") - ] = False - # Checkpoint decryption - decrypted_config_file: A[ - Optional[str], "The path of the decrypted config file.", NS("model") - ] = None - decrypted_draft_config_file: A[ - Optional[str], "The path of the decrypted draft config file.", NS("model") - ] = None - checkpoint_engine_wait_weights_before_ready: A[ - bool, - "If set, the server will wait for initial weights to be loaded via checkpoint-engine or other update methods before serving inference requests.", - NS("model"), - ] = False - - # ------------------------------------------------------------------------- - # Prefill delayer - # ------------------------------------------------------------------------- - enable_prefill_delayer: A[ - bool, - "Enable prefill delayer for DP attention to reduce idle time.", - NS("schedule"), - ] = False - prefill_delayer_max_delay_passes: A[ - int, "Maximum forward passes to delay prefill.", NS("schedule") - ] = 30 - prefill_delayer_token_usage_low_watermark: A[ - Optional[float], - "Token usage low watermark for prefill delayer.", - NS("schedule"), - ] = None - prefill_delayer_forward_passes_buckets: A[ - Optional[List[float]], - "Custom buckets for prefill delayer forward passes histogram. 0 and max_delay_passes-1 will be auto-added.", - NS("schedule"), - ] = None - prefill_delayer_wait_seconds_buckets: A[ - Optional[List[float]], - "Custom buckets for prefill delayer wait seconds histogram. 0 will be auto-added.", - NS("schedule"), - ] = None - prefill_delayer_queue_min_ratio: A[ - Optional[float], - ( - "Opt-in to the adaptive queue-based delay trigger (independent of the " - "slot-based one). Delays prefill until the waiting queue reaches " - "min(running_req * ratio, prefill_max_requests), falling back to the " - "observed max_prefill_bs when no request limit is set. Unset (default) " - "keeps the original slot-only behavior. Typical: 0.1 ~ 0.5." - ), - NS("schedule"), - ] = None - prefill_delayer_max_delay_ms: A[ - Optional[float], - ( - "Wall-clock cap (ms) on a single queue-trigger delay; once exceeded, " - "prefill is force-released to bound worst-case TTFT. Only consulted " - "when --prefill-delayer-queue-min-ratio is set. Typical: 1000 ~ " - "5000; defaults to 5000 if unset." - ), - NS("schedule"), - ] = None - - # ------------------------------------------------------------------------- - # Min free slots delay (prefill refill batching) - # ------------------------------------------------------------------------- - min_free_slots_delay: A[ - Optional[int], - ( - "Hold new prefills until at least N running-request slots have freed " - "up, so they are admitted in one batch instead of one at a time. " - "Useful when each admission is disproportionately expensive, e.g. " - "speculative decoding with a separate draft prefill pass. An " - "explicit value always wins, capped by max-running-requests " - "(1 disables). When unset, DFlash workloads auto-enable the " - "formula; other workloads stay disabled. Not supported with " - "pipeline parallelism." - ), - NS("schedule"), - ] = None - - # ------------------------------------------------------------------------- - # Deterministic inference - # ------------------------------------------------------------------------- - enable_deterministic_inference: A[ - bool, - "Enable deterministic inference mode with batch invariant ops.", - NS("exec.deterministic"), - ] = False - rl_on_policy_target: A[ - Optional[str], - Arg( - help="The training system that SGLang needs to match for true on-policy.", - choices=RL_ON_POLICY_TARGET_CHOICES, - ), - NS("exec.deterministic"), - ] = None - - # ------------------------------------------------------------------------- - # KV canary - # ------------------------------------------------------------------------- - kv_canary: A[ - str, - Arg( - help="KV cache canary mode. 'none' disables the canary (default). 'log' prints them while the server keeps running (production-safe). 'raise' fails the server on the first detected mismatch (CI lane).", - choices=["none", "log", "raise"], - ), - NS("observability"), - ] = "none" - kv_canary_real_data: A[str, NS("observability")] = "none" - kv_canary_sweep_interval: A[ - int, "Every N forward steps, run a full-pool sweep.", NS("observability") - ] = 0 - - # ------------------------------------------------------------------------- - # Dynamic batch tokenizer - # ------------------------------------------------------------------------- - enable_dynamic_batch_tokenizer: A[ - bool, - "Enable async dynamic batch tokenizer for improved performance when multiple requests arrive concurrently.", - NS("serving"), - ] = False - dynamic_batch_tokenizer_batch_size: A[ - int, - "[Only used if --enable-dynamic-batch-tokenizer is set] Maximum batch size for dynamic batch tokenizer.", - NS("serving"), - ] = 32 - dynamic_batch_tokenizer_batch_timeout: A[ - float, - "[Only used if --enable-dynamic-batch-tokenizer is set] Timeout in seconds for batching tokenization requests.", - NS("serving"), - ] = 0.002 - enable_tokenizer_batch_encode: A[ - bool, - "Enable batch tokenization for improved performance when processing multiple text inputs. Do not use with image inputs, pre-tokenized input_ids, or input_embeds.", - NS("serving"), - ] = False - disable_tokenizer_batch_decode: A[ - bool, - "Disable batch decoding when decoding multiple completions.", - NS("serving"), - ] = False - - # ------------------------------------------------------------------------- - # Debug tensor dumps - # ------------------------------------------------------------------------- - debug_tensor_dump_output_folder: A[ - Optional[str], - "The output folder for dumping tensors. In Eagle mode, tensor outputs from draft and target models are stored in separate subdirectories ('draft' and 'target').", - NS("observability"), - ] = None - # None means dump all layers. - debug_tensor_dump_layers: A[ - Optional[List[int]], - "The layer ids to dump. Dump all layers if not specified.", - NS("observability"), - ] = None - # TODO(guoyuhong): clean the old dumper code. - debug_tensor_dump_input_file: A[ - Optional[str], "The input filename for dumping tensors", NS("observability") - ] = None - - # ------------------------------------------------------------------------- - # Misc runtime features - # ------------------------------------------------------------------------- - enable_memory_saver: A[ - bool, - "Allow saving memory using release_memory_occupation and resume_memory_occupation", - NS("exec.features"), - ] = False - enable_weights_cpu_backup: A[ - bool, - "Save model weights (both main model and draft model, if any) to CPU memory during release_weights_occupation and resume_weights_occupation", - NS("exec.features"), - ] = False - enable_draft_weights_cpu_backup: A[ - bool, - "Save draft model weights to CPU memory during release_weights_occupation and resume_weights_occupation", - NS("exec.features"), - ] = False - enable_custom_logit_processor: A[ - bool, - "Enable users to pass custom logit processors to the server (disabled by default for security)", - NS("exec.features"), - ] = False - enable_return_hidden_states: A[ - bool, - "Enable returning full hidden states with responses. Equivalent to " - "`--return-hidden-states-mode full`.", - NS("exec.features"), - ] = False - return_hidden_states_mode: A[ - Optional[str], - Arg( - help="Set the maximum hidden-state return mode supported by the " - "server. `last` allows requests with return_hidden_states=False or " - "`last`; `full` also allows return_hidden_states=True.", - choices=["last", "full"], - ), - NS("exec.features"), - ] = None - enable_return_routed_experts: A[ - bool, - "Enable returning routed experts of each layer with responses.", - NS("exec.features"), - ] = False - enable_return_indexer_topk: A[ - bool, - "Enable returning indexer topk indices of layers with indexer with responses.", - NS("exec.features"), - ] = False - disable_outlines_disk_cache: A[ - bool, - "Disable disk cache of outlines to avoid possible crashes related to file system or high concurrency.", - NS("exec.features"), - ] = False - enable_mis: A[ - bool, - "Enable Multi-Item Scoring optimization. Combines query and multiple items into a single sequence for efficient batch processing. Requires --attention-backend flashinfer; auto-disables CUDA graph, radix cache, and chunked prefill.", - NS("exec.features"), - ] = False - - # ------------------------------------------------------------------------- - # Weight cache - # ------------------------------------------------------------------------- - weight_cache_mode: A[ - str, - Arg( - help="Weight cache mode. 'off': normal disk loading. " - "'daemon': launch weight cache daemon (holds weights in GPU memory). " - "Engine-spawned daemons are co-terminal with the engine and do NOT " - "persist across restarts, so this alone does not speed up restart " - "(the first start is slower). For fast recovery, run the standalone " - "daemon (python -m sglang.srt.weight_cache.daemon) and connect with " - "'client'. 'client': connect to existing daemon and load via IPC.", - choices=["off", "daemon", "client"], - ), - NS("model"), - ] = "off" - weight_cache_socket: A[ - Optional[str], - Arg( - help="Unix socket path for weight cache daemon (client mode)." - "If not set, derives the path from SGLANG_WEIGHT_CACHE_SOCKET_TEMPLATE " - "using the caller's physical GPU UUID.", - ), - NS("model"), - ] = None - weight_cache_timeout: A[ - int, - Arg( - help="Timeout in seconds for weight cache daemon readiness (default: 1800).", - ), - NS("model"), - ] = 1800 - - # ------------------------------------------------------------------------- - # Custom hooks, probe, and plugins - # ------------------------------------------------------------------------- - forward_hooks: A[ - Optional[List[dict[str, Any]]], - Arg( - help="JSON-formatted forward hook specifications to attach to the model.", - type_parser=json_list_type, - ), - NS("observability"), - ] = None - msprobe_dump_config: A[ - Optional[str], - "The path of the JSON configuration file for msProbe. If specified, enables msProbe dump.", - NS("observability"), - ] = None - def __post_init__(self): """Construction leaves the record at what the caller asked for. @@ -4175,6 +701,47 @@ class ServerArgs: return remote_instance_transfer_engine_of(resolving_view(self), load_format) +# The namespaces whose *input* fields make up the record. A namespace declares +# its input and derived fields side by side; only the input half is collected, +# which is why this is a call and not a base-class list. +# +# A set, not an order: `collect_input_fields` orders by `field_order.py`, which +# is what keeps the positional constructor stable. + +_INPUT_NAMESPACES = [ + Model, + ExecDeterministic, + ExecDllm, + ExecOffload, + ExecOverlap, + Lora, + Disagg, + Spec, + ExecMoe, + ExecComm, + ExecGraph, + ExecMamba, + ExecKernel, + Observability, + Device, + Parallel, + Memory, + Schedule, + ExecFeatures, + Mm, + Serving, +] + +_annotations, _defaults, _namespaces = collect_input_fields(_INPUT_NAMESPACES) +ServerArgs.__annotations__ = {**_annotations, **ServerArgs.__annotations__} +# The assembled record has no base classes, so it carries the map the classes +# used to answer through their `_NS_PATH`. +ServerArgs._NS_BY_FIELD = _namespaces +for _name, _value in _defaults.items(): + setattr(ServerArgs, _name, _value) +ServerArgs = dataclasses.dataclass(ServerArgs) + + # -------------------------------------------------------------------------- # Module-level ServerArgs helpers and runtime shims. # --------------------------------------------------------------------------