Cheng Wan
ed82def55f
Second of four; stacked on #38046. Mechanical relocation plus one design change that the relocation makes possible. **Review by checking the identity proofs at the bottom** -- nothing here is meant to change behaviour. ## The declarations move `ServerArgs` carried all 487 declarations in one 4,462-line file, each tagged with an `NS("...")` marker naming the namespace it belongs to -- structure supplied by annotation, in a file a namespace away from the `arg_groups/*_hook.py` that resolves it. They move to `arg_groups/fields/`: one module per top-level namespace, one class per leaf namespace (21 of them, `exec.moe` becomes `exec_.py::ExecMoe`). The class carries the `_NS_PATH` it stands for, so the module a field is declared in *is* its namespace and the marker is redundant -- `namespace_of` reads the declaring class instead. `NS` stays for the one case a class cannot express: a single ad-hoc dataclass whose fields span namespaces, which is what the config-bag tests build. Two things travel with the declarations. The `*_CHOICES` lists and the `add_*_choices` adders that extend them move to `arg_groups/choices.py`, since the fields naming them can no longer import from `server_args` without a cycle; `server_args` re-exports all of them, because out-of-tree plugins have always reached them there. And five fields whose only annotation element was the namespace marker become plain annotations -- `A` is `Annotated`, which needs two arguments, so stripping the marker would have left them invalid. `server_args.py` goes from 4,458 lines to about 1,000. ## The record is assembled, not inherited Inheriting the namespace classes would make the record's contents a property of which classes happen to appear in a base list. That is correct only while every namespace declares nothing but operator input, and it stops being correct the moment a derived field is declared: `attn_tp_size` belongs in `parallel.py` next to the leaves it is derived from, and inheriting `Parallel` would put it on the record -- where it is neither input nor safe, since the record is what crosses a process boundary and a derived width pickled to a subprocess is a stamp that elastic scale-up will not refresh. `collect_input_fields` takes the classes that declare input and returns their annotations, defaults and namespaces. Each source's annotations are resolved in its own module and handed on as type objects; carried across as text they would be re-evaluated where they land, and the composing module deliberately imports none of the names the declarations use. A namespace can now declare both halves side by side, and which half reaches the record is one readable call rather than an invariant spread across a base-class list. Nothing is registered on the derived side yet -- this is what makes it possible. `ServerArgs` is still one flat dataclass with 494 attributes, so `server_args.tp_size`, `ServerArgs(model_path=..., tp_size=8)`, pickling to a subprocess and every existing call site are untouched. ### Field order is a contract, so it is written down A dataclass turns field order into a positional constructor signature, and collecting whole namespaces groups fields that used to be interleaved. Keeping `model_path` first is not enough: `ServerArgs("dummy", "/tmp/tokenizer")` would set `load_format="/tmp/tokenizer"` and leave `tokenizer_path=None`, which then selects an invalid model loader -- silently, at a call site that did not change. So `arg_groups/field_order.py` records the order the record had before the split, and `collect_input_fields` orders what it collects by it. A field the record declares that the frozen order does not name goes after it, in declaration order -- the only backward-compatible place for a new field anyway, so a new declaration needs no edit there. The list is a compatibility record and nothing else reads it; the namespace a field belongs to is still the module it is declared in. ## Verification Four ways, all against the base commit: | check | result | |---|---| | `namespace_of` map, field by field | 494 / 494, **0 differences** | | CLI surface (options, defaults, choices, actions) | 507 / 507, **0 differences** | | field order, name by name | 494 / 494, **identical to the base** | | resolution result, 24 launch shapes x 489 fields | **0 differences** | | names importable from `sglang.srt.server_args` | nothing lost | Plus a full registered-unit sweep (648 files) against the stack's merge-base: 19 failures on both sides, the same 19, none of them config.
168 lines
7.2 KiB
Python
168 lines
7.2 KiB
Python
"""Config fields of the ``disagg`` namespace.
|
|
|
|
One class per namespace. The class *is* the namespace: a field declared here
|
|
lands in the ``disagg`` bag, which is what ``get_disagg()`` returns, so a reader
|
|
spells it exactly as before. ``ServerArgs`` composes these classes, so the
|
|
record stays one flat object -- the split moves where declarations live, not
|
|
how config is shaped at runtime.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import dataclasses
|
|
from typing import (
|
|
List,
|
|
Literal,
|
|
Optional,
|
|
)
|
|
|
|
from sglang.srt.arg_groups.arg_utils import (
|
|
A,
|
|
Arg,
|
|
)
|
|
from sglang.srt.arg_groups.choices import DISAGG_TRANSFER_BACKEND_CHOICES
|
|
from sglang.srt.utils.common import json_list_type
|
|
|
|
|
|
@dataclasses.dataclass
|
|
class Disagg:
|
|
"""Namespace ``disagg``."""
|
|
|
|
_NS_PATH = "disagg"
|
|
|
|
# Decoupled speculative decoding: draft and verify run as
|
|
# separate engines, currently connected by a ZMQ IPC mesh.
|
|
decoupled_spec_bind_endpoint: A[
|
|
Optional[str],
|
|
"ZMQ endpoint this engine binds for its inbound channel in decoupled "
|
|
"speculative decoding (verifier: result PULL; drafter: control PULL).",
|
|
] = None
|
|
decoupled_spec_connect_endpoints: A[
|
|
Optional[List[str]],
|
|
Arg(
|
|
help="Peer inbound (bind) endpoints to connect to, ordered by peer "
|
|
"rank, for decoupled speculative decoding.",
|
|
type_parser=json_list_type,
|
|
),
|
|
] = None
|
|
decoupled_spec_rank: A[
|
|
Optional[int],
|
|
"This engine's rank within its own role space (verifier-rank or "
|
|
"drafter-rank) for decoupled speculative decoding.",
|
|
] = None
|
|
decoupled_spec_role: A[
|
|
Literal["null", "verifier", "drafter"],
|
|
"Role in decoupled speculative decoding: 'null' disables it, 'verifier' "
|
|
"runs the target/verify half, 'drafter' runs the draft half.",
|
|
] = "null"
|
|
|
|
# -------------------------------------------------------------------------
|
|
# PD disaggregation
|
|
# -------------------------------------------------------------------------
|
|
disaggregation_mode: A[
|
|
Literal["null", "prefill", "decode"],
|
|
'Only used for PD disaggregation. "prefill" for prefill-only server, and "decode" for decode-only server. If not specified, it is not PD disaggregated',
|
|
] = "null"
|
|
disaggregation_transfer_backend: A[
|
|
str,
|
|
Arg(
|
|
help="The backend for disaggregation transfer. Default is mooncake.",
|
|
choices=DISAGG_TRANSFER_BACKEND_CHOICES,
|
|
),
|
|
] = "mooncake"
|
|
disaggregation_bootstrap_port: A[
|
|
int, "Bootstrap server port on the prefill server. Default is 8998."
|
|
] = 8998
|
|
disaggregation_ib_device: A[
|
|
Optional[str],
|
|
'The InfiniBand devices for disaggregation transfer. Supports a single device (e.g., --disaggregation-ib-device mlx5_0), a shared comma-separated list (e.g., --disaggregation-ib-device mlx5_0,mlx5_1), a per-GPU JSON mapping (e.g., --disaggregation-ib-device \'{"0": "mlx5_0,mlx5_1", "1": "mlx5_2"}\'), or a path to a JSON file containing that mapping. Default is None, which triggers automatic device detection when mooncake backend is enabled.',
|
|
] = None
|
|
disaggregation_decode_enable_radix_cache: A[
|
|
bool,
|
|
"Enable radix cache on decode server (PD mode). Caches KV prefixes to avoid redundant transfers. Incompatible with --enable-hisparse, speculative decoding, and --disaggregation-transfer-backend fake.",
|
|
] = False
|
|
disaggregation_decode_enable_offload_kvcache: A[
|
|
bool, "Enable async KV cache offloading on decode server (PD mode)."
|
|
] = False
|
|
disaggregation_decode_retraction_backup: A[
|
|
Optional[str],
|
|
Arg(
|
|
help=(
|
|
"Storage backend for KV preserved across PD decode retraction. "
|
|
"'cpu_tensor' uses per-request CPU tensors. 'host_pool' uses "
|
|
"a reserved HiCache pool and does not fall back on exhaustion. "
|
|
"If omitted, the backend is inferred from the decode KV pool."
|
|
),
|
|
choices=["cpu_tensor", "host_pool"],
|
|
),
|
|
] = None
|
|
num_reserved_decode_tokens: A[
|
|
int,
|
|
"Number of decode tokens that will have memory reserved when adding new request to the running batch.",
|
|
] = 512
|
|
disaggregation_decode_extra_slots: A[
|
|
Optional[int],
|
|
"Number of extra decode req_to_token slots pre-allocated for in-transfer requests (PD mode). If unset, defaults to 0 (or 2x the per-worker running batch for small batches).",
|
|
] = None
|
|
disaggregation_decode_polling_interval: A[
|
|
int,
|
|
"The interval to poll requests in decode server. Can be set to >1 to reduce the overhead of this.",
|
|
] = 1
|
|
optimistic_prefill_attempts: A[
|
|
int, "Number of optimistic prefill forward passes that skip the bootstrap wait."
|
|
] = 0
|
|
|
|
# -------------------------------------------------------------------------
|
|
# Encode prefill disaggregation
|
|
# -------------------------------------------------------------------------
|
|
encoder_only: A[
|
|
bool,
|
|
"For MLLM with an encoder, launch an encoder-only server",
|
|
] = False
|
|
language_only: A[
|
|
bool,
|
|
"For VLM, load weights for the language model only.",
|
|
] = False
|
|
language_model_only: A[
|
|
bool,
|
|
"Skip the multimodal encoder entirely: its weights are never loaded and the "
|
|
"tower is never built, freeing that GPU memory for KV cache. Multimodal "
|
|
"requests are rejected. Unlike --language-only this is a standalone mode, "
|
|
"not part of encoder/decoder disaggregation.",
|
|
] = False
|
|
encoder_transfer_backend: A[
|
|
str,
|
|
Arg(
|
|
help="The backend for encoder disaggregation transfer. Auto selects a model- and TP-aware backend.",
|
|
choices=["auto", "zmq_to_scheduler", "zmq_to_tokenizer", "mooncake"],
|
|
),
|
|
] = "auto"
|
|
encoder_urls: A[List[str], "List of encoder server urls."] = dataclasses.field(
|
|
default_factory=list
|
|
)
|
|
encoder_bootstrap_port: A[
|
|
int,
|
|
"Port for the EncoderBootstrapServer that runs in the language-only tokenizer manager process. Encoders register here, and language-only receivers fetch the current URL list from here.",
|
|
] = 8997
|
|
encoder_register_urls: A[
|
|
List[str],
|
|
"One or more EncoderBootstrapServer URLs to register this encoder with on startup, for dynamic encoder discovery. Example: --encoder-register-urls http://prefill0:8997 http://prefill1:8997. Used with --encoder-only servers.",
|
|
] = dataclasses.field(default_factory=list)
|
|
enable_adaptive_dispatch_to_encoder: A[
|
|
bool,
|
|
"When enabled, adaptively dispatch: multi-image requests go to encoder in language_only epd mode, single-image requests are processed locally.",
|
|
] = False
|
|
|
|
# -------------------------------------------------------------------------
|
|
# PD-Multiplexing
|
|
# -------------------------------------------------------------------------
|
|
enable_pdmux: A[
|
|
bool,
|
|
"Enable PD-Multiplexing, PD running on greenctx stream.",
|
|
] = False
|
|
pdmux_config_path: A[
|
|
Optional[str],
|
|
"The path of the PD-Multiplexing config file.",
|
|
] = None
|
|
sm_group_num: A[int, "Number of sm partition groups."] = 8
|