Cheng Wan
b99175dc7d
Last of four; stacked on #38048. The record is the operator's input; the bags are what is in effect. A reader that takes the record and reads a field off it gets the input, which is the wrong one of the two whenever resolution decided something -- and the mistake is silent, because for most fields and most launches the two agree. Several of these files already read both ways, sometimes in the same expression: ```python get_tokenizer( get_serving().tokenizer_path, tokenizer_mode=server_args.tokenizer_mode, # the input, not the decision ... ) ``` Sixty-odd files convert. Record field reads in runtime code go from 199 to 11. Nine parameters that the conversion emptied are dropped along with the argument at every call site -- the dead-parameter ratchet is what names them. ### "Runs after its process publishes" is a per-entry-point claim Most converted reads sit in the serving and model-executor layers, which only exist after publication, or in the two subprocess entry points, which publish first thing. Three places are not like that, and they keep reading the record they were handed: - **`HttpServerEngineAdapter`** launches the server as a *child*. The parent resolves the record and never publishes, so the adapter's own reads -- the launch banner, the API key in its readiness loop, the TP width in `update_weights_from_tensor` -- are of `self.server_args`. A bag read here fails closed in a bare process, or answers for an unrelated engine in one that happens to have published. - **`serve_grpc`** reads its sidecar port before the integrated servicer builds the `Engine` that publishes. The comment above that line already said so and already bound `cfg = resolving_view(server_args)` for it; the sidecar port and the port it derives from read `cfg`. - **`initialize_dp_attention`** runs from callers whose publish is not guaranteed, so its one predicate stays on the resolution view. `ROLE_NAMESPACE_SETS["dp_controller"]` gains `observability` and `serving`, because the controller's metrics gate, tracing setup and worker-port broadcast now read those namespaces. Under `SGLANG_ROLE_NAMESPACES=enforce` that set is what the process may read, so a conversion that reaches a new namespace has to widen it in the same change. ## Three things worth a reviewer's attention **Eleven reads were `getattr(record, "field", default)`.** An AST scan for attribute access does not see those, so the census that said "43 readers" was counting the shape it could match rather than the thing it was after. `incremental_streaming_output` was read that way twice, and the transcription tests were the only reason it surfaced. **Not every record read is a bag read waiting to happen.** A multimodal processor's `base_gpu_id` is the instance's, not the process's: two engines in one process keep different ones, and `test_publishing_another_config_does_not_move_the_device` exists to say so. It stays on the record while `rl_on_policy_target` beside it moves. `RequestMetricsExporter` is the same shape -- it is handed the directory it writes to, and a test builds several with different ones. `configure_logger` is a third: 17 call sites, one of which passes an `argparse.Namespace`, so it is not a global-context reader at all. Those eleven remaining reads are the ones with a reason. **The fixtures move with the code.** Tests that hung config off a mock manager now publish a record, which is what the serving layer reads; where a test states a value it says so with `override_server_args` instead of assigning through the mock. `test_hisparse_unit` is the last of them: it stubbed a `server_args` onto a fake scheduler to say the decode radix cache was off, and the value it was standing in for is the published default, so the stub goes and the class publishes. ## Two things CI caught that a local sweep could not **`unittest.TestCase.enterContext` is Python 3.11+.** The converted fixtures used it at 18 sites; `requires-python` is `>=3.10` and CI runs 3.10, so every one of them raised `AttributeError` there while passing on a newer local interpreter. They call `enter_override(self, ...)` now -- a four-line helper in `sglang/test/test_utils.py` over the override's own `install()` / `restore()`. **A batched sweep cannot see a missing publish.** Three fixtures needed a published config and did not have one; each *passed* inside a shard where some other file had published, and failed when run alone. The affected cases are `test_serving_completions` (which set `incremental_streaming_output` on the mock manager's record, where nothing reads it now), `test_qwen3_vl_feature_materialization` (same shape for `mm_enable_dp_encoder`), and the two Qwen Rust tests -- whose fixture already carried the comment `# Non-auto: get_resolved_model_impl would choke on a SimpleNamespace` next to the `model_impl` it sets, which is exactly what happened once `get_mm_processor_cls` started reading that value from the bag. Its `publish` mirrors `model_impl` now, like the four fields it already mirrored. ## Verification A full registered-unit sweep (648 files) against this stack's merge-base: 19 failures on both sides, the same 19, none of them config. That sweep is what caught 23 failures the file-scoped runs missed -- and, later, that the narrower 139-file list did not even contain the files this change reaches. It is also what caught the `test_hisparse_unit` fixture above: the file passes inside a shard where something else published, and fails when it is run on its own, which is why every failing file is re-run alone before it is counted.
125 lines
4.2 KiB
Python
125 lines
4.2 KiB
Python
# SPDX-License-Identifier: Apache-2.0
|
|
# SPDX-FileCopyrightText: Copyright contributors to the SGLang project
|
|
"""ms_runner launch MindSpore distributed modules."""
|
|
|
|
import logging
|
|
import multiprocessing as mp
|
|
import os
|
|
import sys
|
|
from pathlib import Path
|
|
|
|
import mindspore as ms
|
|
import torch
|
|
from mindspore._c_expression import GroupOptions
|
|
from mindspore.communication import create_group
|
|
|
|
from sglang.srt.distributed.parallel_state import _groups
|
|
from sglang.srt.runtime_context import (
|
|
get_parallel,
|
|
get_serving,
|
|
)
|
|
|
|
logger = logging.getLogger(__name__)
|
|
|
|
|
|
class _Tmp:
|
|
def __init__(self):
|
|
self.sched_p = None
|
|
|
|
def set_sched_process(self, p):
|
|
self.sched_p = p
|
|
|
|
def __del__(self):
|
|
if self.sched_p:
|
|
self.sched_p.kill()
|
|
|
|
|
|
_tmp = _Tmp()
|
|
|
|
|
|
def _get_host_and_ip(distributed_init_method):
|
|
try:
|
|
_, ip_str, port_str = distributed_init_method.split(":")
|
|
ip = ip_str.split("/")[-1]
|
|
port = int(port_str)
|
|
except Exception as e:
|
|
raise RuntimeError(
|
|
"Cannot get host and port information from %s, error: %s!"
|
|
% (distributed_init_method, str(e))
|
|
)
|
|
|
|
return ip, port
|
|
|
|
|
|
def run_scheduler_init(rank, local_rank, world_size, master_addr, master_port):
|
|
with open(str(Path() / "schedule.log"), "w") as scheduler_f:
|
|
# For Python outputs.
|
|
sys.stdout = scheduler_f
|
|
sys.stderr = scheduler_f
|
|
# For C++ outputs.
|
|
os.dup2(scheduler_f.fileno(), 1)
|
|
os.dup2(scheduler_f.fileno(), 2)
|
|
os.environ["DEVICE_ID"] = str(local_rank)
|
|
os.environ["MS_WORKER_NUM"] = str(world_size)
|
|
os.environ["MS_ROLE"] = "MS_SCHED"
|
|
os.environ["MS_NODE_ID"] = str(rank)
|
|
os.environ["MS_SCHED_HOST"] = str(master_addr)
|
|
os.environ["MS_SCHED_PORT"] = str(master_port)
|
|
# This function is blocked until the whole cluster exits.
|
|
ms.communication.init()
|
|
|
|
|
|
def set_ms_parallel_env(rank, local_rank, world_size, init_method):
|
|
master_addr, master_port = _get_host_and_ip(init_method)
|
|
# change port avoiding port conflicts with torch
|
|
master_port = master_port + 35 if master_port < 65500 else master_port - 35
|
|
if not os.getenv("MS_ROLE"):
|
|
if rank == 0:
|
|
# Create a subprocess for scheduler of MindSpore, just for internal collaboration, not for collective communication
|
|
sched_p = mp.Process(
|
|
target=run_scheduler_init,
|
|
args=(rank, local_rank, world_size, master_addr, master_port),
|
|
)
|
|
sched_p.start()
|
|
global _tmp
|
|
_tmp.set_sched_process(sched_p)
|
|
|
|
os.environ["DEVICE_ID"] = str(local_rank)
|
|
os.environ["MS_WORKER_NUM"] = str(world_size)
|
|
os.environ["MS_ROLE"] = "MS_WORKER"
|
|
os.environ["MS_NODE_ID"] = str(rank)
|
|
os.environ["MS_SCHED_HOST"] = str(master_addr)
|
|
os.environ["MS_SCHED_PORT"] = str(master_port)
|
|
|
|
|
|
def reuse_hccl_comm():
|
|
for group_name, group in _groups.items():
|
|
# Torch ProcessGroupHccl
|
|
device_group = group().device_group
|
|
hccl_comm_handle = device_group._get_backend(torch.device("npu")).get_hccl_comm(
|
|
group().local_rank
|
|
)
|
|
logger.info(
|
|
f"MindSpore reuse torch group: {device_group}, group_name: {group_name}, local rank: {group().local_rank},"
|
|
f"hccl communicator handle: {hex(hccl_comm_handle)}",
|
|
)
|
|
# Create MS communication group by hccl comm handle to reuse Torch group.
|
|
group_options = GroupOptions()
|
|
group_options.hccl_config = {"hccl_comm": hccl_comm_handle}
|
|
create_group(group_name, group().ranks, group_options)
|
|
|
|
|
|
def init_ms_distributed(world_size, rank, local_rank, port):
|
|
if get_parallel().dist_init_addr:
|
|
dist_init_method = f"tcp://{get_parallel().dist_init_addr}"
|
|
else:
|
|
dist_init_method = f"tcp://{get_serving().host}:{port}"
|
|
set_ms_parallel_env(rank, local_rank, world_size, dist_init_method)
|
|
|
|
ms.set_context(infer_boost="on", jit_level="O0")
|
|
ms.set_context(mode=ms.context.PYNATIVE_MODE)
|
|
ms.set_device("Ascend", local_rank)
|
|
ms.communication.init("hccl")
|
|
# After distributed job is initialized, reuse hccl comms for MindSpore.
|
|
reuse_hccl_comm()
|