Cheng Wan
c87a2ced12
`test_bag_values_match_server_args` asserted `bag == field`. That holds today only because construction resolves in place; step 12 keeps the record raw, and the plan doc calls this test out as one that becomes **false by design** for every field resolution fills in. Rewritten against the resolved projection, which is the half that survives: the bag carries what resolution produced. The `bag == field` assertion stays as one line at the end, labelled as the tripwire -- when it starts failing for a resolution-written leaf, the flip has landed and the bag is the only place the effective value lives. The reference is an independent resolution of the same raw input (a fresh, never-published record) rather than `resolved_server_args_dict()`, which reads `vars(server_args)` back and therefore only restates the published instance. And the record goes through the real pipeline on a real mini config, published through `publish()`: the dummy-model path returns at the dummy boundary with every sampled leaf still raw, so the old comparison was raw==raw and vacuous (both Codex catches). Reproducibility (#34094) licenses the sibling as a stand-in for the pipeline output. The sample admits only leaves resolution writes on this input on both CI device shapes (attention_backend, page_size, chunked_prefill_size, mem_fraction_static), and the raw-differs guard asserts it per leaf -- a default-count threshold let supplied inputs like `model_path` (no dataclass default, so any path "differs") stand in for resolution work. Passthrough leaves (host, hicache_ratio, moe_runner_backend, model_path) move to a separate projection smoke that claims only what it checks: publish projected an unchanged field into its namespace. Between the two resolutions the test restores environ and the EnvField none-flags, so the sibling resolves the same pristine input rather than the first resolution's leftovers. And the class runs its body exactly once, like the other dual-resolve harnesses: a CI retry re-enters after the first attempt leaked process state, which is the hazard the pristine snapshot exists to rule out. docs(skill): a supplied-instance read is not automatically safe The whole-object rule said "keep the supplied-instance contract; don't rewrite the parameter reads unless the field is runtime-mutated". That is the right rule for the *object* and the wrong stopping point for the *field*: after step 12 the record carries the user's raw input, so `server_args.page_size` inside a runner-owned constructor reads the CLI default rather than the effective value. The rule now names that second case as step-12 debt with a guard attached (`test_supplied_instance_exposure_ratchet.py` fails on a new pair, so the decision is made when the read is written), and names the two shapes that stay parameter-form on purpose: a helper the resolution pipeline calls with a `resolved_view`, and a factory whose contract is "build X from the record you are handed".
Test and Continuous Integration (CI) System in SGLang
This page covers principles and essentials: folder layout, how to run tests, registration, and suite selection. For complete references, see the skill guides:
- Writing tests — templates, fixtures, model selection, complete suite tables, checklist:
.claude/skills/write-sglang-test/SKILL.md - CI pipeline internals — stage flow diagrams, fast-fail layers, gating, partitioning, execution modes, debugging failures:
.claude/skills/ci-workflow-guide/SKILL.md
CI Pipeline Overview
The CI pipeline runs in three sequential stages: A (pre-flight, ~3 min) → B (basic, ~30 min) → C (advanced, ~30 min). Kernel and multimodal-gen tests run in parallel with stage B. For details on stage gating, fast-fail mechanisms, execution modes (PR vs scheduled vs /rerun-stage), and debugging CI failures, see the CI workflow guide.
Folder Organization
registered/: CI test files, auto-discovered byrun_suite.py. Most tests live here. JIT kernel tests are an exception (see below).manual/: Non-CI tests for local debugging or special setups.run_suite.py: CI runner — scansregistered/and JIT kernel directories.
The system supports both unittest and pytest. The launcher runs python filename.py -f with failfast enabled by default.
Make sure your file ends with exactly one of:
# for unittest
if __name__ == "__main__":
unittest.main()
# for pytest
if __name__ == "__main__":
import sys
sys.exit(pytest.main([__file__]))
Do not add custom argparse or modify sys.argv before these calls — the CI runner appends -f for failfast.
Run Tests Locally
# Single file
python3 test/registered/core/test_srt_endpoint.py
# Single test method
python3 test/registered/core/test_srt_endpoint.py TestSRTEndpoint.test_simple_decode
# Single JIT kernel test
python3 test/registered/jit/test_add_constant.py
# Run a suite
python3 test/run_suite.py --hw cpu --suite base-a-test-cpu
python3 test/run_suite.py --hw cuda --suite base-a-test-1-gpu-small
# Nightly tests (CUDA nightly suites take no --nightly; the stage is in the name)
python3 test/run_suite.py --hw cuda --suite nightly-test-1-gpu-large
# With auto-partitioning (for parallel CI jobs)
python3 test/run_suite.py --hw cuda --suite base-b-test-1-gpu-small \
--auto-partition-id 0 --auto-partition-size 4
CI Registration
Every CI-discovered test file must call a registration function at module level:
from sglang.test.ci.ci_register import register_cuda_ci
register_cuda_ci(est_time=80, stage="base-b", runner_config="1-gpu-small")
Parameters: est_time (seconds), stage + runner_config (target stage and runner pool from scripts/ci/runner_configs.yml), nightly=True (nightly-only), disabled="reason" (temporarily disable).
Keep est_time, stage, runner_config as literal values — run_suite.py collects them by AST parsing.
JIT kernel correctness tests and benchmarks live under test/registered/jit/, same as other registered tests (their helpers stay alongside the kernel source under python/sglang/kernels/jit/ and are imported by absolute path):
- Correctness tests:
test/registered/jit/test_*.py→base-b-kernel-unit-test-1-gpu-large - Benchmarks:
test/registered/jit/benchmark/bench_*.py→base-b-kernel-benchmark-test-1-gpu-large
Choosing a Suite
Use the lightest suite that meets your test's needs. Full suite tables are in the write-sglang-test skill.
| Need | Suite |
|---|---|
| No GPU required | base-a-test-cpu |
| Small GPU (fits 5090, 32GB) | base-b-test-1-gpu-small (most tests go here) |
| Large GPU memory or Hopper features | base-b-test-1-gpu-large |
| JIT kernel correctness | base-b-kernel-unit-test-1-gpu-large |
| JIT kernel benchmarks | base-b-kernel-benchmark-test-1-gpu-large |
| Multi-GPU (2/4/8) | base-b-test-2-gpu-large, base-c-test-* |
| Long-running or experimental | nightly-* suites |
Steps for Adding a Test
See the write-sglang-test skill for templates, fixtures, model selection, and a complete checklist.
Multi-Hardware Backends
This README mostly describes the NVIDIA GPU CI pipeline. Other hardware backends (AMD, NPU) follow the same practices and use the multi-backend registry system. A scheduled job summarizes test coverage across all backends; here is an example run.
Tips
- Learn from existing examples in test/registered.
- Reuse servers — launching is expensive. Share one server across many test methods via
setUpClass. - Use as few GPUs as possible. Prefer 1-GPU runners.
- Each test file should take < 500 seconds; split if longer.
- Each GitHub Actions job should take < 30 minutes; split if longer.
- If tests are too slow for per-commit, consider nightly suites.
Other Notes
Adding New Models to Nightly CI
- Text models: Extend the global model list variables in
test_utils.py. - VLMs: Extend the
MODEL_THRESHOLDSdictionary intest/registered/eval/test_vlms_mmmu_eval.py.