sglang/test

Test and Continuous Integration (CI) System in SGLang

This page covers principles and essentials: folder layout, how to run tests, registration, and suite selection. For complete references, see the skill guides:

• Writing tests — templates, fixtures, model selection, complete suite tables, checklist: .claude/skills/write-sglang-test/SKILL.md
• CI pipeline internals — stage flow diagrams, fail-fast layers, gating, partitioning, execution modes, debugging failures: .claude/skills/ci-workflow-guide/SKILL.md

CI Pipeline Overview

The CI pipeline runs in three sequential stages: A (pre-flight, ~3 min) → B (basic, ~30 min) → C (advanced, ~30 min). Kernel and multimodal-gen tests run in parallel with stage B. For details on stage gating, fail-fast mechanisms, execution modes (PR vs scheduled vs manual dispatch), and debugging CI failures, see the CI workflow guide.

Folder Organization

• registered/: CI test files, including kernel tests and benchmarks, auto-discovered by run_suite.py.
• manual/: Non-CI tests for local debugging or special setups.
• run_suite.py: CI runner — scans registered/ recursively.

The system supports both unittest and pytest. The launcher runs python filename.py -f with failfast enabled by default.

Make sure your file ends with exactly one of:

# for unittest
if __name__ == "__main__":
    unittest.main()

# for pytest
if __name__ == "__main__":
    import sys
    sys.exit(pytest.main([__file__]))

Do not add custom argparse or modify sys.argv before these calls — the CI runner appends -f for failfast.

Run Tests Locally

# Single file
python3 test/registered/core/test_srt_endpoint.py

# Single test method
python3 test/registered/core/test_srt_endpoint.py TestSRTEndpoint.test_simple_decode

# Single JIT kernel test
python3 test/registered/kernels/ops/elementwise/test_add_constant.py

# Run a suite
python3 test/run_suite.py --hw cpu --suite base-a-test-cpu
python3 test/run_suite.py --hw cuda --suite base-a-test-1-gpu-small

# Nightly tests (CUDA nightly suites take no --nightly; the stage is in the name)
python3 test/run_suite.py --hw cuda --suite nightly-test-1-gpu-large

# With auto-partitioning (for parallel CI jobs)
python3 test/run_suite.py --hw cuda --suite base-b-test-1-gpu-small \
    --auto-partition-id 0 --auto-partition-size 4

CI Registration

Every CI-discovered test file must call a registration function at module level:

from sglang.test.ci.ci_register import register_cuda_ci

register_cuda_ci(est_time=80, stage="base-b", runner_config="1-gpu-small")

Parameters: est_time (seconds), stage + runner_config (target stage and runner pool from scripts/ci/runner_configs.yml), nightly=True (nightly-only), disabled="reason" (temporarily disable).

Keep est_time, stage, runner_config as literal values — run_suite.py collects them by AST parsing.

Directories under test/registered/ group tests by topic and are free-form (lora/, hicache/, disaggregation/, perf/, ...); unit tests cover one srt module, so they mirror the source tree under unit/. What a test costs, which stage gates it and which runner it needs are declared by its register_*_ci call -- including hardware, which is expressed by one or more register_*_ci calls and never by a new top-level directory. Kernel tests use test/registered/kernels/{ops,benchmark}/<group>/, retaining the established plural kernels root.

Diffusion workflows also enter through test/run_suite.py; registered bridge files preserve their case-level pytest partitioning until the remaining diffusion cases are moved out of the package test-support tree.

Kernel correctness tests and benchmarks use the established plural kernels root and mirror the operator group under python/sglang/kernels/ops/. Helpers stay alongside the kernel source under python/sglang/kernels/jit/ and are imported by absolute path:

• Correctness tests: test/registered/kernels/ops/<group>/test_*.py → base-b-kernel-unit-test-1-gpu-large
• Benchmarks: test/registered/kernels/benchmark/<group>/bench_*.py → base-b-kernel-benchmark-test-1-gpu-large

Choosing a Suite

Use the lightest suite that meets your test's needs. Full suite tables are in the write-sglang-test skill.

Need	Suite
No GPU required	base-a-test-cpu
Small GPU (fits 5090, 32GB)	base-b-test-1-gpu-small (most tests go here)
Large GPU memory or Hopper features	base-b-test-1-gpu-large
JIT kernel correctness	base-b-kernel-unit-test-1-gpu-large
JIT kernel benchmarks	base-b-kernel-benchmark-test-1-gpu-large
Multi-GPU (2/4/8)	base-b-test-2-gpu-large, base-c-test-*
Long-running or experimental	nightly-* suites

Steps for Adding a Test

See the write-sglang-test skill for templates, fixtures, model selection, and a complete checklist.

Before adding a registered test, identify the production change that would make it fail. Prefer extending an existing fixture/server launch over adding another file. The incremental admission check applies these ratchets to new or modified registered tests:

• Temporary disabled= registrations and unconditional skips must reference an issue and include until YYYY-MM-DD; expired entries fail lint.
• A file registered on CUDA plus another accelerator must place a nearby backend-specific: comment above the extra registration and name the path or failure mode that only that backend can catch.
• Default PR registrations are limited to 1,200 estimated weighted accelerator-seconds per backend (est_time * GPU count). Move larger matrices to extra/nightly, or document a nearby ci-cost-override: rationale.

Multi-Hardware Backends

This README mostly describes the NVIDIA GPU CI pipeline. Other hardware backends (AMD, NPU) follow the same practices and use the multi-backend registry system. A scheduled job summarizes test coverage across all backends; here is an example run.

Tips

• Learn from existing examples in test/registered.
• Reuse servers — launching is expensive. Share one server across many test methods via setUpClass.
• Use as few GPUs as possible. Prefer 1-GPU runners.
• Each test file should take < 500 seconds; split if longer.
• Each GitHub Actions job should take < 30 minutes; split if longer.
• If tests are too slow for per-commit, consider nightly suites.

Other Notes

Adding New Models to Nightly CI

• Text models: Extend the global model list variables in test_utils.py.
• VLMs: Extend the MODEL_THRESHOLDS dictionary in test/registered/accuracy/models/test_vlms_mmmu_eval.py.