[CI] Add ci-workflow-guide skill and consolidate CI docs (#21429)
This commit is contained in:
Liangsheng Yin
@@ -5,7 +5,7 @@ description: Guide for writing SGLang CI/UT tests. Covers CustomTestCase, CI reg
|
||||
|
||||
# Writing SGLang CI / UT Tests
|
||||
|
||||
**Before or while applying this skill, read [`test/README.md`](../../../test/README.md)** in the repo root’s `test/` tree. It documents the three-stage CI pipeline, suite naming, how tests are discovered and executed, file layout, and practical tips that complement the sections below.
|
||||
This skill covers **how to write and register tests**. For CI pipeline internals (stage ordering, fast-fail, gating, partitioning, debugging CI failures), see the [CI workflow guide](../ci-workflow-guide/SKILL.md).
|
||||
|
||||
## Core Rules
|
||||
|
||||
@@ -48,18 +48,74 @@ Defined in `python/sglang/test/test_utils.py`:
|
||||
| `DEFAULT_SMALL_EMBEDDING_MODEL_NAME_FOR_TEST` | — | Embedding tests |
|
||||
| `DEFAULT_SMALL_VLM_MODEL_NAME_FOR_TEST` | — | Vision-language tests |
|
||||
|
||||
### All CI suites
|
||||
### Naming Conventions
|
||||
|
||||
| Suite | Runner | Scenario |
|
||||
|-------|--------|----------|
|
||||
| `stage-a-test-cpu` | CPU | CPU unit tests |
|
||||
| `stage-b-test-1-gpu-small` | 1× 5090 (32GB) | Small model tests (1B, 8B) |
|
||||
| `stage-b-test-1-gpu-large` | 1× H100 (80GB) | Large model tests or performance related |
|
||||
| `stage-b-test-2-gpu-large` | 2× H100 | TP=2 tests |
|
||||
| `stage-c-test-4-gpu-h100` | 4× H100 | TP=4 / EP tests |
|
||||
| `stage-c-test-8-gpu-h200` | 8× H200 | Large-scale multi-GPU |
|
||||
| `nightly-1-gpu` | 1 GPU | Nightly-only |
|
||||
| `nightly-8-gpu` | 8 GPU | Nightly-only |
|
||||
- **Suite**: `stage-{a,b,c}-test-{gpu_count}-gpu-{hardware}` (e.g., `stage-b-test-1-gpu-small`)
|
||||
- **CI runner**: `{gpu_count}-gpu-{hardware}` (e.g., `1-gpu-5090`, `4-gpu-h100`, `8-gpu-h200`)
|
||||
|
||||
### All CI Suites
|
||||
|
||||
#### Per-commit (CUDA)
|
||||
|
||||
| Suite | Runner (label) | Description |
|
||||
|-------|----------------|-------------|
|
||||
| `stage-a-test-1-gpu-small` | `1-gpu-5090` | Quick checks on a small NVIDIA GPU before heavier stages |
|
||||
| `stage-a-test-cpu` | `ubuntu-latest` | CPU-only unit tests |
|
||||
| `stage-b-test-1-gpu-small` | `1-gpu-5090` | Core engine tests that fit a 5090-class card |
|
||||
| `stage-b-test-1-gpu-large` | `1-gpu-h100` | Tests that need H100-class memory or kernels (e.g. FA3) |
|
||||
| `stage-b-test-2-gpu-large` | `2-gpu-h100` | Two-GPU correctness and parallelism (TP/PP) on H100 |
|
||||
| `stage-b-test-4-gpu-b200` | `4-gpu-b200` | Early Blackwell coverage (SM100+ paths) on four GPUs |
|
||||
| `stage-b-kernel-unit-1-gpu-large` | `1-gpu-h100` | JIT kernel correctness tests under `python/sglang/jit_kernel/tests/` |
|
||||
| `stage-b-kernel-unit-8-gpu-h200` | `8-gpu-h200` | Multi-GPU JIT kernel correctness tests under `python/sglang/jit_kernel/tests/` |
|
||||
| `stage-b-kernel-benchmark-1-gpu-large` | `1-gpu-h100` | JIT kernel benchmark files under `python/sglang/jit_kernel/benchmark/` |
|
||||
| `stage-c-test-4-gpu-h100` | `4-gpu-h100` | Large 4-GPU H100 integration and scaling tests |
|
||||
| `stage-c-test-8-gpu-h200` | `8-gpu-h200` | Large 8-GPU H200 runs for big models and parallelism |
|
||||
| `stage-c-test-8-gpu-h20` | `8-gpu-h20` | Large 8-GPU H20 runs for big models |
|
||||
| `stage-c-test-deepep-4-gpu-h100` | `4-gpu-h100` | DeepEP expert-parallel and networking on four H100s |
|
||||
| `stage-c-test-deepep-8-gpu-h200` | `8-gpu-h200` | DeepEP at 8-GPU H200 scale |
|
||||
| `stage-c-test-8-gpu-b200` | `8-gpu-b200` | 8-GPU B200 suite (registered but not yet wired to a workflow) |
|
||||
| `stage-c-test-4-gpu-b200` | `4-gpu-b200` | 4-GPU B200 suite for large models on Blackwell |
|
||||
| `stage-c-test-4-gpu-gb200` | `4-gpu-gb200` | 4-GPU GB200 suite for large models on Grace Blackwell |
|
||||
|
||||
#### Per-commit (AMD)
|
||||
|
||||
| Suite | Runner (label) | Description |
|
||||
|-------|----------------|-------------|
|
||||
| `stage-a-test-1-gpu-small-amd` | `linux-mi325-1gpu-sglang` | Quick checks on one MI325-class GPU |
|
||||
| `stage-b-test-1-gpu-small-amd` | `linux-mi325-1gpu-sglang` | Core 1-GPU AMD tests (14 partitions) |
|
||||
| `stage-b-test-1-gpu-small-amd-nondeterministic` | `linux-mi325-1gpu-sglang` | Non-deterministic 1-GPU AMD tests |
|
||||
| `stage-b-test-1-gpu-small-amd-mi35x` | `linux-mi35x-gpu-1` | 1-GPU tests on MI35x hardware |
|
||||
| `stage-b-test-1-gpu-large-amd` | `linux-mi325-1gpu-sglang` | Large 1-GPU AMD tests (2 partitions) |
|
||||
| `stage-b-test-2-gpu-large-amd` | `linux-mi325-2gpu-sglang` | 2-GPU ROCm correctness and parallel setups |
|
||||
| `stage-b-test-large-8-gpu-35x-disaggregation-amd` | `linux-mi35x-gpu-8.fabric` | PD disaggregation and RDMA on 8×MI35x fabric |
|
||||
| `stage-c-test-4-gpu-amd` | `linux-mi325-4gpu-sglang` | 4-GPU AMD integration (2 partitions) |
|
||||
| `stage-c-test-large-8-gpu-amd` | `linux-mi325-8gpu-sglang` | 8-GPU MI325 scaling and integration |
|
||||
| `stage-c-test-large-8-gpu-amd-mi35x` | `linux-mi35x-gpu-8` | 8-GPU MI35x scaling (2 partitions) |
|
||||
|
||||
#### Nightly
|
||||
|
||||
Nightly suites are listed in `NIGHTLY_SUITES` in [`test/run_suite.py`](../../../test/run_suite.py). They run via `nightly-test-nvidia.yml` and `nightly-test-amd.yml`, not `pr-test.yml`. Examples:
|
||||
|
||||
- `nightly-1-gpu` (CUDA)
|
||||
- `nightly-kernel-1-gpu` (CUDA, JIT kernel full grids)
|
||||
- `nightly-kernel-8-gpu-h200` (CUDA, multi-GPU JIT kernel nightly)
|
||||
- `nightly-8-gpu-h200` (CUDA)
|
||||
- `nightly-eval-vlm-2-gpu` (CUDA)
|
||||
- `nightly-amd` (AMD)
|
||||
- `nightly-amd-8-gpu-mi35x` (AMD)
|
||||
|
||||
> **Note**: Multimodal diffusion uses `python/sglang/multimodal_gen/test/run_suite.py`, not `test/run_suite.py`.
|
||||
|
||||
### Choosing a Suite
|
||||
|
||||
Use the lightest suite that meets your test's needs:
|
||||
|
||||
- **No GPU required** → `stage-a-test-cpu`
|
||||
- **Most small GPU tests** → `stage-b-test-1-gpu-small` (default choice)
|
||||
- **Need H100 memory or Hopper features** → `stage-b-test-1-gpu-large`
|
||||
- **JIT kernel correctness** → `stage-b-kernel-unit-1-gpu-large`
|
||||
- **JIT kernel benchmarks** → `stage-b-kernel-benchmark-1-gpu-large`
|
||||
- **Multi-GPU** → only when the test actually needs multiple GPUs
|
||||
|
||||
---
|
||||
|
||||
@@ -236,24 +292,59 @@ Available fixtures in `python/sglang/test/server_fixtures/`:
|
||||
Every CI-discovered test file must call a registration function at module level:
|
||||
|
||||
```python
|
||||
from sglang.test.ci.ci_register import register_cuda_ci
|
||||
from sglang.test.ci.ci_register import (
|
||||
register_cuda_ci,
|
||||
register_amd_ci,
|
||||
register_cpu_ci,
|
||||
register_npu_ci,
|
||||
)
|
||||
|
||||
register_cuda_ci(est_time=60, suite="stage-b-test-1-gpu-small")
|
||||
# Per-commit test (small 1-gpu, runs on 5090)
|
||||
register_cuda_ci(est_time=80, suite="stage-b-test-1-gpu-small")
|
||||
|
||||
# Per-commit test (large 1-gpu, runs on H100)
|
||||
register_cuda_ci(est_time=120, suite="stage-b-test-1-gpu-large")
|
||||
|
||||
# Nightly-only test
|
||||
register_cuda_ci(est_time=200, suite="nightly-1-gpu", nightly=True)
|
||||
|
||||
# Multi-backend test (only when testing backend-specific code paths)
|
||||
register_cuda_ci(est_time=80, suite="stage-a-test-1-gpu-small")
|
||||
register_amd_ci(est_time=120, suite="stage-a-test-1-gpu-small-amd")
|
||||
register_npu_ci(est_time=400, suite="nightly-8-npu-a3", nightly=True)
|
||||
|
||||
# Temporarily disabled test
|
||||
register_cuda_ci(est_time=80, suite="stage-b-test-1-gpu-small", disabled="flaky - see #12345")
|
||||
```
|
||||
|
||||
Parameters:
|
||||
- `est_time`: estimated runtime in seconds (used for CI partitioning)
|
||||
- `suite`: which CI suite to run in (see suite table above)
|
||||
- `suite`: which CI suite to run in (see suite tables above)
|
||||
- `nightly=True`: for nightly-only tests (default `False` = per-commit)
|
||||
- `disabled="reason"`: temporarily disable with explanation
|
||||
|
||||
Only add `register_amd_ci` / `register_cpu_ci` when the test exercises backend-specific code paths.
|
||||
**Key principle**: Only add `register_amd_ci` / `register_npu_ci` when the test exercises backend-specific code paths. Common E2E tests just need `register_cuda_ci` — duplicating across backends wastes CI time.
|
||||
|
||||
For JIT kernel files:
|
||||
- Place correctness tests in `python/sglang/jit_kernel/tests/`
|
||||
- Place benchmarks in `python/sglang/jit_kernel/benchmark/`
|
||||
- Use `register_cuda_ci` with kernel suites such as `stage-b-kernel-unit-1-gpu-large`, `stage-b-kernel-benchmark-1-gpu-large`, and optionally `nightly-kernel-1-gpu`
|
||||
- Keep `est_time` and `suite` as literal values because `test/run_suite.py` collects them by AST parsing
|
||||
### JIT Kernel Registration
|
||||
|
||||
JIT kernel files live outside `test/registered/` but still use registration:
|
||||
|
||||
```python
|
||||
from sglang.test.ci.ci_register import register_cuda_ci
|
||||
|
||||
# Correctness tests in python/sglang/jit_kernel/tests/
|
||||
register_cuda_ci(est_time=30, suite="stage-b-kernel-unit-1-gpu-large")
|
||||
register_cuda_ci(est_time=120, suite="stage-b-kernel-unit-8-gpu-h200")
|
||||
|
||||
# Benchmarks in python/sglang/jit_kernel/benchmark/
|
||||
register_cuda_ci(est_time=6, suite="stage-b-kernel-benchmark-1-gpu-large")
|
||||
|
||||
# Optional nightly registration
|
||||
register_cuda_ci(est_time=120, suite="nightly-kernel-1-gpu", nightly=True)
|
||||
register_cuda_ci(est_time=120, suite="nightly-kernel-8-gpu-h200", nightly=True)
|
||||
```
|
||||
|
||||
Keep `est_time` and `suite` as **literal values** — `run_suite.py` collects them by AST parsing
|
||||
|
||||
---
|
||||
|
||||
|
||||
Reference in New Issue
Block a user