[CI] Rename basic CI stage-a/b/c -> base-a/b/c for symmetry with extra CI (#25420)
This commit is contained in:
Liangsheng Yin
@@ -11,7 +11,7 @@ This skill covers the CI **infrastructure** layer — how tests are dispatched,
|
||||
|
||||
## Naming Conventions
|
||||
|
||||
- **Suite**: `stage-{a,b,c}-test-{gpu_count}-gpu-{hardware}` (e.g., `stage-b-test-1-gpu-small`)
|
||||
- **Suite**: `base-{a,b,c}-test-{gpu_count}-gpu-{hardware}` (e.g., `base-b-test-1-gpu-small`)
|
||||
- **Test group**: Directory-level registered test group under `test/registered/` (e.g., `hicache` maps to `test/registered/hicache/test_*.py`)
|
||||
- **CI runner**: `{gpu_count}-gpu-{hardware}` (e.g., `1-gpu-5090`, `4-gpu-h100`, `8-gpu-h200`)
|
||||
|
||||
@@ -23,7 +23,7 @@ This skill covers the CI **infrastructure** layer — how tests are dispatched,
|
||||
|------|------|
|
||||
| `.github/workflows/pr-test.yml` | Main workflow — all stages, jobs, conditions, matrix definitions |
|
||||
| `.github/workflows/pr-gate.yml` | PR gating: draft check, `run-ci` label, per-user rate limiting |
|
||||
| `.github/actions/check-stage-health/action.yml` | Cross-job fast-fail: queries API for any failed job |
|
||||
| `.github/actions/check-pr-test-health/action.yml` | Cross-job fast-fail: queries API for any failed job |
|
||||
| `.github/actions/wait-for-jobs/action.yml` | Stage gating: polls API until stage jobs complete |
|
||||
| `.github/actions/check-maintenance/action.yml` | Maintenance mode check |
|
||||
| `test/run_suite.py` | Suite runner: collects, filters, partitions, executes tests |
|
||||
@@ -49,53 +49,53 @@ This skill covers the CI **infrastructure** layer — how tests are dispatched,
|
||||
│ │
|
||||
▼ │
|
||||
┌─────────────────────────────────────┐ │
|
||||
│ Stage A (~3 min) │ │
|
||||
│ Base A (~3 min) │ │
|
||||
│ pre-flight check │ │
|
||||
│ │ │
|
||||
│ ┌─────────────────────────────┐ │ │
|
||||
│ │ stage-a-test-1-gpu-small │ │ │
|
||||
│ │ base-a-test-1-gpu-small │ │ │
|
||||
│ │ (small GPUs) │ │ │
|
||||
│ └─────────────────────────────┘ │ │
|
||||
│ ┌─────────────────────────────┐ │ │
|
||||
│ │ stage-a-test-cpu │ │ │
|
||||
│ │ base-a-test-cpu │ │ │
|
||||
│ │ (CPU) │ │ │
|
||||
│ └─────────────────────────────┘ │ │
|
||||
└──────┬──────────────────────────────┘ │
|
||||
│ │
|
||||
▼ ▼
|
||||
┌─────────────────────────────────────┐ ┌──────────────────────────┐
|
||||
│ Stage B (~30 min) │ │ kernel test │
|
||||
│ basic tests │ └──────────────────────────┘
|
||||
│ Base B (~30 min) │ │ kernel test │
|
||||
│ base tests │ └──────────────────────────┘
|
||||
│ │ ┌──────────────────────────┐
|
||||
│ ┌─────────────────────────────┐ │ │ multimodal gen test │
|
||||
│ │ stage-b-test-1-gpu-small │ │ └──────────────────────────┘
|
||||
│ │ base-b-test-1-gpu-small │ │ └──────────────────────────┘
|
||||
│ │ (small GPUs, e.g. 5090) │ │
|
||||
│ └─────────────────────────────┘ │
|
||||
│ ┌─────────────────────────────┐ │
|
||||
│ │ stage-b-test-1-gpu-large │ │
|
||||
│ │ base-b-test-1-gpu-large │ │
|
||||
│ │ (large GPUs, e.g. H100) │ │
|
||||
│ └─────────────────────────────┘ │
|
||||
│ ┌─────────────────────────────┐ │
|
||||
│ │ stage-b-test-2-gpu-large │ │
|
||||
│ │ base-b-test-2-gpu-large │ │
|
||||
│ │ (large GPUs, e.g. H100) │ │
|
||||
│ └─────────────────────────────┘ │
|
||||
└──────┬──────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────────────────────────┐
|
||||
│ Stage C (~30 min) │
|
||||
│ Base C (~30 min) │
|
||||
│ advanced tests │
|
||||
│ │
|
||||
│ ┌─────────────────────────────┐ │
|
||||
│ │ stage-c-test-4-gpu-h100 │ │
|
||||
│ │ base-c-test-4-gpu-h100 │ │
|
||||
│ │ (H100 GPUs) │ │
|
||||
│ └─────────────────────────────┘ │
|
||||
│ ┌─────────────────────────────┐ │
|
||||
│ │ stage-c-test-8-gpu-h200 │ │
|
||||
│ │ base-c-test-8-gpu-h200 │ │
|
||||
│ │ (8 x H200 GPUs) │ │
|
||||
│ └─────────────────────────────┘ │
|
||||
│ ┌─────────────────────────────┐ │
|
||||
│ │ stage-c-test-4-gpu-b200 │ │
|
||||
│ │ base-c-test-4-gpu-b200 │ │
|
||||
│ │ (4 x B200 GPUs) │ │
|
||||
│ └─────────────────────────────┘ │
|
||||
│ ┌─────────────────────────────┐ │
|
||||
@@ -112,9 +112,9 @@ This skill covers the CI **infrastructure** layer — how tests are dispatched,
|
||||
└─────────────────────────────────────┘
|
||||
```
|
||||
|
||||
**Every stage test job** includes a `check-stage-health` step after checkout — if any job in the run has already failed, the job fast-fails (red X) with a root cause annotation.
|
||||
**Every stage test job** includes a `check-pr-test-health` step after checkout — if any job in the run has already failed, the job fast-fails (red X) with a root cause annotation.
|
||||
|
||||
**Scheduled runs** skip `wait-for-stage-*` jobs, running all stages in parallel. Fast-fail is also disabled.
|
||||
**Scheduled runs** skip `wait-for-base-*` jobs, running all stages in parallel. Fast-fail is also disabled.
|
||||
|
||||
---
|
||||
|
||||
@@ -126,13 +126,13 @@ This skill covers the CI **infrastructure** layer — how tests are dispatched,
|
||||
|-------|-----------|-------------|----------------------|
|
||||
| **1. Test method → file** | `unittest -f` (failfast) | One test method fails → entire test file stops immediately | Yes |
|
||||
| **2. File → suite** | `run_unittest_files()` default | One test file fails → entire suite stops (`--continue-on-error` off) | Yes |
|
||||
| **3. Job → job (same stage)** | `check-stage-health` action | One job fails → other waiting jobs in same stage fast-fail (red X) | Yes |
|
||||
| **4. Stage → stage (cross-stage)** | `wait-for-stage` + `needs` | Stage A fails → stage B/C jobs skip entirely (never get a runner) | Yes (wait jobs skipped) |
|
||||
| **3. Job → job (same stage)** | `check-pr-test-health` action | One job fails → other waiting jobs in same stage fast-fail (red X) | Yes |
|
||||
| **4. Stage → stage (cross-stage)** | `wait-for-base-*` + `needs` | Base A fails → base B/C jobs skip entirely (never get a runner) | Yes (wait jobs skipped) |
|
||||
|
||||
- **Layer 1**: `-f` flag appended to all `python3 -m pytest` / `unittest` invocations in `ci_utils.py`
|
||||
- **Layer 2**: `--continue-on-error` flag in `run_suite.py` — off for PRs, on for scheduled runs
|
||||
- **Layer 3**: `check-stage-health` auto-detects `schedule` event and skips; filters out cascade failures to show only root cause jobs
|
||||
- **Layer 4**: `wait-for-stage-*` jobs are conditioned on `github.event_name == 'pull_request'` — skipped for scheduled runs
|
||||
- **Layer 3**: `check-pr-test-health` auto-detects `schedule` event and skips; filters out cascade failures to show only root cause jobs
|
||||
- **Layer 4**: `wait-for-base-*` jobs are conditioned on `github.event_name == 'pull_request'` — skipped for scheduled runs
|
||||
|
||||
---
|
||||
|
||||
@@ -140,8 +140,8 @@ This skill covers the CI **infrastructure** layer — how tests are dispatched,
|
||||
|
||||
| Aspect | PR (`pull_request`) | Scheduled (`cron`, every 6h) | `/rerun-stage` (`workflow_dispatch`) |
|
||||
|--------|---------------------|------------------------------|--------------------------------------|
|
||||
| **Stage ordering** | Sequential: A → B → C via `wait-for-stage-*` | Parallel (all at once) | Single target stage only |
|
||||
| **Cross-job fast-fail** | Yes (`check-stage-health`) | Yes | Yes |
|
||||
| **Stage ordering** | Sequential: A → B → C via `wait-for-base-*` | Parallel (all at once) | Single target stage only |
|
||||
| **Cross-job fast-fail** | Yes (`check-pr-test-health`) | Yes | Yes |
|
||||
| **continue-on-error** | No (stop at first failure within suite) | Yes (run all tests) | No |
|
||||
| **Retry** | Enabled | Enabled | Enabled |
|
||||
| **max_parallel** | 3 (default), 14 if `high priority` label | 14 | 3 (default), 14 if `high priority` |
|
||||
@@ -152,23 +152,23 @@ This skill covers the CI **infrastructure** layer — how tests are dispatched,
|
||||
|
||||
## Stage Gating (`wait-for-jobs` action)
|
||||
|
||||
`wait-for-stage-a` and `wait-for-stage-b` are lightweight `ubuntu-latest` jobs that poll the GitHub Actions API.
|
||||
`wait-for-base-a` and `wait-for-base-b` are lightweight `ubuntu-latest` jobs that poll the GitHub Actions API.
|
||||
|
||||
**How it works:**
|
||||
1. Calls `listJobsForWorkflowRun` to list all jobs in the current run
|
||||
2. Matches jobs by exact name or prefix (for matrix jobs, e.g., `stage-b-test-1-gpu-small (3)`)
|
||||
2. Matches jobs by exact name or prefix (for matrix jobs, e.g., `base-b-test-1-gpu-small (3)`)
|
||||
3. If any matched job has `conclusion === 'failure'` → fail immediately (fast-fail)
|
||||
4. If all matched jobs are completed and count matches `expected_count` → success
|
||||
5. Otherwise → sleep `poll-interval-seconds` (default: 60s) and retry
|
||||
6. Timeout after `max-wait-minutes` (240 min for stage-a, 480 min for stage-b)
|
||||
6. Timeout after `max-wait-minutes` (240 min for base-a, 480 min for base-b)
|
||||
|
||||
**Job specs example** (stage-b):
|
||||
**Job specs example** (base-b):
|
||||
```json
|
||||
[
|
||||
{"prefix": "stage-b-test-1-gpu-small", "expected_count": 8},
|
||||
{"prefix": "stage-b-test-1-gpu-large", "expected_count": 14},
|
||||
{"prefix": "stage-b-test-2-gpu-large", "expected_count": 4},
|
||||
{"prefix": "stage-b-test-4-gpu-b200", "expected_count": 1}
|
||||
{"prefix": "base-b-test-1-gpu-small", "expected_count": 8},
|
||||
{"prefix": "base-b-test-1-gpu-large", "expected_count": 14},
|
||||
{"prefix": "base-b-test-2-gpu-large", "expected_count": 4},
|
||||
{"prefix": "base-b-test-4-gpu-b200", "expected_count": 1}
|
||||
]
|
||||
```
|
||||
|
||||
@@ -178,17 +178,17 @@ This skill covers the CI **infrastructure** layer — how tests are dispatched,
|
||||
|
||||
---
|
||||
|
||||
## Cross-Job Fast-Fail (`check-stage-health` action)
|
||||
## Cross-Job Fast-Fail (`check-pr-test-health` action)
|
||||
|
||||
Composite action called after checkout in every stage test job (21 jobs total across `pr-test.yml`, `pr-test-multimodal-gen.yml`, `pr-test-sgl-kernel.yml`, `pr-test-jit-kernel.yml`).
|
||||
|
||||
**How it works:**
|
||||
1. Queries `listJobsForWorkflowRun` for the current workflow run
|
||||
2. Filters for **root cause failures only** — jobs with `conclusion === 'failure'` whose failing step is NOT `check-stage-health` (excludes cascade failures)
|
||||
2. Filters for **root cause failures only** — jobs with `conclusion === 'failure'` whose failing step is NOT `check-pr-test-health` (excludes cascade failures)
|
||||
3. If root cause failures found → calls `core.setFailed()` with the list of root cause job names
|
||||
4. If none → does nothing (step succeeds)
|
||||
|
||||
**Cascade filtering**: When job A fast-fails due to health check, it also has `conclusion: failure`. Without filtering, job B would list both the original failure AND job A's fast-fail. The filter checks each failed job's `steps` array — if the failing step name contains `check-stage-health` or `Check stage health`, it's excluded from the root cause list.
|
||||
**Cascade filtering**: When job A fast-fails due to health check, it also has `conclusion: failure`. Without filtering, job B would list both the original failure AND job A's fast-fail. The filter checks each failed job's `steps` array — if the failing step name contains `check-pr-test-health` or `Check PR test health`, it's excluded from the root cause list.
|
||||
|
||||
**Usage pattern:**
|
||||
```yaml
|
||||
@@ -197,8 +197,8 @@ steps:
|
||||
uses: actions/checkout@v4
|
||||
...
|
||||
|
||||
- uses: ./.github/actions/check-stage-health
|
||||
id: stage-health
|
||||
- uses: ./.github/actions/check-pr-test-health
|
||||
id: pr-test-health
|
||||
|
||||
- name: Install dependencies # skipped automatically if health check failed
|
||||
... # (default if: success() is false)
|
||||
@@ -213,7 +213,7 @@ steps:
|
||||
|
||||
**Error message example:**
|
||||
```
|
||||
Fast-fail: skipping — root cause job(s): stage-b-test-1-gpu-small (0), stage-b-test-1-gpu-small (1)
|
||||
Fast-fail: skipping — root cause job(s): base-b-test-1-gpu-small (0), base-b-test-1-gpu-small (1)
|
||||
```
|
||||
|
||||
---
|
||||
@@ -270,27 +270,26 @@ Large suites are split across matrix jobs using the **LPT (Longest Processing Ti
|
||||
|
||||
| Suite | Partitions | Runner | max_parallel |
|
||||
|-------|-----------|--------|-------------|
|
||||
| `stage-a-test-1-gpu-small` | 1 (no matrix) | `1-gpu-5090` | — |
|
||||
| `stage-a-test-cpu` | 4 | `ubuntu-latest` | — |
|
||||
| `stage-b-test-1-gpu-small` | 8 | `1-gpu-5090` | 8 |
|
||||
| `stage-b-test-1-gpu-large` | 14 | `1-gpu-h100` | dynamic (3 or 14) |
|
||||
| `stage-b-test-2-gpu-large` | 4 | `2-gpu-h100` | — |
|
||||
| `stage-b-test-4-gpu-b200` | 1 (no matrix) | `4-gpu-b200` | — |
|
||||
| `stage-b-kernel-unit-1-gpu-large` | 1 (no matrix) | `1-gpu-h100` | — |
|
||||
| `stage-b-kernel-unit-1-gpu-b200` | 1 (no matrix) | `4-gpu-b200` | — |
|
||||
| `stage-b-kernel-unit-8-gpu-h200` | 1 (no matrix) | `8-gpu-h200` | — |
|
||||
| `stage-b-kernel-benchmark-1-gpu-large` | 1 (no matrix) | `1-gpu-h100` | — |
|
||||
| `stage-c-test-4-gpu-h100` | 3 | `4-gpu-h100` | — |
|
||||
| `stage-c-test-8-gpu-h200` | 4 | `8-gpu-h200` | — |
|
||||
| `stage-c-test-8-gpu-h20` | 2 | `8-gpu-h20` | — |
|
||||
| `stage-c-test-deepep-4-gpu-h100` | 1 (no matrix) | `4-gpu-h100` | — |
|
||||
| `stage-c-test-deepep-8-gpu-h200` | 1 (no matrix) | `8-gpu-h200` | — |
|
||||
| `stage-c-test-4-gpu-b200` | 3 | `4-gpu-b200` | — |
|
||||
| `stage-c-test-4-gpu-b200-small` | 3 | `4-gpu-b200-low-disk` | — |
|
||||
| `stage-c-test-8-gpu-b200` | registered only | `8-gpu-b200` | — |
|
||||
| `stage-c-test-4-gpu-gb200` | registered only | `4-gpu-gb200` | — |
|
||||
| `base-a-test-1-gpu-small` | 1 (no matrix) | `1-gpu-5090` | — |
|
||||
| `base-a-test-cpu` | 4 | `ubuntu-latest` | — |
|
||||
| `base-b-test-1-gpu-small` | 8 | `1-gpu-5090` | 8 |
|
||||
| `base-b-test-1-gpu-large` | 14 | `1-gpu-h100` | dynamic (3 or 14) |
|
||||
| `base-b-test-2-gpu-large` | 4 | `2-gpu-h100` | — |
|
||||
| `base-b-test-4-gpu-b200` | 1 (no matrix) | `4-gpu-b200` | — |
|
||||
| `base-b-kernel-unit-1-gpu-large` | 1 (no matrix) | `1-gpu-h100` | — |
|
||||
| `base-b-kernel-unit-1-gpu-b200` | 1 (no matrix) | `4-gpu-b200` | — |
|
||||
| `base-b-kernel-unit-8-gpu-h200` | 1 (no matrix) | `8-gpu-h200` | — |
|
||||
| `base-b-kernel-benchmark-1-gpu-large` | 1 (no matrix) | `1-gpu-h100` | — |
|
||||
| `base-c-test-4-gpu-h100` | 3 | `4-gpu-h100` | — |
|
||||
| `base-c-test-8-gpu-h200` | 4 | `8-gpu-h200` | — |
|
||||
| `base-c-test-8-gpu-h20` | 2 | `8-gpu-h20` | — |
|
||||
| `base-c-test-deepep-4-gpu-h100` | 1 (no matrix) | `4-gpu-h100` | — |
|
||||
| `base-c-test-4-gpu-b200` | 3 | `4-gpu-b200` | — |
|
||||
| `base-c-test-4-gpu-b200-small` | 3 | `4-gpu-b200-low-disk` | — |
|
||||
| `base-c-test-8-gpu-b200` | registered only | `8-gpu-b200` | — |
|
||||
| `base-c-test-4-gpu-gb200` | registered only | `4-gpu-gb200` | — |
|
||||
|
||||
> **Note**: Kernel suites (`stage-b-kernel-*`) run via `pr-test-jit-kernel.yml` and `pr-test-sgl-kernel.yml`, not the main `pr-test.yml`. `stage-c-test-8-gpu-b200` is registered in `test/run_suite.py` but not wired to PR CI. The GB200 job is currently commented out in `pr-test.yml` until a company-owned runner is provisioned. Multimodal diffusion uses `python/sglang/multimodal_gen/test/run_suite.py`, not `test/run_suite.py`.
|
||||
> **Note**: Kernel suites (`base-b-kernel-*`) run via `pr-test-jit-kernel.yml` and `pr-test-sgl-kernel.yml`, not the main `pr-test.yml`. `base-c-test-8-gpu-b200` is registered in `test/run_suite.py` but not wired to PR CI. The GB200 job is currently commented out in `pr-test.yml` until a company-owned runner is provisioned. Multimodal diffusion uses `python/sglang/multimodal_gen/test/run_suite.py`, not `test/run_suite.py`.
|
||||
|
||||
**Workflow usage:**
|
||||
```yaml
|
||||
@@ -298,7 +297,7 @@ strategy:
|
||||
matrix:
|
||||
partition: [0, 1, 2, 3, 4, 5, 6, 7]
|
||||
steps:
|
||||
- run: python3 run_suite.py --hw cuda --suite stage-b-test-1-gpu-small \
|
||||
- run: python3 run_suite.py --hw cuda --suite base-b-test-1-gpu-small \
|
||||
--auto-partition-id ${{ matrix.partition }} --auto-partition-size 8
|
||||
```
|
||||
|
||||
@@ -320,7 +319,7 @@ Determines which test suites to run based on file changes.
|
||||
|
||||
| Output | Triggers |
|
||||
|--------|----------|
|
||||
| `main_package` | Stage A/B/C test suites |
|
||||
| `main_package` | Base A/B/C test suites |
|
||||
| `sgl_kernel` | Kernel wheel builds + kernel test suites; also switches B200 jobs to kernel-build runner labels outside `target_stage` mode |
|
||||
| `jit_kernel` | JIT kernel test workflow |
|
||||
| `multimodal_gen` | Multimodal-gen test workflow |
|
||||
@@ -348,10 +347,10 @@ group: pr-test-{event_name}-{branch}-{pr_sha}-{stage}
|
||||
|
||||
## How To: Add a New Stage Job
|
||||
|
||||
1. Define the job in `pr-test.yml` with `needs: [check-changes, call-gate, wait-for-stage-X, ...]`
|
||||
1. Define the job in `pr-test.yml` with `needs: [check-changes, call-gate, wait-for-base-X, ...]`
|
||||
2. Copy the `if:` condition pattern from an existing same-stage job (handles `target_stage`, `schedule`, `main_package`)
|
||||
3. Add `checkout` step
|
||||
4. Add `check-stage-health` step (after checkout) — if any prior job failed, `core.setFailed()` fires and all subsequent steps auto-skip via default `if: success()`
|
||||
4. Add `check-pr-test-health` step (after checkout) — if any prior job failed, `core.setFailed()` fires and all subsequent steps auto-skip via default `if: success()`
|
||||
5. Add `check-maintenance` step
|
||||
6. Add `download-artifact` step if `sgl_kernel` changed
|
||||
7. Add `install dependencies` step
|
||||
@@ -359,7 +358,7 @@ group: pr-test-{event_name}-{branch}-{pr_sha}-{stage}
|
||||
9. Add `upload-cuda-coredumps` step with `if: always()`
|
||||
10. Register the suite name in `PER_COMMIT_SUITES` in `test/run_suite.py`
|
||||
11. If using matrix, add `--auto-partition-id` and `--auto-partition-size` to the run command
|
||||
12. **Update `wait-for-stage-X`** job spec with the new job name and `expected_count` (if matrix)
|
||||
12. **Update `wait-for-base-X`** job spec with the new job name and `expected_count` (if matrix)
|
||||
13. **Add the job to `pr-test-finish.needs`** list
|
||||
|
||||
---
|
||||
@@ -368,8 +367,8 @@ group: pr-test-{event_name}-{branch}-{pr_sha}-{stage}
|
||||
|
||||
| Symptom | Likely cause | What to check |
|
||||
|---------|-------------|---------------|
|
||||
| All stage-B/C jobs green but steps skipped | Earlier job failed, `check-stage-health` triggered | Find the actual failed job (red X) |
|
||||
| `wait-for-stage-b` timeout | `expected_count` doesn't match matrix size | Verify job spec counts match `matrix:` array length |
|
||||
| All stage-B/C jobs green but steps skipped | Earlier job failed, `check-pr-test-health` triggered | Find the actual failed job (red X) |
|
||||
| `wait-for-base-b` timeout | `expected_count` doesn't match matrix size | Verify job spec counts match `matrix:` array length |
|
||||
| `pr-test-finish` fails but all jobs green | A job was `cancelled` (counts as failure in finish) | Check concurrency cancellation |
|
||||
| Tests pass locally but fail in CI | Partition assignment, runner GPU type, or `est_time` inaccuracy | Check which partition the test lands in; verify runner label |
|
||||
| Flaky test retried and passed | Retriable failure (accuracy/perf) | Check `[CI Retry]` markers in job logs |
|
||||
|
||||
Reference in New Issue
Block a user