[CI] Gate /rerun-test on commenter trust and remove /rerun-stage (#35750)
This commit is contained in:
Liangsheng Yin
@@ -139,7 +139,7 @@ This skill covers the CI **infrastructure** layer — how tests are dispatched,
|
||||
|
||||
## Execution Modes
|
||||
|
||||
| Aspect | PR (`pull_request`) | Scheduled (`cron`, every 6h) | `/rerun-stage` (`workflow_dispatch`) |
|
||||
| Aspect | PR (`pull_request`) | Scheduled (`cron`, every 6h) | Manual dispatch (`workflow_dispatch`) |
|
||||
|--------|---------------------|------------------------------|--------------------------------------|
|
||||
| **Stage ordering** | Sequential: A → B → C via `wait-for-base-*` | Parallel (all at once) | Single target stage only |
|
||||
| **Cross-job fast-fail** | Yes (`check-pr-test-health`) | Yes | Yes |
|
||||
@@ -175,7 +175,7 @@ This skill covers the CI **infrastructure** layer — how tests are dispatched,
|
||||
|
||||
> **Critical**: `expected_count` must match the matrix size. If you add/remove matrix entries, update the wait job's spec accordingly.
|
||||
|
||||
**PR only**: Condition `github.event_name == 'pull_request' && !inputs.target_stage` — scheduled runs and `/rerun-stage` skip these entirely, allowing parallel execution.
|
||||
**PR only**: Condition `github.event_name == 'pull_request' && !inputs.target_stage` — scheduled runs and manual dispatches skip these entirely, allowing parallel execution.
|
||||
|
||||
---
|
||||
|
||||
@@ -341,7 +341,7 @@ group: pr-test-{event_name}-{branch}-{pr_sha}-{stage}
|
||||
|---------|--------|---------|
|
||||
| `event_name` | `github.event_name` | Prevents scheduled runs colliding with fork PRs named `main` |
|
||||
| `branch` | `github.head_ref \|\| github.ref_name` | Per-branch isolation |
|
||||
| `pr_sha` | `inputs.pr_head_sha \|\| 'current'` | Isolates `/rerun-stage` from main runs |
|
||||
| `pr_sha` | `inputs.pr_head_sha \|\| 'current'` | Isolates manual dispatches from main runs |
|
||||
| `stage` | `inputs.target_stage \|\| 'all'` | Allows parallel stage dispatches |
|
||||
|
||||
`cancel-in-progress: true` for `pull_request` events (new push cancels old run), `false` for `workflow_call`.
|
||||
@@ -388,7 +388,6 @@ group: pr-test-{event_name}-{branch}-{pr_sha}-{stage}
|
||||
| `/rerun-failed-ci` | Reruns failed jobs in the latest workflow run |
|
||||
| `/tag-and-rerun-ci` | Adds `run-ci` label + reruns failed |
|
||||
| `/tag-and-rerun-ci extra` | Adds both `run-ci` and `run-ci-extra` labels + reruns failed |
|
||||
| `/rerun-stage <stage>` | Deprecated; posts deprecation notice |
|
||||
| `/rerun-test <test-file> [<test-file> ...]` | Reruns specific test file(s) via `rerun-test.yml`. A file arg containing a glob metacharacter (`*`, `?`, `[...]`) expands against `test/registered/` and the multimodal test dir to every matching `test_*.py` (e.g. `/rerun-test test_*backend*.py` — wrap in backticks so GitHub doesn't italicize the `*`); matches are deduped, grouped by dispatch shape, and can't carry a `::test` selector. No match → single ⛔ reply, nothing dispatched. Each reply echoes its originating command (`Results for …`) so concurrent commands stay distinguishable |
|
||||
| `/rerun-group <group> [<group> ...]` | Expands registered test groups, then reuses `/rerun-test` |
|
||||
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -8,5 +8,27 @@ This folder contains tools and workflows for automating maintenance tasks.
|
||||
Maintainers can directly edit the file to add entries with `"reason": "custom override"`.
|
||||
Maintainers can also run `update_ci_permission.py` to update it with some auto rules (e.g., top contributors in the last 90 days get full permissions).
|
||||
|
||||
Recognized permission keys:
|
||||
|
||||
| Key | Grants |
|
||||
| --- | --- |
|
||||
| `can_tag_run_ci_label` | `/tag-run-ci-label`, `/tag-and-rerun-ci` |
|
||||
| `can_rerun_failed_ci` | `/rerun-failed-ci`, `/tag-and-rerun-ci` |
|
||||
| `cooldown_interval_minutes` | rate limit in `pr-gate.yml`; `0` also grants `/rerun-test`, `/rerun-group` |
|
||||
|
||||
`/rerun-test` and `/rerun-group` are gated on the commenter alone: either
|
||||
`cooldown_interval_minutes: 0`, or `write`/`admin` permission on the repo. Where
|
||||
the PR comes from makes no difference, and authoring it grants nothing.
|
||||
|
||||
Those are the same two signals `pr-gate.yml` already uses to waive its rate
|
||||
limit, and that is the point -- a selective rerun dispatches `rerun-test.yml`
|
||||
directly, which never passes through `pr-gate.yml`, so it bypasses the rate limit
|
||||
by construction. Anyone allowed to run one is therefore unthrottled in practice,
|
||||
which is exactly what a zero cooldown already declares.
|
||||
|
||||
Set a cooldown deliberately: `0` lets the holder run PR-head code on the
|
||||
self-hosted GPU runners, and raising it above `0` takes that away again along
|
||||
with their rate-limit waiver.
|
||||
|
||||
## Others
|
||||
- `MAINTAINER.md` defines the code maintenance model.
|
||||
|
||||
@@ -35,6 +35,9 @@ Permissions are assigned according to the following rules:
|
||||
- For all other cases, preserve the original configuration unchanged.
|
||||
3. All other users receive no permissions and a 120-minute cooldown (they are omitted from the file).
|
||||
|
||||
`cooldown_interval_minutes` is not only a rate limit: 0 also authorizes
|
||||
`/rerun-test` and `/rerun-group`, which never pass through `pr-gate.yml`.
|
||||
|
||||
Usage:
|
||||
export GH_TOKEN="your_github_token"
|
||||
python3 update_ci_permission.py
|
||||
@@ -203,7 +206,6 @@ def main():
|
||||
new_permissions[user] = {
|
||||
"can_tag_run_ci_label": True,
|
||||
"can_rerun_failed_ci": True,
|
||||
"can_rerun_stage": True,
|
||||
"cooldown_interval_minutes": 0,
|
||||
"reason": "top contributor",
|
||||
}
|
||||
@@ -221,7 +223,6 @@ def main():
|
||||
new_permissions[user] = {
|
||||
"can_tag_run_ci_label": True,
|
||||
"can_rerun_failed_ci": True,
|
||||
"can_rerun_stage": True,
|
||||
"cooldown_interval_minutes": 60,
|
||||
"reason": "custom override",
|
||||
}
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
name: PR Test ROCm 7.2 (AMD)
|
||||
# Dynamic run-name for /rerun-stage commands to enable URL lookup
|
||||
# Dynamic run-name for manual stage dispatches to enable URL lookup
|
||||
# Format: "[stage-name] sha" for fork PRs, "[stage-name]" for non-fork, default for normal runs
|
||||
run-name: ${{ (inputs.target_stage || inputs.target_stage_select) && (inputs.pr_head_sha && format('[{0}] {1}', inputs.target_stage || inputs.target_stage_select, inputs.pr_head_sha) || format('[{0}]', inputs.target_stage || inputs.target_stage_select)) || '' }}
|
||||
|
||||
@@ -56,7 +56,7 @@ on:
|
||||
type: string
|
||||
default: ""
|
||||
pr_head_sha:
|
||||
description: "PR head SHA to checkout (for /rerun-stage on fork PRs)"
|
||||
description: "PR head SHA to checkout (for stage dispatches on fork PRs)"
|
||||
required: false
|
||||
type: string
|
||||
default: ""
|
||||
@@ -239,7 +239,7 @@ jobs:
|
||||
# pr-test.yml's `call-pr-test-extra`. On `schedule` (and run_all_tests
|
||||
# dispatch) the extra suite runs on `main` without needing the
|
||||
# `run-ci-extra` label (pr-gate.yml only enforces labels on pull_request
|
||||
# events). Targeted /rerun-stage dispatches (target_stage set) are excluded.
|
||||
# events). Targeted stage dispatches (target_stage set) are excluded.
|
||||
# Not added to `pr-test-amd-rocm720-finish` so the base AMD gate never depends on
|
||||
# the opt-in extra suite.
|
||||
call-pr-test-amd-extra-rocm720:
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
name: PR Test ROCm 7.0 (AMD)
|
||||
# Dynamic run-name for /rerun-stage commands to enable URL lookup
|
||||
# Dynamic run-name for manual stage dispatches to enable URL lookup
|
||||
# Format: "[stage-name] sha" for fork PRs, "[stage-name]" for non-fork, default for normal runs
|
||||
run-name: ${{ (inputs.target_stage || inputs.target_stage_select) && (inputs.pr_head_sha && format('[{0}] {1}', inputs.target_stage || inputs.target_stage_select, inputs.pr_head_sha) || format('[{0}]', inputs.target_stage || inputs.target_stage_select)) || '' }}
|
||||
|
||||
@@ -43,7 +43,7 @@ on:
|
||||
type: string
|
||||
default: ""
|
||||
pr_head_sha:
|
||||
description: "PR head SHA to checkout (for /rerun-stage on fork PRs)"
|
||||
description: "PR head SHA to checkout (for stage dispatches on fork PRs)"
|
||||
required: false
|
||||
type: string
|
||||
default: ""
|
||||
@@ -209,7 +209,7 @@ jobs:
|
||||
# pr-test.yml's `call-pr-test-extra`. On `schedule` (and run_all_tests
|
||||
# dispatch) the extra suite runs on `main` without needing the
|
||||
# `run-ci-extra` label (pr-gate.yml only enforces labels on pull_request
|
||||
# events). Targeted /rerun-stage dispatches (target_stage set) are excluded.
|
||||
# events). Targeted stage dispatches (target_stage set) are excluded.
|
||||
# Not added to `pr-test-amd-finish` so the base AMD gate never depends on
|
||||
# the opt-in extra suite.
|
||||
call-pr-test-amd-extra:
|
||||
|
||||
@@ -72,8 +72,9 @@ env:
|
||||
# every run, so a rerun holding the write token could become everyone's next
|
||||
# comparison baseline. Without the repo var it fails fast instead.
|
||||
|
||||
# Every job below sets its own `permissions`, which replaces rather than merges
|
||||
# with a workflow-level block -- so keep the floor here minimal and grant per job.
|
||||
permissions:
|
||||
actions: write
|
||||
contents: read
|
||||
issues: read
|
||||
|
||||
@@ -93,6 +94,9 @@ jobs:
|
||||
uses: actions/checkout@v4
|
||||
with:
|
||||
ref: ${{ inputs.pr_head_sha || github.sha }}
|
||||
# This checkout can hold a fork's code that the steps below execute;
|
||||
# without this the job token stays in .git/config, readable by it.
|
||||
persist-credentials: false
|
||||
|
||||
- name: Mark runner picked up
|
||||
if: inputs.reply_comment_id != '' && inputs.reply_marker != ''
|
||||
@@ -188,6 +192,9 @@ jobs:
|
||||
uses: actions/checkout@v4
|
||||
with:
|
||||
ref: ${{ inputs.pr_head_sha || github.sha }}
|
||||
# This checkout can hold a fork's code that the steps below execute;
|
||||
# without this the job token stays in .git/config, readable by it.
|
||||
persist-credentials: false
|
||||
|
||||
- name: Mark runner picked up
|
||||
if: inputs.reply_comment_id != '' && inputs.reply_marker != ''
|
||||
@@ -262,6 +269,9 @@ jobs:
|
||||
uses: actions/checkout@v4
|
||||
with:
|
||||
ref: ${{ inputs.pr_head_sha || github.sha }}
|
||||
# This checkout can hold a fork's code that the steps below execute;
|
||||
# without this the job token stays in .git/config, readable by it.
|
||||
persist-credentials: false
|
||||
|
||||
- name: Mark runner picked up
|
||||
if: inputs.reply_comment_id != '' && inputs.reply_marker != ''
|
||||
|
||||
@@ -19,7 +19,6 @@ jobs:
|
||||
(contains(github.event.comment.body, '/tag-run-ci-label') ||
|
||||
contains(github.event.comment.body, '/rerun-failed-ci') ||
|
||||
contains(github.event.comment.body, '/tag-and-rerun-ci') ||
|
||||
contains(github.event.comment.body, '/rerun-stage') ||
|
||||
contains(github.event.comment.body, '/rerun-group') ||
|
||||
contains(github.event.comment.body, '/rerun-test'))
|
||||
runs-on: ubuntu-latest
|
||||
|
||||
@@ -130,7 +130,7 @@ The rerun commands have the following permission rules:
|
||||
| Command | PR author | Other users | Additional rule for fork PRs |
|
||||
| --- | --- | --- | --- |
|
||||
| `/rerun-failed-ci` | Always allowed on their own PR | Must have `can_rerun_failed_ci` in `CI_PERMISSIONS.json` | None |
|
||||
| `/rerun-test`, `/rerun-group` | Automatically receives `can_rerun_test` on their own PR | Must have `can_rerun_test` or the legacy `can_rerun_stage` permission in `CI_PERMISSIONS.json` | The commenter must also have `write` or `admin` repository permission. A fork PR author already has the automatic `can_rerun_test` grant; other fork commenters need both repository permission and a configured selective-rerun permission. |
|
||||
| `/rerun-test`, `/rerun-group` | No special allowance; judged as any other commenter | Must have `cooldown_interval_minutes: 0` in `CI_PERMISSIONS.json`, or `write`/`admin` repository permission | None -- the same rule applies wherever the PR comes from |
|
||||
|
||||
<Warning>
|
||||
Selective reruns do not build or install a PR-local `sglang-kernel` wheel. If a PR changes `python/sglang/kernels/aot/` or code that depends on that AOT change, use the normal/full PR CI workflow. A `/rerun-test` or `/rerun-group` result may use the pinned released kernel and does not validate the combined change.
|
||||
|
||||
@@ -146,15 +146,12 @@ For CI to run on a pull request, it must have the "run-ci" label. Authorized use
|
||||
- `/tag-run-ci-label`: Adds the "run-ci" label. Only **future** commits trigger CI; the current commit is unaffected. Add the `extra` argument (`/tag-run-ci-label extra`) to additionally apply the "run-ci-extra" label, opting the PR into the extra test workflow (`pr-test-extra.yml`).
|
||||
- `/rerun-failed-ci`: Reruns workflows from the latest commit with conclusion **failed, flaky, or skipped**.
|
||||
- `/tag-and-rerun-ci`: Runs both. Use this on a fresh PR to kick off CI on the current commit — `/tag-run-ci-label` alone won't. Accepts the same `extra` argument (`/tag-and-rerun-ci extra`).
|
||||
- `/rerun-stage <stage-name>`: Reruns a single test stage without waiting for its dependencies. Useful for quickly validating a specific test fix instead of waiting ~30 minutes for preceding stages.
|
||||
- `/rerun-test <test-spec> [<test-spec> ...]`: Reruns one or more specific tests directly, bypassing stage boundaries. Each `<test-spec>` is pytest-style `<file>::<TestClass>[.<test_method>]` (the `::TestClass` and `.<test_method>` parts are optional). The handler resolves each spec, groups specs by their registered runner-label, and dispatches one [Rerun Test workflow](https://github.com/sgl-project/sglang/actions/workflows/rerun-test.yml) per group. Examples: `/rerun-test test_srt_endpoint.py`, `/rerun-test registered/core/test_srt_endpoint.py::TestSRTEndpoint.test_simple_decode`, `/rerun-test test_a.py test_b.py` (multiple at once).
|
||||
|
||||
If you have permission, the [Slash Command Handler](https://github.com/sgl-project/sglang/actions/workflows/slash-command-handler.yml) will run your command and react with a 👍 to your comment. It may take up to a few minutes for the reaction to appear. Here’s a usage [example](https://github.com/sgl-project/sglang/pull/14253#issuecomment-3599509302).
|
||||
|
||||
To avoid spamming a PR with too many `/rerun-failed-ci` comments, you can also trigger the command by editing an existing comment and adding any suffix (e.g., `/rerun-failed-ci try again`).
|
||||
|
||||
Example of rerunning a single test stage: `/rerun-stage unit-test-backend-4-gpu`.
|
||||
|
||||
If you don’t have permission, please ask maintainers to trigger CI for you.
|
||||
|
||||
### CI rate limits
|
||||
|
||||
@@ -1033,31 +1033,25 @@ def _check_rerun_test_permissions(gh_repo, pr, comment, user_perms, command_name
|
||||
"""
|
||||
Check permissions shared by /rerun-test and /rerun-group.
|
||||
"""
|
||||
# SECURITY: These commands check out and execute code from the PR branch on
|
||||
# self-hosted GPU runners, so fork PRs require a trusted collaborator.
|
||||
is_fork = pr.head.repo is None or pr.head.repo.owner.login != gh_repo.owner.login
|
||||
if is_fork:
|
||||
commenter = comment.user.login
|
||||
perm = gh_repo.get_collaborator_permission(commenter)
|
||||
if perm not in ("admin", "write"):
|
||||
print(f"Permission denied: /{command_name} on fork PR by {commenter}.")
|
||||
comment.create_reaction("confused")
|
||||
pr.create_issue_comment(
|
||||
f"⛔ `/{command_name}` is not available for fork PRs unless the commenter "
|
||||
"has write permission on the repo.\n\n"
|
||||
"Please ask a maintainer to run this command, or use the normal CI flow."
|
||||
)
|
||||
return False
|
||||
print(f"Fork PR, but commenter {commenter} has write+ permission. Proceeding.")
|
||||
# A rerun dispatches rerun-test.yml, which never passes through pr-gate.yml,
|
||||
# so it is unthrottled either way; gate on what pr-gate waives the limit for.
|
||||
if user_perms.get("cooldown_interval_minutes") == 0:
|
||||
return True
|
||||
|
||||
if not (
|
||||
user_perms.get("can_rerun_test", False)
|
||||
or user_perms.get("can_rerun_stage", False)
|
||||
):
|
||||
print("Permission denied: neither can_rerun_test nor can_rerun_stage is true.")
|
||||
return False
|
||||
commenter = comment.user.login
|
||||
perm = gh_repo.get_collaborator_permission(commenter)
|
||||
if perm in ("admin", "write"):
|
||||
print(f"Commenter {commenter} has write+ permission. Proceeding.")
|
||||
return True
|
||||
|
||||
return True
|
||||
print(f"Permission denied: /{command_name} by {commenter} (permission: {perm}).")
|
||||
comment.create_reaction("confused")
|
||||
pr.create_issue_comment(
|
||||
f"⛔ `/{command_name}` requires `cooldown_interval_minutes: 0` in "
|
||||
"`.github/CI_PERMISSIONS.json`, or write permission on the repo.\n\n"
|
||||
"Please ask a maintainer to run this command, or use the normal CI flow."
|
||||
)
|
||||
return False
|
||||
|
||||
|
||||
def handle_rerun_test(
|
||||
@@ -1320,10 +1314,10 @@ def main():
|
||||
pr = repo.get_pull(pr_number)
|
||||
comment = repo.get_issue(pr_number).get_comment(comment_id)
|
||||
|
||||
# PR authors can always rerun failed CI and rerun individual UTs on their own PRs,
|
||||
# even if they are not listed in CI_PERMISSIONS.json.
|
||||
# PR authors can always rerun failed CI on their own PRs, even if they are not
|
||||
# listed in CI_PERMISSIONS.json.
|
||||
# Note: /tag-run-ci-label still requires CI_PERMISSIONS.json.
|
||||
# Note: /rerun-test is blocked entirely for fork PRs in handle_rerun_test() itself.
|
||||
# Authorship grants nothing for /rerun-test; that gate reads the commenter.
|
||||
if pr.user.login == user_login:
|
||||
if user_perms is None:
|
||||
print(
|
||||
@@ -1336,11 +1330,11 @@ def main():
|
||||
f"User {user_login} is the PR author and has existing CI permissions."
|
||||
)
|
||||
user_perms["can_rerun_failed_ci"] = True
|
||||
user_perms["can_rerun_test"] = True
|
||||
|
||||
if not user_perms:
|
||||
print(f"User {user_login} does not have any configured permissions. Exiting.")
|
||||
return
|
||||
# No early exit on a missing entry: /rerun-test also gates on repo permission,
|
||||
# so a write-holder absent from the file must still reach its handler.
|
||||
if user_perms is None:
|
||||
user_perms = {}
|
||||
|
||||
# 4. Parse Command and Execute
|
||||
first_line = comment_body.split("\n")[0].strip()
|
||||
@@ -1380,31 +1374,6 @@ def main():
|
||||
else:
|
||||
print("Combined command finished, but no actions were taken.")
|
||||
|
||||
elif first_line.startswith("/rerun-stage"):
|
||||
print("/rerun-stage is deprecated; posting deprecation notice.")
|
||||
comment.create_reaction("-1")
|
||||
pr.create_issue_comment(
|
||||
"⚠️ **`/rerun-stage` has been deprecated.**\n\n"
|
||||
"Stage granularity is too coarse — a stage usually doesn't map to one "
|
||||
"feature, so rerunning a stage re-pays the cost of unrelated tests. "
|
||||
"If you don't know which exact test files to rerun, you shouldn't be "
|
||||
"using `/rerun-stage` or `/rerun-test` in the first place.\n\n"
|
||||
"**Use one of these instead:**\n"
|
||||
"- **Selective tests** (you know exactly which files to rerun):\n"
|
||||
" ```\n"
|
||||
" /rerun-test test_foo.py test_bar.py\n"
|
||||
" ```\n"
|
||||
"- **Rerun only failed jobs**:\n"
|
||||
" ```\n"
|
||||
" /rerun-failed-ci\n"
|
||||
" ```\n"
|
||||
"- **Full CI rerun** (with extra coverage): add the `run-ci` or "
|
||||
"`run-ci-extra` label and push a new commit (or use `/tag-and-rerun-ci`).\n\n"
|
||||
"**AMD CI**: stage-level dispatch is still available via "
|
||||
"Actions UI → *PR Test ROCm 7.2 (AMD)* (default) / *PR Test ROCm 7.0 (AMD)* → "
|
||||
"*Run workflow* → pick a stage from the dropdown."
|
||||
)
|
||||
|
||||
elif first_line.startswith("/rerun-group"):
|
||||
group_names = first_line.split()[1:]
|
||||
handle_rerun_group(
|
||||
|
||||
+1
-1
@@ -7,7 +7,7 @@ This page covers principles and essentials: folder layout, how to run tests, reg
|
||||
|
||||
## CI Pipeline Overview
|
||||
|
||||
The CI pipeline runs in three sequential stages: **A** (pre-flight, ~3 min) → **B** (basic, ~30 min) → **C** (advanced, ~30 min). Kernel and multimodal-gen tests run in parallel with stage B. For details on stage gating, fast-fail mechanisms, execution modes (PR vs scheduled vs `/rerun-stage`), and debugging CI failures, see the [CI workflow guide](../.claude/skills/ci-workflow-guide/SKILL.md).
|
||||
The CI pipeline runs in three sequential stages: **A** (pre-flight, ~3 min) → **B** (basic, ~30 min) → **C** (advanced, ~30 min). Kernel and multimodal-gen tests run in parallel with stage B. For details on stage gating, fast-fail mechanisms, execution modes (PR vs scheduled vs manual dispatch), and debugging CI failures, see the [CI workflow guide](../.claude/skills/ci-workflow-guide/SKILL.md).
|
||||
|
||||
## Folder Organization
|
||||
|
||||
|
||||
Reference in New Issue
Block a user