Re-land #23109: rebase-required mode + fix for grep-no-match abort (#24180)

.github/MAINTAINER.md

@@ -150,5 +150,18 @@ When the CI is unhealthy (e.g., the scheduled pr-test on `main` is broken for co
 
 Maintenance mode ends when `pr-test.yml` is all green on `main` and the issue is closed.
 
+### Rebase-Required Mode
+When a major update lands on `main` and all open PRs must rebase before CI can run (without fully pausing CI), add a line of the form `MIN_BASE_SHA: <sha>` to the body of issue #21065. **The rebase check is enforced regardless of whether the issue is open or closed** — you do not need to enter full maintenance mode (open the issue) to use this gate; just editing the body to include the directive is enough. While the directive is present:
+- CI is allowed to run only for PRs whose branch already contains `<sha>` (GitHub compare API status `ahead` or `identical` — i.e., the PR has `<sha>` in its history).
+- PRs that are `behind` or `diverged` from `<sha>` are blocked with a "rebase required" error until they rebase onto the latest `main`.
+- The `bypass-maintenance` label still bypasses this check for CI-fix PRs.
+
+Notes:
+- Only the **first** `MIN_BASE_SHA:` line in the issue body is read.
+- The SHA must be 7-40 hex characters; malformed values are ignored (with a warning in the job summary).
+- Avoid pasting the directive inside a fenced code block in the issue body — the parser does not skip code fences and may match example snippets.
+
+Remove the directive from the issue body to lift the rebase requirement (closing the issue does NOT lift it on its own).
+
 ## Suspending Permissions
 If a Merge Oncall bypasses checks to merge a PR that breaks the `main` branch, merges a non-CI-fix PR during CI Maintenance Mode, or repeatedly breaks the CI due to various reasons, their privileges will be suspended for at least two days, depending on the severity of the incident.

.github/actions/check-maintenance/action.yml

@@ -1,5 +1,5 @@
 name: Check Maintenance Mode
-description: Blocks CI when maintenance mode is active (issue #21065 is open), unless the PR has the bypass-maintenance label, or env PR_TEST_BYPASS_MAINTENANCE_ON_MAIN=true (PR Test workflow on main only). Merging non-CI-fix PRs is prohibited during maintenance mode; in severe cases, merge permissions may be revoked.
+description: Blocks CI in two independent modes driven by issue #21065. (1) Full-pause: when the issue is open. (2) Rebase-required: whenever the issue body contains a `MIN_BASE_SHA: <sha>` directive — enforced regardless of whether the issue is open or closed, so maintainers can require all PRs to rebase past a specific commit without having to open the maintenance issue. Both modes are bypassed by the `bypass-maintenance` label on the PR, or by env PR_TEST_BYPASS_MAINTENANCE_ON_MAIN=true (PR Test workflow on main only). Merging non-CI-fix PRs is prohibited during full-pause; in severe cases, merge permissions may be revoked.
 
 inputs:
   github-token:
@@ -18,6 +18,7 @@ runs:
         MAINTENANCE_ISSUE=21065
         REPO="${{ github.repository }}"
         PR_NUMBER="${{ github.event.pull_request.number }}"
+        PR_HEAD_SHA="${{ github.event.pull_request.head.sha }}"
 
         # PR Test workflow only: scheduled runs and runs on main (dispatch / workflow_call) set this env
         if [[ "${PR_TEST_BYPASS_MAINTENANCE_ON_MAIN:-}" == "true" ]]; then
@@ -25,39 +26,143 @@ runs:
           exit 0
         fi
 
-        # Check if maintenance issue is open (fail-open: if API errors, allow CI to proceed)
-        ISSUE_STATE=$(gh issue view "$MAINTENANCE_ISSUE" --repo "$REPO" --json state --jq '.state' 2>/dev/null || echo "UNKNOWN")
+        # Use curl + jq instead of `gh` because self-hosted GPU runners
+        # don't have the gh CLI installed. Without this, the action
+        # silently fell through to "Proceeding with CI" on every GPU
+        # job — the rebase gate only fired on ubuntu-latest jobs that
+        # had gh pre-installed. curl+jq are reliably available on every
+        # Linux runner.
+        gh_api() {
+          local path="$1"
+          local err_file="$2"
+          curl --silent --show-error --fail \
+            --max-time 30 \
+            -H "Authorization: Bearer $GH_TOKEN" \
+            -H "Accept: application/vnd.github+json" \
+            -H "X-GitHub-Api-Version: 2022-11-28" \
+            "https://api.github.com/$path" 2>"$err_file"
+        }
 
-        if [[ "$ISSUE_STATE" != "OPEN" ]]; then
-          echo "✅ Maintenance mode is OFF. Proceeding with CI."
+        # Fetch issue state and body. Fail-open: if we can't read the issue
+        # (network blip, missing token scope), CI proceeds.
+        ERR_FILE=$(mktemp)
+        ISSUE_JSON=$(gh_api "repos/$REPO/issues/$MAINTENANCE_ISSUE" "$ERR_FILE" || true)
+        if [[ -z "$ISSUE_JSON" ]]; then
+          echo "⚠️  Issue fetch returned empty. curl stderr was:"
+          cat "$ERR_FILE" || true
+        fi
+        rm -f "$ERR_FILE"
+        ISSUE_STATE=$(printf '%s' "$ISSUE_JSON" | jq -r '.state // "UNKNOWN"' 2>/dev/null || echo "UNKNOWN")
+        # Issues API returns state in lowercase ("open"/"closed"); normalize
+        # to uppercase so existing comparisons against "OPEN" still work.
+        ISSUE_STATE=$(printf '%s' "$ISSUE_STATE" | tr '[:lower:]' '[:upper:]')
+        ISSUE_BODY=$(printf '%s' "$ISSUE_JSON" | jq -r '.body // ""' 2>/dev/null || echo "")
+        echo "DEBUG: ISSUE_STATE=$ISSUE_STATE  body_length=${#ISSUE_BODY}"
+
+        # Parse optional `MIN_BASE_SHA: <sha>` directive from the issue body
+        # (first occurrence wins). Whenever this directive is present, the
+        # rebase check is enforced regardless of whether the issue is open
+        # or closed — so maintainers can require all PRs to rebase past a
+        # specific commit without having to open the maintenance issue.
+        # `grep` exits 1 on no-match, which under `set -eo pipefail` (the
+        # default for `shell: bash` composite steps) would abort the whole
+        # script — silently failing every PR whose issue body has no
+        # MIN_BASE_SHA line. Wrap grep in a brace group with `|| true` so
+        # an empty match falls through cleanly to "no directive set".
+        MIN_BASE_SHA=$(
+          { printf '%s' "$ISSUE_BODY" | tr -d '\r' | grep -iE '^[[:space:]]*`?MIN_BASE_SHA`?[[:space:]]*[:=]' || true; } \
+            | head -n1 | sed -E 's/.*[:=][[:space:]]*//; s/`//g' | awk '{print $1}'
+        )
+        if [[ -n "$MIN_BASE_SHA" ]] && ! [[ "$MIN_BASE_SHA" =~ ^[a-fA-F0-9]{7,40}$ ]]; then
+          WARN="⚠️  Ignoring malformed MIN_BASE_SHA directive in issue #$MAINTENANCE_ISSUE: '$MIN_BASE_SHA' (must be 7-40 hex chars)"
+          echo "$WARN"
+          echo "$WARN" >> "$GITHUB_STEP_SUMMARY"
+          MIN_BASE_SHA=""
+        fi
+
+        # If neither gate is active (no MIN_BASE_SHA, issue not open), nothing to do.
+        if [[ -z "$MIN_BASE_SHA" && "$ISSUE_STATE" != "OPEN" ]]; then
+          echo "✅ Maintenance mode is OFF and no MIN_BASE_SHA directive. Proceeding with CI."
           exit 0
         fi
 
-        # For PRs, check if bypass-maintenance label is present
+        # bypass-maintenance label bypasses both gates. PR labels live on
+        # the issue resource (PRs are issues with extra metadata in the
+        # GH API), so we hit the same /issues/{n} endpoint.
         if [[ -n "$PR_NUMBER" ]]; then
-          HAS_BYPASS=$(gh pr view "$PR_NUMBER" --repo "$REPO" --json labels --jq '[.labels[].name] | map(select(. == "bypass-maintenance")) | length' 2>/dev/null || echo "0")
-          if [[ "$HAS_BYPASS" -gt 0 ]]; then
-            echo "✅ PR #$PR_NUMBER has 'bypass-maintenance' label. Bypassing maintenance mode."
+          ERR_FILE=$(mktemp)
+          PR_JSON=$(gh_api "repos/$REPO/issues/$PR_NUMBER" "$ERR_FILE" || true)
+          rm -f "$ERR_FILE"
+          HAS_BYPASS=$(printf '%s' "$PR_JSON" | jq -r '[.labels[]?.name] | map(select(. == "bypass-maintenance")) | length' 2>/dev/null || echo "0")
+          if [[ "${HAS_BYPASS:-0}" -gt 0 ]]; then
+            echo "✅ PR #$PR_NUMBER has 'bypass-maintenance' label. Bypassing maintenance + rebase checks."
             exit 0
           fi
         fi
 
-        MSG=$(printf "%s\n" \
-          "## ⚠️ CI Maintenance Mode is Active" \
-          "The CI infrastructure is currently under maintenance." \
-          "All PR CI runs are paused until maintenance is complete." \
-          "**Merging non-CI-fix PRs is prohibited during maintenance mode.** In severe cases, merge permissions may be revoked." \
-          "You might also experience unexpected failures during this period." \
-          "The team is working on the issue and will update the status as soon as possible." \
-          "" \
-          "What should you do?" \
-          "- **Do NOT merge non-CI-fix PRs** until maintenance mode is lifted" \
-          "- Check back later (~12 hours)" \
-          "- Follow CI Maintenance Mode issue: https://github.com/$REPO/issues/$MAINTENANCE_ISSUE for status updates")
+        # Rebase-required gate (independent of issue open/closed state).
+        if [[ -n "$MIN_BASE_SHA" ]]; then
+          if [[ -z "$PR_NUMBER" || -z "$PR_HEAD_SHA" ]]; then
+            echo "✅ Not a PR context; skipping rebase check."
+          else
+            # Use GitHub compare API: status is "ahead"/"identical" when MIN_BASE_SHA is reachable from PR head.
+            ERR_FILE=$(mktemp)
+            COMPARE_JSON=$(gh_api "repos/$REPO/compare/$MIN_BASE_SHA...$PR_HEAD_SHA" "$ERR_FILE" || true)
+            if [[ -z "$COMPARE_JSON" ]]; then
+              echo "⚠️  Compare API failed. curl stderr was:"
+              cat "$ERR_FILE" || true
+            fi
+            rm -f "$ERR_FILE"
+            COMPARE_STATUS=$(printf '%s' "$COMPARE_JSON" | jq -r '.status // "UNKNOWN"' 2>/dev/null || echo "UNKNOWN")
+            COMPARE_STATUS="${COMPARE_STATUS:-UNKNOWN}"
 
-        echo "$MSG" >> "$GITHUB_STEP_SUMMARY"
-        while IFS= read -r line; do
-          echo "::error::$line"
-        done <<< "$MSG"
+            case "$COMPARE_STATUS" in
+              ahead|identical)
+                echo "✅ PR #$PR_NUMBER contains required base ${MIN_BASE_SHA:0:12} ($COMPARE_STATUS)."
+                ;;
+              UNKNOWN)
+                echo "⚠️  Could not determine rebase status via GitHub API; fail-open, allowing rebase check to pass."
+                ;;
+              *)
+                MSG=$(printf "%s\n" \
+                  "## ⚠️ Rebase Required Before CI Can Run" \
+                  "A major update has landed on \`main\`. All PRs must rebase onto the latest \`main\` before CI will run." \
+                  "Required base commit: \`${MIN_BASE_SHA:0:12}\` (your PR is \`$COMPARE_STATUS\` relative to this commit)." \
+                  "" \
+                  "What should you do?" \
+                  "- Rebase your branch onto the latest \`main\` and push again" \
+                  "- Follow CI Maintenance Mode issue: https://github.com/$REPO/issues/$MAINTENANCE_ISSUE for context" \
+                  "- CI-fix PRs may request the \`bypass-maintenance\` label to skip this check")
+                echo "$MSG" >> "$GITHUB_STEP_SUMMARY"
+                while IFS= read -r line; do
+                  echo "::error::$line"
+                done <<< "$MSG"
+                exit 1
+                ;;
+            esac
+          fi
+        fi
 
-        exit 1
+        # Full-pause maintenance gate (only when issue is open).
+        if [[ "$ISSUE_STATE" == "OPEN" ]]; then
+          MSG=$(printf "%s\n" \
+            "## ⚠️ CI Maintenance Mode is Active" \
+            "The CI infrastructure is currently under maintenance." \
+            "All PR CI runs are paused until maintenance is complete." \
+            "**Merging non-CI-fix PRs is prohibited during maintenance mode.** In severe cases, merge permissions may be revoked." \
+            "You might also experience unexpected failures during this period." \
+            "The team is working on the issue and will update the status as soon as possible." \
+            "" \
+            "What should you do?" \
+            "- **Do NOT merge non-CI-fix PRs** until maintenance mode is lifted" \
+            "- Check back later (~12 hours)" \
+            "- Follow CI Maintenance Mode issue: https://github.com/$REPO/issues/$MAINTENANCE_ISSUE for status updates")
+
+          echo "$MSG" >> "$GITHUB_STEP_SUMMARY"
+          while IFS= read -r line; do
+            echo "::error::$line"
+          done <<< "$MSG"
+          exit 1
+        fi
+
+        echo "✅ Rebase check passed; full-pause not active. Proceeding with CI."

scripts/ci/utils/slash_command_handler.py

@@ -13,6 +13,126 @@ from github import Auth, Github
 PERMISSIONS_FILE_PATH = ".github/CI_PERMISSIONS.json"
 
 
+MAINTENANCE_ISSUE_NUMBER = 21065
+
+
+def _check_rebase_gate(gh_repo, pr, token):
+    """
+    Pre-dispatch gate mirroring `.github/actions/check-maintenance/action.yml`.
+
+    Without this, /rerun-stage and /rerun-test would dispatch a workflow_run
+    on a PR that's behind a required base, the action would catch it, and
+    every job in the run would fail at the gate — wasting runner time and
+    producing N error annotations instead of one comment. Pre-checking here
+    short-circuits the dispatch and posts a single explanatory comment.
+
+    Mirrors the action's two independent modes driven by issue #21065:
+      (1) Full-pause: maintenance issue is OPEN
+      (2) Rebase-required: issue body contains `MIN_BASE_SHA: <sha>`
+    Both bypassed by the `bypass-maintenance` PR label.
+
+    Returns (allowed: bool, message: Optional[str]). When allowed=False,
+    caller MUST post `message` to the PR and skip dispatch.
+    Fail-open on API errors (matches the action's behavior).
+    """
+    headers = {
+        "Authorization": f"Bearer {token}",
+        "Accept": "application/vnd.github+json",
+        "X-GitHub-Api-Version": "2022-11-28",
+    }
+    repo_full_name = gh_repo.full_name
+
+    try:
+        issue_resp = requests.get(
+            f"https://api.github.com/repos/{repo_full_name}/issues/{MAINTENANCE_ISSUE_NUMBER}",
+            headers=headers,
+            timeout=15,
+        )
+        if issue_resp.status_code != 200:
+            print(
+                f"check_rebase_gate: issue fetch returned {issue_resp.status_code}; fail-open"
+            )
+            return True, None
+        issue_data = issue_resp.json()
+    except Exception as e:
+        print(f"check_rebase_gate: issue fetch failed ({e}); fail-open")
+        return True, None
+
+    issue_state = (issue_data.get("state") or "").lower()
+    issue_body = issue_data.get("body") or ""
+
+    min_base_sha = None
+    # First MIN_BASE_SHA: <sha> line wins. Match the action's parser:
+    # tolerate optional backticks and either ':' or '=' separator.
+    for line in issue_body.replace("\r", "").split("\n"):
+        m = re.match(
+            r"^\s*`?MIN_BASE_SHA`?\s*[:=]\s*`?([A-Fa-f0-9]+)`?",
+            line,
+        )
+        if m:
+            candidate = m.group(1)
+            if 7 <= len(candidate) <= 40:
+                min_base_sha = candidate
+            break
+
+    gate_active = (issue_state == "open") or bool(min_base_sha)
+    if not gate_active:
+        return True, None
+
+    bypass = any(
+        (lbl.name if hasattr(lbl, "name") else lbl.get("name")) == "bypass-maintenance"
+        for lbl in pr.get_labels()
+    )
+    if bypass:
+        print("check_rebase_gate: PR has bypass-maintenance label; allowing dispatch")
+        return True, None
+
+    if issue_state == "open":
+        msg = (
+            "## ⚠️ CI Maintenance Mode is Active\n"
+            "The CI infrastructure is currently under maintenance. "
+            "All PR CI runs are paused until maintenance is complete. "
+            "**Merging non-CI-fix PRs is prohibited during maintenance mode.**\n\n"
+            f"Follow [issue #{MAINTENANCE_ISSUE_NUMBER}]"
+            f"(https://github.com/{repo_full_name}/issues/{MAINTENANCE_ISSUE_NUMBER}) "
+            "for status updates. Re-run was not dispatched."
+        )
+        return False, msg
+
+    # MIN_BASE_SHA set, issue not OPEN — check rebase status.
+    pr_head_sha = pr.head.sha
+    try:
+        compare_resp = requests.get(
+            f"https://api.github.com/repos/{repo_full_name}/compare/{min_base_sha}...{pr_head_sha}",
+            headers=headers,
+            timeout=15,
+        )
+        if compare_resp.status_code != 200:
+            print(
+                f"check_rebase_gate: compare API returned {compare_resp.status_code}; fail-open"
+            )
+            return True, None
+        status = compare_resp.json().get("status", "unknown")
+    except Exception as e:
+        print(f"check_rebase_gate: compare API failed ({e}); fail-open")
+        return True, None
+
+    if status in ("ahead", "identical"):
+        return True, None
+
+    msg = (
+        "## ⚠️ Rebase Required Before Re-run\n"
+        f"A major update has landed on `main`. Your PR is `{status}` relative "
+        f"to required base commit `{min_base_sha[:12]}`.\n\n"
+        "**Re-run was not dispatched.** What to do:\n"
+        "- Rebase your branch onto the latest `main` and push again\n"
+        f"- Follow [issue #{MAINTENANCE_ISSUE_NUMBER}]"
+        f"(https://github.com/{repo_full_name}/issues/{MAINTENANCE_ISSUE_NUMBER}) for context\n"
+        "- CI-fix PRs may request the `bypass-maintenance` label to skip this check"
+    )
+    return False, msg
+
+
 def find_workflow_run_url(
     gh_repo,
     workflow_id,
@@ -345,6 +465,12 @@ def handle_rerun_stage(
         )
         return False
 
+    allowed, gate_msg = _check_rebase_gate(gh_repo, pr, token)
+    if not allowed:
+        comment.create_reaction("confused")
+        pr.create_issue_comment(gate_msg)
+        return False
+
     try:
         # Get the appropriate workflow based on stage type
         workflow_name = "PR Test (AMD)" if is_amd_stage else "PR Test"
@@ -938,6 +1064,12 @@ def handle_rerun_test(
         )
         return False
 
+    allowed, gate_msg = _check_rebase_gate(gh_repo, pr, token)
+    if not allowed:
+        comment.create_reaction("confused")
+        pr.create_issue_comment(gate_msg)
+        return False
+
     # Phase 1: Resolve all specs
     resolved = []
     resolve_failures = []