minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 13 Packages Projects Releases Wiki Activity

[CI] Replace the Lark queue-digest card with a daily queue-timeline chart (#38380)

Browse Source
This commit is contained in:
Liangsheng Yin
2026-09-07 18:28:34 -07:00
committed by GitHub
parent c72cae201e
commit b8a81f055d
5 changed files with 258 additions and 188 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/ci_monitor/README.md
+2 -2
View File
@@ -7,8 +7,8 @@ Scripts used by [.github/workflows/ci-failure-monitor.yml](../../.github/workflo
1. **Failures Analyzer** (`ci_failures_analysis.py`): Tracks consecutive failures, identifies flaky jobs, and monitors runner health across PR Test / Nightly workflows (Nvidia, AMD, Intel, XPU, NPU).
2. **Lark Notifier** (`lark_notify.py`): Posts CUDA CI health cards to a Lark group through an incoming webhook (`LARK_WEBHOOK` secret). Stdlib only. Three subcommands:
- `ci-status --run-id N`: one card per finished scheduled run of the Nvidia nightly / weekly / scheduled pr-test. The first attempt lists its failed jobs; a rerun (attempt N > 1) is compared with attempt N-1 of the same run (fixed by rerun / still failing). Triggered by `workflow_run`.
- `runner-health --state-file F`: per-pool online / offline counts for the primary CUDA labels (`N-gpu-h100|h200|h20|5090|b200|b300|gb200|gb300|a10`). Posts only on degraded / recovered transitions plus an hourly reminder while degraded; state is carried between runs via `actions/cache`. Needs an admin PAT to list runners.
- `queue-digest --hours 8 --only-if-slow`: per-pool queue time p50 / p90 / max over the window plus currently queued jobs, posted only when some pool's p90 exceeds `--slow-minutes` (default 30). Links to the latest Runner Utilization Report run.
- `runner-health --state-file F`: per-pool online / offline counts for the primary CUDA labels (`N-gpu-h100|h200|h20|5090|b200|b300|gb200|gb300`). Posts only on degraded / recovered transitions plus a reminder every `--remind-hours` (6h in CI) while degraded; state is carried between runs via `actions/cache`. Needs an admin PAT to list runners.
- `queue-timeline --series-file F`: one card a day charting the hourly queue backlog and wait p90 across the CUDA pools, naming the pool behind each peak. The series comes from `runner_utilization_report.py --queue-series-out`, so this runs inside runner-utilization.yml rather than here -- a 24h scan of its own would cost a PAT's whole hourly API budget.
All subcommands accept `--dry-run` to print the card JSON instead of posting.