minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 13 Packages Projects Releases Wiki Activity

Add job and runner failure monitor workflow for CI (#13104)

Browse Source
This commit is contained in:
Douglas Yang
2025-11-12 14:04:50 -08:00
committed by GitHub
parent 2cdde3d46d
commit 03a7e6f4db
3 changed files with 1106 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/ci_monitor/README.md
+65 -1
View File
@@ -2,11 +2,12 @@
> **Note**: This README.md is primarily generated by Claude 4 with some manual adjustments.
A comprehensive toolkit to analyze CI failures and performance trends for the SGLang project. This toolkit includes three main tools:
A comprehensive toolkit to analyze CI failures and performance trends for the SGLang project. This toolkit includes four main tools:
1. **CI Analyzer** (`ci_analyzer.py`): Analyzes CI failures and provides detailed failure pattern analysis
2. **Performance Analyzer** (`ci_analyzer_perf.py`): Tracks performance metrics over time and generates trend charts
3. **Test Balance Analyzer** (`ci_analyzer_balance.py`): Analyzes test time gaps between elapsed and estimated times to help balance CI
4. **Failures Analyzer** (`ci_failures_analysis.py`): Tracks consecutive failures, identifies flaky jobs, and monitors runner health
## Features
@@ -36,6 +37,14 @@ A comprehensive toolkit to analyze CI failures and performance trends for the SG
- **CSV Export**: Export analysis results in CSV format for easy review
- **GitHub Integration**: Generate GitHub Actions summaries with recommendations
### Failures Analyzer (`ci_failures_analysis.py`)
- **Consecutive Failure Tracking**: Identify jobs currently failing
- **Runner Health Monitoring**: Track runner failure rates and identify problematic infrastructure
- **Alert System**: Automatic alerts for consecutive failures and runner problems
- **Instance Tracking**: Monitor specific runner instances for targeted remediation
- **GitHub Integration**: Generate comprehensive summaries with actionable recommendations
- **JSON Export**: Export detailed analysis data for further processing
### Common Features
- **Automated Monitoring**: GitHub Actions workflow for continuous CI and performance monitoring
@@ -133,6 +142,52 @@ python ci_analyzer_balance.py --token YOUR_GITHUB_TOKEN --limit 1000
python ci_analyzer_balance.py --token YOUR_GITHUB_TOKEN --limit 500 --output my_balance_analysis.json
```
### Failures Analyzer
#### Quick Start
```bash
# Set token as environment variable (recommended for security)
export GITHUB_TOKEN="your_token_here"
# Quick test with recent runs
python ci_failures_analysis.py --token $GITHUB_TOKEN --limit 50 --threshold 2
# Standard analysis (same as automated workflow)
python ci_failures_analysis.py --token $GITHUB_TOKEN --limit 300 --threshold 2
# Deep analysis
python ci_failures_analysis.py --token $GITHUB_TOKEN --limit 500 --threshold 3
```
#### Understanding the Output
The script generates a **2-section report**:
**Section 1: Currently Broken Jobs (Active Consecutive Failures)**
- Shows consecutive failure streaks
- These need immediate attention
**Section 2: Runner Health Analysis**
- Shows which runners have high failure rates
- Helps identify infrastructure vs code issues
#### Alert Types
**Job Alerts (Consecutive Failures):**
- Triggered when a job fails ≥ threshold times in a row
- Example: threshold=2, job fails 3 times → ALERT
**Runner Alerts:**
- **Runner Health**: Runner has >30% failure rate with ≥2 different jobs failing
- **Runner Instance**: Specific instance has >50% failure rate with ≥3 jobs
#### Output Files
- **Console**: Human-readable 3-section report (always generated)
- **JSON**: Detailed data (optional, only if `--output` is specified)
- **GitHub Summary**: Markdown (automatically generated in GitHub Actions)
**Important**: Make sure your GitHub token has `repo` and `workflow` permissions, otherwise you'll get 404 errors.
## Data Collection Strategies
@@ -227,6 +282,15 @@ Use `--start-date` and `--end-date` parameters to get **ALL** CI runs within a s
| `--limit` | 1000 | Number of CI runs to analyze |
| `--output` | test_balance_report.json | Output JSON file for detailed analysis data |
### Failures Analyzer Parameters
| Parameter | Default | Description |
|-----------|---------|-------------|
| `--token` | Required | GitHub Personal Access Token |
| `--limit` | 500 | Number of workflow runs to analyze |
| `--threshold` | 3 | Alert threshold for consecutive failures |
| `--output` | None | Output JSON file (optional, only writes if specified) |
## Getting GitHub Token
1. Go to [GitHub Settings > Personal Access Tokens](https://github.com/settings/tokens)