[CI] Add ci monitor balance workflow (#11962)

scripts/ci_monitor/README.md

@@ -2,10 +2,11 @@
 
 > **Note**: This README.md is primarily generated by Claude 4 with some manual adjustments.
 
-A comprehensive toolkit to analyze CI failures and performance trends for the SGLang project. This toolkit includes two main tools:
+A comprehensive toolkit to analyze CI failures and performance trends for the SGLang project. This toolkit includes three main tools:
 
 1. **CI Analyzer** (`ci_analyzer.py`): Analyzes CI failures and provides detailed failure pattern analysis
 2. **Performance Analyzer** (`ci_analyzer_perf.py`): Tracks performance metrics over time and generates trend charts
+3. **Test Balance Analyzer** (`ci_analyzer_balance.py`): Analyzes test time gaps between elapsed and estimated times to help balance CI
 
 ## Features
 
@@ -26,6 +27,15 @@ A comprehensive toolkit to analyze CI failures and performance trends for the SG
 - **Comprehensive Metrics**: Track output throughput, E2E latency, TTFT, accept length, and more
 - **Time-Based Sampling**: Intelligent sampling strategy to cover extended time periods (up to 30 days) with limited API calls
 
+### Test Balance Analyzer (`ci_analyzer_balance.py`)
+- **Time Gap Analysis**: Identify GPU tests with large gaps between elapsed and estimated times
+- **CI Balancing**: Help optimize CI by identifying tests that need time adjustments
+- **Gap Tracking**: Track maximum time gaps for each test across multiple CI runs
+- **PR Test Focus**: Only analyzes GPU jobs from pr-test.yml workflow (excludes AMD and other workflows)
+- **Ranking System**: Sort tests by time gap severity to prioritize adjustments
+- **CSV Export**: Export analysis results in CSV format for easy review
+- **GitHub Integration**: Generate GitHub Actions summaries with recommendations
+
 ### Common Features
 - **Automated Monitoring**: GitHub Actions workflow for continuous CI and performance monitoring
 
@@ -45,6 +55,13 @@ Additional dependencies required for chart generation:
 pip install requests matplotlib pandas
 ```
 
+### For Test Balance Analyzer
+No additional dependencies required beyond Python standard library and `requests`:
+
+```bash
+pip install requests
+```
+
 ## Usage
 
 ### CI Analyzer
@@ -97,6 +114,25 @@ python ci_analyzer_perf.py --token YOUR_GITHUB_TOKEN --start-date $(date -d '7 d
 python ci_analyzer_perf.py --token YOUR_GITHUB_TOKEN --limit 1000 --upload-to-github
 ```
 
+### Test Balance Analyzer
+
+#### Basic Usage
+
+```bash
+# Analyze PR Test GPU job time gaps from recent CI runs
+python ci_analyzer_balance.py --token YOUR_GITHUB_TOKEN
+```
+
+#### Advanced Usage
+
+```bash
+# Analyze last 1000 PR Test GPU CI runs for comprehensive test balance analysis
+python ci_analyzer_balance.py --token YOUR_GITHUB_TOKEN --limit 1000
+
+# Custom output file
+python ci_analyzer_balance.py --token YOUR_GITHUB_TOKEN --limit 500 --output my_balance_analysis.json
+```
+
 **Important**: Make sure your GitHub token has `repo` and `workflow` permissions, otherwise you'll get 404 errors.
 
 ## Data Collection Strategies
@@ -183,6 +219,14 @@ Use `--start-date` and `--end-date` parameters to get **ALL** CI runs within a s
 | `--end-date` | None | End date for date range query (YYYY-MM-DD format) |
 | `--upload-to-github` | False | Upload results to sglang-bot/sglang-ci-data repository |
 
+### Test Balance Analyzer Parameters
+
+| Parameter | Default | Description |
+|-----------|---------|-------------|
+| `--token` | Required | GitHub Personal Access Token |
+| `--limit` | 1000 | Number of CI runs to analyze |
+| `--output` | test_balance_report.json | Output JSON file for detailed analysis data |
+
 ## Getting GitHub Token
 
 1. Go to [GitHub Settings > Personal Access Tokens](https://github.com/settings/tokens)