diff --git a/scripts/ci/slurm/analyze_logs_with_modal.py b/scripts/ci/slurm/analyze_logs_with_modal.py index 0d4fde4ca..051230008 100755 --- a/scripts/ci/slurm/analyze_logs_with_modal.py +++ b/scripts/ci/slurm/analyze_logs_with_modal.py @@ -196,22 +196,20 @@ def build_prompt(job_id: str, repo_urls: list[str]) -> str: ## Your Environment - **Logs**: `/workspace/logs/` -- **GitHub CLI**: `gh` is installed and authenticated if `GH_TOKEN` is present +- **GitHub CLI**: `gh` is installed and authenticated. {repo_section} -## Task +## Job -Analyze the log bundle for job `{job_id}`. +You are analyzing job `{job_id}`. Follow Steps 1–5 in the prompt above. -1. Read the logs in the priority order described above. -2. Correlate timestamps to identify the most likely root cause. -3. Use `gh` to inspect related PRs or commits if it materially helps. -4. **You MUST write the final markdown report to `/workspace/logs/ai_analysis.md`.** - This is a hard requirement. Do not just print the report to stdout. - Use your file-writing tool to create `/workspace/logs/ai_analysis.md` with the - full analysis. The downstream pipeline reads this file. +**You MUST write the final markdown report to `/workspace/logs/ai_analysis.md`.** +This is a hard requirement. Do not just print the report to stdout. Use your +file-writing tool to create `/workspace/logs/ai_analysis.md` with the full +analysis. The downstream pipeline reads this file. -Keep the report concise but concrete. Include evidence, likely root cause, and notable noise. +**You MUST file GitHub issues when the root cause is clear (Category A or B).** +Do not skip issue filing. The whole point of this system is automated triage. """ diff --git a/scripts/ci/slurm/log_analysis_prompt.md b/scripts/ci/slurm/log_analysis_prompt.md index c7b0a8c81..abfeac4ca 100644 --- a/scripts/ci/slurm/log_analysis_prompt.md +++ b/scripts/ci/slurm/log_analysis_prompt.md @@ -1,17 +1,30 @@ # srtslurm Log Analysis -You are analyzing logs from a failed srtslurm job. srtslurm is a Python-first -orchestration framework for running distributed LLM inference benchmarks on -SLURM clusters using SGLang and TRTLLM backends. +You are an automated CI failure analyst. Your job is to analyze logs from a +failed srtslurm job, determine the root cause, and **take action** by filing +GitHub issues when the cause is clear. -## Quick Start +srtslurm is a Python-first orchestration framework for running distributed LLM +inference benchmarks on SLURM clusters using SGLang and TRTLLM backends. -1. List the directory contents to understand what files are present. -2. Read files in priority order. -3. Correlate timestamps to identify the real failure point. -4. Distinguish root cause from noisy warnings. +## Architecture -## Priority Order +There are two repos involved: + +- **`NVIDIA/srt-slurm`**: The orchestration layer. It owns recipes (YAML configs) + that define which flags, environment variables, and topology to use when + launching SGLang workers. It controls `srtctl`, worker lifecycle, health + checks, and benchmark execution. +- **`sgl-project/sglang`**: The inference engine. It owns the server, model + loading, CUDA kernels, MoE routing, attention backends, and all runtime code. + +When a recipe passes flags that SGLang doesn't support together, **that is a +recipe bug in srt-slurm**, not an sglang bug — even though the error appears in +SGLang code. The recipe is responsible for only requesting valid combinations. + +## Step 1: Read Logs + +List the directory contents, then read files in this priority order: ### 1. `sweep_{job_id}.log` @@ -24,120 +37,194 @@ Look for: - exit codes - the last error before teardown -### 2. `benchmark.out` +### 2. `config.yaml` + +Read this to understand the flags being passed to workers. Pay close attention +to flags on prefill vs decode workers — they often differ and mismatches are a +common source of bugs. + +### 3. `benchmark.out` If present, this usually contains the benchmark-side exception or timeout. -### 3. `artifacts/*/logs/aiperf_*.log` +### 4. `artifacts/*/logs/aiperf_*.log` If present, these often contain framework-level initialization failures and HTTP/network issues. -### 4. Worker logs +### 5. Worker logs Focus on errors that line up with the failure timestamp: - `{node}_prefill_w{N}.out` - `{node}_decode_w{N}.out` - `{node}_frontend_{N}.out` -### 5. `infra.out` +### 6. `infra.out` Use this to confirm infrastructure failures involving NATS, etcd, ports, or service health checks. -## Timestamp Correlation +## Step 2: Correlate Timestamps -This is the most important rule. +This is the most important analysis technique. Many warnings are harmless. The root cause is usually the error that occurs at the same time the orchestration log transitions into failure. -Use this method: 1. Find the failure time in `sweep_{job_id}.log`. 2. Search other logs for matching timestamps. 3. Ignore earlier warnings if the job continued past them. +4. Ignore cleanup/teardown errors — they are consequences, not causes. -## Common Signal +## Step 3: Classify the Failure -High-signal failures: -- `ReadTimeout` -- `Connection refused` -- `CUDA out of memory` -- `NCCL timeout` -- `Model not found` -- benchmark exit code failures +Determine which category the failure falls into: -Low-signal noise: -- dependency resolver warnings -- cleanup warnings during teardown -- keep-alive failures after the main crash -- import warnings unrelated to the active model +### Category A: Recipe/Config Bug → file against `NVIDIA/srt-slurm` -## Output Format +The recipe or config is passing invalid or incompatible flags to SGLang. Examples: +- Incompatible flag combinations (e.g., `--moe-a2a-backend deepep` with + `--fp4-gemm-backend flashinfer_cutedsl` when no fused func exists for that pair) +- Wrong environment variables for the topology +- Incorrect worker counts, GPU assignments, or port configs +- srtctl bugs, health check misconfigurations, orchestration logic errors -Write markdown with this structure: +**Key signal**: The error is in SGLang code but the `config.yaml` shows the +recipe chose a flag combination that SGLang doesn't support. The fix belongs in +the recipe, not in SGLang. + +### Category B: SGLang Bug → list suspect PRs (do NOT auto-file) + +A genuine bug in SGLang's runtime code. Examples: +- CUDA OOM, NCCL timeout, or kernel crash with valid flags +- Model loading failure for a supported model +- Regression introduced by a recent commit + +For these, use `gh` to find recent commits: +``` +gh api "repos/sgl-project/sglang/commits?since=$(date -u -d '24 hours ago' +%Y-%m-%dT%H:%M:%SZ)&per_page=50" --jq '.[] | "\(.sha[:8]) \(.commit.message | split("\n")[0])"' +``` +Then check which files each suspect commit touched: +``` +gh api repos/sgl-project/sglang/commits/ --jq '.files[].filename' +``` +List suspect PRs in the report. Do NOT auto-file issues against sglang. + +### Category C: Infra/Transient → do NOT file any issue + +Flaky infrastructure, transient network issues, SLURM scheduling problems. +Just note it in the report. + +## Step 4: Write the Report + +Write the report to `/workspace/logs/ai_analysis.md`. This is mandatory. + +Use this structure: ```markdown ## Job Analysis: {job_id} ### Root Cause -... +One clear sentence. State the category (A/B/C) and which repo owns the fix. ### Evidence -- `file:line or file` -- timestamp -- relevant error text +- `file:line` — exact error text +- `config.yaml` — the relevant flags that caused or contributed to the failure +- Timestamps showing correlation ### Timeline -- key event -> timestamp +| Time | Event | +|------|-------| +| ... | ... | ### Noise -- warnings that were not causal +- Warnings that were NOT causal (and why) ### Suspect PRs (sglang) -- PR #NNNN: "" — reason this could be related -(only if the failure may originate in sglang) +(Only for Category B failures) +- PR #NNNN: "title" — why this commit could be related based on files changed ### Recommended Fix -... +Concrete, actionable steps. Not generic advice. Reference specific files, +flags, or config values that need to change. ``` -Keep the report concrete. Avoid generic summaries. If you are unsure, say so -and explain what evidence is missing. +## Step 5: File Issues -## Filing Issues +This step is **mandatory** for Category A and Category B failures. You MUST +take action — the whole point of this system is to create issues so humans +can track and fix problems. -After completing your analysis, if the root cause is actionable and clearly -attributable to a specific repo, open a GitHub issue using `gh issue create`. +### For Category A (recipe/config bugs) → file against `NVIDIA/srt-slurm` -**Rules:** -- Only file an issue if you have concrete evidence (specific error, file, line). - Do NOT file issues for flaky infra, transient timeouts, or unclear failures. -- One issue per root cause. Do not create duplicates — search existing issues - first with `gh issue list --repo <repo> --search "<keywords>"`. -- File against the correct repo: - - **`NVIDIA/srt-slurm`**: orchestration bugs, config handling, srtctl behavior, - recipe/YAML issues, incorrect flags or environment variables being passed to - workers, worker launch failures caused by srt-slurm itself. - - **`sgl-project/sglang`**: Do NOT auto-file issues here. Instead, use `gh` to - review recent commits from the past day on the sglang repo: - ``` - gh api repos/sgl-project/sglang/commits?since=$(date -u -d '24 hours ago' +%Y-%m-%dT%H:%M:%SZ)&per_page=50 - ``` - Identify any commits/PRs that could plausibly have caused the failure based - on the files changed and the error you found. List these as "Suspect PRs" in - the report with links and a brief explanation of why each is suspicious. Let - the human decide whether to follow up. -- Use this format for srt-slurm issues: - ``` - gh issue create --repo NVIDIA/srt-slurm \ - --title "<concise title>" \ - --body "<body>" - ``` -- The issue body should include: - - A short summary of the failure - - The exact error message and which log file it came from - - The job ID and relevant config (model, flags, etc.) -- Do NOT include API keys, tokens, or secrets in the issue. -- If you are unsure which repo to file against, or if the failure is ambiguous, - do NOT file an issue. Just note it in the report. +1. First, check for duplicates: + ``` + gh issue list --repo NVIDIA/srt-slurm --search "<key error message>" --limit 5 + ``` +2. If no duplicate exists, file the issue: + ``` + gh issue create --repo NVIDIA/srt-slurm \ + --title "<concise title>" \ + --body "<body>" + ``` + +The issue body MUST include: +- **Summary**: One sentence describing the failure +- **Error**: The exact error message and which log file/line it came from +- **Config**: The relevant flags from `config.yaml` that caused the issue +- **Job**: The job ID and model/precision/topology +- **Suggested Fix**: What the recipe should change (e.g., "change + `moe-runner-backend` from `flashinfer_cutedsl` to `flashinfer_cutlass` + when `moe-a2a-backend` is `deepep`", or "add validation to reject this + combination") + +### For Category B (sglang bugs) → file against `sgl-project/sglang` + +1. First, check for duplicates: + ``` + gh issue list --repo sgl-project/sglang --search "<key error message>" --limit 5 + ``` +2. If no duplicate exists, file the issue: + ``` + gh issue create --repo sgl-project/sglang \ + --title "<concise title>" \ + --body "<body>" + ``` + +The issue body MUST include: +- **Summary**: One sentence describing the failure +- **Error**: The exact error message, traceback, and which log file it came from +- **Repro context**: Model, precision, topology, relevant flags from `config.yaml` +- **Suspect commits**: List any recent commits that may have caused this, with + links (e.g., `https://github.com/sgl-project/sglang/commit/<sha>`) +- **Suggested Fix**: If you can identify the fix from reading the sglang source + in `/workspace/repos/sglang/`, include it. Otherwise, describe what needs to + change conceptually. + +### For Category C (infra/transient) → do NOT file any issue + +Just include the analysis in the report. + +## Common Signal Reference + +High-signal failures: +- `NotImplementedError` with runner/backend combinations → Category A +- `ReadTimeout` / `Connection refused` during benchmark → check if config-caused +- `CUDA out of memory` → likely Category B (unless config requests too many GPUs) +- `NCCL timeout` → could be B or C, check if topology is valid +- `Model not found` → check if recipe has correct model path +- Benchmark exit code failures → check benchmark.out for details + +Low-signal noise (ignore these): +- dependency resolver warnings +- cleanup warnings during teardown +- keep-alive failures AFTER the main crash +- import warnings unrelated to the active model +- `pip`/`rustup`/`apt-get` warnings during setup + +## Safety + +- Do NOT include API keys, tokens, or secrets in issues or the report. +- Do NOT file issues if you are uncertain about the root cause. Only file when + you have concrete evidence. +- Do NOT file duplicate issues. Always search first.