minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 13 Packages Projects Releases Wiki Activity

[CI] add Precision Regression Test on Nightly Run CI (#26902)

Browse Source
This commit is contained in:
Jared Wen
2026-06-14 12:43:57 +08:00
committed by GitHub
parent a3fd5c24be
commit 5da3b37a9d
8 changed files with 2160 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.github/workflows/nightly-test-nvidia.yml
+35
View File
@@ -27,6 +27,7 @@ on:
- 'nightly-test-kernel-1-gpu-h100' - 'nightly-test-kernel-1-gpu-h100'
- 'nightly-test-diffusion-comparison' - 'nightly-test-diffusion-comparison'
- 'nightly-test-kernel-8-gpu-h200' - 'nightly-test-kernel-8-gpu-h200'
- 'nightly-test-precision-8-gpu-h200'
workflow_call: workflow_call:
inputs: inputs:
ref: ref:
@@ -604,6 +605,39 @@ jobs:
- uses: ./.github/actions/upload-cuda-coredumps - uses: ./.github/actions/upload-cuda-coredumps
if: failure() if: failure()
# Nightly precision regression - per-layer hidden state comparison
nightly-test-precision-8-gpu-h200:
if: github.repository == 'sgl-project/sglang' && (inputs.job_filter == '' || inputs.job_filter == 'all' || inputs.job_filter == 'nightly-test-precision-8-gpu-h200')
runs-on: 8-gpu-h200
steps:
- name: Checkout code
uses: actions/checkout@v4
with:
ref: ${{ inputs.ref || github.ref }}
- uses: ./.github/actions/check-maintenance
- name: Install dependencies
run: |
bash scripts/ci/cuda/ci_install_dependency.sh
- name: Run precision regression test
timeout-minutes: 120
env:
SGLANG_PRECISION_BASELINE_DIR: /tmp/sglang_precision_baselines
# Required: the test errors if SGLANG_PRECISION_HF_REPO is unset (no
# local-only mode). Set the var + the HF_TOKEN_PRECISION_STORE secret.
SGLANG_PRECISION_HF_REPO: ${{ vars.SGLANG_PRECISION_HF_REPO }}
SGLANG_PRECISION_HF_REVISION: ${{ vars.SGLANG_PRECISION_HF_REVISION || 'main' }}
HF_TOKEN: ${{ secrets.HF_TOKEN_PRECISION_STORE }}
SGLANG_PRECISION_COMMIT: ${{ github.sha }}
run: |
cd test
python3 run_suite.py --hw cuda --suite nightly-precision-8-gpu-h200 --nightly --continue-on-error --timeout-per-file 3600
- uses: ./.github/actions/upload-cuda-coredumps
if: failure()
# Consolidate performance metrics from all jobs # Consolidate performance metrics from all jobs
consolidate-metrics: consolidate-metrics:
if: github.repository == 'sgl-project/sglang' && always() if: github.repository == 'sgl-project/sglang' && always()
@@ -662,6 +696,7 @@ jobs:
- nightly-test-perf-4-gpu-b200 - nightly-test-perf-4-gpu-b200
- nightly-test-specialized-8-gpu-b200 - nightly-test-specialized-8-gpu-b200
- nightly-test-diffusion-comparison - nightly-test-diffusion-comparison
- nightly-test-precision-8-gpu-h200
- consolidate-metrics - consolidate-metrics
runs-on: ubuntu-latest runs-on: ubuntu-latest
steps: steps:
docs_new/docs.json
+2 -1
View File
@@ -855,7 +855,8 @@
"cookbook/base/reference/server_arguments" "cookbook/base/reference/server_arguments"
] ]
}, },
"docs/references/post_training_integration" "docs/references/post_training_integration",
"docs/references/nightly_precision_regression"
] ]
} }
] ]
docs_new/docs/references/nightly_precision_regression.mdx
+372
View File
@@ -0,0 +1,372 @@
# Nightly Precision Regression Testing
## Overview
The nightly precision regression framework detects silent numerical regressions in the SGLang serving engine by comparing **per-layer hidden states** between consecutive runs. It runs as a nightly CI job on 8×H200 GPUs and can also be invoked locally for development and debugging.
The framework operates on a **rolling-baseline** model:
1. **Baseline creation or comparison:** Launch the server, send a fixed prompt, dump per-layer hidden states to disk. If a previous baseline exists, compare the new tensors against it using the SGLang tensor comparator. If the comparison passes, the new tensors become the updated baseline.
2. On the first run (or when the capture shape changes), the dumped tensors are saved as a new baseline with no comparison.
Baselines are stored locally on disk and synced to a **HuggingFace dataset** so they survive across CI runners and can be shared across machines. The HF dataset store is **required** — the test errors if `SGLANG_PRECISION_HF_REPO` is unset.
---
## How It Works
### Step-by-step flow
```
┌──────────────────────────────────────────────────────────────┐
│ 1. Resolve model config (layer count, capture layers) │
│ ↓ │
│ 2. Compute capture_signature (schema, layers, TP, filter) │
│ ↓ │
│ 3. Fetch baseline from HF dataset (signature-matched) │
│ ↓ │
│ 4. Launch SGLang server with DUMPER enabled │
│ ↓ │
│ 5. POST /dumper/configure (set layer filter + cleanup) │
│ ↓ │
│ 6. POST /v1/chat/completions (fixed prompt, 2 tokens, │
│ ignore_eos=true to force decode path) │
│ ↓ │
│ 7. Kill server; assert decode tensors were captured │
│ ↓ │
│ 8. Baseline exists (with matching signature)? │
│ ├── YES → Run comparator → pass/fail │
│ │ ├── PASS → update baseline, push to HF │
│ │ └── FAIL → push diagnostics to HF │
│ └── NO → copy today's tensors as initial baseline │
│ → push to HF as "baseline_established" │
│ ↓ │
│ 9. Report summary (stdout + GitHub Step Summary) │
└──────────────────────────────────────────────────────────────┘
```
### Key components
| Component | File | Purpose |
|-----------|------|---------|
| Test entry point | `test/registered/debug_utils/test_nightly_precision_regression.py` | Orchestrates server launch, dump, compare, and reporting |
| HF baseline store | `python/sglang/test/precision_baseline_store.py` | Push / fetch / prune baselines on a HuggingFace dataset |
| Tensor comparator | `python/sglang/srt/debug_utils/comparator/` | Compares two directories of `.pt` tensors, emits JSONL report |
| Dumper infrastructure | `python/sglang/srt/debug_utils/dumper.py` | Captures per-layer hidden states at runtime |
| CI workflow | `.github/workflows/nightly-test-nvidia.yml` | Schedules the nightly job on 8×H200 |
---
## What Gets Dumped and Compared
### Strided layer capture
Not every layer is dumped — the framework uses a **strided capture** to reduce I/O and storage overhead. By default, it captures:
- Layer 0 (always)
- The last layer (always)
- Every 8th layer in between (configurable via `LAYER_CAPTURE_STRIDE`)
The layer count is resolved automatically from the model's HuggingFace `config.json` (`num_hidden_layers` or `num_layers`). If resolution fails, all layers are captured as a safe fallback.
The dumper filter is built dynamically as a regex matching only the selected layer indices, e.g.:
```
match(r'^non_intrusive__model\.layers\.(0|7|15|23)\.inputs\.1$', name)
```
### Decode-path verification
The test generates **2 tokens** with `ignore_eos=True` to ensure the model's decode path is exercised. After the dump, `_assert_decode_captured()` verifies that tensors from the decode step were actually captured (not just prefill). If only prefill tensors are found, the test fails immediately — this catches misconfigurations where `--max-total-tokens` is too low for the decode loop to run.
### Comparator
The comparator computes **relative differences** (`rel_diff`) for each tensor and checks them against a configurable threshold (default `1e-3`). For tensor-parallel models, the `--override-dims` flag tells the comparator how to reduce across TP ranks before comparing:
```
--override-dims ^non_intrusive__model\.layers\.\d+\.inputs\.1$:bs h[tp:partial]
```
This sums partial TP contributions along the hidden dimension before computing the diff, so the comparison is semantically correct even with TP > 1.
If the comparator returns exit code 0 but compared **zero layers** (baseline/target name mismatch), the test fails with a diagnostic message rather than silently passing.
### Capture signature
A `capture_signature` (SHA-1 hash of schema version, max_tokens, ignore_eos, TP size, and dumper filter) is computed per run. The HF store uses this signature during fetch to ensure only baselines with an identical capture shape are considered. If the signature changes (e.g. you add layers to the capture set or change TP), the framework establishes a fresh baseline instead of erroring on incompatible tensors.
---
## Environment Variables
| Variable | Default | Description |
|----------|---------|-------------|
| `SGLANG_PRECISION_MODELS` | `zai-org/GLM-5.1-FP8` | Comma-separated HuggingFace model IDs to test |
| `SGLANG_PRECISION_BASELINE_DIR` | `/tmp/sglang_precision_baselines` | Local directory for baseline tensors |
| `SGLANG_PRECISION_DIFF_THRESHOLD` | `1e-3` | Per-tensor relative diff threshold |
| `SGLANG_PRECISION_FORCE_UPDATE` | `0` | Set to `1` to skip comparison and unconditionally refresh baseline |
| `SGLANG_PRECISION_COMMIT` | _(auto-detected from git)_ | Override the sglang commit SHA tagged on push |
| `SGLANG_PRECISION_HF_REPO` | _(required)_ | HuggingFace dataset repo for cross-runner baseline storage |
| `SGLANG_PRECISION_HF_REVISION` | `main` | Branch/revision of the HF dataset |
| `HF_TOKEN` | _(required in CI)_ | HuggingFace token with write access to the dataset |
---
## CI Integration
### Workflow job
The nightly job `nightly-test-precision-8-gpu-h200` is defined in `.github/workflows/nightly-test-nvidia.yml` and runs on an 8-GPU H200 runner. It is included in the nightly suite via `test/run_suite.py`.
Key CI configuration:
```yaml
- name: Run precision regression test
timeout-minutes: 120
env:
SGLANG_PRECISION_BASELINE_DIR: /tmp/sglang_precision_baselines
SGLANG_PRECISION_HF_REPO: ${{ vars.SGLANG_PRECISION_HF_REPO }}
SGLANG_PRECISION_HF_REVISION: ${{ vars.SGLANG_PRECISION_HF_REVISION || 'main' }}
HF_TOKEN: ${{ secrets.HF_TOKEN_PRECISION_STORE }}
SGLANG_PRECISION_COMMIT: ${{ github.sha }}
run: |
cd test
python3 run_suite.py --hw cuda --suite nightly-precision-8-gpu-h200 --nightly --continue-on-error --timeout-per-file 3600
```
### Required GitHub secrets/variables
| Name | Type | Purpose |
|------|------|---------|
| `SGLANG_PRECISION_HF_REPO` | Repository variable | HF dataset repo ID (e.g. `org/sglang-precision-baselines`) — **required**, the test errors if unset |
| `SGLANG_PRECISION_HF_REVISION` | Repository variable (optional) | Dataset branch (defaults to `main`) |
| `HF_TOKEN_PRECISION_STORE` | Repository secret | HF token with write access to the dataset |
### GitHub Step Summary
When running in CI, the test writes a Markdown table to the GitHub Actions job summary showing each model's status (`PASSED`, `FAILED`, `BASELINE_ESTABLISHED`, or `ERROR`).
---
## HF Dataset Storage Layout
Baselines are organized in the HF dataset as:
```
<model_sanitized>/<YYYY>/<MM>/<DD>/run-<sha7>/
├── meta.json # Run metadata (model, commit, hardware, thresholds, stats)
├── comparator_report.jsonl # Per-tensor comparison results
└── tensors/
├── layer_0_inputs_1.pt
├── layer_7_inputs_1.pt
└── ...
```
A top-level `manifest.jsonl` tracks all runs with one JSON object per line. Each manifest row carries a `capture_signature` field so that fetch selects only baselines with a matching capture shape.
The `prune_old_runs()` function (callable manually) retains daily runs for 30 days and keeps one run per week beyond that window.
---
## How to Add a New Model
### Option A: Add to the default model list (CI)
Edit the default in `test/registered/debug_utils/test_nightly_precision_regression.py`:
```python
DEFAULT_MODELS_FOR_NIGHTLY_PRECISION = "zai-org/GLM-5.1-FP8,your-org/your-model"
```
Or set the `SGLANG_PRECISION_MODELS` environment variable in the CI workflow to override the default.
### Option B: Run locally for a specific model
```bash
export SGLANG_PRECISION_MODELS="your-org/your-model"
export SGLANG_PRECISION_BASELINE_DIR="/tmp/my_precision_baselines"
export SGLANG_PRECISION_DIFF_THRESHOLD="1e-3"
export SGLANG_PRECISION_HF_REPO="your-org/sglang-precision-baselines"
export HF_TOKEN="hf_..."
cd test
python3 -m pytest registered/debug_utils/test_nightly_precision_regression.py -v
```
### Step-by-step: adding a model to the nightly CI
1. **Verify the model works with the dumper.** Run locally first to ensure hidden states are captured correctly:
```bash
export SGLANG_PRECISION_MODELS="your-org/your-model"
export SGLANG_PRECISION_BASELINE_DIR="/tmp/test_baselines"
export SGLANG_PRECISION_HF_REPO="your-org/sglang-precision-baselines"
export HF_TOKEN="hf_..."
export SGLANG_PRECISION_FORCE_UPDATE="1" # first run: establish baseline
cd test
python3 -m pytest registered/debug_utils/test_nightly_precision_regression.py -v -k test_precision
```
2. **Run a comparison pass** (remove `FORCE_UPDATE`):
```bash
unset SGLANG_PRECISION_FORCE_UPDATE
python3 -m pytest registered/debug_utils/test_nightly_precision_regression.py -v -k test_precision
```
This should report `PASSED` if the engine is numerically stable for the model.
3. **Set the tensor-parallelism size.** If the model requires TP > 1, the test harness defaults to `tp_size=8` for all models. To customize, modify the `ModelLaunchSettings` construction in the test or pass extra server arguments:
```python
# In setUpClass or via env-driven logic
cls.models = [ModelLaunchSettings("your-org/your-model", tp_size=4)]
```
4. **Adjust the diff threshold if needed.** FP8 or quantized models may exhibit larger numerical differences. Set `SGLANG_PRECISION_DIFF_THRESHOLD` to an appropriate value (e.g., `1e-2` for FP8).
5. **Add to the default model list** or configure `SGLANG_PRECISION_MODELS` in the CI workflow.
### Considerations for model-specific adjustments
| Concern | How to handle |
|---------|--------------|
| TP size != 8 | Override `tp_size` in `ModelLaunchSettings` or add model-specific logic |
| Quantized models (FP8, GPTQ) | Loosen `SGLANG_PRECISION_DIFF_THRESHOLD` (e.g., `1e-2`) |
| Model needs extra server args | Pass them via `ModelLaunchSettings(model, extra_args=["--quantization", "fp8"])` |
| Model needs different prompt | Modify `PROMPT` constant or make it model-configurable |
| MoE models with TP partial sums | Already handled by `--override-dims` (`bs h[tp:partial]`) |
| Fewer/more capture layers | Adjust `LAYER_CAPTURE_STRIDE` (default 8); set lower for smaller models |
| Decode not captured | Ensure `--max-total-tokens` is well above the scheduler's decode reservation (default 512); the test uses 4096 |
---
## Running Locally
### Prerequisites
- SGLang installed in development mode
- GPUs matching the model's requirements
- `huggingface_hub` installed
- A **HuggingFace dataset** for baseline storage and a write-capable `HF_TOKEN`. The HF store is **mandatory** — `SGLANG_PRECISION_HF_REPO` must be set or the test will error at startup. This is because the nightly CI runners are ephemeral (no persistent local disk), so baselines must survive across runs via the HF dataset. There is currently no local-only fallback.
### Quick local test
```bash
# All three are required — the test errors if SGLANG_PRECISION_HF_REPO is unset.
export SGLANG_PRECISION_MODELS="Qwen/Qwen2.5-0.5B-Instruct"
export SGLANG_PRECISION_BASELINE_DIR="/tmp/precision_baselines"
export SGLANG_PRECISION_HF_REPO="your-org/sglang-precision-baselines"
export HF_TOKEN="hf_..."
# First run: establish baseline
cd test
python3 -m pytest registered/debug_utils/test_nightly_precision_regression.py -v
# Second run: compare against baseline
python3 -m pytest registered/debug_utils/test_nightly_precision_regression.py -v
```
### Force-refresh a baseline
```bash
export SGLANG_PRECISION_FORCE_UPDATE="1"
python3 -m pytest registered/debug_utils/test_nightly_precision_regression.py -v
```
---
## Interpreting Results
### Status codes
| Status | Meaning |
|--------|---------|
| `BASELINE_ESTABLISHED` | No prior baseline with a matching signature existed; today's tensors saved as the new baseline |
| `PASSED` | All per-layer hidden states are within the diff threshold; baseline updated |
| `FAILED` | One or more layers exceeded the diff threshold, or 0 layers were compared (baseline/target mismatch); diagnostic data pushed to HF |
| `ERROR` | Server launch, inference, or comparison encountered an unexpected error |
### Output example
```
============================================================
Nightly Precision Regression Summary
============================================================
Model Status Details
------------------------------------------------------------
zai-org/GLM-5.1-FP8 PASSED comparison ok, baseline updated
Qwen/Qwen2.5-0.5B-Instruct FAILED tensor=layer_23.inputs_1 rel_diff=0.0152
============================================================
```
### When a failure is detected
1. The comparator output is saved to `/tmp/nightly_precision_<model>_*.log`
2. The failing tensors and comparator report are pushed to the HF dataset with `pass_label="failed"` for offline diagnosis
3. The GitHub Step Summary includes the failure details
4. The CI job exits with a non-zero status
---
## Baseline Management
### Local baselines
Baselines are stored at:
```
$SGLANG_PRECISION_BASELINE_DIR/<model_sanitized>/nightly_precision/*.pt
```
A `baseline_meta.json` next to the tensors records the timestamp and commit that produced the baseline.
### HF dataset baselines
- **Fetch:** At test start, if no local baseline exists, the latest signature-matched baseline is downloaded from the HF dataset.
- **Push:** After each run, tensors and metadata are uploaded to the dataset.
- **Prune:** Use `prune_old_runs()` to garbage-collect old baselines (keeps 30 days of daily runs, one per week after that).
### Refreshing a stale baseline
If an intentional numerical change (e.g., kernel optimization, model refactor) causes a comparison failure:
1. Verify the change is intentional
2. Set `SGLANG_PRECISION_FORCE_UPDATE=1` and run the test once to establish a new baseline
3. Commit any necessary threshold adjustments
If you change the capture configuration (stride, TP size, etc.), the `capture_signature` will differ and the framework automatically establishes a fresh baseline — no manual intervention needed.
---
## Known Limitations
### Baseline drift
The framework uses a **rolling baseline**: every successful comparison updates the baseline to the current run's tensors. This means the reference shifts forward each day. While individual day-to-day diffs stay within the configured threshold, tiny numerical differences can **accumulate over time**, causing the baseline to silently drift away from the original golden values.
**Implications:**
- The framework detects **regressions** (a sudden, large numerical change between consecutive runs), not **absolute accuracy** relative to a fixed reference.
- Over weeks or months, the cumulative drift may become significant enough to mask a real regression that happened gradually, or to cause a false-positive failure when the drift eventually crosses the threshold.
**Mitigation strategies (not yet implemented):**
- Periodically re-establish a fresh anchor baseline from a known-good reference commit.
- Track the cumulative drift in the manifest metadata and alert when it exceeds a long-term budget.
- Compare against a fixed "epoch" baseline in addition to the rolling one.
### No local-only mode
The test requires a HuggingFace dataset (`SGLANG_PRECISION_HF_REPO`) and a write-capable `HF_TOKEN`. There is no local-only fallback. This is by design — CI runners have no persistent local disk, so the HF dataset is the only way to carry baselines across runs. If you need to run the test locally, you must set up a HF dataset (even a private one) and provide the corresponding token.
---
## File Reference
| File | Role |
|------|------|
| `test/registered/debug_utils/test_nightly_precision_regression.py` | Main test — server lifecycle, dump, compare, report |
| `python/sglang/test/precision_baseline_store.py` | HF dataset store — push, fetch, prune baselines |
| `python/sglang/srt/debug_utils/comparator/` | Tensor comparison engine |
| `python/sglang/srt/debug_utils/dumper.py` | Runtime hidden-state capture |
| `.github/workflows/nightly-test-nvidia.yml` | CI workflow definition |
| `test/run_suite.py` | Test suite registration (includes `nightly-precision-8-gpu-h200`) |
python/sglang/test/precision_baseline_store.py
+395
View File
@@ -0,0 +1,395 @@
"""HF dataset store for nightly precision-regression baselines.
Layout: ``<model>/<YYYY>/<MM>/<DD>/run-<sglang_sha7>/{meta.json,
comparator_report.jsonl, tensors/*.pt}``. Root ``manifest.jsonl`` has one
row per run; rows carry a ``push_index`` so fetch picks the latest push
regardless of file order (prune may rewrite the file).
"""
from __future__ import annotations
import json
import os
import shutil
import tempfile
import time
from dataclasses import dataclass
from datetime import datetime, timezone
from pathlib import Path
from typing import Any, Callable, Optional, TypeVar
from huggingface_hub import HfApi, hf_hub_download, snapshot_download
from huggingface_hub.errors import (
EntryNotFoundError,
HfHubHTTPError,
RepositoryNotFoundError,
)
@dataclass
class HfStoreConfig:
repo: str
revision: str = "main"
@classmethod
def from_env(cls) -> HfStoreConfig:
repo = os.environ.get("SGLANG_PRECISION_HF_REPO")
if not repo:
raise RuntimeError(
"SGLANG_PRECISION_HF_REPO is not set. The precision baseline "
"store is required (there is no local-only mode); set the repo "
"and HF_TOKEN_PRECISION_STORE."
)
revision = os.environ.get("SGLANG_PRECISION_HF_REVISION", "main")
return cls(repo=repo, revision=revision)
def _sanitize_model_name(model: str) -> str:
return model.replace("/", "__").replace(" ", "_")
def _today_path() -> tuple[str, str]:
now = datetime.now(timezone.utc)
return now.strftime("%Y-%m-%d"), now.strftime("%Y/%m/%d")
def _push_index() -> int:
return time.time_ns()
_T = TypeVar("_T")
def _with_retries(
op: Callable[[], _T],
*,
what: str,
attempts: int = 3,
base_delay: float = 2.0,
) -> _T:
"""Exponential backoff on 429/5xx; auth/404 raise immediately."""
last_exc: Optional[BaseException] = None
for attempt in range(1, attempts + 1):
try:
return op()
except HfHubHTTPError as e:
status = getattr(getattr(e, "response", None), "status_code", None)
transient = status is None or status == 429 or 500 <= status < 600
if not transient or attempt == attempts:
raise
last_exc = e
time.sleep(base_delay * (2 ** (attempt - 1)))
raise RuntimeError(f"unreachable retry exit for {what}: {last_exc}")
def _row_recency_key(row: dict[str, Any], fallback_index: int) -> tuple[int, int]:
# Fall back to file position for legacy rows that predate push_index.
explicit = row.get("push_index")
try:
explicit_i = int(explicit) if explicit is not None else -1
except (TypeError, ValueError):
explicit_i = -1
return (explicit_i, fallback_index)
def _select_latest_run(
rows: list[dict[str, Any]],
*,
model: str,
capture_signature: Optional[str] = None,
) -> Optional[str]:
# A baseline is only comparable to a target with the same capture shape, so
# when a signature is given, mismatched (incl. legacy unsigned) rows are
# skipped — fetch then returns None and the caller establishes a fresh one
# instead of erroring on incompatible tensors.
candidates: list[tuple[tuple[int, int], dict[str, Any]]] = []
for idx, row in enumerate(rows):
if row.get("model") != model:
continue
if (
capture_signature is not None
and row.get("capture_signature") != capture_signature
):
continue
if "run_path" not in row:
continue
candidates.append((_row_recency_key(row, idx), row))
if not candidates:
return None
candidates.sort(key=lambda kv: kv[0])
return candidates[-1][1]["run_path"]
def fetch_latest_baseline(
*,
config: HfStoreConfig,
model: str,
target_tensors_dir: Path,
capture_signature: Optional[str] = None,
) -> Optional[str]:
# Tensors land flat in target_tensors_dir (no enclosing tensors/) so the
# caller can treat it like a fresh dump dir.
rows, _ = _read_manifest(config)
run_path = _select_latest_run(
rows, model=model, capture_signature=capture_signature
)
if run_path is None:
return None
snapshot_root = _with_retries(
lambda: snapshot_download(
repo_id=config.repo,
repo_type="dataset",
revision=config.revision,
allow_patterns=[f"{run_path}/tensors/*"],
),
what="snapshot download",
)
src = Path(snapshot_root) / run_path / "tensors"
if not src.exists():
return None
target_tensors_dir.mkdir(parents=True, exist_ok=True)
for fp in src.iterdir():
if fp.is_file():
shutil.copy2(fp, target_tensors_dir / fp.name)
return run_path
def _read_manifest(config: HfStoreConfig) -> tuple[list[dict[str, Any]], str]:
# Skip corrupt rows rather than bricking the store on a partial write.
try:
manifest_local = _with_retries(
lambda: hf_hub_download(
repo_id=config.repo,
repo_type="dataset",
filename="manifest.jsonl",
revision=config.revision,
),
what="manifest fetch",
)
except (EntryNotFoundError, RepositoryNotFoundError):
return [], ""
text = Path(manifest_local).read_text(encoding="utf-8")
rows: list[dict[str, Any]] = []
for line in text.splitlines():
if not line.strip():
continue
try:
rows.append(json.loads(line))
except json.JSONDecodeError:
continue
return rows, text
_MANIFEST_PROMOTE_KEYS = (
"hardware",
"tp_size",
"pass_label",
"capture_signature",
"num_layers_compared",
"num_layers_passed",
"num_layers_failed",
"max_rel_diff",
"ci_run_id",
)
def push_run(
*,
config: HfStoreConfig,
model: str,
sglang_commit: str,
today_tensors_dir: Path,
meta: dict[str, Any],
comparator_report: Optional[Path] = None,
force: bool = False,
) -> str:
# Dedup: same model+date+sha → skip tensor upload but still refresh meta
# + comparator_report + append a new manifest row, so pass-1 baseline and
# pass-2 stats both land. force=True re-uploads tensors too.
api = HfApi()
date_str, date_path = _today_path()
model_sanitized = _sanitize_model_name(model)
sha7 = (
sglang_commit[:7] if sglang_commit and sglang_commit != "unknown" else "no_sha"
)
run_path = f"{model_sanitized}/{date_path}/run-{sha7}"
existing_rows, existing_text = _read_manifest(config)
tensors_already_present = any(r.get("run_path") == run_path for r in existing_rows)
skip_tensors = tensors_already_present and not force
with tempfile.TemporaryDirectory() as stage_dir:
stage = Path(stage_dir)
run_dir = stage / "run"
run_dir.mkdir(parents=True)
if not skip_tensors:
tensors_out = run_dir / "tensors"
tensors_out.mkdir()
for fp in today_tensors_dir.iterdir():
if fp.is_file() and fp.suffix == ".pt":
shutil.copy2(fp, tensors_out / fp.name)
(run_dir / "meta.json").write_text(json.dumps(meta, indent=2), encoding="utf-8")
if comparator_report is not None and comparator_report.exists():
shutil.copy2(comparator_report, run_dir / "comparator_report.jsonl")
commit_msg = (
f"refresh meta {sha7} for {model} on {date_str}"
if skip_tensors
else f"add run {sha7} for {model} on {date_str}"
)
_with_retries(
lambda: api.upload_folder(
repo_id=config.repo,
repo_type="dataset",
revision=config.revision,
folder_path=str(run_dir),
path_in_repo=run_path,
commit_message=commit_msg,
),
what="upload_folder",
)
manifest_row = {
"date": date_str,
"model": model,
"run_path": run_path,
"sglang_commit": sglang_commit,
"push_index": _push_index(),
**{k: meta.get(k) for k in _MANIFEST_PROMOTE_KEYS if k in meta},
}
new_manifest_path: Optional[str] = None
try:
with tempfile.NamedTemporaryFile(
"w", suffix=".jsonl", delete=False, encoding="utf-8"
) as tmp_out:
tmp_out.write(existing_text)
if existing_text and not existing_text.endswith("\n"):
tmp_out.write("\n")
tmp_out.write(json.dumps(manifest_row) + "\n")
new_manifest_path = tmp_out.name
_with_retries(
lambda: api.upload_file(
path_or_fileobj=new_manifest_path,
path_in_repo="manifest.jsonl",
repo_id=config.repo,
repo_type="dataset",
revision=config.revision,
commit_message=f"manifest += {sha7} {date_str}",
),
what="manifest upload",
)
finally:
if new_manifest_path and os.path.exists(new_manifest_path):
os.unlink(new_manifest_path)
return run_path
def prune_old_runs(
*,
config: HfStoreConfig,
model: Optional[str] = None,
keep_days: int = 30,
weekly_archive: bool = True,
dry_run: bool = True,
) -> dict[str, list[str]]:
# dry_run defaults True because model=None+keep_days=0 would wipe the
# store. Live mode rewrites the manifest before deleting folders so a
# mid-run failure leaves manifest pointing at the kept rows only.
api = HfApi()
rows, _ = _read_manifest(config)
if not rows:
return {"kept": [], "pruned": []}
cutoff_date = datetime.now(timezone.utc).date()
def _row_date(row: dict[str, Any]) -> Optional[datetime]:
try:
return datetime.strptime(row["date"], "%Y-%m-%d").replace(
tzinfo=timezone.utc
)
except (KeyError, ValueError):
return None
kept_rows: list[dict[str, Any]] = []
pruned_rows: list[dict[str, Any]] = []
by_model_week: dict[tuple[str, int, int], list[dict[str, Any]]] = {}
for row in rows:
if model is not None and row.get("model") != model:
kept_rows.append(row)
continue
dt = _row_date(row)
if dt is None:
kept_rows.append(row)
continue
age_days = (cutoff_date - dt.date()).days
if age_days <= keep_days:
kept_rows.append(row)
continue
iso_year, iso_week, _ = dt.isocalendar()
by_model_week.setdefault((row.get("model", ""), iso_year, iso_week), []).append(
row
)
for week_rows in by_model_week.values():
week_rows.sort(key=lambda r: r.get("date", ""))
if weekly_archive and week_rows:
kept_rows.append(week_rows[-1])
pruned_rows.extend(week_rows[:-1])
else:
pruned_rows.extend(week_rows)
kept_rows.sort(key=lambda r: (r.get("date", ""), r.get("model", "")))
report = {
"kept": [r["run_path"] for r in kept_rows if "run_path" in r],
"pruned": [r["run_path"] for r in pruned_rows if "run_path" in r],
}
if dry_run or not pruned_rows:
return report
rewritten: Optional[str] = None
try:
with tempfile.NamedTemporaryFile(
"w", suffix=".jsonl", delete=False, encoding="utf-8"
) as tmp_out:
for r in kept_rows:
tmp_out.write(json.dumps(r) + "\n")
rewritten = tmp_out.name
_with_retries(
lambda: api.upload_file(
path_or_fileobj=rewritten,
path_in_repo="manifest.jsonl",
repo_id=config.repo,
repo_type="dataset",
revision=config.revision,
commit_message=f"manifest -= {len(pruned_rows)} pruned",
),
what="manifest rewrite (prune)",
)
finally:
if rewritten and os.path.exists(rewritten):
os.unlink(rewritten)
for r in pruned_rows:
rp = r.get("run_path")
if not rp:
continue
try:
api.delete_folder(
repo_id=config.repo,
repo_type="dataset",
path_in_repo=rp,
revision=config.revision,
commit_message=f"prune {rp}",
)
except Exception:
# Folder may already be missing; manifest no longer points at it.
pass
return report
scripts/ci/utils/slash_command_handler.py
+1
View File
@@ -728,6 +728,7 @@ _LEGACY_SUITE_TO_RUNNER_CONFIG = {
"nightly-8-gpu-common": "8-gpu-h200", "nightly-8-gpu-common": "8-gpu-h200",
"nightly-8-gpu-h200": "8-gpu-h200", "nightly-8-gpu-h200": "8-gpu-h200",
"nightly-kernel-8-gpu-h200": "8-gpu-h200", "nightly-kernel-8-gpu-h200": "8-gpu-h200",
"nightly-precision-8-gpu-h200": "8-gpu-h200",
"nightly-8-gpu-h20": "8-gpu-h20", "nightly-8-gpu-h20": "8-gpu-h20",
"nightly-8-gpu-b200": "8-gpu-b200", "nightly-8-gpu-b200": "8-gpu-b200",
"weekly-8-gpu-h200": "8-gpu-h200", "weekly-8-gpu-h200": "8-gpu-h200",
test/registered/debug_utils/test_nightly_precision_regression.py
+749
View File
@@ -0,0 +1,749 @@
"""Nightly precision regression CI test: dump per-layer hidden states and
compare day-over-day against a rolling baseline.
Env knobs:
SGLANG_PRECISION_MODELS comma-separated model ids (default GLM-5.1-FP8)
SGLANG_PRECISION_BASELINE_DIR local baseline dir
SGLANG_PRECISION_DIFF_THRESHOLD per-tensor rel_diff cutoff (default 1e-3)
SGLANG_PRECISION_FORCE_UPDATE=1 skip comparison, refresh baseline
SGLANG_PRECISION_COMMIT override sglang sha (7-40 hex) tagged on push
SGLANG_PRECISION_HF_REPO required HF dataset repo for cross-runner
baseline storage; see precision_baseline_store
"""
from __future__ import annotations
import hashlib
import json
import math
import os
import re
import shutil
import subprocess
import sys
import tempfile
import unittest
import warnings
from datetime import datetime, timezone
from pathlib import Path
from typing import Any, Optional
import requests
from sglang.srt.utils import kill_process_tree
from sglang.test.ci.ci_register import register_cuda_ci
from sglang.test.test_utils import (
DEFAULT_URL_FOR_TEST,
ModelLaunchSettings,
is_in_ci,
parse_models,
popen_launch_server,
write_github_step_summary,
)
# Soft dep: missing huggingface_hub → import fails loudly in setUpClass.
try:
from sglang.test import precision_baseline_store as _hfs
except Exception: # pragma: no cover
_hfs = None
register_cuda_ci(est_time=3600, suite="nightly-precision-8-gpu-h200", nightly=True)
DEFAULT_MODELS_FOR_NIGHTLY_PRECISION = "zai-org/GLM-5.1-FP8"
DEFAULT_DIFF_THRESHOLD = 1e-3
# Fallback when the layer count can't be resolved: never silently shrink coverage.
DUMPER_FILTER_ALL_LAYERS = (
r"match(r'^non_intrusive__model\.layers\.\d+\.inputs\.1$', name)"
)
LAYER_CAPTURE_STRIDE = 8
MAX_TOKENS = 2
SCHEMA_VERSION = 3
EXP_NAME = "nightly_precision"
PROMPT = "The capital of France is"
NIGHTLY_PRECISION_SERVER_TIMEOUT = 3600
def _sanitize_model_name(model: str) -> str:
return model.replace("/", "__").replace(" ", "_")
def _select_capture_layers(
num_layers: int, stride: int = LAYER_CAPTURE_STRIDE
) -> list[int]:
# First + last + every stride-th: the residual stream is cumulative, so the
# last layer still reflects drift originating in any earlier layer.
layers = set(range(0, num_layers, stride))
layers.add(0)
layers.add(num_layers - 1)
return sorted(i for i in layers if 0 <= i < num_layers)
def _build_dumper_filter(capture_layers: Optional[list[int]]) -> str:
if not capture_layers:
return DUMPER_FILTER_ALL_LAYERS
# The dumper evals this with builtins stripped (no int()/arithmetic), so the
# layer subset is baked into the regex; longest-first avoids ambiguous matches.
alt = "|".join(
str(i) for i in sorted(capture_layers, key=lambda x: (-len(str(x)), x))
)
return rf"match(r'^non_intrusive__model\.layers\.({alt})\.inputs\.1$', name)"
def _resolve_num_layers(model_path: str) -> Optional[int]:
# None on any failure -> caller captures every layer rather than a wrong subset.
try:
from transformers import AutoConfig
config = AutoConfig.from_pretrained(model_path, trust_remote_code=True)
except Exception as e:
warnings.warn(f"could not load config for {model_path}: {e}")
return None
text_config = getattr(config, "text_config", None) # VL configs nest LM dims here
for obj in (text_config, config):
if obj is None:
continue
for attr in ("num_hidden_layers", "num_layers"):
n = getattr(obj, attr, None)
if isinstance(n, int) and n > 0:
return n
warnings.warn(f"could not read layer count from config for {model_path}")
return None
def _assert_decode_captured(exp_dir: Path, *, tp_size: int) -> None:
# One dump per (layer, rank) == prefill only: decode never ran, which would
# pass the comparison while silently halving coverage. Fail loudly instead.
pts = list(exp_dir.glob("*.pt"))
if not pts:
raise AssertionError(f"no .pt dumps produced in {exp_dir}")
layers: set[str] = set()
steps: set[str] = set()
for p in pts:
for kv in p.stem.split("___"):
if kv.startswith("layer_id="):
layers.add(kv[len("layer_id=") :])
elif kv.startswith("step="):
steps.add(kv[len("step=") :])
prefill_only = len(layers) * tp_size
if len(pts) <= prefill_only:
raise AssertionError(
f"decode path not captured: {len(pts)} .pt files for {len(layers)} "
f"layers x {tp_size} tp (== {prefill_only}, prefill-only); "
f"steps={sorted(steps)}. The model generated no decode tokens "
f"(check --max-total-tokens vs the decode reservation, max_tokens, "
f"and ignore_eos)."
)
def _capture_signature(dump_cfg: dict[str, Any], tp_size: int) -> str:
# Identifies the dump shape; fetch only compares against baselines with the
# same signature, so changing layers/forwards/tp re-establishes cleanly
# instead of erroring against incompatible tensors.
raw = "|".join(
str(x)
for x in (
SCHEMA_VERSION,
dump_cfg["max_tokens"],
dump_cfg["ignore_eos"],
tp_size,
dump_cfg["dumper_filter"],
)
)
return hashlib.sha1(raw.encode()).hexdigest()[:12]
_SHA_RE = re.compile(r"^[0-9a-fA-F]{7,40}$")
def _get_git_commit() -> str:
val = os.environ.get("SGLANG_PRECISION_COMMIT", "").strip()
if _SHA_RE.match(val):
return val
try:
return (
subprocess.check_output(["git", "rev-parse", "--short", "HEAD"], timeout=10)
.decode()
.strip()
)
except Exception:
return "unknown"
_NVIDIA_SMI_NAME_RE = re.compile(r"NVIDIA\s+([A-Za-z0-9]+)")
def _collect_runtime_context() -> dict[str, Any]:
# Each probe is independently guarded — missing nvidia-smi/torch never blocks.
ctx: dict[str, Any] = {}
try:
import torch
ctx["torch_version"] = torch.__version__
if torch.version.cuda:
ctx["cuda_version"] = torch.version.cuda
if torch.cuda.is_available():
ctx["num_gpus"] = torch.cuda.device_count()
except Exception:
pass
try:
import sglang
ctx["sglang_version"] = getattr(sglang, "__version__", None)
except Exception:
pass
try:
out = subprocess.check_output(
["nvidia-smi", "-L"], stderr=subprocess.DEVNULL, timeout=5
).decode()
m = _NVIDIA_SMI_NAME_RE.search(out)
if m:
ctx["hardware"] = m.group(1)
except Exception:
pass
return ctx
def _collect_ci_context() -> dict[str, Any]:
ctx: dict[str, Any] = {}
run_id = os.environ.get("GITHUB_RUN_ID")
if run_id:
ctx["ci_run_id"] = run_id
server = os.environ.get("GITHUB_SERVER_URL", "https://github.com")
repo = os.environ.get("GITHUB_REPOSITORY", "")
if repo:
ctx["ci_run_url"] = f"{server}/{repo}/actions/runs/{run_id}"
for env_key, meta_key in (
("GITHUB_ACTOR", "ci_actor"),
("GITHUB_REF", "git_ref"),
("GITHUB_WORKFLOW", "ci_workflow"),
):
v = os.environ.get(env_key)
if v:
ctx[meta_key] = v
return ctx
def _parse_comparator_stats(stdout: str) -> dict[str, Any]:
# Only passing records contribute to max/mean — failed records' rel_diff
# would corrupt the headline. NaN/inf is dropped before aggregation.
n_total = 0
n_passed = 0
n_failed = 0
rel_diffs: list[float] = []
abs_diffs: list[float] = []
failing: list[str] = []
def _record_failure(name: str) -> None:
nonlocal n_failed
n_failed += 1
if len(failing) < 10:
failing.append(name)
def _safe_float(v: Any) -> Optional[float]:
if v is None:
return None
try:
f = float(v)
except (TypeError, ValueError):
return None
return f if math.isfinite(f) else None
for line in stdout.strip().splitlines():
try:
rec = json.loads(line)
except json.JSONDecodeError:
continue
if rec.get("type") != "comparison_tensor":
continue
n_total += 1
name = rec.get("name", "?")
diff = rec.get("diff") or {}
if rec.get("errors"):
_record_failure(name)
continue
if not diff.get("passed", True):
_record_failure(name)
continue
bad_rep = next(
(c for c in rec.get("replicated_checks", []) if not c.get("passed", True)),
None,
)
if bad_rep is not None:
_record_failure(name)
continue
rel = _safe_float(diff.get("rel_diff"))
if rel is not None:
rel_diffs.append(rel)
abs_ = _safe_float(diff.get("abs_diff"))
if abs_ is not None:
abs_diffs.append(abs_)
n_passed += 1
out: dict[str, Any] = {
"num_layers_compared": n_total,
"num_layers_passed": n_passed,
"num_layers_failed": n_failed,
}
if rel_diffs:
out["max_rel_diff"] = max(rel_diffs)
out["mean_rel_diff"] = sum(rel_diffs) / len(rel_diffs)
if abs_diffs:
out["max_abs_diff"] = max(abs_diffs)
if failing:
out["failing_layers"] = failing
return out
class TestNightlyPrecisionRegression(unittest.TestCase):
@classmethod
def setUpClass(cls):
models_str = os.environ.get(
"SGLANG_PRECISION_MODELS", DEFAULT_MODELS_FOR_NIGHTLY_PRECISION
)
cls.models = [
ModelLaunchSettings(m, tp_size=8) for m in parse_models(models_str)
]
cls.baseline_dir = Path(
os.environ.get(
"SGLANG_PRECISION_BASELINE_DIR", "/tmp/sglang_precision_baselines"
)
)
cls.baseline_dir.mkdir(parents=True, exist_ok=True)
cls.diff_threshold = float(
os.environ.get(
"SGLANG_PRECISION_DIFF_THRESHOLD", str(DEFAULT_DIFF_THRESHOLD)
)
)
cls.force_update = os.environ.get("SGLANG_PRECISION_FORCE_UPDATE", "0") == "1"
cls.base_url = DEFAULT_URL_FOR_TEST
if _hfs is None:
raise RuntimeError(
"precision baseline store unavailable: could not import "
"sglang.test.precision_baseline_store"
)
# Raises if SGLANG_PRECISION_HF_REPO is unset — the test requires a
# remote baseline store, there is no local-only mode.
cls.hf_cfg = _hfs.HfStoreConfig.from_env()
def test_precision_all_models(self):
warnings.filterwarnings(
"ignore", category=ResourceWarning, message="unclosed.*socket"
)
all_results = []
for model_setup in self.models:
with self.subTest(model=model_setup.model_path):
try:
result = _test_one_model(
model_setup=model_setup,
baseline_dir=self.baseline_dir,
diff_threshold=self.diff_threshold,
force_update=self.force_update,
base_url=self.base_url,
hf_cfg=self.hf_cfg,
)
all_results.append(result)
except Exception as e:
all_results.append((model_setup.model_path, "ERROR", str(e)))
_report_summary(all_results)
failed = [r for r in all_results if r[1] == "FAILED"]
errored = [r for r in all_results if r[1] == "ERROR"]
if failed or errored:
msg = "Nightly precision regression failures:\n"
for model, status, details in failed + errored:
msg += f" {model}: {status} - {details}\n"
self.fail(msg)
def _test_one_model(
*,
model_setup: ModelLaunchSettings,
baseline_dir: Path,
diff_threshold: float,
force_update: bool,
base_url: str,
hf_cfg,
):
model = model_setup.model_path
model_dir_name = _sanitize_model_name(model)
model_baseline_dir = baseline_dir / model_dir_name
baseline_exp_dir = model_baseline_dir / EXP_NAME
# Resolve the capture shape once and reuse it for the dump request and every
# meta push, so the manifest always reflects exactly what was dumped.
num_layers = _resolve_num_layers(model)
capture_layers = _select_capture_layers(num_layers) if num_layers else None
dump_cfg = {
"max_tokens": MAX_TOKENS,
"ignore_eos": True,
"dumper_filter": _build_dumper_filter(capture_layers),
"num_hidden_layers": num_layers,
"capture_layers": capture_layers,
}
dump_cfg["capture_signature"] = _capture_signature(dump_cfg, model_setup.tp_size)
_maybe_hf_fetch(
hf_cfg=hf_cfg,
model=model,
baseline_exp_dir=baseline_exp_dir,
capture_signature=dump_cfg["capture_signature"],
)
with tempfile.TemporaryDirectory() as today_tmp:
today_dump_dir = Path(today_tmp)
_run_server_and_dump(
model_setup=model_setup,
dump_dir=today_dump_dir,
base_url=base_url,
dumper_filter=dump_cfg["dumper_filter"],
max_tokens=dump_cfg["max_tokens"],
ignore_eos=dump_cfg["ignore_eos"],
)
today_exp_dir = today_dump_dir / EXP_NAME
_assert_decode_captured(today_exp_dir, tp_size=model_setup.tp_size)
has_baseline = baseline_exp_dir.exists() and any(baseline_exp_dir.glob("*.pt"))
if has_baseline and not force_update:
result = _run_comparator(
baseline=baseline_exp_dir,
target=today_exp_dir,
threshold=diff_threshold,
)
debug_file = _save_comparator_output(
stdout=result.stdout, stderr=result.stderr, prefix=model_dir_name
)
print(f"Comparator output for {model}: {debug_file}")
report_path = model_baseline_dir / "comparator_report.jsonl"
report_path.parent.mkdir(parents=True, exist_ok=True)
report_path.write_text(result.stdout, encoding="utf-8")
comparator_stats = _parse_comparator_stats(result.stdout)
if comparator_stats.get("num_layers_compared", 0) == 0:
# A clean returncode with nothing compared means the baseline
# and target tensor names never lined up — fail loudly rather
# than pass on an empty comparison.
return (
model,
"FAILED",
"comparator compared 0 layers (baseline/target name mismatch?)",
)
if result.returncode == 0:
_update_baseline(model_baseline_dir, today_exp_dir)
_maybe_hf_push(
hf_cfg=hf_cfg,
model=model,
model_setup=model_setup,
tensors_dir=baseline_exp_dir,
pass_label="passed",
diff_threshold=diff_threshold,
dump_cfg=dump_cfg,
comparator_report=report_path,
comparator_stats=comparator_stats,
)
return (model, "PASSED", "comparison ok, baseline updated")
# FAILED: push today's tensors as pass_label="failed" so the
# diagnostic diff survives on HF without rerunning the comparator.
_maybe_hf_push(
hf_cfg=hf_cfg,
model=model,
model_setup=model_setup,
tensors_dir=today_exp_dir,
pass_label="failed",
diff_threshold=diff_threshold,
dump_cfg=dump_cfg,
comparator_report=report_path,
comparator_stats=comparator_stats,
)
summary = _extract_diff_summary(result.stdout)
return (model, "FAILED", summary)
else:
_update_baseline(model_baseline_dir, today_exp_dir)
_maybe_hf_push(
hf_cfg=hf_cfg,
model=model,
model_setup=model_setup,
tensors_dir=baseline_exp_dir,
pass_label="baseline_established",
diff_threshold=diff_threshold,
dump_cfg=dump_cfg,
comparator_report=None,
comparator_stats=None,
)
reason = "forced update" if force_update else "first run"
return (model, "BASELINE_ESTABLISHED", reason)
def _maybe_hf_fetch(
*, hf_cfg, model: str, baseline_exp_dir: Path, capture_signature: str
) -> None:
if baseline_exp_dir.exists() and any(baseline_exp_dir.glob("*.pt")):
return
try:
src = _hfs.fetch_latest_baseline(
config=hf_cfg,
model=model,
target_tensors_dir=baseline_exp_dir,
capture_signature=capture_signature,
)
if src is not None:
print(f"[hf-store] restored baseline for {model} from {src}", flush=True)
except Exception as e:
msg = f"[hf-store] fetch failed for {model}: {e}"
if os.environ.get("GITHUB_RUN_ID"):
raise RuntimeError(msg) from e
warnings.warn(msg)
def _maybe_hf_push(
*,
hf_cfg,
model: str,
model_setup: ModelLaunchSettings,
tensors_dir: Path,
pass_label: str,
diff_threshold: float,
dump_cfg: dict[str, Any],
comparator_report: Optional[Path] = None,
comparator_stats: Optional[dict[str, Any]] = None,
) -> None:
# Under CI a push failure raises rather than warns, so a misconfigured
# store can't quietly mask the regression-detection guarantee.
pt_files = list(tensors_dir.glob("*.pt"))
if not pt_files:
return
try:
meta: dict[str, Any] = {
"schema_version": SCHEMA_VERSION,
"timestamp_utc": datetime.now(timezone.utc).isoformat(),
"model": model,
"sglang_commit": _get_git_commit(),
"tp_size": model_setup.tp_size,
"prompt": PROMPT,
"max_tokens": dump_cfg["max_tokens"],
"ignore_eos": dump_cfg["ignore_eos"],
"temperature": 0,
"diff_threshold": diff_threshold,
"dumper_filter": dump_cfg["dumper_filter"],
"num_hidden_layers": dump_cfg.get("num_hidden_layers"),
"capture_layers": dump_cfg.get("capture_layers"),
"capture_signature": dump_cfg.get("capture_signature"),
"num_tensor_files": len(pt_files),
"pass_label": pass_label,
"source": "test_nightly_precision_regression.py",
}
meta.update(_collect_runtime_context())
meta.update(_collect_ci_context())
if comparator_stats:
meta.update(comparator_stats)
run_path = _hfs.push_run(
config=hf_cfg,
model=model,
sglang_commit=meta["sglang_commit"],
today_tensors_dir=tensors_dir,
meta=meta,
comparator_report=comparator_report,
)
print(
f"[hf-store] {pass_label} {len(pt_files)} tensors for {model} -> {run_path}",
flush=True,
)
except Exception as e:
msg = f"[hf-store] push failed for {model}: {e}"
if os.environ.get("GITHUB_RUN_ID"):
raise RuntimeError(msg) from e
warnings.warn(msg)
def _run_server_and_dump(
*,
model_setup: ModelLaunchSettings,
dump_dir: Path,
base_url: str,
dumper_filter: str,
max_tokens: int,
ignore_eos: bool,
):
env: dict[str, str] = {
**os.environ,
"DUMPER_DIR": str(dump_dir),
"DUMPER_EXP_NAME": EXP_NAME,
"DUMPER_SERVER_PORT": "reuse",
"DUMPER_NON_INTRUSIVE_MODE": "all",
}
server_args: list[str] = list(model_setup.extra_args or []) + [
# Below the scheduler's decode-token reservation (default 512) the KV
# pool clamps max_new_tokens to 0, so decode never runs and max_tokens>1
# has no effect. Keep it well above that.
"--max-total-tokens",
"4096",
"--mem-fraction-static",
"0.9",
"--disable-cuda-graph",
"--disable-piecewise-cuda-graph",
"--disable-radix-cache",
]
proc = popen_launch_server(
model_setup.model_path,
base_url,
timeout=NIGHTLY_PRECISION_SERVER_TIMEOUT,
other_args=server_args,
env=env,
)
try:
requests.post(
f"{base_url}/dumper/configure",
json={
"enable": True,
"filter": dumper_filter,
"cleanup_previous": True,
},
timeout=60,
).raise_for_status()
resp = requests.post(
f"{base_url}/v1/chat/completions",
json={
"model": model_setup.model_path,
"messages": [{"role": "user", "content": PROMPT}],
"max_tokens": max_tokens,
"temperature": 0,
# Without this the model may EOS on the first token and never
# enter the decode loop, leaving the decode path uncaptured.
"ignore_eos": ignore_eos,
},
timeout=600,
)
if resp.status_code != 200:
raise RuntimeError(f"Chat completions failed: {resp.text}")
finally:
kill_process_tree(proc.pid)
def _run_comparator(
*, baseline: Path, target: Path, threshold: float
) -> subprocess.CompletedProcess[str]:
cmd: list[str] = [
sys.executable,
"-m",
"sglang.srt.debug_utils.comparator",
"--baseline-path",
str(baseline),
"--target-path",
str(target),
"--diff-threshold",
str(threshold),
"--output-format",
"json",
"--allow-skipped-pattern",
"input_ids|positions|seq_lens|req_pool_indices|rids",
# inputs.1 is hidden_states entering the layer (inputs.0 = positions).
# LayerCommunicator defers the cross-layer allreduce, so layer N sees
# per-tp partial sums; bs h[tp:partial] sums across tp on the h axis
# before diffing.
"--override-dims",
r"^non_intrusive__model\.layers\.\d+\.inputs\.1$:bs h[tp:partial]",
]
return subprocess.run(cmd, capture_output=True, text=True, timeout=300)
def _update_baseline(model_baseline_dir: Path, today_exp_dir: Path):
final_dir = model_baseline_dir / EXP_NAME
staging_dir = model_baseline_dir / "_staging"
old_dir = model_baseline_dir / "_old_baseline"
if staging_dir.exists():
shutil.rmtree(staging_dir)
shutil.copytree(today_exp_dir, staging_dir)
meta = {
"timestamp": datetime.now(timezone.utc).isoformat(),
"commit": _get_git_commit(),
}
(staging_dir.parent / "baseline_meta.json").write_text(
json.dumps(meta, indent=2), encoding="utf-8"
)
if final_dir.exists():
if old_dir.exists():
shutil.rmtree(old_dir)
final_dir.rename(old_dir)
staging_dir.rename(final_dir)
if old_dir.exists():
shutil.rmtree(old_dir)
def _extract_diff_summary(stdout: str) -> str:
for line in stdout.strip().splitlines():
try:
record = json.loads(line)
except json.JSONDecodeError:
continue
if record.get("type") != "comparison_tensor":
continue
if record.get("errors"):
name = record.get("name", "unknown")
return f"tensor={name} errored"
diff = record.get("diff") or {}
if diff and not diff.get("passed", True):
name = record.get("name", "unknown")
rel_diff = diff.get("rel_diff", "N/A")
return f"tensor={name} rel_diff={rel_diff}"
bad_replicated = next(
(
c
for c in record.get("replicated_checks", [])
if not c.get("passed", True)
),
None,
)
if bad_replicated is not None:
name = record.get("name", "unknown")
axis = bad_replicated.get("axis", "?")
return f"tensor={name} replicated_check_failed axis={axis}"
return stdout[-200:] if stdout else "no output"
def _save_comparator_output(*, stdout: str, stderr: str, prefix: str) -> Path:
fd, path_str = tempfile.mkstemp(
prefix=f"nightly_precision_{prefix}_", suffix=".log", dir="/tmp"
)
with os.fdopen(fd, "w", encoding="utf-8") as f:
f.write("=== STDOUT ===\n")
f.write(stdout)
f.write("\n=== STDERR ===\n")
f.write(stderr)
return Path(path_str)
def _report_summary(results):
lines = ["\n" + "=" * 60]
lines.append("Nightly Precision Regression Summary")
lines.append("=" * 60)
lines.append(f"{'Model':<45} {'Status':<25} Details")
lines.append("-" * 60)
for model, status, details in results:
lines.append(f"{model:<45} {status:<25} {details[:80]}")
lines.append("=" * 60)
summary = "\n".join(lines)
print(summary, flush=True)
if is_in_ci():
md = "## Nightly Precision Regression\n\n"
md += "| Model | Status | Details |\n"
md += "|-------|--------|--------|\n"
for model, status, details in results:
md += f"| {model} | {status} | {details[:100]} |\n"
write_github_step_summary(md)
if __name__ == "__main__":
unittest.main()
test/registered/unit/test_precision_baseline_store.py
+604
View File
@@ -0,0 +1,604 @@
"""Unit tests for precision_baseline_store — no server, no model loading, no HF network."""
from sglang.test.ci.ci_register import register_cpu_ci
register_cpu_ci(est_time=5, suite="base-a-test-cpu")
import json
import os
import tempfile
import unittest
from datetime import datetime, timezone
from pathlib import Path
from unittest.mock import MagicMock, patch
from sglang.test import precision_baseline_store as hfs
from sglang.test.test_utils import CustomTestCase
# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------
def _make_config() -> hfs.HfStoreConfig:
return hfs.HfStoreConfig(repo="test/repo", revision="main")
def _make_rows(n: int, *, model: str = "org/model", base_index: int = 0) -> list[dict]:
return [
{
"model": model,
"run_path": f"org__model/2025/01/{i:02d}/run-abc123{i}",
"date": f"2025-01-{i + base_index:02d}",
"push_index": (i + base_index) * 1000,
}
for i in range(n)
]
# ---------------------------------------------------------------------------
# Tests
# ---------------------------------------------------------------------------
class TestHfStoreConfig(CustomTestCase):
def test_from_env_reads_required_var(self):
with patch.dict(
os.environ, {"SGLANG_PRECISION_HF_REPO": "my/repo"}, clear=False
):
cfg = hfs.HfStoreConfig.from_env()
self.assertEqual(cfg.repo, "my/repo")
self.assertEqual(cfg.revision, "main")
def test_from_env_reads_optional_revision(self):
with patch.dict(
os.environ,
{
"SGLANG_PRECISION_HF_REPO": "my/repo",
"SGLANG_PRECISION_HF_REVISION": "dev",
},
clear=False,
):
cfg = hfs.HfStoreConfig.from_env()
self.assertEqual(cfg.revision, "dev")
def test_from_env_default_revision(self):
with patch.dict(
os.environ, {"SGLANG_PRECISION_HF_REPO": "my/repo"}, clear=False
):
cfg = hfs.HfStoreConfig.from_env()
self.assertEqual(cfg.revision, "main")
def test_from_env_raises_when_missing(self):
with patch.dict(os.environ, {}, clear=True):
with self.assertRaises(RuntimeError):
hfs.HfStoreConfig.from_env()
class TestSanitizeModelName(CustomTestCase):
def test_slashes_and_spaces(self):
self.assertEqual(hfs._sanitize_model_name("org/model name"), "org__model_name")
def test_no_changes_needed(self):
self.assertEqual(hfs._sanitize_model_name("simple"), "simple")
class TestRowRecencyKey(CustomTestCase):
def test_uses_explicit_push_index(self):
row = {"push_index": 100}
self.assertEqual(hfs._row_recency_key(row, 5), (100, 5))
def test_falls_back_to_index(self):
row = {}
self.assertEqual(hfs._row_recency_key(row, 5), (-1, 5))
def test_invalid_push_index(self):
row = {"push_index": "bad"}
self.assertEqual(hfs._row_recency_key(row, 5), (-1, 5))
def test_none_push_index(self):
row = {"push_index": None}
self.assertEqual(hfs._row_recency_key(row, 5), (-1, 5))
class TestSelectLatestRun(CustomTestCase):
def test_picks_highest_recency(self):
rows = _make_rows(3)
result = hfs._select_latest_run(rows, model="org/model")
self.assertEqual(result, rows[-1]["run_path"])
def test_filters_by_model(self):
rows = [
{"model": "a/model", "run_path": "a", "push_index": 1},
{"model": "b/model", "run_path": "b", "push_index": 2},
]
self.assertEqual(hfs._select_latest_run(rows, model="a/model"), "a")
def test_filters_by_capture_signature(self):
rows = [
{
"model": "org/m",
"run_path": "old",
"capture_signature": "abc123",
"push_index": 1,
},
{
"model": "org/m",
"run_path": "new",
"capture_signature": "def456",
"push_index": 2,
},
]
self.assertEqual(
hfs._select_latest_run(rows, model="org/m", capture_signature="def456"),
"new",
)
def test_returns_none_on_empty(self):
self.assertIsNone(hfs._select_latest_run([], model="org/m"))
def test_skips_rows_without_run_path(self):
rows = [
{"model": "org/m", "push_index": 1},
{"model": "org/m", "run_path": "good", "push_index": 2},
]
self.assertEqual(hfs._select_latest_run(rows, model="org/m"), "good")
def test_returns_none_when_signature_mismatch(self):
rows = [
{
"model": "org/m",
"run_path": "old",
"capture_signature": "abc123",
"push_index": 1,
},
]
self.assertIsNone(
hfs._select_latest_run(rows, model="org/m", capture_signature="zzz")
)
class TestReadManifest(CustomTestCase):
@patch("sglang.test.precision_baseline_store.hf_hub_download")
def test_parses_valid_manifest(self, mock_download):
content = (
'{"model":"a","run_path":"p1","push_index":1}\n'
'{"model":"b","run_path":"p2","push_index":2}\n'
)
tmp = tempfile.NamedTemporaryFile(mode="w", suffix=".jsonl", delete=False)
try:
tmp.write(content)
tmp.close()
mock_download.return_value = tmp.name
rows, text = hfs._read_manifest(_make_config())
finally:
os.unlink(tmp.name)
self.assertEqual(len(rows), 2)
self.assertEqual(rows[0]["model"], "a")
self.assertEqual(text, content)
@patch("sglang.test.precision_baseline_store.hf_hub_download")
def test_skips_blank_and_corrupt_lines(self, mock_download):
content = (
'{"model":"a","run_path":"p1"}\n\nnot-json\n{"model":"b","run_path":"p2"}\n'
)
tmp = tempfile.NamedTemporaryFile(mode="w", suffix=".jsonl", delete=False)
try:
tmp.write(content)
tmp.close()
mock_download.return_value = tmp.name
rows, _ = hfs._read_manifest(_make_config())
finally:
os.unlink(tmp.name)
self.assertEqual(len(rows), 2)
@patch("sglang.test.precision_baseline_store.hf_hub_download")
def test_returns_empty_on_not_found(self, mock_download):
from huggingface_hub.errors import EntryNotFoundError
mock_download.side_effect = EntryNotFoundError("not found")
rows, text = hfs._read_manifest(_make_config())
self.assertEqual(rows, [])
self.assertEqual(text, "")
class TestFetchLatestBaseline(CustomTestCase):
@patch("sglang.test.precision_baseline_store.snapshot_download")
@patch.object(hfs, "_read_manifest")
def test_downloads_and_copies_tensors(self, mock_manifest, mock_snapshot):
rows = [
{
"model": "org/m",
"run_path": "org__m/2025/01/01/run-abc",
"push_index": 1,
}
]
mock_manifest.return_value = (rows, "")
with tempfile.TemporaryDirectory() as snap_dir:
tensors = Path(snap_dir) / "org__m/2025/01/01/run-abc/tensors"
tensors.mkdir(parents=True)
(tensors / "layer0.pt").write_bytes(b"\x00")
mock_snapshot.return_value = snap_dir
with tempfile.TemporaryDirectory() as target:
result = hfs.fetch_latest_baseline(
config=_make_config(),
model="org/m",
target_tensors_dir=Path(target),
)
self.assertEqual(result, "org__m/2025/01/01/run-abc")
@patch.object(hfs, "_read_manifest")
def test_returns_none_when_no_runs(self, mock_manifest):
mock_manifest.return_value = ([], "")
with tempfile.TemporaryDirectory() as target:
result = hfs.fetch_latest_baseline(
config=_make_config(),
model="org/m",
target_tensors_dir=Path(target),
)
self.assertIsNone(result)
@patch("sglang.test.precision_baseline_store.snapshot_download")
@patch.object(hfs, "_read_manifest")
def test_passes_capture_signature(self, mock_manifest, mock_snapshot):
rows = [
{
"model": "org/m",
"run_path": "run_new",
"capture_signature": "sig2",
"push_index": 2,
},
{
"model": "org/m",
"run_path": "run_old",
"capture_signature": "sig1",
"push_index": 1,
},
]
mock_manifest.return_value = (rows, "")
with tempfile.TemporaryDirectory() as snap_dir:
tensors = Path(snap_dir) / "run_new/tensors"
tensors.mkdir(parents=True)
(tensors / "layer0.pt").write_bytes(b"\x00")
mock_snapshot.return_value = snap_dir
with tempfile.TemporaryDirectory() as target:
result = hfs.fetch_latest_baseline(
config=_make_config(),
model="org/m",
target_tensors_dir=Path(target),
capture_signature="sig2",
)
self.assertEqual(result, "run_new")
class TestPushRun(CustomTestCase):
"""push_run deletes its temp manifest file in a finally block, so tests
that inspect the manifest content must capture it via a side_effect on
the mock upload_file *before* push_run cleans up."""
@staticmethod
def _make_push_mocks(mock_manifest, mock_api_cls):
mock_manifest.return_value = ([], "")
mock_api = MagicMock()
mock_api_cls.return_value = mock_api
# Capture manifest text before push_run's finally block deletes it.
captured = []
mock_api.upload_file.side_effect = lambda *a, **kw: captured.append(
Path(kw["path_or_fileobj"]).read_text()
)
return mock_api, captured
@patch("sglang.test.precision_baseline_store.HfApi")
@patch.object(hfs, "_read_manifest")
def test_uploads_tensors_and_manifest(self, mock_manifest, mock_api_cls):
mock_api, captured = self._make_push_mocks(mock_manifest, mock_api_cls)
with tempfile.TemporaryDirectory() as tensor_dir:
(Path(tensor_dir) / "layer0.pt").write_bytes(b"\x01")
meta = {"tp_size": 8, "capture_signature": "abc", "hardware": "H200"}
run_path = hfs.push_run(
config=_make_config(),
model="org/m",
sglang_commit="abc1234567",
today_tensors_dir=Path(tensor_dir),
meta=meta,
)
mock_api.upload_folder.assert_called_once()
mock_api.upload_file.assert_called_once()
row = json.loads(captured[0].strip().splitlines()[-1])
self.assertEqual(row["model"], "org/m")
self.assertEqual(row["capture_signature"], "abc")
self.assertEqual(row["tp_size"], 8)
self.assertTrue(run_path.startswith("org__m/"))
@patch("sglang.test.precision_baseline_store.HfApi")
@patch.object(hfs, "_read_manifest")
def test_skips_existing_tensors_unless_force(self, mock_manifest, mock_api_cls):
# The run_path must match what push_run generates: model/date/sha7.
# _today_path() returns today's date, so build the path accordingly.
today_date, today_date_path = hfs._today_path()
existing_run_path = f"org__m/{today_date_path}/run-abc1234"
existing_row = {
"model": "org/m",
"run_path": existing_run_path,
"date": today_date,
"push_index": 1,
}
mock_manifest.return_value = ([existing_row], json.dumps(existing_row) + "\n")
mock_api = MagicMock()
mock_api_cls.return_value = mock_api
# Capture pt file count before push_run cleans up the temp staging dir.
captured_pt_count = []
mock_api.upload_folder.side_effect = lambda *a, **kw: captured_pt_count.append(
len(list(Path(kw["folder_path"]).rglob("*.pt")))
)
with tempfile.TemporaryDirectory() as tensor_dir:
(Path(tensor_dir) / "layer0.pt").write_bytes(b"\x01")
hfs.push_run(
config=_make_config(),
model="org/m",
sglang_commit="abc1234567",
today_tensors_dir=Path(tensor_dir),
meta={"tp_size": 8},
)
self.assertEqual(captured_pt_count[0], 0)
@patch("sglang.test.precision_baseline_store.HfApi")
@patch.object(hfs, "_read_manifest")
def test_force_re_uploads(self, mock_manifest, mock_api_cls):
# Use today's date so the run_path matches what push_run generates.
today_date, today_date_path = hfs._today_path()
existing_run_path = f"org__m/{today_date_path}/run-abc1234"
existing_row = {
"model": "org/m",
"run_path": existing_run_path,
"date": today_date,
"push_index": 1,
}
mock_manifest.return_value = ([existing_row], json.dumps(existing_row) + "\n")
mock_api = MagicMock()
mock_api_cls.return_value = mock_api
# Capture pt file count before push_run cleans up the temp staging dir.
captured_pt_count = []
mock_api.upload_folder.side_effect = lambda *a, **kw: captured_pt_count.append(
len(list(Path(kw["folder_path"]).rglob("*.pt")))
)
with tempfile.TemporaryDirectory() as tensor_dir:
(Path(tensor_dir) / "layer0.pt").write_bytes(b"\x01")
hfs.push_run(
config=_make_config(),
model="org/m",
sglang_commit="abc1234567",
today_tensors_dir=Path(tensor_dir),
meta={"tp_size": 8},
force=True,
)
self.assertGreater(captured_pt_count[0], 0)
@patch("sglang.test.precision_baseline_store.HfApi")
@patch.object(hfs, "_read_manifest")
def test_manifest_row_promotes_keys(self, mock_manifest, mock_api_cls):
_mock_api, captured = self._make_push_mocks(mock_manifest, mock_api_cls)
with tempfile.TemporaryDirectory() as tensor_dir:
(Path(tensor_dir) / "layer0.pt").write_bytes(b"\x01")
meta = {
"tp_size": 4,
"hardware": "H100",
"capture_signature": "sig1",
"num_layers_compared": 10,
"num_layers_passed": 10,
"num_layers_failed": 0,
"max_rel_diff": 0.001,
"ci_run_id": "12345",
"extra_key_not_promoted": True,
}
hfs.push_run(
config=_make_config(),
model="org/m",
sglang_commit="abc1234567",
today_tensors_dir=Path(tensor_dir),
meta=meta,
)
row = json.loads(captured[0].strip().splitlines()[-1])
for key in hfs._MANIFEST_PROMOTE_KEYS:
if key in meta:
self.assertEqual(
row.get(key),
meta[key],
f"manifest missing promoted key: {key}",
)
self.assertNotIn("extra_key_not_promoted", row)
@patch("sglang.test.precision_baseline_store.HfApi")
@patch.object(hfs, "_read_manifest")
def test_includes_comparator_report(self, mock_manifest, mock_api_cls):
mock_manifest.return_value = ([], "")
mock_api = MagicMock()
mock_api_cls.return_value = mock_api
# Capture file existence before push_run cleans up the temp staging dir.
captured_files = []
mock_api.upload_folder.side_effect = lambda *a, **kw: captured_files.append(
list(Path(kw["folder_path"]).iterdir())
)
with tempfile.TemporaryDirectory() as tensor_dir:
(Path(tensor_dir) / "layer0.pt").write_bytes(b"\x01")
report_path = Path(tensor_dir) / "report.jsonl"
report_path.write_text('{"type":"comparison_tensor"}\n')
hfs.push_run(
config=_make_config(),
model="org/m",
sglang_commit="abc1234567",
today_tensors_dir=Path(tensor_dir),
meta={"tp_size": 8},
comparator_report=report_path,
)
staged_names = [f.name for f in captured_files[0]]
self.assertIn("comparator_report.jsonl", staged_names)
class TestPruneOldRuns(CustomTestCase):
@patch.object(hfs, "_read_manifest")
def test_keeps_recent_runs(self, mock_manifest):
today = datetime.now(timezone.utc).strftime("%Y-%m-%d")
rows = [
{"model": "org/m", "run_path": "recent", "date": today},
]
mock_manifest.return_value = (rows, json.dumps(rows[0]) + "\n")
result = hfs.prune_old_runs(config=_make_config(), keep_days=30)
self.assertIn("recent", result["kept"])
self.assertEqual(result["pruned"], [])
@patch.object(hfs, "_read_manifest")
def test_archives_one_per_week(self, mock_manifest):
rows = [
{"model": "org/m", "run_path": "old1", "date": "2020-01-06"},
{"model": "org/m", "run_path": "old2", "date": "2020-01-07"},
{"model": "org/m", "run_path": "old3", "date": "2020-01-08"},
]
mock_manifest.return_value = (rows, "")
result = hfs.prune_old_runs(
config=_make_config(), keep_days=0, weekly_archive=True, dry_run=True
)
self.assertEqual(len(result["kept"]), 1)
self.assertEqual(result["kept"][0], "old3")
self.assertEqual(len(result["pruned"]), 2)
@patch.object(hfs, "_read_manifest")
def test_prune_without_archive(self, mock_manifest):
rows = [
{"model": "org/m", "run_path": "old1", "date": "2020-01-06"},
{"model": "org/m", "run_path": "old2", "date": "2020-01-07"},
]
mock_manifest.return_value = (rows, "")
result = hfs.prune_old_runs(
config=_make_config(), keep_days=0, weekly_archive=False, dry_run=True
)
self.assertEqual(result["kept"], [])
self.assertEqual(len(result["pruned"]), 2)
@patch("sglang.test.precision_baseline_store.HfApi")
@patch.object(hfs, "_read_manifest")
def test_dry_run_does_not_delete(self, mock_manifest, mock_api_cls):
rows = [
{"model": "org/m", "run_path": "old1", "date": "2020-01-06"},
]
mock_manifest.return_value = (rows, "")
mock_api_cls.return_value = MagicMock()
hfs.prune_old_runs(config=_make_config(), keep_days=0, dry_run=True)
mock_api_cls.return_value.upload_file.assert_not_called()
mock_api_cls.return_value.delete_folder.assert_not_called()
@patch("sglang.test.precision_baseline_store.HfApi")
@patch.object(hfs, "_read_manifest")
def test_live_mode_deletes(self, mock_manifest, mock_api_cls):
rows = [
{"model": "org/m", "run_path": "old1", "date": "2020-01-06"},
{"model": "org/m", "run_path": "old2", "date": "2020-01-07"},
]
mock_manifest.return_value = (rows, "")
mock_api = MagicMock()
mock_api_cls.return_value = mock_api
result = hfs.prune_old_runs(
config=_make_config(), keep_days=0, weekly_archive=True, dry_run=False
)
self.assertEqual(len(result["kept"]), 1)
self.assertEqual(len(result["pruned"]), 1)
mock_api.upload_file.assert_called_once()
mock_api.delete_folder.assert_called_once()
@patch.object(hfs, "_read_manifest")
def test_filters_by_model(self, mock_manifest):
rows = [
{"model": "org/m1", "run_path": "m1_old", "date": "2020-01-06"},
{"model": "org/m2", "run_path": "m2_old", "date": "2020-01-06"},
]
mock_manifest.return_value = (rows, "")
result = hfs.prune_old_runs(
config=_make_config(),
model="org/m1",
keep_days=0,
weekly_archive=False,
dry_run=True,
)
self.assertIn("m2_old", result["kept"])
self.assertIn("m1_old", result["pruned"])
class TestWithRetries(CustomTestCase):
@patch("sglang.test.precision_baseline_store.time")
def test_succeeds_on_first_attempt(self, mock_time):
result = hfs._with_retries(lambda: 42, what="test")
self.assertEqual(result, 42)
mock_time.sleep.assert_not_called()
@patch("sglang.test.precision_baseline_store.time")
def test_retries_on_429(self, mock_time):
from huggingface_hub.errors import HfHubHTTPError
resp_429 = MagicMock()
resp_429.status_code = 429
exc_429 = HfHubHTTPError("rate limited", response=resp_429)
mock_op = MagicMock(side_effect=[exc_429, "ok"])
result = hfs._with_retries(mock_op, what="test", base_delay=0.01)
self.assertEqual(result, "ok")
mock_time.sleep.assert_called_once()
@patch("sglang.test.precision_baseline_store.time")
def test_retries_on_5xx(self, mock_time):
from huggingface_hub.errors import HfHubHTTPError
resp_500 = MagicMock()
resp_500.status_code = 500
exc_500 = HfHubHTTPError("server error", response=resp_500)
mock_op = MagicMock(side_effect=[exc_500, "ok"])
result = hfs._with_retries(mock_op, what="test", base_delay=0.01)
self.assertEqual(result, "ok")
mock_time.sleep.assert_called_once()
@patch("sglang.test.precision_baseline_store.time")
def test_raises_on_auth_error(self, mock_time):
from huggingface_hub.errors import HfHubHTTPError
resp_401 = MagicMock()
resp_401.status_code = 401
exc_401 = HfHubHTTPError("unauthorized", response=resp_401)
mock_op = MagicMock(side_effect=exc_401)
with self.assertRaises(HfHubHTTPError):
hfs._with_retries(mock_op, what="test")
mock_time.sleep.assert_not_called()
@patch("sglang.test.precision_baseline_store.time")
def test_raises_after_max_attempts(self, mock_time):
from huggingface_hub.errors import HfHubHTTPError
resp_500 = MagicMock()
resp_500.status_code = 500
exc = HfHubHTTPError("server error", response=resp_500)
mock_op = MagicMock(side_effect=exc)
with self.assertRaises(HfHubHTTPError):
hfs._with_retries(mock_op, what="test", attempts=2, base_delay=0.001)
self.assertEqual(mock_time.sleep.call_count, 1)
if __name__ == "__main__":
unittest.main()
test/run_suite.py
+2
View File
@@ -119,6 +119,8 @@ NIGHTLY_SUITES = {
"nightly-perf-vlm-2-gpu", "nightly-perf-vlm-2-gpu",
# GB300 (4x B200 NVL4) nightly suite # GB300 (4x B200 NVL4) nightly suite
"nightly-4-gpu-gb300", "nightly-4-gpu-gb300",
# Nightly precision regression (per-layer hidden state comparison)
"nightly-precision-8-gpu-h200",
], ],
HWBackend.AMD: [ HWBackend.AMD: [
"nightly-amd", "nightly-amd",