minke.yu/sglang
1
0
Fork 0
Code Issues Pull Requests Actions 12 Packages Projects Releases Wiki Activity

[CI] Key scheduled CUDA suites by runner_config instead of hand-written jobs (#34186)

Browse Source
This commit is contained in:
Liangsheng Yin
2026-08-09 16:44:53 -07:00
committed by GitHub
parent 4a5d7d3c67
commit 7c90840bad
94 changed files with 531 additions and 869 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.claude/skills/add-jit-kernel/SKILL.md
+6 -6
View File
@@ -489,7 +489,7 @@ if torch.cuda.get_device_capability()[0] < 9:
JIT kernel correctness tests and benchmarks live under `test/registered/kernels/ops/<group>/` and `test/registered/kernels/benchmark/<group>/`, mirroring the wrapper's group under `python/sglang/kernels/ops/` (NOT inside the `sglang` package -- a `register_*_ci(...)` call anywhere under `python/sglang/` is rejected by the `check-no-registered-tests-in-package` pre-commit hook). Only their test-only helpers (e.g. `benchmark/marker.py`) stay alongside the kernel source under `python/sglang/kernels/jit/` and are imported by absolute path. **CI does not run `pytest` in those directories directly.** The unified runner `test/run_suite.py` discovers every `test_*.py` and `bench_*.py` under `test/registered/`, collects `register_*_ci(...)` calls by **statically parsing each file's AST**, and executes the selected suite. Every test file must register at least one CUDA entry or the collector fails its sanity check. JIT kernel correctness tests and benchmarks live under `test/registered/kernels/ops/<group>/` and `test/registered/kernels/benchmark/<group>/`, mirroring the wrapper's group under `python/sglang/kernels/ops/` (NOT inside the `sglang` package -- a `register_*_ci(...)` call anywhere under `python/sglang/` is rejected by the `check-no-registered-tests-in-package` pre-commit hook). Only their test-only helpers (e.g. `benchmark/marker.py`) stay alongside the kernel source under `python/sglang/kernels/jit/` and are imported by absolute path. **CI does not run `pytest` in those directories directly.** The unified runner `test/run_suite.py` discovers every `test_*.py` and `bench_*.py` under `test/registered/`, collects `register_*_ci(...)` calls by **statically parsing each file's AST**, and executes the selected suite. Every test file must register at least one CUDA entry or the collector fails its sanity check.
- **PR / per-commit CUDA suites** (see `test/run_suite.py` → `PER_COMMIT_SUITES`): JIT unit tests use `base-b-kernel-unit-test-1-gpu-large` on H100 and `base-b-kernel-unit-test-4-gpu-b200` on B200/SM100 paths (see `.github/workflows/pr-test-jit-kernel.yml`). Multi-GPU JIT tests use `base-b-kernel-unit-test-8-gpu-h200`. - **PR / per-commit CUDA suites** (see `test/run_suite.py` → `PER_COMMIT_SUITES`): JIT unit tests use `base-b-kernel-unit-test-1-gpu-large` on H100 and `base-b-kernel-unit-test-4-gpu-b200` on B200/SM100 paths (see `.github/workflows/pr-test-jit-kernel.yml`). Multi-GPU JIT tests use `base-b-kernel-unit-test-8-gpu-h200`.
- **Nightly kernel suite**: `nightly-kernel-1-gpu` with `--nightly` — typically used with `SGLANG_JIT_KERNEL_RUN_FULL_TESTS=1` in CI for expanded parameter grids (see `python/sglang/kernels/jit/utils/common.py` → `should_run_full_tests` / `get_ci_test_range`). Wired in `.github/workflows/nightly-test-nvidia.yml` (e.g. `python3 run_suite.py --hw cuda --suite nightly-kernel-1-gpu --nightly --continue-on-error`). - **Nightly kernel suite**: register with `stage="nightly"` plus the `runner_config` of the machine it needs (e.g. `1-gpu-large`), giving the `nightly-test-1-gpu-large` suite. `.github/workflows/nightly-test-nvidia.yml` sets `SGLANG_JIT_KERNEL_RUN_FULL_TESTS=1` for the whole nightly run, so the expanded parameter grids apply automatically (see `python/sglang/kernels/jit/utils/common.py` → `should_run_full_tests` / `get_ci_test_range`). There is no separate kernel-only nightly job: every nightly test on one machine type shares that machine's suite.
Registration pattern (module level, **literal** `est_time`, `stage`, and `runner_config` values — required for AST parsing): Registration pattern (module level, **literal** `est_time`, `stage`, and `runner_config` values — required for AST parsing):
@@ -499,12 +499,12 @@ from sglang.test.ci.ci_register import register_cuda_ci
register_cuda_ci(est_time=30, stage="base-b-kernel-unit", runner_config="1-gpu-large") register_cuda_ci(est_time=30, stage="base-b-kernel-unit", runner_config="1-gpu-large")
# Optional B200/SM100 registration for tests that cover Blackwell-specific code paths # Optional B200/SM100 registration for tests that cover Blackwell-specific code paths
# register_cuda_ci(est_time=30, stage="base-b-kernel-unit", runner_config="4-gpu-b200") # register_cuda_ci(est_time=30, stage="base-b-kernel-unit", runner_config="4-gpu-b200")
# Optional second registration: same file also listed under the nightly kernel suite # Optional second registration: same file also runs nightly, same form,
# (nightly suites use the legacy single-string suite=, not stage/runner_config) # stage is just "nightly" there (and no `nightly=True`)
# register_cuda_ci(est_time=120, suite="nightly-kernel-1-gpu", nightly=True) # register_cuda_ci(est_time=120, stage="nightly", runner_config="1-gpu-large")
``` ```
CI generates the suite name as `{stage}-test-{runner_config}`, so `stage="base-b-kernel-unit", runner_config="1-gpu-large"` becomes the `base-b-kernel-unit-test-1-gpu-large` suite you pass to `run_suite.py` below — don't put the `-test-` infix in `register_cuda_ci`. The single-string `suite=` form is only for nightly/stress/weekly suites. CI generates the suite name as `{stage}-test-{runner_config}`, so `stage="base-b-kernel-unit", runner_config="1-gpu-large"` becomes the `base-b-kernel-unit-test-1-gpu-large` suite you pass to `run_suite.py` below — don't put the `-test-` infix in `register_cuda_ci`. Nightly uses the same shape with `stage="nightly"`; the single-string `suite=` form is left only for `stress` and non-CUDA pools.
Keep `est_time`, `stage`, `runner_config`, and `suite` as literal values. `run_suite.py` collects them from the file AST, so computed values and helper wrappers can break CI discovery. Keep `est_time`, `stage`, `runner_config`, and `suite` as literal values. `run_suite.py` collects them from the file AST, so computed values and helper wrappers can break CI discovery.
@@ -661,7 +661,7 @@ cd test && python3 run_suite.py --hw cuda --suite base-b-kernel-benchmark-test-1
## Troubleshooting ## Troubleshooting
- **`No CI registry found in ...` from `run_suite.py`**: add a module-level `register_cuda_ci(...)` with literal `est_time`, `stage`, and `runner_config` (and optional `nightly=True`); starred args and non-literal values break AST collection - **`No CI registry found in ...` from `run_suite.py`**: add a module-level `register_cuda_ci(...)` with literal `est_time`, `stage`, and `runner_config`; starred args and non-literal values break AST collection
- **JIT compilation fails**: ensure the `.cuh` file is under `python/sglang/kernels/jit/csrc/`; reduce template argument combinations - **JIT compilation fails**: ensure the `.cuh` file is under `python/sglang/kernels/jit/csrc/`; reduce template argument combinations
- **CUDA crash / illegal memory access**: `CUDA_LAUNCH_BLOCKING=1`; `compute-sanitizer --tool memcheck python ...` - **CUDA crash / illegal memory access**: `CUDA_LAUNCH_BLOCKING=1`; `compute-sanitizer --tool memcheck python ...`
- **Unstable benchmark results**: `marker.do_bench` uses CUDA-graph-based timing by default; set `use_cuda_graph=False` only if the kernel can't be captured. `graph_clone_args` defaults to `"all"`; if you narrow it, it must still cover every *read* tensor — reusing a single buffer keeps it L2-hot and skews results. Keep *write* tensors in it too: they are what sets the rotation count, and a shared output buffer stays L2-hot the same way. - **Unstable benchmark results**: `marker.do_bench` uses CUDA-graph-based timing by default; set `use_cuda_graph=False` only if the kernel can't be captured. `graph_clone_args` defaults to `"all"`; if you narrow it, it must still cover every *read* tensor — reusing a single buffer keeps it L2-hot and skews results. Keep *write* tensors in it too: they are what sets the rotation count, and a shared output buffer stays L2-hot the same way.
.claude/skills/write-sglang-test/SKILL.md
+15 -14
View File
@@ -56,7 +56,7 @@ A per-commit suite name is **generated** from registration metadata as `{stage}-
- **`runner_config`** — a runner-pool key from `scripts/ci/runner_configs.yml`, which maps it to the physical runner label (so `1-gpu-large` runs on `1-gpu-h100`). AMD/NPU use their own keys (e.g. `amd`). - **`runner_config`** — a runner-pool key from `scripts/ci/runner_configs.yml`, which maps it to the physical runner label (so `1-gpu-large` runs on `1-gpu-h100`). AMD/NPU use their own keys (e.g. `amd`).
- **Suite** — `register_cuda_ci(stage="base-b", runner_config="1-gpu-small")` → `base-b-test-1-gpu-small`, the name you pass to `run_suite.py --suite`. The `-test-` is just the connector; never put it in `register_*_ci`. - **Suite** — `register_cuda_ci(stage="base-b", runner_config="1-gpu-small")` → `base-b-test-1-gpu-small`, the name you pass to `run_suite.py --suite`. The `-test-` is just the connector; never put it in `register_*_ci`.
> Legacy single-string `suite=` is only for suites that don't fit that shape — nightly/stress/weekly and some AMD/CPU/NPU pools (e.g. `suite="nightly-kernel-1-gpu", nightly=True`). Per-commit tests always use `stage=` + `runner_config=`. > CUDA nightly uses the same shape with `stage="nightly"` (e.g. `stage="nightly", runner_config="1-gpu-large"` → `nightly-test-1-gpu-large`) and **no** `nightly=True` — the stage name carries the cadence, and setting the flag makes the test silently never run. Legacy single-string `suite=` is left only for `stress` and some AMD/CPU/NPU pools.
### All CI Suites ### All CI Suites
@@ -113,13 +113,14 @@ A per-commit suite name is **generated** from registration metadata as `{stage}-
#### Nightly #### Nightly
Nightly suites are listed in `NIGHTLY_SUITES` in [`test/run_suite.py`](../../../test/run_suite.py). They run via `nightly-test-nvidia.yml`, `nightly-test-amd.yml`, and `nightly-test-npu.yml`, not `pr-test.yml`. Examples: Nightly suites are listed in `NIGHTLY_SUITES` in [`test/run_suite.py`](../../../test/run_suite.py). They run via `nightly-test-nvidia.yml`, `nightly-test-amd.yml`, and `nightly-test-npu.yml`, not `pr-test.yml`.
- `nightly-1-gpu` (CUDA) CUDA nightly suites are named `nightly-test-{runner_config}` — one per machine type, holding everything that runs nightly on it. There is no per-purpose split (kernel / eval / perf / precision all share their machine's suite); `auto_partition` splits the work. Examples:
- `nightly-kernel-1-gpu` (CUDA, JIT kernel full grids)
- `nightly-kernel-8-gpu-h200` (CUDA, multi-GPU JIT kernel nightly) - `nightly-test-1-gpu-large` (CUDA)
- `nightly-8-gpu-h200` (CUDA) - `nightly-test-2-gpu-large` (CUDA)
- `nightly-eval-vlm-2-gpu` (CUDA) - `nightly-test-8-gpu-h200` (CUDA)
- `nightly-test-4-gpu-gb300` (CUDA)
- `nightly-amd` (AMD) - `nightly-amd` (AMD)
- `nightly-amd-8-gpu-mi35x` (AMD) - `nightly-amd-8-gpu-mi35x` (AMD)
- `nightly-1-npu-a3` (NPU) - `nightly-1-npu-a3` (NPU)
@@ -330,8 +331,8 @@ register_cuda_ci(est_time=80, suite="base-b-test-1-gpu-small")
# Per-commit test (large 1-gpu, runs on H100) # Per-commit test (large 1-gpu, runs on H100)
register_cuda_ci(est_time=120, suite="base-b-test-1-gpu-large") register_cuda_ci(est_time=120, suite="base-b-test-1-gpu-large")
# Nightly-only test # Nightly-only test (same shape as per-commit, stage is just "nightly")
register_cuda_ci(est_time=200, suite="nightly-1-gpu", nightly=True) register_cuda_ci(est_time=200, stage="nightly", runner_config="1-gpu-large")
# Multi-backend test (only when testing backend-specific code paths) # Multi-backend test (only when testing backend-specific code paths)
register_cuda_ci(est_time=80, suite="base-a-test-1-gpu-small") register_cuda_ci(est_time=80, suite="base-a-test-1-gpu-small")
@@ -345,7 +346,7 @@ register_cuda_ci(est_time=80, suite="base-b-test-1-gpu-small", disabled="flaky -
Parameters: Parameters:
- `est_time`: estimated runtime in seconds (used for CI partitioning) - `est_time`: estimated runtime in seconds (used for CI partitioning)
- `suite`: which CI suite to run in (see suite tables above) - `suite`: which CI suite to run in (see suite tables above)
- `nightly=True`: for nightly-only tests (default `False` = per-commit) - `nightly=True`: legacy cadence flag, for non-CUDA nightly suites only. CUDA nightly uses `stage="nightly"` and must leave this unset
- `disabled="reason"`: temporarily disable with explanation - `disabled="reason"`: temporarily disable with explanation
**Key principle**: Only add `register_amd_ci` / `register_npu_ci` when the test exercises backend-specific code paths. Common E2E tests just need `register_cuda_ci` — duplicating across backends wastes CI time. **Key principle**: Only add `register_amd_ci` / `register_npu_ci` when the test exercises backend-specific code paths. Common E2E tests just need `register_cuda_ci` — duplicating across backends wastes CI time.
@@ -365,12 +366,12 @@ register_cuda_ci(est_time=120, stage="base-b-kernel-unit", runner_config="8-gpu-
# Benchmarks in test/registered/jit/benchmark/ # Benchmarks in test/registered/jit/benchmark/
register_cuda_ci(est_time=6, stage="base-b-kernel-benchmark", runner_config="1-gpu-large") register_cuda_ci(est_time=6, stage="base-b-kernel-benchmark", runner_config="1-gpu-large")
# Optional nightly registration — nightly suites use the legacy single-string suite= # Optional nightly registration — same form, stage is just "nightly"
register_cuda_ci(est_time=120, suite="nightly-kernel-1-gpu", nightly=True) register_cuda_ci(est_time=120, stage="nightly", runner_config="1-gpu-large")
register_cuda_ci(est_time=120, suite="nightly-kernel-8-gpu-h200", nightly=True) register_cuda_ci(est_time=120, stage="nightly", runner_config="8-gpu-h200")
``` ```
The `stage` + `runner_config` calls generate suites like `base-b-kernel-unit-test-1-gpu-large`; nightly keeps the legacy `suite=` string. Keep `est_time`, `stage`, `runner_config`, and `suite` as **literal values** — `run_suite.py` collects them by AST parsing. Every call generates a suite named `{stage}-test-{runner_config}`, e.g. `base-b-kernel-unit-test-1-gpu-large` and `nightly-test-1-gpu-large`. Keep `est_time`, `stage`, `runner_config`, and `suite` as **literal values** — `run_suite.py` collects them by AST parsing.
--- ---
.github/workflows/_pr-test-stage.yml
+63 -2
View File
@@ -63,6 +63,15 @@ on:
type: boolean type: boolean
default: false default: false
scheduled:
description: 'The caller is a nightly or weekly workflow, not a per-commit one. Switches on per-file timeouts derived from est_time, the full jit_kernel grids, metrics upload, and serial shards unless dispatched with full_parallel.'
type: boolean
default: false
job_timeout_minutes:
description: 'Job-level cap, covering install and warmup on top of run_timeout_minutes.'
type: string
default: '240'
# Mirror pr-test.yml top-level env. Reusable workflows do NOT inherit caller's # Mirror pr-test.yml top-level env. Reusable workflows do NOT inherit caller's
# workflow-level env across the workflow_call boundary, so anything pr-test.yml # workflow-level env across the workflow_call boundary, so anything pr-test.yml
# defines must be redeclared here for the called job to see the same context. # defines must be redeclared here for the called job to see the same context.
@@ -74,6 +83,15 @@ env:
SGLANG_ENABLE_ASYNC_ASSERT: ${{ startsWith(inputs.self_name, 'base-a') && 'false' || 'true' }} SGLANG_ENABLE_ASYNC_ASSERT: ${{ startsWith(inputs.self_name, 'base-a') && 'false' || 'true' }}
SGLANG_CUDA_COREDUMP: "1" SGLANG_CUDA_COREDUMP: "1"
SGLANG_JIT_DEEPGEMM_FAST_WARMUP: true SGLANG_JIT_DEEPGEMM_FAST_WARMUP: true
SGLANG_JIT_KERNEL_RUN_FULL_TESTS: ${{ inputs.scheduled && '1' || '0' }}
# is_h200_system() (test_utils.py) raises the server-launch ceiling from 600s
# to 3600s. Only the scheduled 8-gpu-h200 suites load models that need it;
# per-commit stages keep the short ceiling so a hung launch fails fast.
IS_H200: ${{ (inputs.scheduled && inputs.runner_config == '8-gpu-h200') && '1' || '0' }}
# Scheduled suites pull hundreds of GB of checkpoints; the hub's 10s etag
# default times out on a cold cache.
HF_HUB_DOWNLOAD_TIMEOUT: 300
HF_HUB_ETAG_TIMEOUT: 300
SKIP_PR_TEST_HEALTH_CHECK: ${{ (fromJson(inputs.caller_inputs).skip_pr_test_health_check || fromJson(inputs.caller_inputs).test_parallel_dispatch || fromJson(inputs.caller_inputs).run_all_tests) && 'true' || 'false' }} SKIP_PR_TEST_HEALTH_CHECK: ${{ (fromJson(inputs.caller_inputs).skip_pr_test_health_check || fromJson(inputs.caller_inputs).test_parallel_dispatch || fromJson(inputs.caller_inputs).run_all_tests) && 'true' || 'false' }}
PR_TEST_BYPASS_MAINTENANCE_ON_MAIN: ${{ github.ref == 'refs/heads/main' && 'true' || 'false' }} PR_TEST_BYPASS_MAINTENANCE_ON_MAIN: ${{ github.ref == 'refs/heads/main' && 'true' || 'false' }}
USE_VENV: false USE_VENV: false
@@ -90,10 +108,10 @@ jobs:
# $b200_runner (see runner_configs.py --map). rdma_devices is exported # $b200_runner (see runner_configs.py --map). rdma_devices is exported
# below in a setup step via $GITHUB_ENV. # below in a setup step via $GITHUB_ENV.
runs-on: ${{ fromJson(fromJson(inputs.check_changes).runs_on_map)[inputs.runner_config] }} runs-on: ${{ fromJson(fromJson(inputs.check_changes).runs_on_map)[inputs.runner_config] }}
timeout-minutes: 240 timeout-minutes: ${{ fromJson(inputs.job_timeout_minutes) }}
strategy: strategy:
fail-fast: false fail-fast: false
max-parallel: ${{ fromJson(inputs.partitions)[inputs.self_name].max_parallel }} max-parallel: ${{ (inputs.scheduled && !fromJson(inputs.caller_inputs).full_parallel) && 1 || fromJson(inputs.partitions)[inputs.self_name].max_parallel }}
matrix: matrix:
partition: ${{ fromJson(inputs.partitions)[inputs.self_name].arr }} partition: ${{ fromJson(inputs.partitions)[inputs.self_name].arr }}
steps: steps:
@@ -165,16 +183,39 @@ jobs:
curl --fail --silent --show-error --max-time 15 --retry 3 --retry-delay 2 \ curl --fail --silent --show-error --max-time 15 --retry 3 --retry-delay 2 \
"$URL" -o /tmp/partition-model.json "$URL" -o /tmp/partition-model.json
# Only test_nightly_precision_regression.py reads these; the others are
# inert wherever it does not run.
- name: Export precision baseline env
if: inputs.scheduled
env:
BASELINE_HF_TOKEN: ${{ secrets.HF_TOKEN_PRECISION_STORE }}
run: |
{
echo "SGLANG_PRECISION_BASELINE_DIR=/tmp/sglang_precision_baselines"
echo "SGLANG_PRECISION_HF_REPO=${{ vars.SGLANG_PRECISION_HF_REPO }}"
echo "SGLANG_PRECISION_HF_REVISION=${{ vars.SGLANG_PRECISION_HF_REVISION || 'main' }}"
echo "SGLANG_PRECISION_COMMIT=${{ github.sha }}"
echo "SGLANG_PRECISION_FORCE_UPDATE=${{ fromJson(inputs.caller_inputs).force_baseline_update && '1' || '0' }}"
echo "SGLANG_PRECISION_HF_TOKEN=${BASELINE_HF_TOKEN}"
} >> "$GITHUB_ENV"
- name: Run test - name: Run test
timeout-minutes: ${{ fromJson(inputs.run_timeout_minutes) }} timeout-minutes: ${{ fromJson(inputs.run_timeout_minutes) }}
env: env:
CONTINUE_ON_ERROR_FLAG: ${{ fromJson(inputs.check_changes).continue_on_error == 'true' && '--continue-on-error' || '' }} CONTINUE_ON_ERROR_FLAG: ${{ fromJson(inputs.check_changes).continue_on_error == 'true' && '--continue-on-error' || '' }}
RUNNER_LABELS: ${{ fromJson(fromJson(inputs.check_changes).runs_on_map)[inputs.runner_config] }}
GPU_CONFIG: ${{ inputs.runner_config }}
# Read only by test_runai_model_loader.py (nightly 1-gpu); inert elsewhere.
# Left unconditional because an `env:` key cannot be omitted, and an empty
# value would reach the runai streamer as a malformed limit.
RUNAI_STREAMER_MEMORY_LIMIT: 0
run: | run: |
cd test cd test
python3 run_suite.py --hw cuda --suite ${{ inputs.self_name }} \ python3 run_suite.py --hw cuda --suite ${{ inputs.self_name }} \
--auto-partition-id ${{ matrix.partition }} \ --auto-partition-id ${{ matrix.partition }} \
--auto-partition-size ${{ fromJson(inputs.partitions)[inputs.self_name].size }} \ --auto-partition-size ${{ fromJson(inputs.partitions)[inputs.self_name].size }} \
--partition-model-file /tmp/partition-model.json \ --partition-model-file /tmp/partition-model.json \
${{ inputs.scheduled && '--timeout-from-est-time' || '' }} \
${{ inputs.timeout_per_file && format('--timeout-per-file {0}', inputs.timeout_per_file) || '' }} \ ${{ inputs.timeout_per_file && format('--timeout-per-file {0}', inputs.timeout_per_file) || '' }} \
$CONTINUE_ON_ERROR_FLAG $CONTINUE_ON_ERROR_FLAG
@@ -183,6 +224,26 @@ jobs:
timeout-minutes: 10 timeout-minutes: 10
run: python3 -m pytest -q ${{ inputs.extra_pytest_path }} run: python3 -m pytest -q ${{ inputs.extra_pytest_path }}
- name: Collect performance metrics
if: ${{ always() && inputs.scheduled }}
run: |
python3 scripts/ci/utils/save_metrics.py \
--gpu-config ${{ inputs.runner_config }} \
--partition ${{ matrix.partition }} \
--run-id ${{ github.run_id }} \
--output test/metrics-${{ inputs.runner_config }}-partition-${{ matrix.partition }}.json \
--search-dir test/performance_results_8_gpu \
--search-dir test
- name: Upload partition metrics
if: ${{ always() && inputs.scheduled }}
uses: actions/upload-artifact@v4
with:
name: metrics-${{ inputs.runner_config }}-partition-${{ matrix.partition }}
path: test/metrics-${{ inputs.runner_config }}-partition-${{ matrix.partition }}.json
retention-days: 5
if-no-files-found: ignore
- uses: ./.github/actions/upload-cuda-coredumps - uses: ./.github/actions/upload-cuda-coredumps
if: failure() if: failure()
with: with:
.github/workflows/nightly-test-nvidia.yml
+179 -549
View File
@@ -1,3 +1,10 @@
# Nightly CUDA tests. One job per runner_config, running that machine's
# `nightly-test-{runner_config}` suite -- a test reaches a machine by declaring
# `runner_config=` on register_cuda_ci, never by editing this file.
#
# Jobs go through the same _pr-test-stage.yml as the per-commit suites; what a
# scheduled run needs on top is its `scheduled` input, so runner resolution,
# rust-ext reuse and shard sizing are not reimplemented here.
name: Nightly Test (Nvidia) name: Nightly Test (Nvidia)
on: on:
@@ -5,32 +12,28 @@ on:
- cron: '0 14 */2 * *' - cron: '0 14 */2 * *'
workflow_dispatch: workflow_dispatch:
inputs: inputs:
job_filter: runner_filter:
description: 'Select which job to run (leave empty or "all" to run all jobs)' description: 'Select which runner_config to run (leave empty or "all" to run all)'
required: false required: false
type: choice type: choice
default: 'all' default: 'all'
options: options:
- 'all' - 'all'
- 'nightly-test-general-1-gpu-h100' - '1-gpu-large'
- 'nightly-test-general-4-gpu-h100' - '2-gpu-large'
- 'nightly-test-general-8-gpu-h200' - '4-gpu-h100'
- 'nightly-test-general-8-gpu-h20' - '4-gpu-b200'
- 'nightly-test-general-8-gpu-b200' - '4-gpu-gb300'
- 'nightly-test-text-accuracy-2-gpu-h100' - '8-gpu-h200'
- 'nightly-test-text-perf-2-gpu-h100' - '8-gpu-b200'
- 'nightly-test-vlm-accuracy-2-gpu-h100' - 'diffusion'
- 'nightly-test-vlm-perf-2-gpu-h100' full_parallel:
- 'nightly-test-perf-4-gpu-b200' description: 'Run all shards of a job at once (faster, but competes with per-commit CI for machines). Off by default: one shard at a time.'
- 'nightly-test-perf-8-gpu-b200' required: false
- 'nightly-test-specialized-8-gpu-b200' type: boolean
- 'nightly-test-perf-4-gpu-gb300' default: false
- 'nightly-test-kernel-1-gpu-h100'
- 'nightly-test-diffusion'
- 'nightly-test-kernel-8-gpu-h200'
- 'nightly-test-precision-8-gpu-h200'
force_baseline_update: force_baseline_update:
description: 'precision job only: refresh the rolling baseline instead of comparing (sets SGLANG_PRECISION_FORCE_UPDATE=1). Dispatch once after an intentional forward-path dtype/precision change stales the baseline; later scheduled runs compare against it and go green.' description: 'Refresh the precision rolling baseline instead of comparing. Dispatch once after an intentional forward-path precision change stales it; later runs compare against the new one.'
required: false required: false
type: boolean type: boolean
default: false default: false
@@ -41,499 +44,160 @@ on:
required: false required: false
type: string type: string
default: '' default: ''
job_filter: runner_filter:
description: 'Select which job to run (leave empty or "all" to run all jobs)' description: 'Select which runner_config to run (leave empty or "all" to run all)'
required: false required: false
type: string type: string
default: 'all' default: 'all'
full_parallel:
description: 'Run all shards of a job at once (faster, but competes with per-commit CI for machines). Off by default: one shard at a time.'
required: false
type: boolean
default: false
concurrency: concurrency:
group: nightly-test-nvidia-${{ inputs.ref || github.ref }} group: nightly-test-nvidia-${{ inputs.ref || github.ref }}
cancel-in-progress: ${{ github.event_name != 'workflow_call' }} cancel-in-progress: ${{ github.event_name != 'workflow_call' }}
permissions:
actions: write
contents: read
issues: read
pull-requests: read
jobs:
# run_all_tests skips the paths-filter, so main_package is 'true' and
# sgl_kernel stays empty: every test runs, and no job waits on a wheel this
# workflow never builds. pr_test_yml points back here so shard sizing reads
# this file's own run_timeout_minutes.
check-changes:
uses: ./.github/workflows/_pr-test-check-changes.yml
with:
git_ref: ${{ inputs.ref || '' }}
pr_test_yml: '.github/workflows/nightly-test-nvidia.yml'
run_all_tests: true
force_continue_on_error: true
secrets: inherit
nightly-1-gpu-large:
needs: check-changes
if: github.repository == 'sgl-project/sglang' && (inputs.runner_filter == '' || inputs.runner_filter == 'all' || inputs.runner_filter == '1-gpu-large')
uses: ./.github/workflows/_pr-test-stage.yml
with:
self_name: nightly-test-1-gpu-large
runner_config: 1-gpu-large
check_changes: ${{ toJson(needs.check-changes.outputs) }}
caller_inputs: ${{ toJson(inputs) }}
partitions: ${{ needs.check-changes.outputs.partitions }}
run_timeout_minutes: '120'
job_timeout_minutes: '180'
scheduled: true
secrets: inherit
nightly-2-gpu-large:
needs: check-changes
if: github.repository == 'sgl-project/sglang' && (inputs.runner_filter == '' || inputs.runner_filter == 'all' || inputs.runner_filter == '2-gpu-large')
uses: ./.github/workflows/_pr-test-stage.yml
with:
self_name: nightly-test-2-gpu-large
runner_config: 2-gpu-large
check_changes: ${{ toJson(needs.check-changes.outputs) }}
caller_inputs: ${{ toJson(inputs) }}
partitions: ${{ needs.check-changes.outputs.partitions }}
run_timeout_minutes: '240'
job_timeout_minutes: '300'
scheduled: true
secrets: inherit
nightly-4-gpu-h100:
needs: check-changes
if: github.repository == 'sgl-project/sglang' && (inputs.runner_filter == '' || inputs.runner_filter == 'all' || inputs.runner_filter == '4-gpu-h100')
uses: ./.github/workflows/_pr-test-stage.yml
with:
self_name: nightly-test-4-gpu-h100
runner_config: 4-gpu-h100
check_changes: ${{ toJson(needs.check-changes.outputs) }}
caller_inputs: ${{ toJson(inputs) }}
partitions: ${{ needs.check-changes.outputs.partitions }}
run_timeout_minutes: '120'
job_timeout_minutes: '180'
scheduled: true
secrets: inherit
nightly-4-gpu-b200:
needs: check-changes
if: github.repository == 'sgl-project/sglang' && (inputs.runner_filter == '' || inputs.runner_filter == 'all' || inputs.runner_filter == '4-gpu-b200')
uses: ./.github/workflows/_pr-test-stage.yml
with:
self_name: nightly-test-4-gpu-b200
runner_config: 4-gpu-b200
check_changes: ${{ toJson(needs.check-changes.outputs) }}
caller_inputs: ${{ toJson(inputs) }}
partitions: ${{ needs.check-changes.outputs.partitions }}
run_timeout_minutes: '150'
job_timeout_minutes: '210'
scheduled: true
secrets: inherit
nightly-4-gpu-gb300:
needs: check-changes
if: github.repository == 'sgl-project/sglang' && (inputs.runner_filter == '' || inputs.runner_filter == 'all' || inputs.runner_filter == '4-gpu-gb300')
uses: ./.github/workflows/_pr-test-stage.yml
with:
self_name: nightly-test-4-gpu-gb300
runner_config: 4-gpu-gb300
check_changes: ${{ toJson(needs.check-changes.outputs) }}
caller_inputs: ${{ toJson(inputs) }}
partitions: ${{ needs.check-changes.outputs.partitions }}
run_timeout_minutes: '360'
job_timeout_minutes: '420'
scheduled: true
# aarch64: the rust-ext cache key is x86_64-only, same reason base-c skips it.
skip_prebuilt_rust_ext: true
secrets: inherit
nightly-8-gpu-h200:
needs: check-changes
if: github.repository == 'sgl-project/sglang' && (inputs.runner_filter == '' || inputs.runner_filter == 'all' || inputs.runner_filter == '8-gpu-h200')
uses: ./.github/workflows/_pr-test-stage.yml
with:
self_name: nightly-test-8-gpu-h200
runner_config: 8-gpu-h200
check_changes: ${{ toJson(needs.check-changes.outputs) }}
caller_inputs: ${{ toJson(inputs) }}
partitions: ${{ needs.check-changes.outputs.partitions }}
run_timeout_minutes: '300'
job_timeout_minutes: '360'
scheduled: true
secrets: inherit
nightly-8-gpu-b200:
needs: check-changes
if: github.repository == 'sgl-project/sglang' && (inputs.runner_filter == '' || inputs.runner_filter == 'all' || inputs.runner_filter == '8-gpu-b200')
uses: ./.github/workflows/_pr-test-stage.yml
with:
self_name: nightly-test-8-gpu-b200
runner_config: 8-gpu-b200
check_changes: ${{ toJson(needs.check-changes.outputs) }}
caller_inputs: ${{ toJson(inputs) }}
partitions: ${{ needs.check-changes.outputs.partitions }}
run_timeout_minutes: '360'
job_timeout_minutes: '420'
scheduled: true
secrets: inherit
# Hand-written because it is not a registry suite: it drives run_comparison.py
# and publishes a dashboard rather than running test files.
nightly-test-diffusion:
if: github.repository == 'sgl-project/sglang' && (inputs.runner_filter == '' || inputs.runner_filter == 'all' || inputs.runner_filter == 'diffusion')
runs-on: 4-gpu-h100
timeout-minutes: 300
env: env:
SGLANG_IS_IN_CI: true SGLANG_IS_IN_CI: true
SGLANG_ENABLE_ASYNC_ASSERT: true SGLANG_ENABLE_ASYNC_ASSERT: true
SGLANG_CUDA_COREDUMP: "1" SGLANG_CUDA_COREDUMP: "1"
HF_HUB_DOWNLOAD_TIMEOUT: 300 HF_HUB_DOWNLOAD_TIMEOUT: 300
HF_HUB_ETAG_TIMEOUT: 300 HF_HUB_ETAG_TIMEOUT: 300
jobs:
# General tests - 1 GPU
nightly-test-general-1-gpu-h100:
if: github.repository == 'sgl-project/sglang' && (inputs.job_filter == '' || inputs.job_filter == 'all' || inputs.job_filter == 'nightly-test-general-1-gpu-h100')
runs-on: 1-gpu-h100
steps:
- name: Checkout code
uses: actions/checkout@v4
with:
ref: ${{ inputs.ref || github.ref }}
- uses: ./.github/actions/check-maintenance
- name: Install dependencies
run: |
bash scripts/ci/cuda/ci_install_dependency.sh
- name: Run test
timeout-minutes: 60
env:
RUNAI_STREAMER_MEMORY_LIMIT: 0
run: |
cd test
python3 run_suite.py --hw cuda --suite nightly-1-gpu --nightly --continue-on-error
- uses: ./.github/actions/upload-cuda-coredumps
if: failure()
# JIT kernel full unit tests (expanded parameter ranges via SGLANG_JIT_KERNEL_RUN_FULL_TESTS)
nightly-test-kernel-1-gpu-h100:
if: github.repository == 'sgl-project/sglang' && (inputs.job_filter == '' || inputs.job_filter == 'all' || inputs.job_filter == 'nightly-test-kernel-1-gpu-h100')
runs-on: 1-gpu-h100
timeout-minutes: 60
env:
# Full jit_kernel test grids (see sglang.kernels.jit.utils.should_run_full_tests)
SGLANG_JIT_KERNEL_RUN_FULL_TESTS: "1"
# Match pr-test-jit-kernel workflow for consistent JIT warmup behavior
SGLANG_JIT_DEEPGEMM_FAST_WARMUP: true
# Allow maintenance bypass on default branch (same semantics as PR JIT workflow)
PR_TEST_BYPASS_MAINTENANCE_ON_MAIN: ${{ github.ref == 'refs/heads/main' && 'true' || 'false' }}
steps:
- name: Checkout code
uses: actions/checkout@v4
with:
ref: ${{ inputs.ref || github.ref }}
- uses: ./.github/actions/check-maintenance
- name: Install dependencies
timeout-minutes: 20
run: |
bash scripts/ci/cuda/ci_install_dependency.sh
- name: Run jit kernel nightly suite
timeout-minutes: 60
run: |
cd test
python3 run_suite.py --hw cuda --suite nightly-kernel-1-gpu --nightly --continue-on-error
- uses: ./.github/actions/upload-cuda-coredumps
if: failure()
nightly-test-kernel-8-gpu-h200:
if: github.repository == 'sgl-project/sglang' && (inputs.job_filter == '' || inputs.job_filter == 'all' || inputs.job_filter == 'nightly-test-kernel-8-gpu-h200')
runs-on: 8-gpu-h200
timeout-minutes: 240
env:
SGLANG_JIT_KERNEL_RUN_FULL_TESTS: "1"
SGLANG_JIT_DEEPGEMM_FAST_WARMUP: true
PR_TEST_BYPASS_MAINTENANCE_ON_MAIN: ${{ github.ref == 'refs/heads/main' && 'true' || 'false' }}
steps:
- name: Checkout code
uses: actions/checkout@v4
with:
ref: ${{ inputs.ref || github.ref }}
- uses: ./.github/actions/check-maintenance
- name: Install dependencies
timeout-minutes: 20
run: |
bash scripts/ci/cuda/ci_install_dependency.sh
- name: Run multi-GPU jit kernel nightly suite
timeout-minutes: 90
run: |
cd test
# Full grids run ~7x the in-CI parametrizations per world size; the
# default 1200s per-file budget only fits the reduced PR sweep.
python3 run_suite.py --hw cuda --suite nightly-kernel-8-gpu-h200 --nightly --continue-on-error --timeout-per-file 3600
- uses: ./.github/actions/upload-cuda-coredumps
if: failure()
# General tests - 4 GPU H100
nightly-test-general-4-gpu-h100:
if: github.repository == 'sgl-project/sglang' && (inputs.job_filter == '' || inputs.job_filter == 'all' || inputs.job_filter == 'nightly-test-general-4-gpu-h100')
runs-on: 4-gpu-h100
steps:
- name: Checkout code
uses: actions/checkout@v4
with:
ref: ${{ inputs.ref || github.ref }}
- uses: ./.github/actions/check-maintenance
- name: Install dependencies
run: |
bash scripts/ci/cuda/ci_install_dependency.sh
- name: Run test
timeout-minutes: 60
run: |
cd test
python3 run_suite.py --hw cuda --suite nightly-4-gpu --nightly --continue-on-error
- uses: ./.github/actions/upload-cuda-coredumps
if: failure()
# General tests - 8 GPU H200
nightly-test-general-8-gpu-h200:
if: github.repository == 'sgl-project/sglang' && (inputs.job_filter == '' || inputs.job_filter == 'all' || inputs.job_filter == 'nightly-test-general-8-gpu-h200')
runs-on: 8-gpu-h200
strategy:
fail-fast: false
max-parallel: 2
matrix:
partition: [0, 1, 2, 3]
env:
RUNNER_LABELS: 8-gpu-h200
steps:
- name: Checkout code
uses: actions/checkout@v4
with:
ref: ${{ inputs.ref || github.ref }}
- uses: ./.github/actions/check-maintenance
- name: Install dependencies
run: |
bash scripts/ci/cuda/ci_install_dependency.sh
- name: Run common 8-GPU model tests
if: always()
timeout-minutes: 300
env:
GPU_CONFIG: "8-gpu-h200"
IS_H200: "1"
run: |
cd test
python3 run_suite.py --hw cuda --suite nightly-8-gpu-common --nightly --timeout-per-file=18000 --continue-on-error --auto-partition-id=${{ matrix.partition }} --auto-partition-size=4
- name: Run test
timeout-minutes: 30
env:
GPU_CONFIG: "8-gpu-h200"
run: |
cd test
python3 run_suite.py --hw cuda --suite nightly-8-gpu-h200 --nightly --continue-on-error --auto-partition-id=${{ matrix.partition }} --auto-partition-size=4
- name: Collect performance metrics
if: always()
run: |
python3 scripts/ci/utils/save_metrics.py \
--gpu-config 8-gpu-h200 \
--partition ${{ matrix.partition }} \
--run-id ${{ github.run_id }} \
--output test/metrics-8gpu-h200-partition-${{ matrix.partition }}.json \
--search-dir test/performance_results_8_gpu \
--search-dir test
- name: Upload partition metrics
if: always()
uses: actions/upload-artifact@v4
with:
name: metrics-8gpu-h200-partition-${{ matrix.partition }}
path: test/metrics-8gpu-h200-partition-${{ matrix.partition }}.json
retention-days: 5
if-no-files-found: ignore
- uses: ./.github/actions/upload-cuda-coredumps
if: failure()
with:
artifact-suffix: ${{ matrix.partition }}
# General tests - 8 GPU H20
nightly-test-general-8-gpu-h20:
if: github.repository == 'sgl-project/sglang' && (inputs.job_filter == '' || inputs.job_filter == 'all' || inputs.job_filter == 'nightly-test-general-8-gpu-h20')
runs-on: 8-gpu-h20
env:
SGLANG_CI_RDMA_ALL_DEVICES: "mlx5_1,mlx5_2,mlx5_3,mlx5_4"
steps:
- name: Checkout code
uses: actions/checkout@v4
with:
ref: ${{ inputs.ref || github.ref }}
- uses: ./.github/actions/check-maintenance
- name: Install dependencies
run: |
bash scripts/ci/cuda/ci_install_dependency.sh
- name: Run test
timeout-minutes: 30
env:
GPU_CONFIG: "8-gpu-h20"
run: |
cd test
python3 run_suite.py --hw cuda --suite nightly-8-gpu-h20 --nightly --continue-on-error
- uses: ./.github/actions/upload-cuda-coredumps
if: failure()
# General tests - 8 GPU B200
nightly-test-general-8-gpu-b200:
if: github.repository == 'sgl-project/sglang' && (inputs.job_filter == '' || inputs.job_filter == 'all' || inputs.job_filter == 'nightly-test-general-8-gpu-b200')
runs-on: 8-gpu-b200
strategy:
fail-fast: false
max-parallel: 2
matrix:
partition: [0, 1, 2, 3]
steps:
- name: Checkout code
uses: actions/checkout@v4
with:
ref: ${{ inputs.ref || github.ref }}
- uses: ./.github/actions/check-maintenance
- name: Install dependencies
run: |
bash scripts/ci/cuda/ci_install_dependency.sh
- name: Run common 8-GPU model tests
if: always()
timeout-minutes: 200
env:
GPU_CONFIG: "8-gpu-b200"
run: |
cd test
python3 run_suite.py --hw cuda --suite nightly-8-gpu-common --nightly --timeout-per-file=12000 --continue-on-error --auto-partition-id=${{ matrix.partition }} --auto-partition-size=4
- name: Collect performance metrics
if: always()
run: |
python3 scripts/ci/utils/save_metrics.py \
--gpu-config 8-gpu-b200 \
--partition ${{ matrix.partition }} \
--run-id ${{ github.run_id }} \
--output test/metrics-8gpu-b200-partition-${{ matrix.partition }}.json \
--search-dir test/performance_results_8_gpu \
--search-dir test
- name: Upload partition metrics
if: always()
uses: actions/upload-artifact@v4
with:
name: metrics-8gpu-b200-partition-${{ matrix.partition }}
path: test/metrics-8gpu-b200-partition-${{ matrix.partition }}.json
retention-days: 5
if-no-files-found: ignore
- uses: ./.github/actions/upload-cuda-coredumps
if: failure()
with:
artifact-suffix: ${{ matrix.partition }}
# Text model accuracy tests
nightly-test-text-accuracy-2-gpu-h100:
if: github.repository == 'sgl-project/sglang' && (inputs.job_filter == '' || inputs.job_filter == 'all' || inputs.job_filter == 'nightly-test-text-accuracy-2-gpu-h100')
runs-on: 2-gpu-h100
steps:
- name: Checkout code
uses: actions/checkout@v4
with:
ref: ${{ inputs.ref || github.ref }}
- uses: ./.github/actions/check-maintenance
- name: Install dependencies
run: |
bash scripts/ci/cuda/ci_install_dependency.sh
- name: Run eval test for text models
timeout-minutes: 120
run: |
cd test
python3 run_suite.py --hw cuda --suite nightly-eval-text-2-gpu --nightly --continue-on-error --timeout-per-file 4500
- uses: ./.github/actions/upload-cuda-coredumps
if: failure()
# Text model performance tests
nightly-test-text-perf-2-gpu-h100:
if: github.repository == 'sgl-project/sglang' && (inputs.job_filter == '' || inputs.job_filter == 'all' || inputs.job_filter == 'nightly-test-text-perf-2-gpu-h100')
runs-on: 2-gpu-h100
steps:
- name: Checkout code
uses: actions/checkout@v4
with:
ref: ${{ inputs.ref || github.ref }}
- uses: ./.github/actions/check-maintenance
- name: Install dependencies
run: |
bash scripts/ci/cuda/ci_install_dependency.sh
- name: Run performance test for text models
timeout-minutes: 30
env:
GPU_CONFIG: "2-gpu-h100"
run: |
cd test
rm -rf performance_results_text_models/
python3 run_suite.py --hw cuda --suite nightly-perf-text-2-gpu --nightly --continue-on-error --timeout-per-file 3600
- uses: ./.github/actions/upload-cuda-coredumps
if: failure()
# VLM accuracy tests
nightly-test-vlm-accuracy-2-gpu-h100:
if: github.repository == 'sgl-project/sglang' && (inputs.job_filter == '' || inputs.job_filter == 'all' || inputs.job_filter == 'nightly-test-vlm-accuracy-2-gpu-h100')
runs-on: 2-gpu-h100
steps:
- name: Checkout code
uses: actions/checkout@v4
with:
ref: ${{ inputs.ref || github.ref }}
- uses: ./.github/actions/check-maintenance
- name: Install dependencies
run: |
bash scripts/ci/cuda/ci_install_dependency.sh
- name: Run eval test for VLM models (fixed MMMU-100)
timeout-minutes: 120
run: |
cd test
python3 run_suite.py --hw cuda --suite nightly-eval-vlm-2-gpu --nightly --continue-on-error --timeout-per-file 9000
- uses: ./.github/actions/upload-cuda-coredumps
if: failure()
# VLM performance tests
nightly-test-vlm-perf-2-gpu-h100:
if: github.repository == 'sgl-project/sglang' && (inputs.job_filter == '' || inputs.job_filter == 'all' || inputs.job_filter == 'nightly-test-vlm-perf-2-gpu-h100')
runs-on: 2-gpu-h100
steps:
- name: Checkout code
uses: actions/checkout@v4
with:
ref: ${{ inputs.ref || github.ref }}
- uses: ./.github/actions/check-maintenance
- name: Install dependencies
run: |
bash scripts/ci/cuda/ci_install_dependency.sh
- name: Run perf test for VLM models (MMMU)
timeout-minutes: 30
env:
GPU_CONFIG: "2-gpu-h100"
run: |
cd test
rm -rf performance_results_vlms/
python3 run_suite.py --hw cuda --suite nightly-perf-vlm-2-gpu --nightly --continue-on-error --timeout-per-file 3600
- uses: ./.github/actions/upload-cuda-coredumps
if: failure()
# B200 Performance tests - 4 GPU
nightly-test-perf-4-gpu-b200:
if: github.repository == 'sgl-project/sglang' && (inputs.job_filter == '' || inputs.job_filter == 'all' || inputs.job_filter == 'nightly-test-perf-4-gpu-b200')
runs-on: 4-gpu-b200
steps:
- name: Checkout code
uses: actions/checkout@v4
with:
ref: ${{ inputs.ref || github.ref }}
- uses: ./.github/actions/check-maintenance
- name: Install dependencies
run: |
bash scripts/ci/cuda/ci_install_dependency.sh
- name: Run test
timeout-minutes: 200
env:
GPU_CONFIG: "4-gpu-b200"
run: |
cd test
python3 run_suite.py --hw cuda --suite nightly-4-gpu-b200 --nightly --continue-on-error --timeout-per-file 12000
- uses: ./.github/actions/upload-cuda-coredumps
if: failure()
# GB300 (Grace-Blackwell NVL4) performance tests - 4 GPU (ARM64)
nightly-test-perf-4-gpu-gb300:
if: github.repository == 'sgl-project/sglang' && (inputs.job_filter == '' || inputs.job_filter == 'all' || inputs.job_filter == 'nightly-test-perf-4-gpu-gb300')
name: nightly-test-perf-4-gpu-gb300 (${{ matrix.model }})
runs-on: 4-gpu-gb300-nightly
strategy:
fail-fast: false
matrix:
include:
- model: glm5-nvfp4
suite: nightly-4-gpu-gb300-glm5-nvfp4
- model: qwen35-fp8
suite: nightly-4-gpu-gb300-qwen35-fp8
- model: deepseek-v4-pro-fp4
suite: nightly-4-gpu-gb300-deepseek-v4-pro-fp4
- model: kimi-k25-nvfp4
suite: nightly-4-gpu-gb300-kimi-k25-nvfp4
steps:
- name: Checkout code
uses: actions/checkout@v4
with:
ref: ${{ inputs.ref || github.ref }}
- uses: ./.github/actions/check-maintenance
- name: Install dependencies
run: |
bash scripts/ci/cuda/ci_install_dependency.sh
- name: Run test
timeout-minutes: 600
env:
GPU_CONFIG: "4-gpu-gb300"
run: |
cd test
python3 run_suite.py --hw cuda --suite ${{ matrix.suite }} --nightly --continue-on-error --timeout-per-file 7200
- uses: ./.github/actions/upload-cuda-coredumps
if: failure()
# Specialized B200 tests - 8 GPU, for specific backends and configs
nightly-test-specialized-8-gpu-b200:
if: github.repository == 'sgl-project/sglang' && (inputs.job_filter == '' || inputs.job_filter == 'all' || inputs.job_filter == 'nightly-test-perf-8-gpu-b200' || inputs.job_filter == 'nightly-test-specialized-8-gpu-b200')
runs-on: 8-gpu-b200
env:
RUNNER_LABELS: 8-gpu-b200
steps:
- name: Checkout code
uses: actions/checkout@v4
with:
ref: ${{ inputs.ref || github.ref }}
- uses: ./.github/actions/check-maintenance
- name: Install dependencies
run: |
bash scripts/ci/cuda/ci_install_dependency.sh
- name: Run test
timeout-minutes: 60
env:
GPU_CONFIG: "8-gpu-b200"
run: |
cd test
python3 run_suite.py --hw cuda --suite nightly-8-gpu-b200 --nightly --continue-on-error --timeout-per-file 2400
- uses: ./.github/actions/upload-cuda-coredumps
if: failure()
# SGLang-Diffusion nightly benchmark
nightly-test-diffusion:
if: github.repository == 'sgl-project/sglang' && (inputs.job_filter == '' || inputs.job_filter == 'all' || inputs.job_filter == 'nightly-test-diffusion')
runs-on: 4-gpu-h100
timeout-minutes: 300
steps: steps:
- name: Checkout code - name: Checkout code
uses: actions/checkout@v4 uses: actions/checkout@v4
@@ -593,47 +257,19 @@ jobs:
- uses: ./.github/actions/upload-cuda-coredumps - uses: ./.github/actions/upload-cuda-coredumps
if: failure() if: failure()
# Nightly precision regression - per-layer hidden state comparison
nightly-test-precision-8-gpu-h200:
if: github.repository == 'sgl-project/sglang' && (inputs.job_filter == '' || inputs.job_filter == 'all' || inputs.job_filter == 'nightly-test-precision-8-gpu-h200')
runs-on: 8-gpu-h200
steps:
- name: Checkout code
uses: actions/checkout@v4
with:
ref: ${{ inputs.ref || github.ref }}
- uses: ./.github/actions/check-maintenance
- name: Install dependencies
run: |
bash scripts/ci/cuda/ci_install_dependency.sh
- name: Run precision regression test
timeout-minutes: 120
env:
SGLANG_PRECISION_BASELINE_DIR: /tmp/sglang_precision_baselines
# Required: the test errors if SGLANG_PRECISION_HF_REPO is unset (no
# local-only mode). Set the var + the HF_TOKEN_PRECISION_STORE secret.
SGLANG_PRECISION_HF_REPO: ${{ vars.SGLANG_PRECISION_HF_REPO }}
SGLANG_PRECISION_HF_REVISION: ${{ vars.SGLANG_PRECISION_HF_REVISION || 'main' }}
HF_TOKEN: ${{ secrets.HF_TOKEN_PRECISION_STORE }}
SGLANG_PRECISION_COMMIT: ${{ github.sha }}
# '0' on scheduled runs (input unset); '1' only on a manual baseline-refresh dispatch.
SGLANG_PRECISION_FORCE_UPDATE: ${{ inputs.force_baseline_update && '1' || '0' }}
run: |
cd test
python3 run_suite.py --hw cuda --suite nightly-precision-8-gpu-h200 --nightly --continue-on-error --timeout-per-file 3600
- uses: ./.github/actions/upload-cuda-coredumps
if: failure()
# Consolidate performance metrics from all jobs # Consolidate performance metrics from all jobs
consolidate-metrics: consolidate-metrics:
if: github.repository == 'sgl-project/sglang' && always() if: github.repository == 'sgl-project/sglang' && always()
# Every scheduled stage uploads metrics now, so all of them must finish
# before the download step globs `*metrics-*`.
needs: needs:
- nightly-test-general-8-gpu-h200 - nightly-1-gpu-large
- nightly-test-general-8-gpu-b200 - nightly-2-gpu-large
- nightly-4-gpu-h100
- nightly-4-gpu-b200
- nightly-4-gpu-gb300
- nightly-8-gpu-h200
- nightly-8-gpu-b200
runs-on: ubuntu-latest runs-on: ubuntu-latest
steps: steps:
- name: Checkout code - name: Checkout code
@@ -674,20 +310,14 @@ jobs:
check-all-jobs: check-all-jobs:
if: github.repository == 'sgl-project/sglang' && always() if: github.repository == 'sgl-project/sglang' && always()
needs: needs:
- nightly-test-general-1-gpu-h100 - nightly-1-gpu-large
- nightly-test-general-4-gpu-h100 - nightly-2-gpu-large
- nightly-test-general-8-gpu-h200 - nightly-4-gpu-h100
- nightly-test-general-8-gpu-h20 - nightly-4-gpu-b200
- nightly-test-general-8-gpu-b200 - nightly-4-gpu-gb300
- nightly-test-text-accuracy-2-gpu-h100 - nightly-8-gpu-h200
- nightly-test-text-perf-2-gpu-h100 - nightly-8-gpu-b200
- nightly-test-vlm-accuracy-2-gpu-h100
- nightly-test-vlm-perf-2-gpu-h100
- nightly-test-perf-4-gpu-b200
- nightly-test-specialized-8-gpu-b200
- nightly-test-perf-4-gpu-gb300
- nightly-test-diffusion - nightly-test-diffusion
- nightly-test-precision-8-gpu-h200
- consolidate-metrics - consolidate-metrics
runs-on: ubuntu-latest runs-on: ubuntu-latest
steps: steps:
.github/workflows/weekly-test-nvidia.yml
+39 -30
View File
@@ -1,3 +1,5 @@
# Weekly CUDA tests. Same shape as nightly-test-nvidia.yml (see its header); a
# test joins by declaring `stage="weekly"` plus a runner_config.
name: Weekly Test (Nvidia) name: Weekly Test (Nvidia)
on: on:
@@ -5,46 +7,53 @@ on:
- cron: '0 0 * * 0' # Run every Sunday at midnight UTC - cron: '0 0 * * 0' # Run every Sunday at midnight UTC
workflow_dispatch: workflow_dispatch:
inputs: inputs:
job_filter: runner_filter:
description: 'Select which job to run (leave empty or "all" to run all jobs)' description: 'Select which runner_config to run (leave empty or "all" to run all)'
required: false required: false
type: choice type: choice
default: 'all' default: 'all'
options: options:
- 'all' - 'all'
- 'weekly-test-8-gpu-h200' - '8-gpu-h200'
full_parallel:
description: 'Run all shards of a job at once (faster, but competes with per-commit CI for machines). Off by default: one shard at a time.'
required: false
type: boolean
default: false
concurrency: concurrency:
group: weekly-test-nvidia-${{ github.ref }} group: weekly-test-nvidia-${{ github.ref }}
cancel-in-progress: true cancel-in-progress: true
env: permissions:
SGLANG_IS_IN_CI: true actions: write
SGLANG_ENABLE_ASYNC_ASSERT: true contents: read
HF_HUB_DOWNLOAD_TIMEOUT: 300 issues: read
HF_HUB_ETAG_TIMEOUT: 300 pull-requests: read
jobs: jobs:
# Weekly tests - 8 GPU H200 # run_all_tests skips the paths-filter, so main_package is 'true' and
# sgl_kernel stays empty: every test runs, and no job waits on a wheel this
# workflow never builds.
check-changes:
uses: ./.github/workflows/_pr-test-check-changes.yml
with:
pr_test_yml: '.github/workflows/weekly-test-nvidia.yml'
run_all_tests: true
force_continue_on_error: true
secrets: inherit
weekly-test-8-gpu-h200: weekly-test-8-gpu-h200:
if: github.repository == 'sgl-project/sglang' && (inputs.job_filter == '' || inputs.job_filter == 'all' || inputs.job_filter == 'weekly-test-8-gpu-h200') needs: check-changes
runs-on: 8-gpu-h200 if: github.repository == 'sgl-project/sglang' && (inputs.runner_filter == '' || inputs.runner_filter == 'all' || inputs.runner_filter == '8-gpu-h200')
timeout-minutes: 120 uses: ./.github/workflows/_pr-test-stage.yml
env: with:
RUNNER_LABELS: 8-gpu-h200 self_name: weekly-test-8-gpu-h200
steps: runner_config: 8-gpu-h200
- name: Checkout code check_changes: ${{ toJson(needs.check-changes.outputs) }}
uses: actions/checkout@v4 caller_inputs: ${{ toJson(inputs) }}
partitions: ${{ needs.check-changes.outputs.partitions }}
- name: Install dependencies run_timeout_minutes: '240'
run: | job_timeout_minutes: '300'
bash scripts/ci/cuda/ci_install_dependency.sh scheduled: true
secrets: inherit
- name: Run weekly 8-GPU H200 tests
timeout-minutes: 120
env:
GPU_CONFIG: "8-gpu-h200"
IS_H200: "1"
run: |
cd test
python3 run_suite.py --hw cuda --suite weekly-8-gpu-h200 --nightly --continue-on-error --timeout-per-file 7200
docs/docs/references/nightly_precision_regression.mdx
+21 -18
View File
@@ -108,7 +108,7 @@ A `capture_signature` (SHA-1 hash of schema version, max_tokens, ignore_eos, TP
| `SGLANG_PRECISION_COMMIT` | _(auto-detected from git)_ | Override the sglang commit SHA tagged on push | | `SGLANG_PRECISION_COMMIT` | _(auto-detected from git)_ | Override the sglang commit SHA tagged on push |
| `SGLANG_PRECISION_HF_REPO` | _(required)_ | HuggingFace dataset repo for cross-runner baseline storage | | `SGLANG_PRECISION_HF_REPO` | _(required)_ | HuggingFace dataset repo for cross-runner baseline storage |
| `SGLANG_PRECISION_HF_REVISION` | `main` | Branch/revision of the HF dataset | | `SGLANG_PRECISION_HF_REVISION` | `main` | Branch/revision of the HF dataset |
| `HF_TOKEN` | _(required in CI)_ | HuggingFace token with write access to the dataset | | `SGLANG_PRECISION_HF_TOKEN` | _(required in CI)_ | HuggingFace token with write access to the dataset. Kept off `HF_TOKEN`, which already carries the runner's gated-model read token |
--- ---
@@ -116,31 +116,34 @@ A `capture_signature` (SHA-1 hash of schema version, max_tokens, ignore_eos, TP
### Workflow job ### Workflow job
The nightly job `nightly-test-precision-8-gpu-h200` is defined in `.github/workflows/nightly-test-nvidia.yml` and runs on an 8-GPU H200 runner. It is included in the nightly suite via `test/run_suite.py`. The test is registered on the `nightly-8-gpu-h200` stage in `.github/workflows/nightly-test-nvidia.yml`, which runs through `_pr-test-stage.yml` like every other CUDA stage. The baseline env below is exported on every scheduled stage; only this test reads it.
Key CI configuration: Key CI configuration:
```yaml ```yaml
- name: Run precision regression test - name: Export precision baseline env
timeout-minutes: 120 if: inputs.scheduled
env: env:
SGLANG_PRECISION_BASELINE_DIR: /tmp/sglang_precision_baselines BASELINE_HF_TOKEN: ${{ secrets.HF_TOKEN_PRECISION_STORE }}
SGLANG_PRECISION_HF_REPO: ${{ vars.SGLANG_PRECISION_HF_REPO }}
SGLANG_PRECISION_HF_REVISION: ${{ vars.SGLANG_PRECISION_HF_REVISION || 'main' }}
HF_TOKEN: ${{ secrets.HF_TOKEN_PRECISION_STORE }}
SGLANG_PRECISION_COMMIT: ${{ github.sha }}
run: | run: |
cd test {
python3 run_suite.py --hw cuda --suite nightly-precision-8-gpu-h200 --nightly --continue-on-error --timeout-per-file 3600 echo "SGLANG_PRECISION_BASELINE_DIR=/tmp/sglang_precision_baselines"
echo "SGLANG_PRECISION_HF_REPO=${{ vars.SGLANG_PRECISION_HF_REPO }}"
echo "SGLANG_PRECISION_HF_REVISION=${{ vars.SGLANG_PRECISION_HF_REVISION || 'main' }}"
echo "SGLANG_PRECISION_COMMIT=${{ github.sha }}"
echo "SGLANG_PRECISION_HF_TOKEN=${BASELINE_HF_TOKEN}"
} >> "$GITHUB_ENV"
``` ```
`SGLANG_PRECISION_HF_TOKEN` rather than `HF_TOKEN`: the latter already carries the runner's gated-model read token, and overwriting it would turn every gated model on the job into a 401.
### Required GitHub secrets/variables ### Required GitHub secrets/variables
| Name | Type | Purpose | | Name | Type | Purpose |
|------|------|---------| |------|------|---------|
| `SGLANG_PRECISION_HF_REPO` | Repository variable | HF dataset repo ID (e.g. `org/sglang-precision-baselines`) — **required**, the test errors if unset | | `SGLANG_PRECISION_HF_REPO` | Repository variable | HF dataset repo ID (e.g. `org/sglang-precision-baselines`) — **required**, the test errors if unset |
| `SGLANG_PRECISION_HF_REVISION` | Repository variable (optional) | Dataset branch (defaults to `main`) | | `SGLANG_PRECISION_HF_REVISION` | Repository variable (optional) | Dataset branch (defaults to `main`) |
| `HF_TOKEN_PRECISION_STORE` | Repository secret | HF token with write access to the dataset | | `HF_TOKEN_PRECISION_STORE` | Repository secret | HF token with write access to the dataset; exported to the job as `SGLANG_PRECISION_HF_TOKEN` |
### GitHub Step Summary ### GitHub Step Summary
@@ -187,7 +190,7 @@ export SGLANG_PRECISION_MODELS="your-org/your-model"
export SGLANG_PRECISION_BASELINE_DIR="/tmp/my_precision_baselines" export SGLANG_PRECISION_BASELINE_DIR="/tmp/my_precision_baselines"
export SGLANG_PRECISION_DIFF_THRESHOLD="1e-3" export SGLANG_PRECISION_DIFF_THRESHOLD="1e-3"
export SGLANG_PRECISION_HF_REPO="your-org/sglang-precision-baselines" export SGLANG_PRECISION_HF_REPO="your-org/sglang-precision-baselines"
export HF_TOKEN="hf_..." export SGLANG_PRECISION_HF_TOKEN="hf_..."
cd test cd test
python3 -m pytest registered/debug_utils/test_nightly_precision_regression.py -v python3 -m pytest registered/debug_utils/test_nightly_precision_regression.py -v
@@ -201,7 +204,7 @@ python3 -m pytest registered/debug_utils/test_nightly_precision_regression.py -v
export SGLANG_PRECISION_MODELS="your-org/your-model" export SGLANG_PRECISION_MODELS="your-org/your-model"
export SGLANG_PRECISION_BASELINE_DIR="/tmp/test_baselines" export SGLANG_PRECISION_BASELINE_DIR="/tmp/test_baselines"
export SGLANG_PRECISION_HF_REPO="your-org/sglang-precision-baselines" export SGLANG_PRECISION_HF_REPO="your-org/sglang-precision-baselines"
export HF_TOKEN="hf_..." export SGLANG_PRECISION_HF_TOKEN="hf_..."
export SGLANG_PRECISION_FORCE_UPDATE="1" # first run: establish baseline export SGLANG_PRECISION_FORCE_UPDATE="1" # first run: establish baseline
cd test cd test
@@ -249,7 +252,7 @@ python3 -m pytest registered/debug_utils/test_nightly_precision_regression.py -v
- SGLang installed in development mode - SGLang installed in development mode
- GPUs matching the model's requirements - GPUs matching the model's requirements
- `huggingface_hub` installed - `huggingface_hub` installed
- A **HuggingFace dataset** for baseline storage and a write-capable `HF_TOKEN`. The HF store is **mandatory** — `SGLANG_PRECISION_HF_REPO` must be set or the test will error at startup. This is because the nightly CI runners are ephemeral (no persistent local disk), so baselines must survive across runs via the HF dataset. There is currently no local-only fallback. - A **HuggingFace dataset** for baseline storage and a write-capable `SGLANG_PRECISION_HF_TOKEN`. The HF store is **mandatory** — `SGLANG_PRECISION_HF_REPO` must be set or the test will error at startup. This is because the nightly CI runners are ephemeral (no persistent local disk), so baselines must survive across runs via the HF dataset. There is currently no local-only fallback.
### Quick local test ### Quick local test
@@ -258,7 +261,7 @@ python3 -m pytest registered/debug_utils/test_nightly_precision_regression.py -v
export SGLANG_PRECISION_MODELS="Qwen/Qwen2.5-0.5B-Instruct" export SGLANG_PRECISION_MODELS="Qwen/Qwen2.5-0.5B-Instruct"
export SGLANG_PRECISION_BASELINE_DIR="/tmp/precision_baselines" export SGLANG_PRECISION_BASELINE_DIR="/tmp/precision_baselines"
export SGLANG_PRECISION_HF_REPO="your-org/sglang-precision-baselines" export SGLANG_PRECISION_HF_REPO="your-org/sglang-precision-baselines"
export HF_TOKEN="hf_..." export SGLANG_PRECISION_HF_TOKEN="hf_..."
# First run: establish baseline # First run: establish baseline
cd test cd test
@@ -356,7 +359,7 @@ The framework uses a **rolling baseline**: every successful comparison updates t
### No local-only mode ### No local-only mode
The test requires a HuggingFace dataset (`SGLANG_PRECISION_HF_REPO`) and a write-capable `HF_TOKEN`. There is no local-only fallback. This is by design — CI runners have no persistent local disk, so the HF dataset is the only way to carry baselines across runs. If you need to run the test locally, you must set up a HF dataset (even a private one) and provide the corresponding token. The test requires a HuggingFace dataset (`SGLANG_PRECISION_HF_REPO`) and a write-capable `SGLANG_PRECISION_HF_TOKEN`. There is no local-only fallback. This is by design — CI runners have no persistent local disk, so the HF dataset is the only way to carry baselines across runs. If you need to run the test locally, you must set up a HF dataset (even a private one) and provide the corresponding token.
--- ---
@@ -369,4 +372,4 @@ The test requires a HuggingFace dataset (`SGLANG_PRECISION_HF_REPO`) and a write
| `python/sglang/srt/debug_utils/comparator/` | Tensor comparison engine | | `python/sglang/srt/debug_utils/comparator/` | Tensor comparison engine |
| `python/sglang/srt/debug_utils/dumper.py` | Runtime hidden-state capture | | `python/sglang/srt/debug_utils/dumper.py` | Runtime hidden-state capture |
| `.github/workflows/nightly-test-nvidia.yml` | CI workflow definition | | `.github/workflows/nightly-test-nvidia.yml` | CI workflow definition |
| `test/run_suite.py` | Test suite registration (includes `nightly-precision-8-gpu-h200`) | | `test/run_suite.py` | Test suite registration (includes `nightly-test-8-gpu-h200`) |
python/sglang/test/ci/ci_utils.py
+28 -9
View File
@@ -137,9 +137,23 @@ def _repo_relative_path(p: str) -> str:
return p[idx + len(marker) :] if idx >= 0 else p return p[idx + len(marker) :] if idx >= 0 else p
# Slow-run variance is largely additive (cold HF cache, slow server launch), so
# the multiplier alone under-provisions at both ends: test_encoder_dp runs
# 200-426s but once took over 1185s against a 1.5x budget of 765s, and
# test_lora_deepseek_v3_base_logprob_diff (est 1800) landed on exactly 1.5 * est.
# Every file gets the same absolute slack on top of the proportional one.
DERIVED_TIMEOUT_SLACK = 1800.0
DERIVED_TIMEOUT_FACTOR = 1.5
def derive_timeout_per_file(est_time: float) -> float:
est = float(est_time)
return max(est * DERIVED_TIMEOUT_FACTOR, est + DERIVED_TIMEOUT_SLACK)
def run_unittest_files( def run_unittest_files(
files: Union[List[TestFile], List[CIRegistry]], files: Union[List[TestFile], List[CIRegistry]],
timeout_per_file: float, timeout_per_file: Optional[float] = None,
continue_on_error: bool = False, continue_on_error: bool = False,
enable_retry: bool = False, enable_retry: bool = False,
max_attempts: int = 2, max_attempts: int = 2,
@@ -150,7 +164,8 @@ def run_unittest_files(
Args: Args:
files: List of TestFile objects to run files: List of TestFile objects to run
timeout_per_file: Timeout in seconds for each test file timeout_per_file: Fixed timeout in seconds for every test file, or None
to derive each file's budget from its own est_time.
continue_on_error: If True, continue running remaining tests even if one fails. continue_on_error: If True, continue running remaining tests even if one fails.
If False, stop at first failure (default behavior for PR tests). If False, stop at first failure (default behavior for PR tests).
enable_retry: If True, retry failed tests that appear to be accuracy/performance enable_retry: If True, retry failed tests that appear to be accuracy/performance
@@ -178,6 +193,12 @@ def run_unittest_files(
# FIXME: remove this branch after migrating all tests to use CIRegistry # FIXME: remove this branch after migrating all tests to use CIRegistry
filename, estimated_time = file.name, file.estimated_time filename, estimated_time = file.name, file.estimated_time
file_timeout = (
timeout_per_file
if timeout_per_file is not None
else derive_timeout_per_file(estimated_time)
)
process = None process = None
output_lines = [] output_lines = []
@@ -235,7 +256,7 @@ def run_unittest_files(
run_one_file, run_one_file,
args=(filename,), args=(filename,),
kwargs={"capture_output": enable_retry}, kwargs={"capture_output": enable_retry},
timeout=timeout_per_file, timeout=file_timeout,
) )
if ret_code == 0: if ret_code == 0:
@@ -281,24 +302,22 @@ def run_unittest_files(
# TimeoutError aborts run_one_file before its elapsed write; # TimeoutError aborts run_one_file before its elapsed write;
# record the timeout cap as an upper bound so the file still # record the timeout cap as an upper bound so the file still
# appears in the TIMINGS block below. # appears in the TIMINGS block below.
file_elapsed[filename] = float(timeout_per_file) file_elapsed[filename] = float(file_timeout)
# Retry once on timeout: usually a stuck server / hung device. # Retry once on timeout: usually a stuck server / hung device.
# A real hang times out again and is reported. # A real hang times out again and is reported.
if enable_retry and attempt < max_attempts: if enable_retry and attempt < max_attempts:
logger.info( logger.info(
f"\n[CI Retry] {filename} timed out after " f"\n[CI Retry] {filename} timed out after "
f"{timeout_per_file}s; waiting {retry_wait_seconds}s " f"{file_timeout}s; waiting {retry_wait_seconds}s "
f"before retry (attempt {attempt + 1}/{max_attempts})\n" f"before retry (attempt {attempt + 1}/{max_attempts})\n"
) )
time.sleep(retry_wait_seconds) time.sleep(retry_wait_seconds)
attempt += 1 attempt += 1
continue continue
logger.info( logger.info(f"\n✗ TIMEOUT: {filename} after {file_timeout} seconds\n")
f"\n✗ TIMEOUT: {filename} after {timeout_per_file} seconds\n"
)
if was_retried: if was_retried:
retried_tests.append((filename, attempt, "timeout")) retried_tests.append((filename, attempt, "timeout"))
failed_tests.append((filename, f"timeout after {timeout_per_file}s")) failed_tests.append((filename, f"timeout after {file_timeout}s"))
break break
if not file_passed: if not file_passed:
python/sglang/test/precision_baseline_store.py
+15 -3
View File
@@ -26,6 +26,16 @@ from huggingface_hub.errors import (
) )
def _store_token() -> Optional[str]:
"""Write token for the baseline dataset repo.
Deliberately not HF_TOKEN: that name already carries the runner's
gated-model read token, so writing the store token there would shadow it
and turn every gated model on the job into a 401.
"""
return os.environ.get("SGLANG_PRECISION_HF_TOKEN") or None
@dataclass @dataclass
class HfStoreConfig: class HfStoreConfig:
repo: str repo: str
@@ -38,7 +48,7 @@ class HfStoreConfig:
raise RuntimeError( raise RuntimeError(
"SGLANG_PRECISION_HF_REPO is not set. The precision baseline " "SGLANG_PRECISION_HF_REPO is not set. The precision baseline "
"store is required (there is no local-only mode); set the repo " "store is required (there is no local-only mode); set the repo "
"and HF_TOKEN_PRECISION_STORE." "and SGLANG_PRECISION_HF_TOKEN."
) )
revision = os.environ.get("SGLANG_PRECISION_HF_REVISION", "main") revision = os.environ.get("SGLANG_PRECISION_HF_REVISION", "main")
return cls(repo=repo, revision=revision) return cls(repo=repo, revision=revision)
@@ -148,6 +158,7 @@ def fetch_latest_baseline(
repo_type="dataset", repo_type="dataset",
revision=config.revision, revision=config.revision,
allow_patterns=[f"{run_path}/tensors/*"], allow_patterns=[f"{run_path}/tensors/*"],
token=_store_token(),
), ),
what="snapshot download", what="snapshot download",
) )
@@ -171,6 +182,7 @@ def _read_manifest(config: HfStoreConfig) -> tuple[list[dict[str, Any]], str]:
repo_type="dataset", repo_type="dataset",
filename="manifest.jsonl", filename="manifest.jsonl",
revision=config.revision, revision=config.revision,
token=_store_token(),
), ),
what="manifest fetch", what="manifest fetch",
) )
@@ -215,7 +227,7 @@ def push_run(
# Dedup: same model+date+sha → skip tensor upload but still refresh meta # Dedup: same model+date+sha → skip tensor upload but still refresh meta
# + comparator_report + append a new manifest row, so pass-1 baseline and # + comparator_report + append a new manifest row, so pass-1 baseline and
# pass-2 stats both land. force=True re-uploads tensors too. # pass-2 stats both land. force=True re-uploads tensors too.
api = HfApi() api = HfApi(token=_store_token())
date_str, date_path = _today_path() date_str, date_path = _today_path()
model_sanitized = _sanitize_model_name(model) model_sanitized = _sanitize_model_name(model)
sha7 = ( sha7 = (
@@ -306,7 +318,7 @@ def prune_old_runs(
# dry_run defaults True because model=None+keep_days=0 would wipe the # dry_run defaults True because model=None+keep_days=0 would wipe the
# store. Live mode rewrites the manifest before deleting folders so a # store. Live mode rewrites the manifest before deleting folders so a
# mid-run failure leaves manifest pointing at the kept rows only. # mid-run failure leaves manifest pointing at the kept rows only.
api = HfApi() api = HfApi(token=_store_token())
rows, _ = _read_manifest(config) rows, _ = _read_manifest(config)
if not rows: if not rows:
return {"kept": [], "pruned": []} return {"kept": [], "pruned": []}
scripts/ci/check_registered_tests.py
+8 -11
View File
@@ -4,11 +4,11 @@ Pre-commit hook: validate CI registry calls under test/registered/.
1. Every test file must contain a CI registry call (register_cuda_ci, 1. Every test file must contain a CI registry call (register_cuda_ci,
register_amd_ci, etc.). register_amd_ci, etc.).
2. A CUDA test must register its PR-test suite via the modern 2. A CUDA test must register its suite via the modern
`stage=`/`runner_config=` form. The legacy single-string `suite=` is reserved `stage=`/`runner_config=` form. The legacy single-string `suite=` is reserved
for the nightly/stress/weekly families (and for AMD/CPU/NPU suites); any other for the stress family (and for AMD/CPU/NPU suites); any other CUDA `suite=`
CUDA `suite=` resolves to a name no PR-test workflow invokes, so the test resolves to a name no workflow invokes, so the test silently never runs.
silently never runs. Two shapes are rejected: Two shapes are rejected:
a. `{stage}-test-{runner_config}` -- the modern name stuffed back into the a. `{stage}-test-{runner_config}` -- the modern name stuffed back into the
legacy form. Reported with the exact stage/runner split to use. legacy form. Reported with the exact stage/runner split to use.
b. an older `{stage}-{runner_config}` PR-test name (e.g. the pre-migration b. an older `{stage}-{runner_config}` PR-test name (e.g. the pre-migration
@@ -33,11 +33,10 @@ import sys
# shape is always expressible (and should be expressed) the modern way. # shape is always expressible (and should be expressed) the modern way.
_MODERN_SHAPE = re.compile(r"^(.+)-test-(.+)$") _MODERN_SHAPE = re.compile(r"^(.+)-test-(.+)$")
# The only suite families a CUDA registry may keep on the legacy single-string # The only CUDA suite family still allowed on the legacy single-string `suite=`
# `suite=` form. Everything else is a PR-test/base stage that must use the # form. Anything else needs stage=/runner_config=, or its effective_suite matches
# modern stage=/runner_config= form (otherwise its effective_suite matches no # no suite any workflow invokes and the test silently never runs.
# suite the PR-test workflows invoke, and the test silently never runs). _LEGACY_CUDA_PREFIXES = ("stress",)
_LEGACY_CUDA_PREFIXES = ("nightly", "stress", "weekly")
def _defines_testcase(tree: ast.AST) -> bool: def _defines_testcase(tree: ast.AST) -> bool:
@@ -118,8 +117,6 @@ def main() -> int:
and r.runner_config is None and r.runner_config is None
): ):
continue continue
# nightly/stress/weekly are the only CUDA suites allowed to stay on
# the legacy single-string form.
if r.suite.split("-", 1)[0] in _LEGACY_CUDA_PREFIXES: if r.suite.split("-", 1)[0] in _LEGACY_CUDA_PREFIXES:
continue continue
m = _MODERN_SHAPE.match(r.suite) m = _MODERN_SHAPE.match(r.suite)
scripts/ci/stage_models_overrides.json
+2 -16
View File
@@ -1,5 +1,5 @@
{ {
"_comment": "Manual overrides for list_stage_models.py. by_file/by_suite ADD models the static scan cannot see (models built dynamically, read from configs, passed via CLI args). deny REMOVES false-positive ids the heuristic mistakes for models. Keys in by_file are repo-relative test paths (e.g. test/registered/foo/test_bar.py). suite_labels maps legacy suite= registrations (no runner_config) to the GH runner label(s) their dispatching workflow hardcodes in runs-on -- a list, because one suite can run on several labels (nightly-8-gpu-common). $b200_runner is the dynamic-b200 placeholder from runner_configs.yml. Deliberately absent: nightly-4-gpu-gb300-* (run as k8s pods, not GHA runners) and nightly-2-gpu (registered but dispatched by no workflow); both stay visible in unmapped_suites.", "_comment": "Manual overrides for list_stage_models.py. by_file/by_suite ADD models the static scan cannot see (models built dynamically, read from configs, passed via CLI args). deny REMOVES false-positive ids the heuristic mistakes for models. Keys in by_file are repo-relative test paths (e.g. test/registered/foo/test_bar.py). suite_labels maps legacy suite= registrations (no runner_config) to the GH runner label(s) their dispatching workflow hardcodes in runs-on -- a list, because one suite can run on several labels. $b200_runner is the dynamic-b200 placeholder from runner_configs.yml. Suites registered with stage=/runner_config= need no entry: their label resolves through runner_configs.yml. Anything unmappable stays visible in unmapped_suites.",
"by_file": {}, "by_file": {},
"by_suite": {}, "by_suite": {},
"suite_labels": { "suite_labels": {
@@ -7,21 +7,7 @@
"base-b-kernel-unit-1-gpu-b200": ["$b200_runner"], "base-b-kernel-unit-1-gpu-b200": ["$b200_runner"],
"base-b-kernel-unit-1-gpu-large": ["1-gpu-h100"], "base-b-kernel-unit-1-gpu-large": ["1-gpu-h100"],
"base-b-kernel-unit-8-gpu-h200": ["8-gpu-h200"], "base-b-kernel-unit-8-gpu-h200": ["8-gpu-h200"],
"nightly-1-gpu": ["1-gpu-h100"], "stress": ["8-gpu-h200"]
"nightly-4-gpu": ["4-gpu-h100"],
"nightly-4-gpu-b200": ["$b200_runner"],
"nightly-8-gpu-b200": ["8-gpu-b200"],
"nightly-8-gpu-common": ["8-gpu-h200", "8-gpu-b200"],
"nightly-8-gpu-h200": ["8-gpu-h200"],
"nightly-eval-text-2-gpu": ["2-gpu-h100"],
"nightly-eval-vlm-2-gpu": ["2-gpu-h100"],
"nightly-kernel-1-gpu": ["1-gpu-h100"],
"nightly-kernel-8-gpu-h200": ["8-gpu-h200"],
"nightly-perf-text-2-gpu": ["2-gpu-h100"],
"nightly-perf-vlm-2-gpu": ["2-gpu-h100"],
"nightly-precision-8-gpu-h200": ["8-gpu-h200"],
"stress": ["8-gpu-h200"],
"weekly-8-gpu-h200": ["8-gpu-h200"]
}, },
"deny": [ "deny": [
"tok/req", "tok/req",
scripts/ci/utils/compute_partitions.py
+5 -2
View File
@@ -128,8 +128,11 @@ def compute_partitions(
in-source `est_time` / `(1.0, 0.0)`. in-source `est_time` / `(1.0, 0.0)`.
`full_parallel=True` lifts the matrix-fanout throttle. `full_parallel=True` lifts the matrix-fanout throttle.
""" """
# Allowlist: stages pr-test.yml dispatches. Stress / weekly / # Allowlist of the stages this workflow dispatches -- what keeps stress /
# nightly-* live in test/registered/ but pr-test doesn't run them. # weekly / nightly out, since CUDA scheduled suites no longer carry
# `nightly=True`. The nightly filter still matters for CPU: some tests sit on
# a dispatched suite with the flag set, so run_suite.py skips them and their
# est_time must not inflate the shard count.
dispatched_suites = set(run_timeouts) | set(_BASE_A_OVERRIDES) dispatched_suites = set(run_timeouts) | set(_BASE_A_OVERRIDES)
suite_tests = defaultdict(list) suite_tests = defaultdict(list)
for t in tests: for t in tests:
scripts/ci/utils/slash_command_handler.py
+11 -59
View File
@@ -702,11 +702,8 @@ def _extract_runner_configs(content):
def _extract_legacy_suites(content): def _extract_legacy_suites(content):
"""Pull every legacy single-string `suite=` from `register_cuda_ci(...)` calls. """Pull every legacy single-string `suite=` from `register_cuda_ci(...)`
calls. Used only to report why such a file is not dispatchable."""
Mirrors _extract_runner_configs for the legacy nightly/weekly shape: a file
may register on multiple pools, so collect all of them rather than the first.
"""
out = [] out = []
for args in re.finditer( for args in re.finditer(
r"^[^#\n]*register_cuda_ci\s*\(([^)]*)\)", content, re.MULTILINE r"^[^#\n]*register_cuda_ci\s*\(([^)]*)\)", content, re.MULTILINE
@@ -717,38 +714,6 @@ def _extract_legacy_suites(content):
return out return out
# Legacy nightly/weekly CUDA suites register with a single-string `suite=`
# instead of `runner_config=`, so they carry no runner metadata of their own.
# Map each to the runner_config in scripts/ci/runner_configs.yml whose hardware
# matches the runner the nightly/weekly pipeline actually uses (see
# .github/workflows/{nightly,weekly}-test-nvidia.yml), so /rerun-test can still
# dispatch a single nightly/weekly test. The runner label, install script,
# timeout and rdma_devices are then resolved from
# runner_configs.yml as usual, keeping that file the single source of truth for
# runner details.
#
# Suites on hardware with no matching runner_config (e.g. nightly-4-gpu-gb300)
# and non-CUDA suites (npu/amd) are intentionally absent and stay
# non-dispatchable until a matching runner_config exists.
_LEGACY_SUITE_TO_RUNNER_CONFIG = {
"nightly-1-gpu": "1-gpu-large",
"nightly-kernel-1-gpu": "1-gpu-large",
"nightly-eval-text-2-gpu": "2-gpu-large",
"nightly-perf-text-2-gpu": "2-gpu-large",
"nightly-eval-vlm-2-gpu": "2-gpu-large",
"nightly-perf-vlm-2-gpu": "2-gpu-large",
"nightly-4-gpu": "4-gpu-h100",
"nightly-4-gpu-b200": "4-gpu-b200",
"nightly-8-gpu-common": ["8-gpu-h200", "8-gpu-b200"],
"nightly-8-gpu-h200": "8-gpu-h200",
"nightly-kernel-8-gpu-h200": "8-gpu-h200",
"nightly-precision-8-gpu-h200": "8-gpu-h200",
"nightly-8-gpu-h20": "8-gpu-h20",
"nightly-8-gpu-b200": "8-gpu-b200",
"weekly-8-gpu-h200": "8-gpu-h200",
}
def _dispatch_err(suite, msg): def _dispatch_err(suite, msg):
"""Build a detect_suite error result for the given suite.""" """Build a detect_suite error result for the given suite."""
return { return {
@@ -811,11 +776,10 @@ def detect_suite(file_path_from_test):
pool it should run on — so this returns a *list* of dispatch dicts, one pool it should run on — so this returns a *list* of dispatch dicts, one
per registration. Runner label, install script, timeout, and rdma_devices per registration. Runner label, install script, timeout, and rdma_devices
are all resolved from scripts/ci/runner_configs.yml — the are all resolved from scripts/ci/runner_configs.yml — the
same single source of truth that drives the main PR test pipeline. same single source of truth that drives the main PR test pipeline. Every
dispatchable CUDA suite, per-commit and scheduled alike, goes through that
Legacy nightly/weekly CUDA suites (single-string `suite=`) are dispatchable one path; the legacy single-string `suite=` carries no runner_config and is
too: each suite name is mapped to the matching runner_config via reported as non-dispatchable.
_LEGACY_SUITE_TO_RUNNER_CONFIG, then resolved the same way.
CPU files yield a single-element list. A file with no recognised (or no CPU files yield a single-element list. A file with no recognised (or no
dispatchable) registration yields a one-element list whose dict has an dispatchable) registration yields a one-element list whose dict has an
@@ -837,19 +801,7 @@ def detect_suite(file_path_from_test):
results.append(_resolve_runner_config(rc, full_path, suite)) results.append(_resolve_runner_config(rc, full_path, suite))
return results return results
# Legacy nightly/weekly CUDA suites: single-string `suite=`, no
# runner_config. Map each mappable suite to its runner_config and resolve.
legacy_suites = _extract_legacy_suites(content) legacy_suites = _extract_legacy_suites(content)
mappable = [s for s in legacy_suites if s in _LEGACY_SUITE_TO_RUNNER_CONFIG]
if mappable:
results = []
for s in mappable:
rcs = _LEGACY_SUITE_TO_RUNNER_CONFIG[s]
if isinstance(rcs, str):
rcs = [rcs]
for rc in rcs:
results.append(_resolve_runner_config(rc, full_path, s))
return results
if re.search(r"^[^#\n]*register_cpu_ci\s*\(", content, re.MULTILINE): if re.search(r"^[^#\n]*register_cpu_ci\s*\(", content, re.MULTILINE):
return [ return [
@@ -869,11 +821,11 @@ def detect_suite(file_path_from_test):
return [ return [
_dispatch_err( _dispatch_err(
suite, suite,
f"Suite `{suite}` in `{full_path}` is not dispatchable via " f"Suite `{suite}` in `{full_path}` is registered with the legacy "
f"/rerun-test. It has no entry in _LEGACY_SUITE_TO_RUNNER_CONFIG " f"single-string `suite=`, which carries no runner_config and so "
f"— either it is a non-CUDA suite (npu/amd) or it runs on " f"is not dispatchable via /rerun-test. Re-register it with "
f"hardware with no matching runner_config in " f"`stage=`/`runner_config=` (CUDA), or dispatch its own "
f"scripts/ci/runner_configs.yml.", f"workflow (npu/amd).",
) )
] ]
test/README.md
+2 -2
View File
@@ -50,8 +50,8 @@ python3 test/registered/jit/test_add_constant.py
python3 test/run_suite.py --hw cpu --suite base-a-test-cpu python3 test/run_suite.py --hw cpu --suite base-a-test-cpu
python3 test/run_suite.py --hw cuda --suite base-a-test-1-gpu-small python3 test/run_suite.py --hw cuda --suite base-a-test-1-gpu-small
# Nightly tests # Nightly tests (CUDA nightly suites take no --nightly; the stage is in the name)
python3 test/run_suite.py --hw cuda --suite nightly-1-gpu --nightly python3 test/run_suite.py --hw cuda --suite nightly-test-1-gpu-large
# With auto-partitioning (for parallel CI jobs) # With auto-partitioning (for parallel CI jobs)
python3 test/run_suite.py --hw cuda --suite base-b-test-1-gpu-small \ python3 test/run_suite.py --hw cuda --suite base-b-test-1-gpu-small \
test/registered/4-gpu-models/test_nvidia_nemotron_3_super_nvfp4.py
+1 -1
View File
@@ -12,7 +12,7 @@ from sglang.test.test_utils import (
popen_launch_server, popen_launch_server,
) )
register_cuda_ci(est_time=810, suite="nightly-4-gpu-b200", nightly=True) register_cuda_ci(est_time=1200, stage="nightly", runner_config="4-gpu-b200")
NEMOTRON_3_SUPER_NVFP4_MODEL = "nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4" NEMOTRON_3_SUPER_NVFP4_MODEL = "nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4"
test/registered/8-gpu-models/test_glm52_fp8.py
+3 -2
View File
@@ -6,8 +6,9 @@ from sglang.test.performance_test_runner import PerformanceTestParams
from sglang.test.run_combined_tests import run_combined_tests from sglang.test.run_combined_tests import run_combined_tests
from sglang.test.test_utils import ModelLaunchSettings from sglang.test.test_utils import ModelLaunchSettings
# Runs on both H200 and B200 via nightly-8-gpu-common suite # Runs on both H200 and B200: registered once per runner_config below
register_cuda_ci(est_time=1800, suite="nightly-8-gpu-common", nightly=True) register_cuda_ci(est_time=2880, stage="nightly", runner_config="8-gpu-h200")
register_cuda_ci(est_time=2880, stage="nightly", runner_config="8-gpu-b200")
GLM_52_FP8_MODEL_PATH = "zai-org/GLM-5.2-FP8" GLM_52_FP8_MODEL_PATH = "zai-org/GLM-5.2-FP8"
test/registered/8-gpu-models/test_glm_46.py
+3 -2
View File
@@ -6,8 +6,9 @@ from sglang.test.performance_test_runner import PerformanceTestParams
from sglang.test.run_combined_tests import run_combined_tests from sglang.test.run_combined_tests import run_combined_tests
from sglang.test.test_utils import ModelLaunchSettings from sglang.test.test_utils import ModelLaunchSettings
# Runs on both H200 and B200 via nightly-8-gpu-common suite # Runs on both H200 and B200: registered once per runner_config below
register_cuda_ci(est_time=1800, suite="nightly-8-gpu-common", nightly=True) register_cuda_ci(est_time=1320, stage="nightly", runner_config="8-gpu-h200")
register_cuda_ci(est_time=1320, stage="nightly", runner_config="8-gpu-b200")
GLM_4_6_MODEL_PATH = "zai-org/GLM-4.6" GLM_4_6_MODEL_PATH = "zai-org/GLM-4.6"
test/registered/8-gpu-models/test_gpt_oss_120b.py
+3 -2
View File
@@ -5,9 +5,10 @@ from sglang.test.performance_test_runner import PerformanceTestParams
from sglang.test.run_combined_tests import run_combined_tests from sglang.test.run_combined_tests import run_combined_tests
from sglang.test.test_utils import ModelLaunchSettings from sglang.test.test_utils import ModelLaunchSettings
# Runs on both H200 and B200 via nightly-8-gpu-common suite # Runs on both H200 and B200: registered once per runner_config below
# Higher est_time due to 6 variants with both performance and accuracy tests # Higher est_time due to 6 variants with both performance and accuracy tests
register_cuda_ci(est_time=1800, suite="nightly-8-gpu-common", nightly=True) register_cuda_ci(est_time=690, stage="nightly", runner_config="8-gpu-h200")
register_cuda_ci(est_time=690, stage="nightly", runner_config="8-gpu-b200")
GPT_OSS_120B_MXFP4_MODEL_PATH = "openai/gpt-oss-120b" GPT_OSS_120B_MXFP4_MODEL_PATH = "openai/gpt-oss-120b"
GPT_OSS_120B_EAGLE3_DRAFT_MODEL_PATH = "lmsys/EAGLE3-gpt-oss-120b-bf16" GPT_OSS_120B_EAGLE3_DRAFT_MODEL_PATH = "lmsys/EAGLE3-gpt-oss-120b-bf16"
test/registered/8-gpu-models/test_inkling_nvfp4_nightly.py
+2 -1
View File
@@ -10,7 +10,8 @@ from sglang.test.test_utils import ModelLaunchSettings, is_blackwell_system
# NVFP4 needs Blackwell FP4 kernels, so this runs on the Blackwell leg of the # NVFP4 needs Blackwell FP4 kernels, so this runs on the Blackwell leg of the
# common 8-GPU suite (Hopper is skipped below). # common 8-GPU suite (Hopper is skipped below).
register_cuda_ci(est_time=3600, suite="nightly-8-gpu-common", nightly=True) register_cuda_ci(est_time=3600, stage="nightly", runner_config="8-gpu-h200")
register_cuda_ci(est_time=3600, stage="nightly", runner_config="8-gpu-b200")
INKLING_NVFP4_MODEL = "thinkingmachines/Inkling-NVFP4" INKLING_NVFP4_MODEL = "thinkingmachines/Inkling-NVFP4"
INKLING_SMALL_NVFP4_MODEL = "thinkingmachines/Inkling-Small-NVFP4" INKLING_SMALL_NVFP4_MODEL = "thinkingmachines/Inkling-Small-NVFP4"
test/registered/8-gpu-models/test_kimi_k25.py
+3 -2
View File
@@ -6,8 +6,9 @@ from sglang.test.performance_test_runner import PerformanceTestParams
from sglang.test.run_combined_tests import run_combined_tests from sglang.test.run_combined_tests import run_combined_tests
from sglang.test.test_utils import ModelLaunchSettings from sglang.test.test_utils import ModelLaunchSettings
# Runs on both H200 and B200 via nightly-8-gpu-common suite # Runs on both H200 and B200: registered once per runner_config below
register_cuda_ci(est_time=3600, suite="nightly-8-gpu-common", nightly=True) register_cuda_ci(est_time=2820, stage="nightly", runner_config="8-gpu-h200")
register_cuda_ci(est_time=2820, stage="nightly", runner_config="8-gpu-b200")
KIMI_K25_MODEL_PATH = "moonshotai/Kimi-K2.5" KIMI_K25_MODEL_PATH = "moonshotai/Kimi-K2.5"
test/registered/8-gpu-models/test_ling_2_6_flash.py
+2 -1
View File
@@ -12,7 +12,8 @@ from sglang.test.ci.ci_register import register_cuda_ci
from sglang.test.kits.eval_accuracy_kit import GSM8KMixin from sglang.test.kits.eval_accuracy_kit import GSM8KMixin
from sglang.test.server_fixtures.default_fixture import DefaultServerBase from sglang.test.server_fixtures.default_fixture import DefaultServerBase
register_cuda_ci(est_time=600, suite="nightly-8-gpu-common", nightly=True) register_cuda_ci(est_time=300, stage="nightly", runner_config="8-gpu-h200")
register_cuda_ci(est_time=300, stage="nightly", runner_config="8-gpu-b200")
class TestLing26Flash(GSM8KMixin, DefaultServerBase): class TestLing26Flash(GSM8KMixin, DefaultServerBase):
test/registered/8-gpu-models/test_llama4.py
+3 -2
View File
@@ -6,8 +6,9 @@ from sglang.test.performance_test_runner import PerformanceTestParams
from sglang.test.run_combined_tests import run_combined_tests from sglang.test.run_combined_tests import run_combined_tests
from sglang.test.test_utils import ModelLaunchSettings from sglang.test.test_utils import ModelLaunchSettings
# Runs on both H200 and B200 via nightly-8-gpu-common suite # Runs on both H200 and B200: registered once per runner_config below
register_cuda_ci(est_time=1800, suite="nightly-8-gpu-common", nightly=True) register_cuda_ci(est_time=1800, stage="nightly", runner_config="8-gpu-h200")
register_cuda_ci(est_time=1800, stage="nightly", runner_config="8-gpu-b200")
LLAMA4_MODEL_PATH = "meta-llama/Llama-4-Scout-17B-16E-Instruct" LLAMA4_MODEL_PATH = "meta-llama/Llama-4-Scout-17B-16E-Instruct"
test/registered/8-gpu-models/test_longcat_flash_lite_fp8.py
+1 -1
View File
@@ -6,7 +6,7 @@ from sglang.test.performance_test_runner import PerformanceTestParams
from sglang.test.run_combined_tests import run_combined_tests from sglang.test.run_combined_tests import run_combined_tests
from sglang.test.test_utils import ModelLaunchSettings from sglang.test.test_utils import ModelLaunchSettings
register_cuda_ci(est_time=1200, suite="nightly-8-gpu-h200", nightly=True) register_cuda_ci(est_time=1200, stage="nightly", runner_config="8-gpu-h200")
# LongCat-Flash-Lite-FP8 is the smallest member of the LongCat family # LongCat-Flash-Lite-FP8 is the smallest member of the LongCat family
# (~138 GB FP8 weights, hidden=3072, 14 layers, 256 routed + 128 zero # (~138 GB FP8 weights, hidden=3072, 14 layers, 256 routed + 128 zero
test/registered/8-gpu-models/test_minimax_m25.py
+3 -2
View File
@@ -6,8 +6,9 @@ from sglang.test.performance_test_runner import PerformanceTestParams
from sglang.test.run_combined_tests import run_combined_tests from sglang.test.run_combined_tests import run_combined_tests
from sglang.test.test_utils import ModelLaunchSettings from sglang.test.test_utils import ModelLaunchSettings
# Runs on both H200 and B200 via nightly-8-gpu-common suite # Runs on both H200 and B200: registered once per runner_config below
register_cuda_ci(est_time=1800, suite="nightly-8-gpu-common", nightly=True) register_cuda_ci(est_time=1860, stage="nightly", runner_config="8-gpu-h200")
register_cuda_ci(est_time=1860, stage="nightly", runner_config="8-gpu-b200")
MINIMAX_M25_MODEL_PATH = "MiniMaxAI/MiniMax-M2.5" MINIMAX_M25_MODEL_PATH = "MiniMaxAI/MiniMax-M2.5"
test/registered/8-gpu-models/test_mistral_large3.py
+3 -2
View File
@@ -8,9 +8,10 @@ from sglang.test.performance_test_runner import PerformanceTestParams
from sglang.test.run_combined_tests import run_combined_tests from sglang.test.run_combined_tests import run_combined_tests
from sglang.test.test_utils import ModelLaunchSettings, is_blackwell_system from sglang.test.test_utils import ModelLaunchSettings, is_blackwell_system
# Runs on both H200 and B200 via nightly-8-gpu-common suite # Runs on both H200 and B200: registered once per runner_config below
# Note: trtllm_mla backend may have hardware-specific behavior # Note: trtllm_mla backend may have hardware-specific behavior
register_cuda_ci(est_time=3000, suite="nightly-8-gpu-common", nightly=True) register_cuda_ci(est_time=3000, stage="nightly", runner_config="8-gpu-h200")
register_cuda_ci(est_time=3000, stage="nightly", runner_config="8-gpu-b200")
MISTRAL_LARGE3_FP8_MODEL_PATH = "mistralai/Mistral-Large-3-675B-Instruct-2512" MISTRAL_LARGE3_FP8_MODEL_PATH = "mistralai/Mistral-Large-3-675B-Instruct-2512"
MISTRAL_LARGE3_NVFP4_MODEL_PATH = "mistralai/Mistral-Large-3-675B-Instruct-2512-NVFP4" MISTRAL_LARGE3_NVFP4_MODEL_PATH = "mistralai/Mistral-Large-3-675B-Instruct-2512-NVFP4"
test/registered/8-gpu-models/test_nvidia_nemotron_3_super_nightly.py
+3 -2
View File
@@ -7,8 +7,9 @@ from sglang.test.performance_test_runner import PerformanceTestParams
from sglang.test.run_combined_tests import run_combined_tests from sglang.test.run_combined_tests import run_combined_tests
from sglang.test.test_utils import ModelLaunchSettings, is_blackwell_system from sglang.test.test_utils import ModelLaunchSettings, is_blackwell_system
# Runs on both Hopper and Blackwell via nightly-8-gpu-common suite # Runs on both Hopper and Blackwell: registered once per runner_config below
register_cuda_ci(est_time=5400, suite="nightly-8-gpu-common", nightly=True) register_cuda_ci(est_time=3360, stage="nightly", runner_config="8-gpu-h200")
register_cuda_ci(est_time=3360, stage="nightly", runner_config="8-gpu-b200")
NEMOTRON_3_SUPER_BF16_MODEL = "nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16" NEMOTRON_3_SUPER_BF16_MODEL = "nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16"
NEMOTRON_3_SUPER_NVFP4_MODEL = "nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4" NEMOTRON_3_SUPER_NVFP4_MODEL = "nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4"
test/registered/8-gpu-models/test_qwen35.py
+3 -2
View File
@@ -6,8 +6,9 @@ from sglang.test.performance_test_runner import PerformanceTestParams
from sglang.test.run_combined_tests import run_combined_tests from sglang.test.run_combined_tests import run_combined_tests
from sglang.test.test_utils import ModelLaunchSettings from sglang.test.test_utils import ModelLaunchSettings
# Runs on both H200 and B200 via nightly-8-gpu-common suite # Runs on both H200 and B200: registered once per runner_config below
register_cuda_ci(est_time=1800, suite="nightly-8-gpu-common", nightly=True) register_cuda_ci(est_time=3000, stage="nightly", runner_config="8-gpu-h200")
register_cuda_ci(est_time=3000, stage="nightly", runner_config="8-gpu-b200")
QWEN35_MODEL_PATH = "Qwen/Qwen3.5-397B-A17B-FP8" QWEN35_MODEL_PATH = "Qwen/Qwen3.5-397B-A17B-FP8"
test/registered/8-gpu-models/test_ring_2_5_1t.py
+2 -1
View File
@@ -5,7 +5,8 @@ from sglang.test.ci.ci_register import register_cuda_ci
from sglang.test.run_combined_tests import run_combined_tests from sglang.test.run_combined_tests import run_combined_tests
from sglang.test.test_utils import ModelLaunchSettings from sglang.test.test_utils import ModelLaunchSettings
register_cuda_ci(est_time=510, suite="nightly-8-gpu-common", nightly=True) register_cuda_ci(est_time=1680, stage="nightly", runner_config="8-gpu-h200")
register_cuda_ci(est_time=1680, stage="nightly", runner_config="8-gpu-b200")
RING_2_5_1T_MODEL_PATH = "inclusionAI/Ring-2.5-1T" RING_2_5_1T_MODEL_PATH = "inclusionAI/Ring-2.5-1T"
test/registered/attention/test_deepseek_v3_deterministic.py
+1 -1
View File
@@ -12,7 +12,7 @@ from sglang.test.test_deterministic_utils import (
TestDeterministicBase, TestDeterministicBase,
) )
register_cuda_ci(est_time=240, suite="nightly-1-gpu", nightly=True) register_cuda_ci(est_time=240, stage="nightly", runner_config="1-gpu-large")
DEEPSEEK_MODEL = "lmsys/sglang-ci-dsv3-test" DEEPSEEK_MODEL = "lmsys/sglang-ci-dsv3-test"
test/registered/attention/test_qwen3_next_deterministic.py
+1 -1
View File
@@ -12,7 +12,7 @@ from sglang.test.test_deterministic_utils import (
TestDeterministicBase, TestDeterministicBase,
) )
register_cuda_ci(est_time=200, suite="nightly-4-gpu", nightly=True) register_cuda_ci(est_time=200, stage="nightly", runner_config="4-gpu-h100")
QWEN3_NEXT = "Qwen/Qwen3-Next-80B-A3B-Instruct" QWEN3_NEXT = "Qwen/Qwen3-Next-80B-A3B-Instruct"
test/registered/backends/test_deepseek_v3_fp4_cutedsl_moe.py
+1 -1
View File
@@ -17,7 +17,7 @@ from sglang.test.test_utils import (
write_github_step_summary, write_github_step_summary,
) )
register_cuda_ci(est_time=450, suite="nightly-4-gpu-b200", nightly=True) register_cuda_ci(est_time=900, stage="nightly", runner_config="4-gpu-b200")
FULL_DEEPSEEK_V3_FP4_MODEL_PATH = "nvidia/DeepSeek-V3-0324-FP4" FULL_DEEPSEEK_V3_FP4_MODEL_PATH = "nvidia/DeepSeek-V3-0324-FP4"
SERVER_LAUNCH_TIMEOUT = 1000 SERVER_LAUNCH_TIMEOUT = 1000
test/registered/backends/test_flashinfer_trtllm_gen_moe_backend.py
+1 -1
View File
@@ -12,7 +12,7 @@ from sglang.test.test_utils import (
popen_launch_server, popen_launch_server,
) )
register_cuda_ci(est_time=800, suite="nightly-4-gpu-b200", nightly=True) register_cuda_ci(est_time=1770, stage="nightly", runner_config="4-gpu-b200")
class FlashinferTrtllmGenMoeBackendFP8Base: class FlashinferTrtllmGenMoeBackendFP8Base:
test/registered/cuda_graph/piecewise/test_pcg_with_speculative_decoding.py
+1 -1
View File
@@ -11,7 +11,7 @@ import unittest
from sglang.test.ci.ci_register import register_cuda_ci from sglang.test.ci.ci_register import register_cuda_ci
from sglang.test.server_fixtures.pcg_spec_fixture import PCGSpecBase from sglang.test.server_fixtures.pcg_spec_fixture import PCGSpecBase
register_cuda_ci(est_time=531, suite="nightly-4-gpu", nightly=True) register_cuda_ci(est_time=130, stage="nightly", runner_config="4-gpu-h100")
class TestPCGWithEAGLE3(PCGSpecBase, unittest.TestCase): class TestPCGWithEAGLE3(PCGSpecBase, unittest.TestCase):
test/registered/cuda_graph/piecewise/test_pcg_with_speculative_decoding_dflash.py
+1 -1
View File
@@ -15,7 +15,7 @@ from sglang.test.test_utils import (
CustomTestCase, CustomTestCase,
) )
register_cuda_ci(est_time=531, suite="nightly-1-gpu", nightly=True) register_cuda_ci(est_time=110, stage="nightly", runner_config="1-gpu-large")
class TestPCGWithDFlash(PCGSpecBase, CustomTestCase): class TestPCGWithDFlash(PCGSpecBase, CustomTestCase):
test/registered/cuda_graph/piecewise/test_pcg_with_speculative_decoding_extra.py
+1 -1
View File
@@ -8,7 +8,7 @@ import unittest
from sglang.test.ci.ci_register import register_cuda_ci from sglang.test.ci.ci_register import register_cuda_ci
from sglang.test.server_fixtures.pcg_spec_fixture import PCGSpecBase from sglang.test.server_fixtures.pcg_spec_fixture import PCGSpecBase
register_cuda_ci(est_time=531, suite="nightly-4-gpu", nightly=True) register_cuda_ci(est_time=450, stage="nightly", runner_config="4-gpu-h100")
class TestPCGWithMTP(PCGSpecBase, unittest.TestCase): class TestPCGWithMTP(PCGSpecBase, unittest.TestCase):
test/registered/cuda_graph/piecewise/test_piecewise_cuda_graph_support_1_gpu.py
+1 -1
View File
@@ -19,7 +19,7 @@ from sglang.test.test_utils import (
) )
# CI Registration # CI Registration
register_cuda_ci(est_time=180, suite="nightly-1-gpu", nightly=True) register_cuda_ci(est_time=250, stage="nightly", runner_config="1-gpu-large")
register_amd_ci(est_time=180, suite="stage-b-test-1-gpu-large-amd") register_amd_ci(est_time=180, suite="stage-b-test-1-gpu-large-amd")
test/registered/dcp/test_qwen3p5_triton_dcp.py
+1 -1
View File
@@ -20,7 +20,7 @@ register_amd_ci(
suite="nightly-amd-accuracy-8-gpu-mi35x-qwen35-triton-dcp", suite="nightly-amd-accuracy-8-gpu-mi35x-qwen35-triton-dcp",
nightly=True, nightly=True,
) )
register_cuda_ci(est_time=4800, suite="nightly-4-gpu-b200", nightly=True) register_cuda_ci(est_time=4800, stage="nightly", runner_config="4-gpu-b200")
QWEN35_MODEL_PATH = os.environ.get("QWEN3_5_MODEL_PATH", "Qwen/Qwen3.5-397B-A17B-FP8") QWEN35_MODEL_PATH = os.environ.get("QWEN3_5_MODEL_PATH", "Qwen/Qwen3.5-397B-A17B-FP8")
SERVER_LAUNCH_TIMEOUT = 4800 SERVER_LAUNCH_TIMEOUT = 4800
test/registered/debug_utils/test_crash_dump.py
+1 -1
View File
@@ -21,7 +21,7 @@ from sglang.test.test_utils import (
popen_launch_server, popen_launch_server,
) )
register_cuda_ci(est_time=40, suite="nightly-1-gpu", nightly=True) register_cuda_ci(est_time=70, stage="nightly", runner_config="1-gpu-large")
register_amd_ci(est_time=40, suite="nightly-amd-1-gpu", nightly=True) register_amd_ci(est_time=40, suite="nightly-amd-1-gpu", nightly=True)
register_cpu_ci(est_time=225, suite="base-c-test-cpu") register_cpu_ci(est_time=225, suite="base-c-test-cpu")
test/registered/debug_utils/test_dumper.py
+1 -1
View File
@@ -50,7 +50,7 @@ from sglang.test.test_utils import (
run_distributed_test, run_distributed_test,
) )
register_cuda_ci(est_time=30, suite="nightly-2-gpu", nightly=True) register_cuda_ci(est_time=30, stage="nightly", runner_config="2-gpu-large")
register_amd_ci(est_time=60, suite="nightly-amd", nightly=True) register_amd_ci(est_time=60, suite="nightly-amd", nightly=True)
test/registered/debug_utils/test_engine_dumper_comparator_e2e.py
+1 -1
View File
@@ -35,7 +35,7 @@ from sglang.test.test_utils import (
popen_launch_server, popen_launch_server,
) )
register_cuda_ci(est_time=300, suite="nightly-4-gpu", nightly=True) register_cuda_ci(est_time=290, stage="nightly", runner_config="4-gpu-h100")
register_amd_ci( register_amd_ci(
est_time=300, est_time=300,
suite="nightly-amd-4-gpu", suite="nightly-amd-4-gpu",
test/registered/debug_utils/test_nightly_precision_regression.py
+3 -1
View File
@@ -9,6 +9,8 @@ Env knobs:
SGLANG_PRECISION_COMMIT override sglang sha (7-40 hex) tagged on push SGLANG_PRECISION_COMMIT override sglang sha (7-40 hex) tagged on push
SGLANG_PRECISION_HF_REPO required HF dataset repo for cross-runner SGLANG_PRECISION_HF_REPO required HF dataset repo for cross-runner
baseline storage; see precision_baseline_store baseline storage; see precision_baseline_store
SGLANG_PRECISION_HF_TOKEN write token for that repo (not HF_TOKEN, which
carries the runner's gated-model read token)
""" """
from __future__ import annotations from __future__ import annotations
@@ -47,7 +49,7 @@ try:
except Exception: # pragma: no cover except Exception: # pragma: no cover
_hfs = None _hfs = None
register_cuda_ci(est_time=3600, suite="nightly-precision-8-gpu-h200", nightly=True) register_cuda_ci(est_time=3600, stage="nightly", runner_config="8-gpu-h200")
DEFAULT_MODELS_FOR_NIGHTLY_PRECISION = "zai-org/GLM-5.2-FP8" DEFAULT_MODELS_FOR_NIGHTLY_PRECISION = "zai-org/GLM-5.2-FP8"
DEFAULT_DIFF_THRESHOLD = 1e-3 DEFAULT_DIFF_THRESHOLD = 1e-3
test/registered/debug_utils/test_soft_watchdog.py
+1 -1
View File
@@ -17,7 +17,7 @@ from sglang.test.test_utils import (
popen_launch_server, popen_launch_server,
) )
register_cuda_ci(est_time=120, suite="nightly-1-gpu", nightly=True) register_cuda_ci(est_time=240, stage="nightly", runner_config="1-gpu-large")
register_amd_ci(est_time=120, suite="nightly-amd-1-gpu", nightly=True) register_amd_ci(est_time=120, suite="nightly-amd-1-gpu", nightly=True)
register_cpu_ci(est_time=622, suite="base-c-test-cpu") register_cpu_ci(est_time=622, suite="base-c-test-cpu")
test/registered/disaggregation/test_kimi_linear_pd_dcp4.py
+1 -1
View File
@@ -19,7 +19,7 @@ from sglang.test.test_utils import (
popen_launch_server, popen_launch_server,
) )
register_cuda_ci(est_time=1200, suite="nightly-8-gpu-b200", nightly=True) register_cuda_ci(est_time=450, stage="nightly", runner_config="8-gpu-b200")
KIMI_LINEAR_MODEL = "moonshotai/Kimi-Linear-48B-A3B-Instruct" KIMI_LINEAR_MODEL = "moonshotai/Kimi-Linear-48B-A3B-Instruct"
PHYSICAL_PAGE_SIZE = 64 PHYSICAL_PAGE_SIZE = 64
test/registered/ep/test_eplb_no_a2a.py
+1 -1
View File
@@ -23,7 +23,7 @@ from sglang.test.test_utils import (
try_cached_model, try_cached_model,
) )
register_cuda_ci(est_time=420, suite="nightly-eval-text-2-gpu", nightly=True) register_cuda_ci(est_time=200, stage="nightly", runner_config="2-gpu-large")
# 72 routed experts + 48 replicas = 120 physical, 60 per rank, so two thirds of # 72 routed experts + 48 replicas = 120 physical, 60 per rank, so two thirds of
# the routed (token, expert) pairs get double-counted when ranks disagree. At 24 # the routed (token, expert) pairs get double-counted when ranks disagree. At 24
test/registered/eval/test_text_models_gsm8k_eval.py
+1 -1
View File
@@ -23,7 +23,7 @@ from sglang.test.test_utils import (
# downloading on cache miss. Use a longer timeout than the default 600s. # downloading on cache miss. Use a longer timeout than the default 600s.
NIGHTLY_EVAL_SERVER_TIMEOUT = 1800 NIGHTLY_EVAL_SERVER_TIMEOUT = 1800
register_cuda_ci(est_time=3600, suite="nightly-eval-text-2-gpu", nightly=True) register_cuda_ci(est_time=2880, stage="nightly", runner_config="2-gpu-large")
MODEL_SCORE_THRESHOLDS = { MODEL_SCORE_THRESHOLDS = {
# sgl-eval (zero-shot chat, \boxed{}, math_verify grading). Thresholds are # sgl-eval (zero-shot chat, \boxed{}, math_verify grading). Thresholds are
test/registered/eval/test_vlms_mmmu_eval.py
+1 -1
View File
@@ -19,7 +19,7 @@ from sglang.test.test_utils import (
# Use a longer timeout than the default 600s. # Use a longer timeout than the default 600s.
NIGHTLY_EVAL_SERVER_TIMEOUT = 1800 NIGHTLY_EVAL_SERVER_TIMEOUT = 1800
register_cuda_ci(est_time=7200, suite="nightly-eval-vlm-2-gpu", nightly=True) register_cuda_ci(est_time=7200, stage="nightly", runner_config="2-gpu-large")
MODEL_THRESHOLDS = { MODEL_THRESHOLDS = {
# Conservative thresholds on 100 MMMU samples, especially for latency thresholds # Conservative thresholds on 100 MMMU samples, especially for latency thresholds
test/registered/gb300/test_deepseek_v4_pro_fp4.py
+1 -3
View File
@@ -6,9 +6,7 @@ from sglang.test.performance_test_runner import PerformanceTestParams
from sglang.test.run_combined_tests import run_combined_tests from sglang.test.run_combined_tests import run_combined_tests
from sglang.test.test_utils import ModelLaunchSettings from sglang.test.test_utils import ModelLaunchSettings
register_cuda_ci( register_cuda_ci(est_time=7200, stage="nightly", runner_config="4-gpu-gb300")
est_time=7200, suite="nightly-4-gpu-gb300-deepseek-v4-pro-fp4", nightly=True
)
MODEL_PATH = "deepseek-ai/DeepSeek-V4-Pro" MODEL_PATH = "deepseek-ai/DeepSeek-V4-Pro"
SERVER_LAUNCH_TIMEOUT = 3600 SERVER_LAUNCH_TIMEOUT = 3600
test/registered/gb300/test_glm52_nvfp4.py
+1 -1
View File
@@ -6,7 +6,7 @@ from sglang.test.performance_test_runner import PerformanceTestParams
from sglang.test.run_combined_tests import run_combined_tests from sglang.test.run_combined_tests import run_combined_tests
from sglang.test.test_utils import ModelLaunchSettings from sglang.test.test_utils import ModelLaunchSettings
register_cuda_ci(est_time=7200, suite="nightly-4-gpu-gb300-glm5-nvfp4", nightly=True) register_cuda_ci(est_time=2280, stage="nightly", runner_config="4-gpu-gb300")
MODEL_PATH = "nvidia/GLM-5.2-NVFP4" MODEL_PATH = "nvidia/GLM-5.2-NVFP4"
test/registered/gb300/test_kimi_k25_nvfp4.py
+1 -3
View File
@@ -6,9 +6,7 @@ from sglang.test.performance_test_runner import PerformanceTestParams
from sglang.test.run_combined_tests import run_combined_tests from sglang.test.run_combined_tests import run_combined_tests
from sglang.test.test_utils import ModelLaunchSettings from sglang.test.test_utils import ModelLaunchSettings
register_cuda_ci( register_cuda_ci(est_time=7200, stage="nightly", runner_config="4-gpu-gb300")
est_time=7200, suite="nightly-4-gpu-gb300-kimi-k25-nvfp4", nightly=True
)
MODEL_PATH = "nvidia/Kimi-K2.5-NVFP4" MODEL_PATH = "nvidia/Kimi-K2.5-NVFP4"
DRAFT_MODEL_PATH = "lightseekorg/kimi-k2.5-eagle3-mla" DRAFT_MODEL_PATH = "lightseekorg/kimi-k2.5-eagle3-mla"
test/registered/gb300/test_qwen35_fp8.py
+1 -1
View File
@@ -6,7 +6,7 @@ from sglang.test.performance_test_runner import PerformanceTestParams
from sglang.test.run_combined_tests import run_combined_tests from sglang.test.run_combined_tests import run_combined_tests
from sglang.test.test_utils import ModelLaunchSettings from sglang.test.test_utils import ModelLaunchSettings
register_cuda_ci(est_time=7200, suite="nightly-4-gpu-gb300-qwen35-fp8", nightly=True) register_cuda_ci(est_time=7200, stage="nightly", runner_config="4-gpu-gb300")
MODEL_PATH = "Qwen/Qwen3.5-397B-A17B-FP8" MODEL_PATH = "Qwen/Qwen3.5-397B-A17B-FP8"
test/registered/kernels/benchmark/kv_canary/bench_plan.py
+1 -1
View File
@@ -25,7 +25,7 @@ from sglang.kernels.ops.kv_canary.verify import VerifyPlan
from sglang.kernels.ops.kv_canary.write import WritePlan from sglang.kernels.ops.kv_canary.write import WritePlan
from sglang.test.ci.ci_register import register_amd_ci, register_cuda_ci from sglang.test.ci.ci_register import register_amd_ci, register_cuda_ci
register_cuda_ci(est_time=900, suite="nightly-kernel-1-gpu", nightly=True) register_cuda_ci(est_time=40, stage="nightly", runner_config="1-gpu-large")
# AMD mirrors the CUDA nightly registration (nightly-only, no per-PR suite). # AMD mirrors the CUDA nightly registration (nightly-only, no per-PR suite).
register_amd_ci(est_time=900, suite="nightly-amd-kernel-1-gpu", nightly=True) register_amd_ci(est_time=900, suite="nightly-amd-kernel-1-gpu", nightly=True)
test/registered/kernels/benchmark/kv_canary/bench_scatter_req_token_ids.py
+1 -1
View File
@@ -16,7 +16,7 @@ from sglang.kernels.ops.kv_canary.scatter_req_token_ids import (
) )
from sglang.test.ci.ci_register import register_amd_ci, register_cuda_ci from sglang.test.ci.ci_register import register_amd_ci, register_cuda_ci
register_cuda_ci(est_time=180, suite="nightly-kernel-1-gpu", nightly=True) register_cuda_ci(est_time=20, stage="nightly", runner_config="1-gpu-large")
# AMD mirrors the CUDA nightly registration (nightly-only, no per-PR suite). # AMD mirrors the CUDA nightly registration (nightly-only, no per-PR suite).
# Note: amd_ci_exec.sh sets SGLANG_IS_IN_CI, so this runs the CI-reduced range # Note: amd_ci_exec.sh sets SGLANG_IS_IN_CI, so this runs the CI-reduced range
# (_BS_AXIS_CI/_SEQ_LEN_AXIS_CI via get_benchmark_range), same as CUDA nightly. # (_BS_AXIS_CI/_SEQ_LEN_AXIS_CI via get_benchmark_range), same as CUDA nightly.
test/registered/kernels/benchmark/kv_canary/bench_verify.py
+1 -1
View File
@@ -32,7 +32,7 @@ from sglang.kernels.ops.kv_canary.verify import (
) )
from sglang.test.ci.ci_register import register_amd_ci, register_cuda_ci from sglang.test.ci.ci_register import register_amd_ci, register_cuda_ci
register_cuda_ci(est_time=900, suite="nightly-kernel-1-gpu", nightly=True) register_cuda_ci(est_time=20, stage="nightly", runner_config="1-gpu-large")
# AMD mirrors the CUDA nightly registration (nightly-only, no per-PR suite). # AMD mirrors the CUDA nightly registration (nightly-only, no per-PR suite).
# Note: amd_ci_exec.sh sets SGLANG_IS_IN_CI, so this runs the CI-reduced range # Note: amd_ci_exec.sh sets SGLANG_IS_IN_CI, so this runs the CI-reduced range
# (build_fast_matrix_cases via get_benchmark_range), same as CUDA nightly. # (build_fast_matrix_cases via get_benchmark_range), same as CUDA nightly.
test/registered/kernels/benchmark/kv_canary/bench_write.py
+1 -1
View File
@@ -30,7 +30,7 @@ from sglang.kernels.ops.kv_canary.verify import (
from sglang.kernels.ops.kv_canary.write import WritePlan, launch_canary_write_kernel from sglang.kernels.ops.kv_canary.write import WritePlan, launch_canary_write_kernel
from sglang.test.ci.ci_register import register_amd_ci, register_cuda_ci from sglang.test.ci.ci_register import register_amd_ci, register_cuda_ci
register_cuda_ci(est_time=900, suite="nightly-kernel-1-gpu", nightly=True) register_cuda_ci(est_time=20, stage="nightly", runner_config="1-gpu-large")
# AMD mirrors the CUDA nightly registration (nightly-only, no per-PR suite). # AMD mirrors the CUDA nightly registration (nightly-only, no per-PR suite).
# Note: amd_ci_exec.sh sets SGLANG_IS_IN_CI, so this runs the CI-reduced range # Note: amd_ci_exec.sh sets SGLANG_IS_IN_CI, so this runs the CI-reduced range
# (build_fast_matrix_cases via get_benchmark_range), same as CUDA nightly. # (build_fast_matrix_cases via get_benchmark_range), same as CUDA nightly.
test/registered/kernels/ops/activation/test_activation.py
+1 -1
View File
@@ -14,7 +14,7 @@ from sglang.test.ci.ci_register import register_amd_ci, register_cuda_ci
register_cuda_ci(est_time=20, stage="base-b-kernel-unit", runner_config="1-gpu-large") register_cuda_ci(est_time=20, stage="base-b-kernel-unit", runner_config="1-gpu-large")
# Nightly is not redundant here: it sets SGLANG_JIT_KERNEL_RUN_FULL_TESTS=1 to expand get_ci_test_range sweeps. # Nightly is not redundant here: it sets SGLANG_JIT_KERNEL_RUN_FULL_TESTS=1 to expand get_ci_test_range sweeps.
register_cuda_ci(est_time=30, suite="nightly-kernel-1-gpu", nightly=True) register_cuda_ci(est_time=20, stage="nightly", runner_config="1-gpu-large")
register_amd_ci(est_time=20, stage="jit-kernel-unit", runner_config="amd") register_amd_ci(est_time=20, stage="jit-kernel-unit", runner_config="amd")
test/registered/kernels/ops/attention/test_cute_dsl_fp8_paged_mqa_logits.py
+1 -1
View File
@@ -14,7 +14,7 @@ from sglang.srt.layers.attention.dsa.utils import (
from sglang.srt.utils import is_sm100_supported from sglang.srt.utils import is_sm100_supported
from sglang.test.ci.ci_register import register_cuda_ci from sglang.test.ci.ci_register import register_cuda_ci
register_cuda_ci(est_time=180, suite="nightly-4-gpu-b200", nightly=True) register_cuda_ci(est_time=180, stage="nightly", runner_config="4-gpu-b200")
BLOCK_KV = 64 BLOCK_KV = 64
HEAD_DIM = 128 HEAD_DIM = 128
test/registered/kernels/ops/attention/test_deepgemm_paged_mqa_logits.py
+1 -1
View File
@@ -17,7 +17,7 @@ from sglang.srt.layers.attention.dsa.utils import (
from sglang.srt.utils import is_sm90_supported, is_sm100_supported from sglang.srt.utils import is_sm90_supported, is_sm100_supported
from sglang.test.ci.ci_register import register_cuda_ci from sglang.test.ci.ci_register import register_cuda_ci
register_cuda_ci(est_time=40, suite="nightly-4-gpu-b200", nightly=True) register_cuda_ci(est_time=40, stage="nightly", runner_config="4-gpu-b200")
BLOCK_KV = 64 BLOCK_KV = 64
HEAD_DIM = 128 HEAD_DIM = 128
test/registered/kernels/ops/attention/test_rope.py
+1 -1
View File
@@ -10,7 +10,7 @@ from sglang.test.ci.ci_register import register_amd_ci, register_cuda_ci
register_cuda_ci(est_time=64, stage="base-b-kernel-unit", runner_config="1-gpu-large") register_cuda_ci(est_time=64, stage="base-b-kernel-unit", runner_config="1-gpu-large")
# Nightly is not redundant here: it sets SGLANG_JIT_KERNEL_RUN_FULL_TESTS=1 to expand get_ci_test_range sweeps. # Nightly is not redundant here: it sets SGLANG_JIT_KERNEL_RUN_FULL_TESTS=1 to expand get_ci_test_range sweeps.
register_cuda_ci(est_time=256, suite="nightly-kernel-1-gpu", nightly=True) register_cuda_ci(est_time=390, stage="nightly", runner_config="1-gpu-large")
register_amd_ci(est_time=64, suite="jit-kernel-unit-test-amd") register_amd_ci(est_time=64, suite="jit-kernel-unit-test-amd")
DEVICE = "cuda" DEVICE = "cuda"
test/registered/kernels/ops/communication/test_custom_all_reduce.py
+1 -5
View File
@@ -46,11 +46,7 @@ register_cuda_ci(
runner_config="8-gpu-h200", runner_config="8-gpu-h200",
) )
# Nightly is not redundant here: it sets SGLANG_JIT_KERNEL_RUN_FULL_TESTS=1 to expand get_ci_test_range sweeps. # Nightly is not redundant here: it sets SGLANG_JIT_KERNEL_RUN_FULL_TESTS=1 to expand get_ci_test_range sweeps.
register_cuda_ci( register_cuda_ci(est_time=110, stage="nightly", runner_config="8-gpu-h200")
est_time=300,
suite="nightly-kernel-8-gpu-h200",
nightly=True,
)
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# Test parameters # Test parameters
test/registered/kernels/ops/communication/test_symm_mem_all_gather.py
+1 -1
View File
@@ -36,7 +36,7 @@ from sglang.test.kernels.utils import multigpu_pytest_main
register_cuda_ci(est_time=240, stage="extra-b", runner_config="8-gpu-h200") register_cuda_ci(est_time=240, stage="extra-b", runner_config="8-gpu-h200")
# Nightly is not redundant here: it sets SGLANG_JIT_KERNEL_RUN_FULL_TESTS=1 to expand get_ci_test_range sweeps. # Nightly is not redundant here: it sets SGLANG_JIT_KERNEL_RUN_FULL_TESTS=1 to expand get_ci_test_range sweeps.
register_cuda_ci(est_time=240, suite="nightly-kernel-8-gpu-h200", nightly=True) register_cuda_ci(est_time=70, stage="nightly", runner_config="8-gpu-h200")
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# Test parameters # Test parameters
test/registered/kernels/ops/diffusion/test_qknorm_rope.py
+1 -1
View File
@@ -10,7 +10,7 @@ from sglang.test.ci.ci_register import register_cuda_ci
register_cuda_ci(est_time=44, stage="base-b-kernel-unit", runner_config="1-gpu-large") register_cuda_ci(est_time=44, stage="base-b-kernel-unit", runner_config="1-gpu-large")
# Nightly is not redundant here: it sets SGLANG_JIT_KERNEL_RUN_FULL_TESTS=1 to expand get_ci_test_range sweeps. # Nightly is not redundant here: it sets SGLANG_JIT_KERNEL_RUN_FULL_TESTS=1 to expand get_ci_test_range sweeps.
register_cuda_ci(est_time=176, suite="nightly-kernel-1-gpu", nightly=True) register_cuda_ci(est_time=220, stage="nightly", runner_config="1-gpu-large")
DEVICE = "cuda" DEVICE = "cuda"
DTYPE = torch.bfloat16 DTYPE = torch.bfloat16
test/registered/kernels/ops/diffusion/test_qwen_image_modulation.py
+1 -1
View File
@@ -14,7 +14,7 @@ from sglang.test.ci.ci_register import register_amd_ci, register_cuda_ci
register_cuda_ci(est_time=15, stage="base-b-kernel-unit", runner_config="1-gpu-large") register_cuda_ci(est_time=15, stage="base-b-kernel-unit", runner_config="1-gpu-large")
# Nightly is not redundant here: it sets SGLANG_JIT_KERNEL_RUN_FULL_TESTS=1 to expand get_ci_test_range sweeps. # Nightly is not redundant here: it sets SGLANG_JIT_KERNEL_RUN_FULL_TESTS=1 to expand get_ci_test_range sweeps.
register_cuda_ci(est_time=120, suite="nightly-kernel-1-gpu", nightly=True) register_cuda_ci(est_time=30, stage="nightly", runner_config="1-gpu-large")
register_amd_ci(est_time=30, suite="nightly-amd-kernel-1-gpu", nightly=True) register_amd_ci(est_time=30, suite="nightly-amd-kernel-1-gpu", nightly=True)
DEVICE = "cuda" DEVICE = "cuda"
test/registered/kernels/ops/diffusion/test_timestep_embedding.py
+1 -1
View File
@@ -18,7 +18,7 @@ from sglang.test.ci.ci_register import register_cuda_ci
register_cuda_ci(est_time=16, stage="base-b-kernel-unit", runner_config="1-gpu-large") register_cuda_ci(est_time=16, stage="base-b-kernel-unit", runner_config="1-gpu-large")
# Nightly is not redundant here: it sets SGLANG_JIT_KERNEL_RUN_FULL_TESTS=1 to expand get_ci_test_range sweeps. # Nightly is not redundant here: it sets SGLANG_JIT_KERNEL_RUN_FULL_TESTS=1 to expand get_ci_test_range sweeps.
register_cuda_ci(est_time=120, suite="nightly-kernel-1-gpu", nightly=True) register_cuda_ci(est_time=20, stage="nightly", runner_config="1-gpu-large")
CORRECTNESS_BATCH_SIZES = get_ci_test_range( CORRECTNESS_BATCH_SIZES = get_ci_test_range(
[1, 2, 8, 128, 256, 512, 1536, 2048, 4096, 11008, 16384], [1, 2, 8, 128, 256, 512, 1536, 2048, 4096, 11008, 16384],
test/registered/kernels/ops/diffusion/test_varlen_pack_pad.py
+1 -1
View File
@@ -17,7 +17,7 @@ from sglang.test.ci.ci_register import register_amd_ci, register_cuda_ci
register_cuda_ci(est_time=10, stage="base-b-kernel-unit", runner_config="1-gpu-large") register_cuda_ci(est_time=10, stage="base-b-kernel-unit", runner_config="1-gpu-large")
# Nightly is not redundant here: it sets SGLANG_JIT_KERNEL_RUN_FULL_TESTS=1 to expand get_ci_test_range sweeps. # Nightly is not redundant here: it sets SGLANG_JIT_KERNEL_RUN_FULL_TESTS=1 to expand get_ci_test_range sweeps.
register_cuda_ci(est_time=60, suite="nightly-kernel-1-gpu", nightly=True) register_cuda_ci(est_time=20, stage="nightly", runner_config="1-gpu-large")
register_amd_ci(est_time=15, suite="nightly-amd-kernel-1-gpu", nightly=True) register_amd_ci(est_time=15, suite="nightly-amd-kernel-1-gpu", nightly=True)
DEVICE = "cuda" DEVICE = "cuda"
test/registered/kernels/ops/diffusion/test_varlen_uspattn_equivalence.py
+1 -1
View File
@@ -31,7 +31,7 @@ from sglang.test.ci.ci_register import register_cuda_ci
register_cuda_ci(est_time=15, stage="base-b-kernel-unit", runner_config="1-gpu-large") register_cuda_ci(est_time=15, stage="base-b-kernel-unit", runner_config="1-gpu-large")
# Nightly is not redundant here: it sets SGLANG_JIT_KERNEL_RUN_FULL_TESTS=1 to expand get_ci_test_range sweeps. # Nightly is not redundant here: it sets SGLANG_JIT_KERNEL_RUN_FULL_TESTS=1 to expand get_ci_test_range sweeps.
register_cuda_ci(est_time=60, suite="nightly-kernel-1-gpu", nightly=True) register_cuda_ci(est_time=20, stage="nightly", runner_config="1-gpu-large")
DEVICE = "cuda" DEVICE = "cuda"
DTYPES = get_ci_test_range([torch.bfloat16, torch.float16], [torch.bfloat16]) DTYPES = get_ci_test_range([torch.bfloat16, torch.float16], [torch.bfloat16])
test/registered/kernels/ops/gemm/test_dsv3_router_gemm.py
+1 -1
View File
@@ -16,7 +16,7 @@ from sglang.test.ci.ci_register import register_cuda_ci
register_cuda_ci(est_time=37, stage="base-b-kernel-unit", runner_config="1-gpu-large") register_cuda_ci(est_time=37, stage="base-b-kernel-unit", runner_config="1-gpu-large")
# Nightly is not redundant here: it sets SGLANG_JIT_KERNEL_RUN_FULL_TESTS=1 to expand get_ci_test_range sweeps. # Nightly is not redundant here: it sets SGLANG_JIT_KERNEL_RUN_FULL_TESTS=1 to expand get_ci_test_range sweeps.
register_cuda_ci(est_time=148, suite="nightly-kernel-1-gpu", nightly=True) register_cuda_ci(est_time=110, stage="nightly", runner_config="1-gpu-large")
HIDDEN_DIMS = [1024, 4096, 5120, 6144, 7168] HIDDEN_DIMS = [1024, 4096, 5120, 6144, 7168]
ROUTER_GEMM_CASES = get_ci_test_range( ROUTER_GEMM_CASES = get_ci_test_range(
test/registered/kernels/ops/kimi_k3/test_collectives.py
+1 -1
View File
@@ -24,7 +24,7 @@ from sglang.test.ci.ci_register import register_cuda_ci
from sglang.test.kernels.utils import multigpu_pytest_main from sglang.test.kernels.utils import multigpu_pytest_main
register_cuda_ci(est_time=240, stage="base-c", runner_config="4-gpu-b200") register_cuda_ci(est_time=240, stage="base-c", runner_config="4-gpu-b200")
register_cuda_ci(est_time=480, suite="nightly-8-gpu-b200", nightly=True) register_cuda_ci(est_time=480, stage="nightly", runner_config="8-gpu-b200")
_HIDDEN_SIZE = 7168 _HIDDEN_SIZE = 7168
_GEMM_AR_K_TOTAL = 12288 _GEMM_AR_K_TOTAL = 12288
test/registered/kernels/ops/kvcache/test_store_cache.py
+1 -1
View File
@@ -10,7 +10,7 @@ from sglang.test.ci.ci_register import register_amd_ci, register_cuda_ci
register_cuda_ci(est_time=28, stage="base-b-kernel-unit", runner_config="1-gpu-large") register_cuda_ci(est_time=28, stage="base-b-kernel-unit", runner_config="1-gpu-large")
# Nightly is not redundant here: it sets SGLANG_JIT_KERNEL_RUN_FULL_TESTS=1 to expand get_ci_test_range sweeps. # Nightly is not redundant here: it sets SGLANG_JIT_KERNEL_RUN_FULL_TESTS=1 to expand get_ci_test_range sweeps.
register_cuda_ci(est_time=120, suite="nightly-kernel-1-gpu", nightly=True) register_cuda_ci(est_time=40, stage="nightly", runner_config="1-gpu-large")
register_amd_ci(est_time=55, stage="jit-kernel-unit", runner_config="amd") register_amd_ci(est_time=55, stage="jit-kernel-unit", runner_config="amd")
BS_LIST = [2**n for n in range(0, 15)] BS_LIST = [2**n for n in range(0, 15)]
test/registered/kernels/ops/layernorm/test_fused_add_rmsnorm.py
+1 -1
View File
@@ -9,7 +9,7 @@ from sglang.test.ci.ci_register import register_cuda_ci
register_cuda_ci(est_time=10, stage="base-b-kernel-unit", runner_config="1-gpu-large") register_cuda_ci(est_time=10, stage="base-b-kernel-unit", runner_config="1-gpu-large")
# Nightly is not redundant here: it sets SGLANG_JIT_KERNEL_RUN_FULL_TESTS=1 to expand get_ci_test_range sweeps. # Nightly is not redundant here: it sets SGLANG_JIT_KERNEL_RUN_FULL_TESTS=1 to expand get_ci_test_range sweeps.
register_cuda_ci(est_time=120, suite="nightly-kernel-1-gpu", nightly=True) register_cuda_ci(est_time=20, stage="nightly", runner_config="1-gpu-large")
def sglang_jit_fused_add_rmsnorm( def sglang_jit_fused_add_rmsnorm(
test/registered/kernels/ops/layernorm/test_qknorm.py
+1 -1
View File
@@ -10,7 +10,7 @@ from sglang.test.ci.ci_register import register_cuda_ci
register_cuda_ci(est_time=37, stage="base-b-kernel-unit", runner_config="1-gpu-large") register_cuda_ci(est_time=37, stage="base-b-kernel-unit", runner_config="1-gpu-large")
# Nightly is not redundant here: it sets SGLANG_JIT_KERNEL_RUN_FULL_TESTS=1 to expand get_ci_test_range sweeps. # Nightly is not redundant here: it sets SGLANG_JIT_KERNEL_RUN_FULL_TESTS=1 to expand get_ci_test_range sweeps.
register_cuda_ci(est_time=148, suite="nightly-kernel-1-gpu", nightly=True) register_cuda_ci(est_time=130, stage="nightly", runner_config="1-gpu-large")
def sglang_aot_qknorm( def sglang_aot_qknorm(
test/registered/kernels/ops/layernorm/test_qknorm_across_heads.py
+1 -1
View File
@@ -10,7 +10,7 @@ from sglang.test.ci.ci_register import register_cuda_ci
register_cuda_ci(est_time=15, stage="base-b-kernel-unit", runner_config="1-gpu-large") register_cuda_ci(est_time=15, stage="base-b-kernel-unit", runner_config="1-gpu-large")
# Nightly is not redundant here: it sets SGLANG_JIT_KERNEL_RUN_FULL_TESTS=1 to expand get_ci_test_range sweeps. # Nightly is not redundant here: it sets SGLANG_JIT_KERNEL_RUN_FULL_TESTS=1 to expand get_ci_test_range sweeps.
register_cuda_ci(est_time=120, suite="nightly-kernel-1-gpu", nightly=True) register_cuda_ci(est_time=20, stage="nightly", runner_config="1-gpu-large")
def sglang_jit_qknorm_across_heads( def sglang_jit_qknorm_across_heads(
test/registered/kernels/ops/layernorm/test_rmsnorm.py
+1 -1
View File
@@ -10,7 +10,7 @@ from sglang.test.ci.ci_register import register_amd_ci, register_cuda_ci
register_cuda_ci(est_time=45, stage="base-b-kernel-unit", runner_config="1-gpu-large") register_cuda_ci(est_time=45, stage="base-b-kernel-unit", runner_config="1-gpu-large")
# Nightly is not redundant here: it sets SGLANG_JIT_KERNEL_RUN_FULL_TESTS=1 to expand get_ci_test_range sweeps. # Nightly is not redundant here: it sets SGLANG_JIT_KERNEL_RUN_FULL_TESTS=1 to expand get_ci_test_range sweeps.
register_cuda_ci(est_time=240, suite="nightly-kernel-1-gpu", nightly=True) register_cuda_ci(est_time=160, stage="nightly", runner_config="1-gpu-large")
register_amd_ci(est_time=45, suite="jit-kernel-unit-test-amd") register_amd_ci(est_time=45, suite="jit-kernel-unit-test-amd")
test/registered/kernels/ops/layernorm/test_rmsnorm_hf.py
+1 -1
View File
@@ -15,7 +15,7 @@ from sglang.test.ci.ci_register import register_amd_ci, register_cuda_ci
register_cuda_ci(est_time=30, stage="base-b-kernel-unit", runner_config="1-gpu-large") register_cuda_ci(est_time=30, stage="base-b-kernel-unit", runner_config="1-gpu-large")
# Nightly is not redundant here: it sets SGLANG_JIT_KERNEL_RUN_FULL_TESTS=1 to expand get_ci_test_range sweeps. # Nightly is not redundant here: it sets SGLANG_JIT_KERNEL_RUN_FULL_TESTS=1 to expand get_ci_test_range sweeps.
register_cuda_ci(est_time=120, suite="nightly-kernel-1-gpu", nightly=True) register_cuda_ci(est_time=100, stage="nightly", runner_config="1-gpu-large")
register_amd_ci(est_time=30, stage="jit-kernel-unit", runner_config="amd") register_amd_ci(est_time=30, stage="jit-kernel-unit", runner_config="amd")
EPS = 1e-5 EPS = 1e-5
test/registered/kernels/ops/moe/test_fp4_moe.py
+1 -1
View File
@@ -16,7 +16,7 @@ from sglang.test.quant_ref_utils import (
dequantize_nvfp4_to_dtype, dequantize_nvfp4_to_dtype,
) )
register_cuda_ci(est_time=300, suite="nightly-4-gpu-b200", nightly=True) register_cuda_ci(est_time=300, stage="nightly", runner_config="4-gpu-b200")
if torch.cuda.get_device_capability() < (10, 0): if torch.cuda.get_device_capability() < (10, 0):
pytest.skip( pytest.skip(
test/registered/kernels/ops/moe/test_fused_topk_deepseek.py
+1 -1
View File
@@ -7,7 +7,7 @@ from sglang.srt.layers.moe.topk import biased_grouped_topk_gpu, biased_grouped_t
from sglang.srt.utils import get_device from sglang.srt.utils import get_device
from sglang.test.ci.ci_register import register_cuda_ci from sglang.test.ci.ci_register import register_cuda_ci
register_cuda_ci(est_time=2, suite="nightly-1-gpu", nightly=True) register_cuda_ci(est_time=40, stage="nightly", runner_config="1-gpu-large")
@pytest.mark.parametrize( @pytest.mark.parametrize(
test/registered/kernels/ops/moe/test_moe_align_block_size.py
+1 -1
View File
@@ -12,7 +12,7 @@ from sglang.test.ci.ci_register import register_cuda_ci
register_cuda_ci(est_time=28, stage="base-b-kernel-unit", runner_config="1-gpu-large") register_cuda_ci(est_time=28, stage="base-b-kernel-unit", runner_config="1-gpu-large")
# Nightly is not redundant here: it sets SGLANG_JIT_KERNEL_RUN_FULL_TESTS=1 to expand get_ci_test_range sweeps. # Nightly is not redundant here: it sets SGLANG_JIT_KERNEL_RUN_FULL_TESTS=1 to expand get_ci_test_range sweeps.
register_cuda_ci(est_time=120, suite="nightly-kernel-1-gpu", nightly=True) register_cuda_ci(est_time=50, stage="nightly", runner_config="1-gpu-large")
def ceil_div(a, b): def ceil_div(a, b):
test/registered/kernels/ops/quantization/test_per_tensor_quant_fp8.py
+1 -1
View File
@@ -13,7 +13,7 @@ from sglang.test.ci.ci_register import register_cuda_ci
register_cuda_ci(est_time=16, stage="base-b-kernel-unit", runner_config="1-gpu-large") register_cuda_ci(est_time=16, stage="base-b-kernel-unit", runner_config="1-gpu-large")
# Nightly is not redundant here: it sets SGLANG_JIT_KERNEL_RUN_FULL_TESTS=1 to expand get_ci_test_range sweeps. # Nightly is not redundant here: it sets SGLANG_JIT_KERNEL_RUN_FULL_TESTS=1 to expand get_ci_test_range sweeps.
register_cuda_ci(est_time=120, suite="nightly-kernel-1-gpu", nightly=True) register_cuda_ci(est_time=20, stage="nightly", runner_config="1-gpu-large")
try: try:
from sglang.srt.utils import is_hip from sglang.srt.utils import is_hip
test/registered/lora/test_lora_deepseek_v3_base_logprob_diff.py
+1 -4
View File
@@ -34,10 +34,7 @@ import sglang as sgl
from sglang.test.ci.ci_register import register_cuda_ci from sglang.test.ci.ci_register import register_cuda_ci
from sglang.test.test_utils import CustomTestCase from sglang.test.test_utils import CustomTestCase
register_cuda_ci( register_cuda_ci(est_time=1800, stage="nightly", runner_config="8-gpu-b200")
est_time=300,
suite="nightly-8-gpu-b200",
)
BASE_MODEL = "deepseek-ai/DeepSeek-V3.1-Base" BASE_MODEL = "deepseek-ai/DeepSeek-V3.1-Base"
LORA_HF_REPO = "yushengsu/lora-diff-DeepSeek-V3.1-Base" LORA_HF_REPO = "yushengsu/lora-diff-DeepSeek-V3.1-Base"
test/registered/lora/test_lora_eviction_policy.py
+1 -1
View File
@@ -27,7 +27,7 @@ from sglang.test.ci.ci_register import (
register_xpu_ci, register_xpu_ci,
) )
register_cuda_ci(est_time=200, suite="nightly-1-gpu", nightly=True) register_cuda_ci(est_time=10, stage="nightly", runner_config="1-gpu-large")
register_amd_ci(est_time=200, suite="nightly-amd-1-gpu", nightly=True) register_amd_ci(est_time=200, suite="nightly-amd-1-gpu", nightly=True)
register_cpu_ci(est_time=6, suite="base-c-test-cpu") register_cpu_ci(est_time=6, suite="base-c-test-cpu")
register_xpu_ci(est_time=10, suite="stage-a-test-1-gpu-xpu") register_xpu_ci(est_time=10, suite="stage-a-test-1-gpu-xpu")
test/registered/lora/test_lora_kimi_k25_logprob_diff.py
+1 -4
View File
@@ -34,10 +34,7 @@ import sglang as sgl
from sglang.test.ci.ci_register import register_cuda_ci from sglang.test.ci.ci_register import register_cuda_ci
from sglang.test.test_utils import CustomTestCase from sglang.test.test_utils import CustomTestCase
register_cuda_ci( register_cuda_ci(est_time=420, stage="nightly", runner_config="8-gpu-b200")
est_time=360,
suite="nightly-8-gpu-b200",
)
BASE_MODEL = "moonshotai/Kimi-K2.5" BASE_MODEL = "moonshotai/Kimi-K2.5"
LORA_HF_REPO = "yushengsu/lora-diff-Kimi-K2.5" LORA_HF_REPO = "yushengsu/lora-diff-Kimi-K2.5"
test/registered/lora/test_lora_openai_compatible.py
+1 -1
View File
@@ -27,7 +27,7 @@ from sglang.test.test_utils import (
popen_launch_server, popen_launch_server,
) )
register_cuda_ci(est_time=150, suite="nightly-1-gpu", nightly=True) register_cuda_ci(est_time=180, stage="nightly", runner_config="1-gpu-large")
register_amd_ci(est_time=150, suite="nightly-amd-1-gpu", nightly=True) register_amd_ci(est_time=150, suite="nightly-amd-1-gpu", nightly=True)
test/registered/model_loading/test_prefetch_checkpoints_multi_gpu.py
+1 -1
View File
@@ -4,7 +4,7 @@ import sglang as sgl
from sglang.test.ci.ci_register import register_cuda_ci from sglang.test.ci.ci_register import register_cuda_ci
from sglang.test.test_utils import CustomTestCase from sglang.test.test_utils import CustomTestCase
register_cuda_ci(est_time=300, suite="nightly-4-gpu") register_cuda_ci(est_time=300, stage="nightly", runner_config="4-gpu-h100")
PROMPTS = [ PROMPTS = [
"Hello, my name is", "Hello, my name is",
test/registered/model_loading/test_runai_model_loader.py
+1 -1
View File
@@ -5,7 +5,7 @@ from sglang.srt.utils.common import temp_set_env
from sglang.test.ci.ci_register import register_cuda_ci from sglang.test.ci.ci_register import register_cuda_ci
from sglang.test.test_utils import CustomTestCase from sglang.test.test_utils import CustomTestCase
register_cuda_ci(est_time=380, suite="nightly-1-gpu", nightly=True) register_cuda_ci(est_time=50, stage="nightly", runner_config="1-gpu-large")
TEST_GCS_MODEL = "gs://vertex-model-garden-public-us/codegemma/codegemma-2b/" TEST_GCS_MODEL = "gs://vertex-model-garden-public-us/codegemma/codegemma-2b/"
test/registered/models_e2e/test_dsa_glm52_cache_layer_split.py
+1 -5
View File
@@ -20,11 +20,7 @@ from sglang.test.server_fixtures.disaggregation_fixture import (
PDDisaggregationServerBase, PDDisaggregationServerBase,
) )
register_cuda_ci( register_cuda_ci(est_time=750, stage="nightly", runner_config="8-gpu-b200")
est_time=450,
suite="nightly-8-gpu-b200",
nightly=True,
)
class TestGLM52DSACacheLayerSplit(PDDisaggregationServerBase, GSM8KMixin): class TestGLM52DSACacheLayerSplit(PDDisaggregationServerBase, GSM8KMixin):
test/registered/models_e2e/test_kimi_linear_unified_memory.py
+1 -1
View File
@@ -32,7 +32,7 @@ from sglang.test.kits.eval_accuracy_kit import GSM8KMixin
from sglang.test.kits.prefix_cache_branching_kit import PrefixCacheBranchingMixin from sglang.test.kits.prefix_cache_branching_kit import PrefixCacheBranchingMixin
from sglang.test.server_fixtures.default_fixture import DefaultServerBase from sglang.test.server_fixtures.default_fixture import DefaultServerBase
register_cuda_ci(est_time=600, suite="nightly-4-gpu", nightly=True) register_cuda_ci(est_time=570, stage="nightly", runner_config="4-gpu-h100")
KIMI_LINEAR_MODEL = "moonshotai/Kimi-Linear-48B-A3B-Instruct" KIMI_LINEAR_MODEL = "moonshotai/Kimi-Linear-48B-A3B-Instruct"
test/registered/moe/test_hybrid_dp_ep_tp_mtp.py
+1 -1
View File
@@ -17,7 +17,7 @@ from sglang.test.test_utils import (
# 60 test classes testing hybrid parallelism configurations # 60 test classes testing hybrid parallelism configurations
# Each test launches server + runs MMLU eval (~90s per test) # Each test launches server + runs MMLU eval (~90s per test)
register_cuda_ci(est_time=5400, suite="weekly-8-gpu-h200", nightly=True) register_cuda_ci(est_time=8160, stage="weekly", runner_config="8-gpu-h200")
class Test00(CustomTestCase): class Test00(CustomTestCase):
test/registered/perf/test_text_models_perf.py
+1 -1
View File
@@ -9,7 +9,7 @@ from sglang.test.test_utils import (
parse_models, parse_models,
) )
register_cuda_ci(est_time=3600, suite="nightly-perf-text-2-gpu", nightly=True) register_cuda_ci(est_time=450, stage="nightly", runner_config="2-gpu-large")
RESULT_DIR = "performance_results_text_models" RESULT_DIR = "performance_results_text_models"
test/registered/perf/test_vlms_perf.py
+1 -1
View File
@@ -11,7 +11,7 @@ from sglang.test.test_utils import (
parse_models, parse_models,
) )
register_cuda_ci(est_time=7200, suite="nightly-perf-vlm-2-gpu", nightly=True) register_cuda_ci(est_time=750, stage="nightly", runner_config="2-gpu-large")
RESULT_DIR = "performance_results_vlms" RESULT_DIR = "performance_results_vlms"
test/registered/radix_cache/unified_radix_tree/test_unified_radix_cache_kl_mimo.py
+1 -1
View File
@@ -23,7 +23,7 @@ MIMO_LAUNCH_TIMEOUT = 3600
# MiMo V2.5 is pre-cached on the eight-H200 runner. The H200-only nightly suite # MiMo V2.5 is pre-cached on the eight-H200 runner. The H200-only nightly suite
# exercises the asymmetric MHA host pool end to end without adding PR CI cost. # exercises the asymmetric MHA host pool end to end without adding PR CI cost.
register_cuda_ci(est_time=1200, suite="nightly-8-gpu-h200", nightly=True) register_cuda_ci(est_time=270, stage="nightly", runner_config="8-gpu-h200")
class TestUnifiedMiMoHiCacheLoadBackKL(CustomTestCase): class TestUnifiedMiMoHiCacheLoadBackKL(CustomTestCase):
test/registered/radix_cache/unified_radix_tree/test_unified_radix_cache_kl_nightly.py
+1 -1
View File
@@ -26,7 +26,7 @@ from sglang.test.test_utils import (
GLM5_MODEL = "zai-org/GLM-5.2-FP8" GLM5_MODEL = "zai-org/GLM-5.2-FP8"
GLM5_LAUNCH_TIMEOUT = 3600 GLM5_LAUNCH_TIMEOUT = 3600
register_cuda_ci(est_time=900, suite="nightly-8-gpu-h200", nightly=True) register_cuda_ci(est_time=690, stage="nightly", runner_config="8-gpu-h200")
class AccuracyTwoPassMixin: class AccuracyTwoPassMixin:
test/registered/rl/test_weight_checker_e2e.py
+1 -1
View File
@@ -34,7 +34,7 @@ from sglang.test.test_utils import (
popen_launch_server, popen_launch_server,
) )
register_cuda_ci(est_time=150, suite="nightly-1-gpu", nightly=True) register_cuda_ci(est_time=60, stage="nightly", runner_config="1-gpu-large")
_MODEL_NAME = "Qwen/Qwen3-0.6B" _MODEL_NAME = "Qwen/Qwen3-0.6B"
# We address the up half via the HF-style unfused name "up_proj.weight". sglang's # We address the up half via the HF-style unfused name "up_proj.weight". sglang's
test/registered/scheduler/test_routing_key_scheduling.py
+1 -1
View File
@@ -20,7 +20,7 @@ from sglang.test.test_utils import (
popen_launch_server, popen_launch_server,
) )
register_cuda_ci(est_time=120, suite="nightly-1-gpu", nightly=True) register_cuda_ci(est_time=50, stage="nightly", runner_config="1-gpu-large")
register_amd_ci(est_time=120, suite="nightly-amd-1-gpu", nightly=True) register_amd_ci(est_time=120, suite="nightly-amd-1-gpu", nightly=True)
register_cpu_ci(est_time=184, suite="base-c-test-cpu") register_cpu_ci(est_time=184, suite="base-c-test-cpu")
test/registered/unit/batch_invariant_ops/test_batch_invariant_ops.py
+1 -1
View File
@@ -11,7 +11,7 @@ from sglang.test.test_utils import CustomTestCase
# Note: MI300 (gfx942) has 64KB shared memory limit but kernel needs 66KB # Note: MI300 (gfx942) has 64KB shared memory limit but kernel needs 66KB
# MI35x (gfx950/CDNA4) may have different limits - testing on MI35x only # MI35x (gfx950/CDNA4) may have different limits - testing on MI35x only
register_cuda_ci(est_time=10, suite="nightly-1-gpu", nightly=True) register_cuda_ci(est_time=20, stage="nightly", runner_config="1-gpu-large")
register_amd_ci(est_time=10, suite="nightly-amd-1-gpu-mi35x", nightly=True) register_amd_ci(est_time=10, suite="nightly-amd-1-gpu-mi35x", nightly=True)
device_type = getattr(torch.accelerator.current_accelerator(), "type", "cpu") device_type = getattr(torch.accelerator.current_accelerator(), "type", "cpu")
test/registered/vlm/test_encoder_dp.py
+1 -1
View File
@@ -7,7 +7,7 @@ from sglang.test.ci.ci_register import register_amd_ci, register_cuda_ci
from sglang.test.kits.mmmu_vlm_kit import MMMUMultiModelTestBase from sglang.test.kits.mmmu_vlm_kit import MMMUMultiModelTestBase
from sglang.test.test_utils import is_in_ci from sglang.test.test_utils import is_in_ci
register_cuda_ci(est_time=500, suite="nightly-4-gpu", nightly=True) register_cuda_ci(est_time=510, stage="nightly", runner_config="4-gpu-h100")
register_amd_ci(est_time=500, suite="nightly-amd-4-gpu", nightly=True) register_amd_ci(est_time=500, suite="nightly-amd-4-gpu", nightly=True)
MODELS = [ MODELS = [
test/run_suite.py
+33 -40
View File
@@ -121,34 +121,16 @@ PER_COMMIT_SUITES = {
# Nightly test suites (run nightly, organized by GPU configuration) # Nightly test suites (run nightly, organized by GPU configuration)
NIGHTLY_SUITES = { NIGHTLY_SUITES = {
HWBackend.CUDA: [ HWBackend.CUDA: [
"nightly-1-gpu", # `stage="nightly"` + a runner_config, same `{stage}-test-{runner_config}`
"nightly-2-gpu", # shape as the per-commit suites. No `nightly=True`: the stage name
"nightly-4-gpu", # carries the cadence; only the legacy suites below still need the flag.
"nightly-4-gpu-b200", "nightly-test-1-gpu-large",
"nightly-8-gpu", "nightly-test-2-gpu-large",
"nightly-8-gpu-h200", "nightly-test-4-gpu-h100",
"nightly-8-gpu-h20", "nightly-test-4-gpu-b200",
"nightly-8-gpu-b200", "nightly-test-4-gpu-gb300",
"nightly-8-gpu-h200-basic", # Basic tests for large models on H200 "nightly-test-8-gpu-h200",
"nightly-8-gpu-b200-basic", # Basic tests for large models on B200 "nightly-test-8-gpu-b200",
"nightly-8-gpu-common", # Common tests that run on both H200 and B200
"nightly-kernel-1-gpu",
"nightly-kernel-8-gpu-h200",
# Eval and perf suites (2-gpu)
"nightly-eval-text-2-gpu",
"nightly-eval-vlm-2-gpu",
"nightly-perf-text-2-gpu",
"nightly-perf-vlm-2-gpu",
# GB300 (4x GB300 NVL4) nightly suites
"nightly-4-gpu-gb300",
"nightly-4-gpu-gb300-deepseek-v4-pro-fp4",
"nightly-4-gpu-gb300-glm5-nvfp4",
"nightly-4-gpu-gb300-kimi-k25",
"nightly-4-gpu-gb300-kimi-k25-nvfp4",
"nightly-4-gpu-gb300-qwen35-fp8",
"nightly-4-gpu-gb300-qwen35-nvfp4",
# Nightly precision regression (per-layer hidden state comparison)
"nightly-precision-8-gpu-h200",
], ],
HWBackend.AMD: [ HWBackend.AMD: [
"nightly-amd", "nightly-amd",
@@ -194,7 +176,9 @@ OTHER_SUITES = {
], ],
HWBackend.CUDA: [ HWBackend.CUDA: [
"stress", "stress",
"weekly-8-gpu-h200", # `stage="weekly"` -- same shape. The three dicts group names for
# readability only; validation reads their union.
"weekly-test-8-gpu-h200",
], ],
} }
@@ -244,14 +228,10 @@ def filter_tests(
if t.backend == hw and t.effective_suite == suite and t.nightly == nightly if t.backend == hw and t.effective_suite == suite and t.nightly == nightly
] ]
valid_suites = ( # Union of all three dicts, not just the per-commit or nightly half:
NIGHTLY_SUITES.get(hw, []) if nightly else PER_COMMIT_SUITES.get(hw, []) # CUDA nightly suites are selected by name alone, without --nightly.
) if suite not in _valid_suites_by_backend().get(hw, set()):
print(f"Warning: Unknown suite {suite} for backend {hw.name}")
if suite not in valid_suites:
print(
f"Warning: Unknown suite {suite} for backend {hw.name}, nightly={nightly}"
)
enabled_tests = [t for t in ci_tests if t.disabled is None] enabled_tests = [t for t in ci_tests if t.disabled is None]
skipped_tests = [t for t in ci_tests if t.disabled is not None] skipped_tests = [t for t in ci_tests if t.disabled is not None]
@@ -369,9 +349,11 @@ def run_a_suite(args):
pretty_print_tests(args, ci_tests, skipped_tests) pretty_print_tests(args, ci_tests, skipped_tests)
# None hands the per-file budget over to est_time (see run_unittest_files).
timeout = None if args.timeout_from_est_time else args.timeout_per_file
# Add extra timeout when retry is enabled # Add extra timeout when retry is enabled
timeout = args.timeout_per_file if timeout is not None and args.enable_retry:
if args.enable_retry:
timeout += args.retry_timeout_increase timeout += args.retry_timeout_increase
return run_unittest_files( return run_unittest_files(
@@ -399,7 +381,10 @@ def main():
parser.add_argument( parser.add_argument(
"--nightly", "--nightly",
action="store_true", action="store_true",
help="Run nightly tests instead of per-commit tests.", help=(
"Include tests registered with nightly=True (AMD/CPU/NPU). CUDA "
"scheduled suites are selected by name and take no flag."
),
) )
parser.add_argument( parser.add_argument(
"--timeout-per-file", "--timeout-per-file",
@@ -407,6 +392,14 @@ def main():
default=1200, default=1200,
help="The time limit for running one file in seconds (default: 1200).", help="The time limit for running one file in seconds (default: 1200).",
) )
parser.add_argument(
"--timeout-from-est-time",
action="store_true",
help=(
"Derive each file's time limit from its own est_time instead of "
"the flat --timeout-per-file, for suites mixing fast and slow tests."
),
)
parser.add_argument( parser.add_argument(
"--continue-on-error", "--continue-on-error",
action="store_true", action="store_true",