diff --git a/.claude/skills/add-jit-kernel/SKILL.md b/.claude/skills/add-jit-kernel/SKILL.md index fef6a1d3d..9789ce7fd 100644 --- a/.claude/skills/add-jit-kernel/SKILL.md +++ b/.claude/skills/add-jit-kernel/SKILL.md @@ -489,7 +489,7 @@ if torch.cuda.get_device_capability()[0] < 9: JIT kernel correctness tests and benchmarks live under `test/registered/kernels/ops//` and `test/registered/kernels/benchmark//`, mirroring the wrapper's group under `python/sglang/kernels/ops/` (NOT inside the `sglang` package -- a `register_*_ci(...)` call anywhere under `python/sglang/` is rejected by the `check-no-registered-tests-in-package` pre-commit hook). Only their test-only helpers (e.g. `benchmark/marker.py`) stay alongside the kernel source under `python/sglang/kernels/jit/` and are imported by absolute path. **CI does not run `pytest` in those directories directly.** The unified runner `test/run_suite.py` discovers every `test_*.py` and `bench_*.py` under `test/registered/`, collects `register_*_ci(...)` calls by **statically parsing each file's AST**, and executes the selected suite. Every test file must register at least one CUDA entry or the collector fails its sanity check. - **PR / per-commit CUDA suites** (see `test/run_suite.py` → `PER_COMMIT_SUITES`): JIT unit tests use `base-b-kernel-unit-test-1-gpu-large` on H100 and `base-b-kernel-unit-test-4-gpu-b200` on B200/SM100 paths (see `.github/workflows/pr-test-jit-kernel.yml`). Multi-GPU JIT tests use `base-b-kernel-unit-test-8-gpu-h200`. -- **Nightly kernel suite**: `nightly-kernel-1-gpu` with `--nightly` — typically used with `SGLANG_JIT_KERNEL_RUN_FULL_TESTS=1` in CI for expanded parameter grids (see `python/sglang/kernels/jit/utils/common.py` → `should_run_full_tests` / `get_ci_test_range`). Wired in `.github/workflows/nightly-test-nvidia.yml` (e.g. `python3 run_suite.py --hw cuda --suite nightly-kernel-1-gpu --nightly --continue-on-error`). +- **Nightly kernel suite**: register with `stage="nightly"` plus the `runner_config` of the machine it needs (e.g. `1-gpu-large`), giving the `nightly-test-1-gpu-large` suite. `.github/workflows/nightly-test-nvidia.yml` sets `SGLANG_JIT_KERNEL_RUN_FULL_TESTS=1` for the whole nightly run, so the expanded parameter grids apply automatically (see `python/sglang/kernels/jit/utils/common.py` → `should_run_full_tests` / `get_ci_test_range`). There is no separate kernel-only nightly job: every nightly test on one machine type shares that machine's suite. Registration pattern (module level, **literal** `est_time`, `stage`, and `runner_config` values — required for AST parsing): @@ -499,12 +499,12 @@ from sglang.test.ci.ci_register import register_cuda_ci register_cuda_ci(est_time=30, stage="base-b-kernel-unit", runner_config="1-gpu-large") # Optional B200/SM100 registration for tests that cover Blackwell-specific code paths # register_cuda_ci(est_time=30, stage="base-b-kernel-unit", runner_config="4-gpu-b200") -# Optional second registration: same file also listed under the nightly kernel suite -# (nightly suites use the legacy single-string suite=, not stage/runner_config) -# register_cuda_ci(est_time=120, suite="nightly-kernel-1-gpu", nightly=True) +# Optional second registration: same file also runs nightly, same form, +# stage is just "nightly" there (and no `nightly=True`) +# register_cuda_ci(est_time=120, stage="nightly", runner_config="1-gpu-large") ``` -CI generates the suite name as `{stage}-test-{runner_config}`, so `stage="base-b-kernel-unit", runner_config="1-gpu-large"` becomes the `base-b-kernel-unit-test-1-gpu-large` suite you pass to `run_suite.py` below — don't put the `-test-` infix in `register_cuda_ci`. The single-string `suite=` form is only for nightly/stress/weekly suites. +CI generates the suite name as `{stage}-test-{runner_config}`, so `stage="base-b-kernel-unit", runner_config="1-gpu-large"` becomes the `base-b-kernel-unit-test-1-gpu-large` suite you pass to `run_suite.py` below — don't put the `-test-` infix in `register_cuda_ci`. Nightly uses the same shape with `stage="nightly"`; the single-string `suite=` form is left only for `stress` and non-CUDA pools. Keep `est_time`, `stage`, `runner_config`, and `suite` as literal values. `run_suite.py` collects them from the file AST, so computed values and helper wrappers can break CI discovery. @@ -661,7 +661,7 @@ cd test && python3 run_suite.py --hw cuda --suite base-b-kernel-benchmark-test-1 ## Troubleshooting -- **`No CI registry found in ...` from `run_suite.py`**: add a module-level `register_cuda_ci(...)` with literal `est_time`, `stage`, and `runner_config` (and optional `nightly=True`); starred args and non-literal values break AST collection +- **`No CI registry found in ...` from `run_suite.py`**: add a module-level `register_cuda_ci(...)` with literal `est_time`, `stage`, and `runner_config`; starred args and non-literal values break AST collection - **JIT compilation fails**: ensure the `.cuh` file is under `python/sglang/kernels/jit/csrc/`; reduce template argument combinations - **CUDA crash / illegal memory access**: `CUDA_LAUNCH_BLOCKING=1`; `compute-sanitizer --tool memcheck python ...` - **Unstable benchmark results**: `marker.do_bench` uses CUDA-graph-based timing by default; set `use_cuda_graph=False` only if the kernel can't be captured. `graph_clone_args` defaults to `"all"`; if you narrow it, it must still cover every *read* tensor — reusing a single buffer keeps it L2-hot and skews results. Keep *write* tensors in it too: they are what sets the rotation count, and a shared output buffer stays L2-hot the same way. diff --git a/.claude/skills/write-sglang-test/SKILL.md b/.claude/skills/write-sglang-test/SKILL.md index 2c22f99c4..1faaf2d9b 100644 --- a/.claude/skills/write-sglang-test/SKILL.md +++ b/.claude/skills/write-sglang-test/SKILL.md @@ -56,7 +56,7 @@ A per-commit suite name is **generated** from registration metadata as `{stage}- - **`runner_config`** — a runner-pool key from `scripts/ci/runner_configs.yml`, which maps it to the physical runner label (so `1-gpu-large` runs on `1-gpu-h100`). AMD/NPU use their own keys (e.g. `amd`). - **Suite** — `register_cuda_ci(stage="base-b", runner_config="1-gpu-small")` → `base-b-test-1-gpu-small`, the name you pass to `run_suite.py --suite`. The `-test-` is just the connector; never put it in `register_*_ci`. -> Legacy single-string `suite=` is only for suites that don't fit that shape — nightly/stress/weekly and some AMD/CPU/NPU pools (e.g. `suite="nightly-kernel-1-gpu", nightly=True`). Per-commit tests always use `stage=` + `runner_config=`. +> CUDA nightly uses the same shape with `stage="nightly"` (e.g. `stage="nightly", runner_config="1-gpu-large"` → `nightly-test-1-gpu-large`) and **no** `nightly=True` — the stage name carries the cadence, and setting the flag makes the test silently never run. Legacy single-string `suite=` is left only for `stress` and some AMD/CPU/NPU pools. ### All CI Suites @@ -113,13 +113,14 @@ A per-commit suite name is **generated** from registration metadata as `{stage}- #### Nightly -Nightly suites are listed in `NIGHTLY_SUITES` in [`test/run_suite.py`](../../../test/run_suite.py). They run via `nightly-test-nvidia.yml`, `nightly-test-amd.yml`, and `nightly-test-npu.yml`, not `pr-test.yml`. Examples: +Nightly suites are listed in `NIGHTLY_SUITES` in [`test/run_suite.py`](../../../test/run_suite.py). They run via `nightly-test-nvidia.yml`, `nightly-test-amd.yml`, and `nightly-test-npu.yml`, not `pr-test.yml`. -- `nightly-1-gpu` (CUDA) -- `nightly-kernel-1-gpu` (CUDA, JIT kernel full grids) -- `nightly-kernel-8-gpu-h200` (CUDA, multi-GPU JIT kernel nightly) -- `nightly-8-gpu-h200` (CUDA) -- `nightly-eval-vlm-2-gpu` (CUDA) +CUDA nightly suites are named `nightly-test-{runner_config}` — one per machine type, holding everything that runs nightly on it. There is no per-purpose split (kernel / eval / perf / precision all share their machine's suite); `auto_partition` splits the work. Examples: + +- `nightly-test-1-gpu-large` (CUDA) +- `nightly-test-2-gpu-large` (CUDA) +- `nightly-test-8-gpu-h200` (CUDA) +- `nightly-test-4-gpu-gb300` (CUDA) - `nightly-amd` (AMD) - `nightly-amd-8-gpu-mi35x` (AMD) - `nightly-1-npu-a3` (NPU) @@ -330,8 +331,8 @@ register_cuda_ci(est_time=80, suite="base-b-test-1-gpu-small") # Per-commit test (large 1-gpu, runs on H100) register_cuda_ci(est_time=120, suite="base-b-test-1-gpu-large") -# Nightly-only test -register_cuda_ci(est_time=200, suite="nightly-1-gpu", nightly=True) +# Nightly-only test (same shape as per-commit, stage is just "nightly") +register_cuda_ci(est_time=200, stage="nightly", runner_config="1-gpu-large") # Multi-backend test (only when testing backend-specific code paths) register_cuda_ci(est_time=80, suite="base-a-test-1-gpu-small") @@ -345,7 +346,7 @@ register_cuda_ci(est_time=80, suite="base-b-test-1-gpu-small", disabled="flaky - Parameters: - `est_time`: estimated runtime in seconds (used for CI partitioning) - `suite`: which CI suite to run in (see suite tables above) -- `nightly=True`: for nightly-only tests (default `False` = per-commit) +- `nightly=True`: legacy cadence flag, for non-CUDA nightly suites only. CUDA nightly uses `stage="nightly"` and must leave this unset - `disabled="reason"`: temporarily disable with explanation **Key principle**: Only add `register_amd_ci` / `register_npu_ci` when the test exercises backend-specific code paths. Common E2E tests just need `register_cuda_ci` — duplicating across backends wastes CI time. @@ -365,12 +366,12 @@ register_cuda_ci(est_time=120, stage="base-b-kernel-unit", runner_config="8-gpu- # Benchmarks in test/registered/jit/benchmark/ register_cuda_ci(est_time=6, stage="base-b-kernel-benchmark", runner_config="1-gpu-large") -# Optional nightly registration — nightly suites use the legacy single-string suite= -register_cuda_ci(est_time=120, suite="nightly-kernel-1-gpu", nightly=True) -register_cuda_ci(est_time=120, suite="nightly-kernel-8-gpu-h200", nightly=True) +# Optional nightly registration — same form, stage is just "nightly" +register_cuda_ci(est_time=120, stage="nightly", runner_config="1-gpu-large") +register_cuda_ci(est_time=120, stage="nightly", runner_config="8-gpu-h200") ``` -The `stage` + `runner_config` calls generate suites like `base-b-kernel-unit-test-1-gpu-large`; nightly keeps the legacy `suite=` string. Keep `est_time`, `stage`, `runner_config`, and `suite` as **literal values** — `run_suite.py` collects them by AST parsing. +Every call generates a suite named `{stage}-test-{runner_config}`, e.g. `base-b-kernel-unit-test-1-gpu-large` and `nightly-test-1-gpu-large`. Keep `est_time`, `stage`, `runner_config`, and `suite` as **literal values** — `run_suite.py` collects them by AST parsing. --- diff --git a/.github/workflows/_pr-test-stage.yml b/.github/workflows/_pr-test-stage.yml index 7005f1f2a..308869df3 100644 --- a/.github/workflows/_pr-test-stage.yml +++ b/.github/workflows/_pr-test-stage.yml @@ -63,6 +63,15 @@ on: type: boolean default: false + scheduled: + description: 'The caller is a nightly or weekly workflow, not a per-commit one. Switches on per-file timeouts derived from est_time, the full jit_kernel grids, metrics upload, and serial shards unless dispatched with full_parallel.' + type: boolean + default: false + job_timeout_minutes: + description: 'Job-level cap, covering install and warmup on top of run_timeout_minutes.' + type: string + default: '240' + # Mirror pr-test.yml top-level env. Reusable workflows do NOT inherit caller's # workflow-level env across the workflow_call boundary, so anything pr-test.yml # defines must be redeclared here for the called job to see the same context. @@ -74,6 +83,15 @@ env: SGLANG_ENABLE_ASYNC_ASSERT: ${{ startsWith(inputs.self_name, 'base-a') && 'false' || 'true' }} SGLANG_CUDA_COREDUMP: "1" SGLANG_JIT_DEEPGEMM_FAST_WARMUP: true + SGLANG_JIT_KERNEL_RUN_FULL_TESTS: ${{ inputs.scheduled && '1' || '0' }} + # is_h200_system() (test_utils.py) raises the server-launch ceiling from 600s + # to 3600s. Only the scheduled 8-gpu-h200 suites load models that need it; + # per-commit stages keep the short ceiling so a hung launch fails fast. + IS_H200: ${{ (inputs.scheduled && inputs.runner_config == '8-gpu-h200') && '1' || '0' }} + # Scheduled suites pull hundreds of GB of checkpoints; the hub's 10s etag + # default times out on a cold cache. + HF_HUB_DOWNLOAD_TIMEOUT: 300 + HF_HUB_ETAG_TIMEOUT: 300 SKIP_PR_TEST_HEALTH_CHECK: ${{ (fromJson(inputs.caller_inputs).skip_pr_test_health_check || fromJson(inputs.caller_inputs).test_parallel_dispatch || fromJson(inputs.caller_inputs).run_all_tests) && 'true' || 'false' }} PR_TEST_BYPASS_MAINTENANCE_ON_MAIN: ${{ github.ref == 'refs/heads/main' && 'true' || 'false' }} USE_VENV: false @@ -90,10 +108,10 @@ jobs: # $b200_runner (see runner_configs.py --map). rdma_devices is exported # below in a setup step via $GITHUB_ENV. runs-on: ${{ fromJson(fromJson(inputs.check_changes).runs_on_map)[inputs.runner_config] }} - timeout-minutes: 240 + timeout-minutes: ${{ fromJson(inputs.job_timeout_minutes) }} strategy: fail-fast: false - max-parallel: ${{ fromJson(inputs.partitions)[inputs.self_name].max_parallel }} + max-parallel: ${{ (inputs.scheduled && !fromJson(inputs.caller_inputs).full_parallel) && 1 || fromJson(inputs.partitions)[inputs.self_name].max_parallel }} matrix: partition: ${{ fromJson(inputs.partitions)[inputs.self_name].arr }} steps: @@ -165,16 +183,39 @@ jobs: curl --fail --silent --show-error --max-time 15 --retry 3 --retry-delay 2 \ "$URL" -o /tmp/partition-model.json + # Only test_nightly_precision_regression.py reads these; the others are + # inert wherever it does not run. + - name: Export precision baseline env + if: inputs.scheduled + env: + BASELINE_HF_TOKEN: ${{ secrets.HF_TOKEN_PRECISION_STORE }} + run: | + { + echo "SGLANG_PRECISION_BASELINE_DIR=/tmp/sglang_precision_baselines" + echo "SGLANG_PRECISION_HF_REPO=${{ vars.SGLANG_PRECISION_HF_REPO }}" + echo "SGLANG_PRECISION_HF_REVISION=${{ vars.SGLANG_PRECISION_HF_REVISION || 'main' }}" + echo "SGLANG_PRECISION_COMMIT=${{ github.sha }}" + echo "SGLANG_PRECISION_FORCE_UPDATE=${{ fromJson(inputs.caller_inputs).force_baseline_update && '1' || '0' }}" + echo "SGLANG_PRECISION_HF_TOKEN=${BASELINE_HF_TOKEN}" + } >> "$GITHUB_ENV" + - name: Run test timeout-minutes: ${{ fromJson(inputs.run_timeout_minutes) }} env: CONTINUE_ON_ERROR_FLAG: ${{ fromJson(inputs.check_changes).continue_on_error == 'true' && '--continue-on-error' || '' }} + RUNNER_LABELS: ${{ fromJson(fromJson(inputs.check_changes).runs_on_map)[inputs.runner_config] }} + GPU_CONFIG: ${{ inputs.runner_config }} + # Read only by test_runai_model_loader.py (nightly 1-gpu); inert elsewhere. + # Left unconditional because an `env:` key cannot be omitted, and an empty + # value would reach the runai streamer as a malformed limit. + RUNAI_STREAMER_MEMORY_LIMIT: 0 run: | cd test python3 run_suite.py --hw cuda --suite ${{ inputs.self_name }} \ --auto-partition-id ${{ matrix.partition }} \ --auto-partition-size ${{ fromJson(inputs.partitions)[inputs.self_name].size }} \ --partition-model-file /tmp/partition-model.json \ + ${{ inputs.scheduled && '--timeout-from-est-time' || '' }} \ ${{ inputs.timeout_per_file && format('--timeout-per-file {0}', inputs.timeout_per_file) || '' }} \ $CONTINUE_ON_ERROR_FLAG @@ -183,6 +224,26 @@ jobs: timeout-minutes: 10 run: python3 -m pytest -q ${{ inputs.extra_pytest_path }} + - name: Collect performance metrics + if: ${{ always() && inputs.scheduled }} + run: | + python3 scripts/ci/utils/save_metrics.py \ + --gpu-config ${{ inputs.runner_config }} \ + --partition ${{ matrix.partition }} \ + --run-id ${{ github.run_id }} \ + --output test/metrics-${{ inputs.runner_config }}-partition-${{ matrix.partition }}.json \ + --search-dir test/performance_results_8_gpu \ + --search-dir test + + - name: Upload partition metrics + if: ${{ always() && inputs.scheduled }} + uses: actions/upload-artifact@v4 + with: + name: metrics-${{ inputs.runner_config }}-partition-${{ matrix.partition }} + path: test/metrics-${{ inputs.runner_config }}-partition-${{ matrix.partition }}.json + retention-days: 5 + if-no-files-found: ignore + - uses: ./.github/actions/upload-cuda-coredumps if: failure() with: diff --git a/.github/workflows/nightly-test-nvidia.yml b/.github/workflows/nightly-test-nvidia.yml index 3901d0d14..5ff529fd7 100644 --- a/.github/workflows/nightly-test-nvidia.yml +++ b/.github/workflows/nightly-test-nvidia.yml @@ -1,3 +1,10 @@ +# Nightly CUDA tests. One job per runner_config, running that machine's +# `nightly-test-{runner_config}` suite -- a test reaches a machine by declaring +# `runner_config=` on register_cuda_ci, never by editing this file. +# +# Jobs go through the same _pr-test-stage.yml as the per-commit suites; what a +# scheduled run needs on top is its `scheduled` input, so runner resolution, +# rust-ext reuse and shard sizing are not reimplemented here. name: Nightly Test (Nvidia) on: @@ -5,32 +12,28 @@ on: - cron: '0 14 */2 * *' workflow_dispatch: inputs: - job_filter: - description: 'Select which job to run (leave empty or "all" to run all jobs)' + runner_filter: + description: 'Select which runner_config to run (leave empty or "all" to run all)' required: false type: choice default: 'all' options: - 'all' - - 'nightly-test-general-1-gpu-h100' - - 'nightly-test-general-4-gpu-h100' - - 'nightly-test-general-8-gpu-h200' - - 'nightly-test-general-8-gpu-h20' - - 'nightly-test-general-8-gpu-b200' - - 'nightly-test-text-accuracy-2-gpu-h100' - - 'nightly-test-text-perf-2-gpu-h100' - - 'nightly-test-vlm-accuracy-2-gpu-h100' - - 'nightly-test-vlm-perf-2-gpu-h100' - - 'nightly-test-perf-4-gpu-b200' - - 'nightly-test-perf-8-gpu-b200' - - 'nightly-test-specialized-8-gpu-b200' - - 'nightly-test-perf-4-gpu-gb300' - - 'nightly-test-kernel-1-gpu-h100' - - 'nightly-test-diffusion' - - 'nightly-test-kernel-8-gpu-h200' - - 'nightly-test-precision-8-gpu-h200' + - '1-gpu-large' + - '2-gpu-large' + - '4-gpu-h100' + - '4-gpu-b200' + - '4-gpu-gb300' + - '8-gpu-h200' + - '8-gpu-b200' + - 'diffusion' + full_parallel: + description: 'Run all shards of a job at once (faster, but competes with per-commit CI for machines). Off by default: one shard at a time.' + required: false + type: boolean + default: false force_baseline_update: - description: 'precision job only: refresh the rolling baseline instead of comparing (sets SGLANG_PRECISION_FORCE_UPDATE=1). Dispatch once after an intentional forward-path dtype/precision change stales the baseline; later scheduled runs compare against it and go green.' + description: 'Refresh the precision rolling baseline instead of comparing. Dispatch once after an intentional forward-path precision change stales it; later runs compare against the new one.' required: false type: boolean default: false @@ -41,499 +44,160 @@ on: required: false type: string default: '' - job_filter: - description: 'Select which job to run (leave empty or "all" to run all jobs)' + runner_filter: + description: 'Select which runner_config to run (leave empty or "all" to run all)' required: false type: string default: 'all' + full_parallel: + description: 'Run all shards of a job at once (faster, but competes with per-commit CI for machines). Off by default: one shard at a time.' + required: false + type: boolean + default: false concurrency: group: nightly-test-nvidia-${{ inputs.ref || github.ref }} cancel-in-progress: ${{ github.event_name != 'workflow_call' }} -env: - SGLANG_IS_IN_CI: true - SGLANG_ENABLE_ASYNC_ASSERT: true - SGLANG_CUDA_COREDUMP: "1" - HF_HUB_DOWNLOAD_TIMEOUT: 300 - HF_HUB_ETAG_TIMEOUT: 300 +permissions: + actions: write + contents: read + issues: read + pull-requests: read jobs: - # General tests - 1 GPU - nightly-test-general-1-gpu-h100: - if: github.repository == 'sgl-project/sglang' && (inputs.job_filter == '' || inputs.job_filter == 'all' || inputs.job_filter == 'nightly-test-general-1-gpu-h100') - runs-on: 1-gpu-h100 - steps: - - name: Checkout code - uses: actions/checkout@v4 - with: - ref: ${{ inputs.ref || github.ref }} - - - uses: ./.github/actions/check-maintenance - - - name: Install dependencies - run: | - bash scripts/ci/cuda/ci_install_dependency.sh - - - name: Run test - timeout-minutes: 60 - env: - RUNAI_STREAMER_MEMORY_LIMIT: 0 - run: | - cd test - python3 run_suite.py --hw cuda --suite nightly-1-gpu --nightly --continue-on-error - - - uses: ./.github/actions/upload-cuda-coredumps - if: failure() - - # JIT kernel full unit tests (expanded parameter ranges via SGLANG_JIT_KERNEL_RUN_FULL_TESTS) - nightly-test-kernel-1-gpu-h100: - if: github.repository == 'sgl-project/sglang' && (inputs.job_filter == '' || inputs.job_filter == 'all' || inputs.job_filter == 'nightly-test-kernel-1-gpu-h100') - runs-on: 1-gpu-h100 - timeout-minutes: 60 - env: - # Full jit_kernel test grids (see sglang.kernels.jit.utils.should_run_full_tests) - SGLANG_JIT_KERNEL_RUN_FULL_TESTS: "1" - # Match pr-test-jit-kernel workflow for consistent JIT warmup behavior - SGLANG_JIT_DEEPGEMM_FAST_WARMUP: true - # Allow maintenance bypass on default branch (same semantics as PR JIT workflow) - PR_TEST_BYPASS_MAINTENANCE_ON_MAIN: ${{ github.ref == 'refs/heads/main' && 'true' || 'false' }} - steps: - - name: Checkout code - uses: actions/checkout@v4 - with: - ref: ${{ inputs.ref || github.ref }} - - - uses: ./.github/actions/check-maintenance - - - name: Install dependencies - timeout-minutes: 20 - run: | - bash scripts/ci/cuda/ci_install_dependency.sh - - - name: Run jit kernel nightly suite - timeout-minutes: 60 - run: | - cd test - python3 run_suite.py --hw cuda --suite nightly-kernel-1-gpu --nightly --continue-on-error - - - uses: ./.github/actions/upload-cuda-coredumps - if: failure() - - nightly-test-kernel-8-gpu-h200: - if: github.repository == 'sgl-project/sglang' && (inputs.job_filter == '' || inputs.job_filter == 'all' || inputs.job_filter == 'nightly-test-kernel-8-gpu-h200') - runs-on: 8-gpu-h200 - timeout-minutes: 240 - env: - SGLANG_JIT_KERNEL_RUN_FULL_TESTS: "1" - SGLANG_JIT_DEEPGEMM_FAST_WARMUP: true - PR_TEST_BYPASS_MAINTENANCE_ON_MAIN: ${{ github.ref == 'refs/heads/main' && 'true' || 'false' }} - steps: - - name: Checkout code - uses: actions/checkout@v4 - with: - ref: ${{ inputs.ref || github.ref }} - - - uses: ./.github/actions/check-maintenance - - - name: Install dependencies - timeout-minutes: 20 - run: | - bash scripts/ci/cuda/ci_install_dependency.sh - - - name: Run multi-GPU jit kernel nightly suite - timeout-minutes: 90 - run: | - cd test - # Full grids run ~7x the in-CI parametrizations per world size; the - # default 1200s per-file budget only fits the reduced PR sweep. - python3 run_suite.py --hw cuda --suite nightly-kernel-8-gpu-h200 --nightly --continue-on-error --timeout-per-file 3600 - - - uses: ./.github/actions/upload-cuda-coredumps - if: failure() - - # General tests - 4 GPU H100 - nightly-test-general-4-gpu-h100: - if: github.repository == 'sgl-project/sglang' && (inputs.job_filter == '' || inputs.job_filter == 'all' || inputs.job_filter == 'nightly-test-general-4-gpu-h100') - runs-on: 4-gpu-h100 - steps: - - name: Checkout code - uses: actions/checkout@v4 - with: - ref: ${{ inputs.ref || github.ref }} - - - uses: ./.github/actions/check-maintenance - - - name: Install dependencies - run: | - bash scripts/ci/cuda/ci_install_dependency.sh - - - name: Run test - timeout-minutes: 60 - run: | - cd test - python3 run_suite.py --hw cuda --suite nightly-4-gpu --nightly --continue-on-error - - - uses: ./.github/actions/upload-cuda-coredumps - if: failure() - - # General tests - 8 GPU H200 - nightly-test-general-8-gpu-h200: - if: github.repository == 'sgl-project/sglang' && (inputs.job_filter == '' || inputs.job_filter == 'all' || inputs.job_filter == 'nightly-test-general-8-gpu-h200') - runs-on: 8-gpu-h200 - strategy: - fail-fast: false - max-parallel: 2 - matrix: - partition: [0, 1, 2, 3] - env: - RUNNER_LABELS: 8-gpu-h200 - steps: - - name: Checkout code - uses: actions/checkout@v4 - with: - ref: ${{ inputs.ref || github.ref }} - - - uses: ./.github/actions/check-maintenance - - - name: Install dependencies - run: | - bash scripts/ci/cuda/ci_install_dependency.sh - - - name: Run common 8-GPU model tests - if: always() - timeout-minutes: 300 - env: - GPU_CONFIG: "8-gpu-h200" - IS_H200: "1" - run: | - cd test - python3 run_suite.py --hw cuda --suite nightly-8-gpu-common --nightly --timeout-per-file=18000 --continue-on-error --auto-partition-id=${{ matrix.partition }} --auto-partition-size=4 - - - name: Run test - timeout-minutes: 30 - env: - GPU_CONFIG: "8-gpu-h200" - run: | - cd test - python3 run_suite.py --hw cuda --suite nightly-8-gpu-h200 --nightly --continue-on-error --auto-partition-id=${{ matrix.partition }} --auto-partition-size=4 - - - name: Collect performance metrics - if: always() - run: | - python3 scripts/ci/utils/save_metrics.py \ - --gpu-config 8-gpu-h200 \ - --partition ${{ matrix.partition }} \ - --run-id ${{ github.run_id }} \ - --output test/metrics-8gpu-h200-partition-${{ matrix.partition }}.json \ - --search-dir test/performance_results_8_gpu \ - --search-dir test - - - name: Upload partition metrics - if: always() - uses: actions/upload-artifact@v4 - with: - name: metrics-8gpu-h200-partition-${{ matrix.partition }} - path: test/metrics-8gpu-h200-partition-${{ matrix.partition }}.json - retention-days: 5 - if-no-files-found: ignore - - - uses: ./.github/actions/upload-cuda-coredumps - if: failure() - with: - artifact-suffix: ${{ matrix.partition }} - - # General tests - 8 GPU H20 - nightly-test-general-8-gpu-h20: - if: github.repository == 'sgl-project/sglang' && (inputs.job_filter == '' || inputs.job_filter == 'all' || inputs.job_filter == 'nightly-test-general-8-gpu-h20') - runs-on: 8-gpu-h20 - env: - SGLANG_CI_RDMA_ALL_DEVICES: "mlx5_1,mlx5_2,mlx5_3,mlx5_4" - steps: - - name: Checkout code - uses: actions/checkout@v4 - with: - ref: ${{ inputs.ref || github.ref }} - - - uses: ./.github/actions/check-maintenance - - - name: Install dependencies - run: | - bash scripts/ci/cuda/ci_install_dependency.sh - - - name: Run test - timeout-minutes: 30 - env: - GPU_CONFIG: "8-gpu-h20" - run: | - cd test - python3 run_suite.py --hw cuda --suite nightly-8-gpu-h20 --nightly --continue-on-error - - - uses: ./.github/actions/upload-cuda-coredumps - if: failure() - - # General tests - 8 GPU B200 - nightly-test-general-8-gpu-b200: - if: github.repository == 'sgl-project/sglang' && (inputs.job_filter == '' || inputs.job_filter == 'all' || inputs.job_filter == 'nightly-test-general-8-gpu-b200') - runs-on: 8-gpu-b200 - strategy: - fail-fast: false - max-parallel: 2 - matrix: - partition: [0, 1, 2, 3] - steps: - - name: Checkout code - uses: actions/checkout@v4 - with: - ref: ${{ inputs.ref || github.ref }} - - - uses: ./.github/actions/check-maintenance - - - name: Install dependencies - run: | - bash scripts/ci/cuda/ci_install_dependency.sh - - - name: Run common 8-GPU model tests - if: always() - timeout-minutes: 200 - env: - GPU_CONFIG: "8-gpu-b200" - run: | - cd test - python3 run_suite.py --hw cuda --suite nightly-8-gpu-common --nightly --timeout-per-file=12000 --continue-on-error --auto-partition-id=${{ matrix.partition }} --auto-partition-size=4 - - - name: Collect performance metrics - if: always() - run: | - python3 scripts/ci/utils/save_metrics.py \ - --gpu-config 8-gpu-b200 \ - --partition ${{ matrix.partition }} \ - --run-id ${{ github.run_id }} \ - --output test/metrics-8gpu-b200-partition-${{ matrix.partition }}.json \ - --search-dir test/performance_results_8_gpu \ - --search-dir test - - - name: Upload partition metrics - if: always() - uses: actions/upload-artifact@v4 - with: - name: metrics-8gpu-b200-partition-${{ matrix.partition }} - path: test/metrics-8gpu-b200-partition-${{ matrix.partition }}.json - retention-days: 5 - if-no-files-found: ignore - - - uses: ./.github/actions/upload-cuda-coredumps - if: failure() - with: - artifact-suffix: ${{ matrix.partition }} - - # Text model accuracy tests - nightly-test-text-accuracy-2-gpu-h100: - if: github.repository == 'sgl-project/sglang' && (inputs.job_filter == '' || inputs.job_filter == 'all' || inputs.job_filter == 'nightly-test-text-accuracy-2-gpu-h100') - runs-on: 2-gpu-h100 - steps: - - name: Checkout code - uses: actions/checkout@v4 - with: - ref: ${{ inputs.ref || github.ref }} - - - uses: ./.github/actions/check-maintenance - - - name: Install dependencies - run: | - bash scripts/ci/cuda/ci_install_dependency.sh - - - name: Run eval test for text models - timeout-minutes: 120 - run: | - cd test - python3 run_suite.py --hw cuda --suite nightly-eval-text-2-gpu --nightly --continue-on-error --timeout-per-file 4500 - - - uses: ./.github/actions/upload-cuda-coredumps - if: failure() - - # Text model performance tests - nightly-test-text-perf-2-gpu-h100: - if: github.repository == 'sgl-project/sglang' && (inputs.job_filter == '' || inputs.job_filter == 'all' || inputs.job_filter == 'nightly-test-text-perf-2-gpu-h100') - runs-on: 2-gpu-h100 - steps: - - name: Checkout code - uses: actions/checkout@v4 - with: - ref: ${{ inputs.ref || github.ref }} - - - uses: ./.github/actions/check-maintenance - - - name: Install dependencies - run: | - bash scripts/ci/cuda/ci_install_dependency.sh - - - name: Run performance test for text models - timeout-minutes: 30 - env: - GPU_CONFIG: "2-gpu-h100" - run: | - cd test - rm -rf performance_results_text_models/ - python3 run_suite.py --hw cuda --suite nightly-perf-text-2-gpu --nightly --continue-on-error --timeout-per-file 3600 - - - uses: ./.github/actions/upload-cuda-coredumps - if: failure() - - # VLM accuracy tests - nightly-test-vlm-accuracy-2-gpu-h100: - if: github.repository == 'sgl-project/sglang' && (inputs.job_filter == '' || inputs.job_filter == 'all' || inputs.job_filter == 'nightly-test-vlm-accuracy-2-gpu-h100') - runs-on: 2-gpu-h100 - steps: - - name: Checkout code - uses: actions/checkout@v4 - with: - ref: ${{ inputs.ref || github.ref }} - - - uses: ./.github/actions/check-maintenance - - - name: Install dependencies - run: | - bash scripts/ci/cuda/ci_install_dependency.sh - - - name: Run eval test for VLM models (fixed MMMU-100) - timeout-minutes: 120 - run: | - cd test - python3 run_suite.py --hw cuda --suite nightly-eval-vlm-2-gpu --nightly --continue-on-error --timeout-per-file 9000 - - - uses: ./.github/actions/upload-cuda-coredumps - if: failure() - - # VLM performance tests - nightly-test-vlm-perf-2-gpu-h100: - if: github.repository == 'sgl-project/sglang' && (inputs.job_filter == '' || inputs.job_filter == 'all' || inputs.job_filter == 'nightly-test-vlm-perf-2-gpu-h100') - runs-on: 2-gpu-h100 - steps: - - name: Checkout code - uses: actions/checkout@v4 - with: - ref: ${{ inputs.ref || github.ref }} - - - uses: ./.github/actions/check-maintenance - - - name: Install dependencies - run: | - bash scripts/ci/cuda/ci_install_dependency.sh - - - name: Run perf test for VLM models (MMMU) - timeout-minutes: 30 - env: - GPU_CONFIG: "2-gpu-h100" - run: | - cd test - rm -rf performance_results_vlms/ - python3 run_suite.py --hw cuda --suite nightly-perf-vlm-2-gpu --nightly --continue-on-error --timeout-per-file 3600 - - - uses: ./.github/actions/upload-cuda-coredumps - if: failure() - - # B200 Performance tests - 4 GPU - nightly-test-perf-4-gpu-b200: - if: github.repository == 'sgl-project/sglang' && (inputs.job_filter == '' || inputs.job_filter == 'all' || inputs.job_filter == 'nightly-test-perf-4-gpu-b200') - runs-on: 4-gpu-b200 - steps: - - name: Checkout code - uses: actions/checkout@v4 - with: - ref: ${{ inputs.ref || github.ref }} - - - uses: ./.github/actions/check-maintenance - - - name: Install dependencies - run: | - bash scripts/ci/cuda/ci_install_dependency.sh - - - name: Run test - timeout-minutes: 200 - env: - GPU_CONFIG: "4-gpu-b200" - run: | - cd test - python3 run_suite.py --hw cuda --suite nightly-4-gpu-b200 --nightly --continue-on-error --timeout-per-file 12000 - - - uses: ./.github/actions/upload-cuda-coredumps - if: failure() - - # GB300 (Grace-Blackwell NVL4) performance tests - 4 GPU (ARM64) - nightly-test-perf-4-gpu-gb300: - if: github.repository == 'sgl-project/sglang' && (inputs.job_filter == '' || inputs.job_filter == 'all' || inputs.job_filter == 'nightly-test-perf-4-gpu-gb300') - name: nightly-test-perf-4-gpu-gb300 (${{ matrix.model }}) - runs-on: 4-gpu-gb300-nightly - strategy: - fail-fast: false - matrix: - include: - - model: glm5-nvfp4 - suite: nightly-4-gpu-gb300-glm5-nvfp4 - - model: qwen35-fp8 - suite: nightly-4-gpu-gb300-qwen35-fp8 - - model: deepseek-v4-pro-fp4 - suite: nightly-4-gpu-gb300-deepseek-v4-pro-fp4 - - model: kimi-k25-nvfp4 - suite: nightly-4-gpu-gb300-kimi-k25-nvfp4 - steps: - - name: Checkout code - uses: actions/checkout@v4 - with: - ref: ${{ inputs.ref || github.ref }} - - - uses: ./.github/actions/check-maintenance - - - name: Install dependencies - run: | - bash scripts/ci/cuda/ci_install_dependency.sh - - - name: Run test - timeout-minutes: 600 - env: - GPU_CONFIG: "4-gpu-gb300" - run: | - cd test - python3 run_suite.py --hw cuda --suite ${{ matrix.suite }} --nightly --continue-on-error --timeout-per-file 7200 - - - uses: ./.github/actions/upload-cuda-coredumps - if: failure() - - # Specialized B200 tests - 8 GPU, for specific backends and configs - nightly-test-specialized-8-gpu-b200: - if: github.repository == 'sgl-project/sglang' && (inputs.job_filter == '' || inputs.job_filter == 'all' || inputs.job_filter == 'nightly-test-perf-8-gpu-b200' || inputs.job_filter == 'nightly-test-specialized-8-gpu-b200') - runs-on: 8-gpu-b200 - env: - RUNNER_LABELS: 8-gpu-b200 - steps: - - name: Checkout code - uses: actions/checkout@v4 - with: - ref: ${{ inputs.ref || github.ref }} - - - uses: ./.github/actions/check-maintenance - - - name: Install dependencies - run: | - bash scripts/ci/cuda/ci_install_dependency.sh - - - name: Run test - timeout-minutes: 60 - env: - GPU_CONFIG: "8-gpu-b200" - run: | - cd test - python3 run_suite.py --hw cuda --suite nightly-8-gpu-b200 --nightly --continue-on-error --timeout-per-file 2400 - - - uses: ./.github/actions/upload-cuda-coredumps - if: failure() - - # SGLang-Diffusion nightly benchmark + # run_all_tests skips the paths-filter, so main_package is 'true' and + # sgl_kernel stays empty: every test runs, and no job waits on a wheel this + # workflow never builds. pr_test_yml points back here so shard sizing reads + # this file's own run_timeout_minutes. + check-changes: + uses: ./.github/workflows/_pr-test-check-changes.yml + with: + git_ref: ${{ inputs.ref || '' }} + pr_test_yml: '.github/workflows/nightly-test-nvidia.yml' + run_all_tests: true + force_continue_on_error: true + secrets: inherit + + nightly-1-gpu-large: + needs: check-changes + if: github.repository == 'sgl-project/sglang' && (inputs.runner_filter == '' || inputs.runner_filter == 'all' || inputs.runner_filter == '1-gpu-large') + uses: ./.github/workflows/_pr-test-stage.yml + with: + self_name: nightly-test-1-gpu-large + runner_config: 1-gpu-large + check_changes: ${{ toJson(needs.check-changes.outputs) }} + caller_inputs: ${{ toJson(inputs) }} + partitions: ${{ needs.check-changes.outputs.partitions }} + run_timeout_minutes: '120' + job_timeout_minutes: '180' + scheduled: true + secrets: inherit + + nightly-2-gpu-large: + needs: check-changes + if: github.repository == 'sgl-project/sglang' && (inputs.runner_filter == '' || inputs.runner_filter == 'all' || inputs.runner_filter == '2-gpu-large') + uses: ./.github/workflows/_pr-test-stage.yml + with: + self_name: nightly-test-2-gpu-large + runner_config: 2-gpu-large + check_changes: ${{ toJson(needs.check-changes.outputs) }} + caller_inputs: ${{ toJson(inputs) }} + partitions: ${{ needs.check-changes.outputs.partitions }} + run_timeout_minutes: '240' + job_timeout_minutes: '300' + scheduled: true + secrets: inherit + + nightly-4-gpu-h100: + needs: check-changes + if: github.repository == 'sgl-project/sglang' && (inputs.runner_filter == '' || inputs.runner_filter == 'all' || inputs.runner_filter == '4-gpu-h100') + uses: ./.github/workflows/_pr-test-stage.yml + with: + self_name: nightly-test-4-gpu-h100 + runner_config: 4-gpu-h100 + check_changes: ${{ toJson(needs.check-changes.outputs) }} + caller_inputs: ${{ toJson(inputs) }} + partitions: ${{ needs.check-changes.outputs.partitions }} + run_timeout_minutes: '120' + job_timeout_minutes: '180' + scheduled: true + secrets: inherit + + nightly-4-gpu-b200: + needs: check-changes + if: github.repository == 'sgl-project/sglang' && (inputs.runner_filter == '' || inputs.runner_filter == 'all' || inputs.runner_filter == '4-gpu-b200') + uses: ./.github/workflows/_pr-test-stage.yml + with: + self_name: nightly-test-4-gpu-b200 + runner_config: 4-gpu-b200 + check_changes: ${{ toJson(needs.check-changes.outputs) }} + caller_inputs: ${{ toJson(inputs) }} + partitions: ${{ needs.check-changes.outputs.partitions }} + run_timeout_minutes: '150' + job_timeout_minutes: '210' + scheduled: true + secrets: inherit + + nightly-4-gpu-gb300: + needs: check-changes + if: github.repository == 'sgl-project/sglang' && (inputs.runner_filter == '' || inputs.runner_filter == 'all' || inputs.runner_filter == '4-gpu-gb300') + uses: ./.github/workflows/_pr-test-stage.yml + with: + self_name: nightly-test-4-gpu-gb300 + runner_config: 4-gpu-gb300 + check_changes: ${{ toJson(needs.check-changes.outputs) }} + caller_inputs: ${{ toJson(inputs) }} + partitions: ${{ needs.check-changes.outputs.partitions }} + run_timeout_minutes: '360' + job_timeout_minutes: '420' + scheduled: true + # aarch64: the rust-ext cache key is x86_64-only, same reason base-c skips it. + skip_prebuilt_rust_ext: true + secrets: inherit + + nightly-8-gpu-h200: + needs: check-changes + if: github.repository == 'sgl-project/sglang' && (inputs.runner_filter == '' || inputs.runner_filter == 'all' || inputs.runner_filter == '8-gpu-h200') + uses: ./.github/workflows/_pr-test-stage.yml + with: + self_name: nightly-test-8-gpu-h200 + runner_config: 8-gpu-h200 + check_changes: ${{ toJson(needs.check-changes.outputs) }} + caller_inputs: ${{ toJson(inputs) }} + partitions: ${{ needs.check-changes.outputs.partitions }} + run_timeout_minutes: '300' + job_timeout_minutes: '360' + scheduled: true + secrets: inherit + + nightly-8-gpu-b200: + needs: check-changes + if: github.repository == 'sgl-project/sglang' && (inputs.runner_filter == '' || inputs.runner_filter == 'all' || inputs.runner_filter == '8-gpu-b200') + uses: ./.github/workflows/_pr-test-stage.yml + with: + self_name: nightly-test-8-gpu-b200 + runner_config: 8-gpu-b200 + check_changes: ${{ toJson(needs.check-changes.outputs) }} + caller_inputs: ${{ toJson(inputs) }} + partitions: ${{ needs.check-changes.outputs.partitions }} + run_timeout_minutes: '360' + job_timeout_minutes: '420' + scheduled: true + secrets: inherit + + # Hand-written because it is not a registry suite: it drives run_comparison.py + # and publishes a dashboard rather than running test files. nightly-test-diffusion: - if: github.repository == 'sgl-project/sglang' && (inputs.job_filter == '' || inputs.job_filter == 'all' || inputs.job_filter == 'nightly-test-diffusion') + if: github.repository == 'sgl-project/sglang' && (inputs.runner_filter == '' || inputs.runner_filter == 'all' || inputs.runner_filter == 'diffusion') runs-on: 4-gpu-h100 timeout-minutes: 300 + env: + SGLANG_IS_IN_CI: true + SGLANG_ENABLE_ASYNC_ASSERT: true + SGLANG_CUDA_COREDUMP: "1" + HF_HUB_DOWNLOAD_TIMEOUT: 300 + HF_HUB_ETAG_TIMEOUT: 300 steps: - name: Checkout code uses: actions/checkout@v4 @@ -593,47 +257,19 @@ jobs: - uses: ./.github/actions/upload-cuda-coredumps if: failure() - # Nightly precision regression - per-layer hidden state comparison - nightly-test-precision-8-gpu-h200: - if: github.repository == 'sgl-project/sglang' && (inputs.job_filter == '' || inputs.job_filter == 'all' || inputs.job_filter == 'nightly-test-precision-8-gpu-h200') - runs-on: 8-gpu-h200 - steps: - - name: Checkout code - uses: actions/checkout@v4 - with: - ref: ${{ inputs.ref || github.ref }} - - - uses: ./.github/actions/check-maintenance - - - name: Install dependencies - run: | - bash scripts/ci/cuda/ci_install_dependency.sh - - - name: Run precision regression test - timeout-minutes: 120 - env: - SGLANG_PRECISION_BASELINE_DIR: /tmp/sglang_precision_baselines - # Required: the test errors if SGLANG_PRECISION_HF_REPO is unset (no - # local-only mode). Set the var + the HF_TOKEN_PRECISION_STORE secret. - SGLANG_PRECISION_HF_REPO: ${{ vars.SGLANG_PRECISION_HF_REPO }} - SGLANG_PRECISION_HF_REVISION: ${{ vars.SGLANG_PRECISION_HF_REVISION || 'main' }} - HF_TOKEN: ${{ secrets.HF_TOKEN_PRECISION_STORE }} - SGLANG_PRECISION_COMMIT: ${{ github.sha }} - # '0' on scheduled runs (input unset); '1' only on a manual baseline-refresh dispatch. - SGLANG_PRECISION_FORCE_UPDATE: ${{ inputs.force_baseline_update && '1' || '0' }} - run: | - cd test - python3 run_suite.py --hw cuda --suite nightly-precision-8-gpu-h200 --nightly --continue-on-error --timeout-per-file 3600 - - - uses: ./.github/actions/upload-cuda-coredumps - if: failure() - # Consolidate performance metrics from all jobs consolidate-metrics: if: github.repository == 'sgl-project/sglang' && always() + # Every scheduled stage uploads metrics now, so all of them must finish + # before the download step globs `*metrics-*`. needs: - - nightly-test-general-8-gpu-h200 - - nightly-test-general-8-gpu-b200 + - nightly-1-gpu-large + - nightly-2-gpu-large + - nightly-4-gpu-h100 + - nightly-4-gpu-b200 + - nightly-4-gpu-gb300 + - nightly-8-gpu-h200 + - nightly-8-gpu-b200 runs-on: ubuntu-latest steps: - name: Checkout code @@ -674,20 +310,14 @@ jobs: check-all-jobs: if: github.repository == 'sgl-project/sglang' && always() needs: - - nightly-test-general-1-gpu-h100 - - nightly-test-general-4-gpu-h100 - - nightly-test-general-8-gpu-h200 - - nightly-test-general-8-gpu-h20 - - nightly-test-general-8-gpu-b200 - - nightly-test-text-accuracy-2-gpu-h100 - - nightly-test-text-perf-2-gpu-h100 - - nightly-test-vlm-accuracy-2-gpu-h100 - - nightly-test-vlm-perf-2-gpu-h100 - - nightly-test-perf-4-gpu-b200 - - nightly-test-specialized-8-gpu-b200 - - nightly-test-perf-4-gpu-gb300 + - nightly-1-gpu-large + - nightly-2-gpu-large + - nightly-4-gpu-h100 + - nightly-4-gpu-b200 + - nightly-4-gpu-gb300 + - nightly-8-gpu-h200 + - nightly-8-gpu-b200 - nightly-test-diffusion - - nightly-test-precision-8-gpu-h200 - consolidate-metrics runs-on: ubuntu-latest steps: diff --git a/.github/workflows/weekly-test-nvidia.yml b/.github/workflows/weekly-test-nvidia.yml index e89705061..416a3a1d8 100644 --- a/.github/workflows/weekly-test-nvidia.yml +++ b/.github/workflows/weekly-test-nvidia.yml @@ -1,3 +1,5 @@ +# Weekly CUDA tests. Same shape as nightly-test-nvidia.yml (see its header); a +# test joins by declaring `stage="weekly"` plus a runner_config. name: Weekly Test (Nvidia) on: @@ -5,46 +7,53 @@ on: - cron: '0 0 * * 0' # Run every Sunday at midnight UTC workflow_dispatch: inputs: - job_filter: - description: 'Select which job to run (leave empty or "all" to run all jobs)' + runner_filter: + description: 'Select which runner_config to run (leave empty or "all" to run all)' required: false type: choice default: 'all' options: - 'all' - - 'weekly-test-8-gpu-h200' + - '8-gpu-h200' + full_parallel: + description: 'Run all shards of a job at once (faster, but competes with per-commit CI for machines). Off by default: one shard at a time.' + required: false + type: boolean + default: false concurrency: group: weekly-test-nvidia-${{ github.ref }} cancel-in-progress: true -env: - SGLANG_IS_IN_CI: true - SGLANG_ENABLE_ASYNC_ASSERT: true - HF_HUB_DOWNLOAD_TIMEOUT: 300 - HF_HUB_ETAG_TIMEOUT: 300 +permissions: + actions: write + contents: read + issues: read + pull-requests: read jobs: - # Weekly tests - 8 GPU H200 + # run_all_tests skips the paths-filter, so main_package is 'true' and + # sgl_kernel stays empty: every test runs, and no job waits on a wheel this + # workflow never builds. + check-changes: + uses: ./.github/workflows/_pr-test-check-changes.yml + with: + pr_test_yml: '.github/workflows/weekly-test-nvidia.yml' + run_all_tests: true + force_continue_on_error: true + secrets: inherit + weekly-test-8-gpu-h200: - if: github.repository == 'sgl-project/sglang' && (inputs.job_filter == '' || inputs.job_filter == 'all' || inputs.job_filter == 'weekly-test-8-gpu-h200') - runs-on: 8-gpu-h200 - timeout-minutes: 120 - env: - RUNNER_LABELS: 8-gpu-h200 - steps: - - name: Checkout code - uses: actions/checkout@v4 - - - name: Install dependencies - run: | - bash scripts/ci/cuda/ci_install_dependency.sh - - - name: Run weekly 8-GPU H200 tests - timeout-minutes: 120 - env: - GPU_CONFIG: "8-gpu-h200" - IS_H200: "1" - run: | - cd test - python3 run_suite.py --hw cuda --suite weekly-8-gpu-h200 --nightly --continue-on-error --timeout-per-file 7200 + needs: check-changes + if: github.repository == 'sgl-project/sglang' && (inputs.runner_filter == '' || inputs.runner_filter == 'all' || inputs.runner_filter == '8-gpu-h200') + uses: ./.github/workflows/_pr-test-stage.yml + with: + self_name: weekly-test-8-gpu-h200 + runner_config: 8-gpu-h200 + check_changes: ${{ toJson(needs.check-changes.outputs) }} + caller_inputs: ${{ toJson(inputs) }} + partitions: ${{ needs.check-changes.outputs.partitions }} + run_timeout_minutes: '240' + job_timeout_minutes: '300' + scheduled: true + secrets: inherit diff --git a/docs/docs/references/nightly_precision_regression.mdx b/docs/docs/references/nightly_precision_regression.mdx index 522cf93ce..678b1ecd5 100644 --- a/docs/docs/references/nightly_precision_regression.mdx +++ b/docs/docs/references/nightly_precision_regression.mdx @@ -108,7 +108,7 @@ A `capture_signature` (SHA-1 hash of schema version, max_tokens, ignore_eos, TP | `SGLANG_PRECISION_COMMIT` | _(auto-detected from git)_ | Override the sglang commit SHA tagged on push | | `SGLANG_PRECISION_HF_REPO` | _(required)_ | HuggingFace dataset repo for cross-runner baseline storage | | `SGLANG_PRECISION_HF_REVISION` | `main` | Branch/revision of the HF dataset | -| `HF_TOKEN` | _(required in CI)_ | HuggingFace token with write access to the dataset | +| `SGLANG_PRECISION_HF_TOKEN` | _(required in CI)_ | HuggingFace token with write access to the dataset. Kept off `HF_TOKEN`, which already carries the runner's gated-model read token | --- @@ -116,31 +116,34 @@ A `capture_signature` (SHA-1 hash of schema version, max_tokens, ignore_eos, TP ### Workflow job -The nightly job `nightly-test-precision-8-gpu-h200` is defined in `.github/workflows/nightly-test-nvidia.yml` and runs on an 8-GPU H200 runner. It is included in the nightly suite via `test/run_suite.py`. +The test is registered on the `nightly-8-gpu-h200` stage in `.github/workflows/nightly-test-nvidia.yml`, which runs through `_pr-test-stage.yml` like every other CUDA stage. The baseline env below is exported on every scheduled stage; only this test reads it. Key CI configuration: ```yaml -- name: Run precision regression test - timeout-minutes: 120 +- name: Export precision baseline env + if: inputs.scheduled env: - SGLANG_PRECISION_BASELINE_DIR: /tmp/sglang_precision_baselines - SGLANG_PRECISION_HF_REPO: ${{ vars.SGLANG_PRECISION_HF_REPO }} - SGLANG_PRECISION_HF_REVISION: ${{ vars.SGLANG_PRECISION_HF_REVISION || 'main' }} - HF_TOKEN: ${{ secrets.HF_TOKEN_PRECISION_STORE }} - SGLANG_PRECISION_COMMIT: ${{ github.sha }} + BASELINE_HF_TOKEN: ${{ secrets.HF_TOKEN_PRECISION_STORE }} run: | - cd test - python3 run_suite.py --hw cuda --suite nightly-precision-8-gpu-h200 --nightly --continue-on-error --timeout-per-file 3600 + { + echo "SGLANG_PRECISION_BASELINE_DIR=/tmp/sglang_precision_baselines" + echo "SGLANG_PRECISION_HF_REPO=${{ vars.SGLANG_PRECISION_HF_REPO }}" + echo "SGLANG_PRECISION_HF_REVISION=${{ vars.SGLANG_PRECISION_HF_REVISION || 'main' }}" + echo "SGLANG_PRECISION_COMMIT=${{ github.sha }}" + echo "SGLANG_PRECISION_HF_TOKEN=${BASELINE_HF_TOKEN}" + } >> "$GITHUB_ENV" ``` +`SGLANG_PRECISION_HF_TOKEN` rather than `HF_TOKEN`: the latter already carries the runner's gated-model read token, and overwriting it would turn every gated model on the job into a 401. + ### Required GitHub secrets/variables | Name | Type | Purpose | |------|------|---------| | `SGLANG_PRECISION_HF_REPO` | Repository variable | HF dataset repo ID (e.g. `org/sglang-precision-baselines`) — **required**, the test errors if unset | | `SGLANG_PRECISION_HF_REVISION` | Repository variable (optional) | Dataset branch (defaults to `main`) | -| `HF_TOKEN_PRECISION_STORE` | Repository secret | HF token with write access to the dataset | +| `HF_TOKEN_PRECISION_STORE` | Repository secret | HF token with write access to the dataset; exported to the job as `SGLANG_PRECISION_HF_TOKEN` | ### GitHub Step Summary @@ -187,7 +190,7 @@ export SGLANG_PRECISION_MODELS="your-org/your-model" export SGLANG_PRECISION_BASELINE_DIR="/tmp/my_precision_baselines" export SGLANG_PRECISION_DIFF_THRESHOLD="1e-3" export SGLANG_PRECISION_HF_REPO="your-org/sglang-precision-baselines" -export HF_TOKEN="hf_..." +export SGLANG_PRECISION_HF_TOKEN="hf_..." cd test python3 -m pytest registered/debug_utils/test_nightly_precision_regression.py -v @@ -201,7 +204,7 @@ python3 -m pytest registered/debug_utils/test_nightly_precision_regression.py -v export SGLANG_PRECISION_MODELS="your-org/your-model" export SGLANG_PRECISION_BASELINE_DIR="/tmp/test_baselines" export SGLANG_PRECISION_HF_REPO="your-org/sglang-precision-baselines" - export HF_TOKEN="hf_..." + export SGLANG_PRECISION_HF_TOKEN="hf_..." export SGLANG_PRECISION_FORCE_UPDATE="1" # first run: establish baseline cd test @@ -249,7 +252,7 @@ python3 -m pytest registered/debug_utils/test_nightly_precision_regression.py -v - SGLang installed in development mode - GPUs matching the model's requirements - `huggingface_hub` installed -- A **HuggingFace dataset** for baseline storage and a write-capable `HF_TOKEN`. The HF store is **mandatory** — `SGLANG_PRECISION_HF_REPO` must be set or the test will error at startup. This is because the nightly CI runners are ephemeral (no persistent local disk), so baselines must survive across runs via the HF dataset. There is currently no local-only fallback. +- A **HuggingFace dataset** for baseline storage and a write-capable `SGLANG_PRECISION_HF_TOKEN`. The HF store is **mandatory** — `SGLANG_PRECISION_HF_REPO` must be set or the test will error at startup. This is because the nightly CI runners are ephemeral (no persistent local disk), so baselines must survive across runs via the HF dataset. There is currently no local-only fallback. ### Quick local test @@ -258,7 +261,7 @@ python3 -m pytest registered/debug_utils/test_nightly_precision_regression.py -v export SGLANG_PRECISION_MODELS="Qwen/Qwen2.5-0.5B-Instruct" export SGLANG_PRECISION_BASELINE_DIR="/tmp/precision_baselines" export SGLANG_PRECISION_HF_REPO="your-org/sglang-precision-baselines" -export HF_TOKEN="hf_..." +export SGLANG_PRECISION_HF_TOKEN="hf_..." # First run: establish baseline cd test @@ -356,7 +359,7 @@ The framework uses a **rolling baseline**: every successful comparison updates t ### No local-only mode -The test requires a HuggingFace dataset (`SGLANG_PRECISION_HF_REPO`) and a write-capable `HF_TOKEN`. There is no local-only fallback. This is by design — CI runners have no persistent local disk, so the HF dataset is the only way to carry baselines across runs. If you need to run the test locally, you must set up a HF dataset (even a private one) and provide the corresponding token. +The test requires a HuggingFace dataset (`SGLANG_PRECISION_HF_REPO`) and a write-capable `SGLANG_PRECISION_HF_TOKEN`. There is no local-only fallback. This is by design — CI runners have no persistent local disk, so the HF dataset is the only way to carry baselines across runs. If you need to run the test locally, you must set up a HF dataset (even a private one) and provide the corresponding token. --- @@ -369,4 +372,4 @@ The test requires a HuggingFace dataset (`SGLANG_PRECISION_HF_REPO`) and a write | `python/sglang/srt/debug_utils/comparator/` | Tensor comparison engine | | `python/sglang/srt/debug_utils/dumper.py` | Runtime hidden-state capture | | `.github/workflows/nightly-test-nvidia.yml` | CI workflow definition | -| `test/run_suite.py` | Test suite registration (includes `nightly-precision-8-gpu-h200`) | +| `test/run_suite.py` | Test suite registration (includes `nightly-test-8-gpu-h200`) | diff --git a/python/sglang/test/ci/ci_utils.py b/python/sglang/test/ci/ci_utils.py index dd8907fd9..8863e4b3d 100644 --- a/python/sglang/test/ci/ci_utils.py +++ b/python/sglang/test/ci/ci_utils.py @@ -137,9 +137,23 @@ def _repo_relative_path(p: str) -> str: return p[idx + len(marker) :] if idx >= 0 else p +# Slow-run variance is largely additive (cold HF cache, slow server launch), so +# the multiplier alone under-provisions at both ends: test_encoder_dp runs +# 200-426s but once took over 1185s against a 1.5x budget of 765s, and +# test_lora_deepseek_v3_base_logprob_diff (est 1800) landed on exactly 1.5 * est. +# Every file gets the same absolute slack on top of the proportional one. +DERIVED_TIMEOUT_SLACK = 1800.0 +DERIVED_TIMEOUT_FACTOR = 1.5 + + +def derive_timeout_per_file(est_time: float) -> float: + est = float(est_time) + return max(est * DERIVED_TIMEOUT_FACTOR, est + DERIVED_TIMEOUT_SLACK) + + def run_unittest_files( files: Union[List[TestFile], List[CIRegistry]], - timeout_per_file: float, + timeout_per_file: Optional[float] = None, continue_on_error: bool = False, enable_retry: bool = False, max_attempts: int = 2, @@ -150,7 +164,8 @@ def run_unittest_files( Args: files: List of TestFile objects to run - timeout_per_file: Timeout in seconds for each test file + timeout_per_file: Fixed timeout in seconds for every test file, or None + to derive each file's budget from its own est_time. continue_on_error: If True, continue running remaining tests even if one fails. If False, stop at first failure (default behavior for PR tests). enable_retry: If True, retry failed tests that appear to be accuracy/performance @@ -178,6 +193,12 @@ def run_unittest_files( # FIXME: remove this branch after migrating all tests to use CIRegistry filename, estimated_time = file.name, file.estimated_time + file_timeout = ( + timeout_per_file + if timeout_per_file is not None + else derive_timeout_per_file(estimated_time) + ) + process = None output_lines = [] @@ -235,7 +256,7 @@ def run_unittest_files( run_one_file, args=(filename,), kwargs={"capture_output": enable_retry}, - timeout=timeout_per_file, + timeout=file_timeout, ) if ret_code == 0: @@ -281,24 +302,22 @@ def run_unittest_files( # TimeoutError aborts run_one_file before its elapsed write; # record the timeout cap as an upper bound so the file still # appears in the TIMINGS block below. - file_elapsed[filename] = float(timeout_per_file) + file_elapsed[filename] = float(file_timeout) # Retry once on timeout: usually a stuck server / hung device. # A real hang times out again and is reported. if enable_retry and attempt < max_attempts: logger.info( f"\n[CI Retry] {filename} timed out after " - f"{timeout_per_file}s; waiting {retry_wait_seconds}s " + f"{file_timeout}s; waiting {retry_wait_seconds}s " f"before retry (attempt {attempt + 1}/{max_attempts})\n" ) time.sleep(retry_wait_seconds) attempt += 1 continue - logger.info( - f"\n✗ TIMEOUT: {filename} after {timeout_per_file} seconds\n" - ) + logger.info(f"\n✗ TIMEOUT: {filename} after {file_timeout} seconds\n") if was_retried: retried_tests.append((filename, attempt, "timeout")) - failed_tests.append((filename, f"timeout after {timeout_per_file}s")) + failed_tests.append((filename, f"timeout after {file_timeout}s")) break if not file_passed: diff --git a/python/sglang/test/precision_baseline_store.py b/python/sglang/test/precision_baseline_store.py index f90ba27f9..fe5c4d647 100644 --- a/python/sglang/test/precision_baseline_store.py +++ b/python/sglang/test/precision_baseline_store.py @@ -26,6 +26,16 @@ from huggingface_hub.errors import ( ) +def _store_token() -> Optional[str]: + """Write token for the baseline dataset repo. + + Deliberately not HF_TOKEN: that name already carries the runner's + gated-model read token, so writing the store token there would shadow it + and turn every gated model on the job into a 401. + """ + return os.environ.get("SGLANG_PRECISION_HF_TOKEN") or None + + @dataclass class HfStoreConfig: repo: str @@ -38,7 +48,7 @@ class HfStoreConfig: raise RuntimeError( "SGLANG_PRECISION_HF_REPO is not set. The precision baseline " "store is required (there is no local-only mode); set the repo " - "and HF_TOKEN_PRECISION_STORE." + "and SGLANG_PRECISION_HF_TOKEN." ) revision = os.environ.get("SGLANG_PRECISION_HF_REVISION", "main") return cls(repo=repo, revision=revision) @@ -148,6 +158,7 @@ def fetch_latest_baseline( repo_type="dataset", revision=config.revision, allow_patterns=[f"{run_path}/tensors/*"], + token=_store_token(), ), what="snapshot download", ) @@ -171,6 +182,7 @@ def _read_manifest(config: HfStoreConfig) -> tuple[list[dict[str, Any]], str]: repo_type="dataset", filename="manifest.jsonl", revision=config.revision, + token=_store_token(), ), what="manifest fetch", ) @@ -215,7 +227,7 @@ def push_run( # Dedup: same model+date+sha → skip tensor upload but still refresh meta # + comparator_report + append a new manifest row, so pass-1 baseline and # pass-2 stats both land. force=True re-uploads tensors too. - api = HfApi() + api = HfApi(token=_store_token()) date_str, date_path = _today_path() model_sanitized = _sanitize_model_name(model) sha7 = ( @@ -306,7 +318,7 @@ def prune_old_runs( # dry_run defaults True because model=None+keep_days=0 would wipe the # store. Live mode rewrites the manifest before deleting folders so a # mid-run failure leaves manifest pointing at the kept rows only. - api = HfApi() + api = HfApi(token=_store_token()) rows, _ = _read_manifest(config) if not rows: return {"kept": [], "pruned": []} diff --git a/scripts/ci/check_registered_tests.py b/scripts/ci/check_registered_tests.py index 963fc74f3..b9e5254d6 100755 --- a/scripts/ci/check_registered_tests.py +++ b/scripts/ci/check_registered_tests.py @@ -4,11 +4,11 @@ Pre-commit hook: validate CI registry calls under test/registered/. 1. Every test file must contain a CI registry call (register_cuda_ci, register_amd_ci, etc.). -2. A CUDA test must register its PR-test suite via the modern +2. A CUDA test must register its suite via the modern `stage=`/`runner_config=` form. The legacy single-string `suite=` is reserved - for the nightly/stress/weekly families (and for AMD/CPU/NPU suites); any other - CUDA `suite=` resolves to a name no PR-test workflow invokes, so the test - silently never runs. Two shapes are rejected: + for the stress family (and for AMD/CPU/NPU suites); any other CUDA `suite=` + resolves to a name no workflow invokes, so the test silently never runs. + Two shapes are rejected: a. `{stage}-test-{runner_config}` -- the modern name stuffed back into the legacy form. Reported with the exact stage/runner split to use. b. an older `{stage}-{runner_config}` PR-test name (e.g. the pre-migration @@ -33,11 +33,10 @@ import sys # shape is always expressible (and should be expressed) the modern way. _MODERN_SHAPE = re.compile(r"^(.+)-test-(.+)$") -# The only suite families a CUDA registry may keep on the legacy single-string -# `suite=` form. Everything else is a PR-test/base stage that must use the -# modern stage=/runner_config= form (otherwise its effective_suite matches no -# suite the PR-test workflows invoke, and the test silently never runs). -_LEGACY_CUDA_PREFIXES = ("nightly", "stress", "weekly") +# The only CUDA suite family still allowed on the legacy single-string `suite=` +# form. Anything else needs stage=/runner_config=, or its effective_suite matches +# no suite any workflow invokes and the test silently never runs. +_LEGACY_CUDA_PREFIXES = ("stress",) def _defines_testcase(tree: ast.AST) -> bool: @@ -118,8 +117,6 @@ def main() -> int: and r.runner_config is None ): continue - # nightly/stress/weekly are the only CUDA suites allowed to stay on - # the legacy single-string form. if r.suite.split("-", 1)[0] in _LEGACY_CUDA_PREFIXES: continue m = _MODERN_SHAPE.match(r.suite) diff --git a/scripts/ci/stage_models_overrides.json b/scripts/ci/stage_models_overrides.json index 2a7be7774..157b1fc23 100644 --- a/scripts/ci/stage_models_overrides.json +++ b/scripts/ci/stage_models_overrides.json @@ -1,5 +1,5 @@ { - "_comment": "Manual overrides for list_stage_models.py. by_file/by_suite ADD models the static scan cannot see (models built dynamically, read from configs, passed via CLI args). deny REMOVES false-positive ids the heuristic mistakes for models. Keys in by_file are repo-relative test paths (e.g. test/registered/foo/test_bar.py). suite_labels maps legacy suite= registrations (no runner_config) to the GH runner label(s) their dispatching workflow hardcodes in runs-on -- a list, because one suite can run on several labels (nightly-8-gpu-common). $b200_runner is the dynamic-b200 placeholder from runner_configs.yml. Deliberately absent: nightly-4-gpu-gb300-* (run as k8s pods, not GHA runners) and nightly-2-gpu (registered but dispatched by no workflow); both stay visible in unmapped_suites.", + "_comment": "Manual overrides for list_stage_models.py. by_file/by_suite ADD models the static scan cannot see (models built dynamically, read from configs, passed via CLI args). deny REMOVES false-positive ids the heuristic mistakes for models. Keys in by_file are repo-relative test paths (e.g. test/registered/foo/test_bar.py). suite_labels maps legacy suite= registrations (no runner_config) to the GH runner label(s) their dispatching workflow hardcodes in runs-on -- a list, because one suite can run on several labels. $b200_runner is the dynamic-b200 placeholder from runner_configs.yml. Suites registered with stage=/runner_config= need no entry: their label resolves through runner_configs.yml. Anything unmappable stays visible in unmapped_suites.", "by_file": {}, "by_suite": {}, "suite_labels": { @@ -7,21 +7,7 @@ "base-b-kernel-unit-1-gpu-b200": ["$b200_runner"], "base-b-kernel-unit-1-gpu-large": ["1-gpu-h100"], "base-b-kernel-unit-8-gpu-h200": ["8-gpu-h200"], - "nightly-1-gpu": ["1-gpu-h100"], - "nightly-4-gpu": ["4-gpu-h100"], - "nightly-4-gpu-b200": ["$b200_runner"], - "nightly-8-gpu-b200": ["8-gpu-b200"], - "nightly-8-gpu-common": ["8-gpu-h200", "8-gpu-b200"], - "nightly-8-gpu-h200": ["8-gpu-h200"], - "nightly-eval-text-2-gpu": ["2-gpu-h100"], - "nightly-eval-vlm-2-gpu": ["2-gpu-h100"], - "nightly-kernel-1-gpu": ["1-gpu-h100"], - "nightly-kernel-8-gpu-h200": ["8-gpu-h200"], - "nightly-perf-text-2-gpu": ["2-gpu-h100"], - "nightly-perf-vlm-2-gpu": ["2-gpu-h100"], - "nightly-precision-8-gpu-h200": ["8-gpu-h200"], - "stress": ["8-gpu-h200"], - "weekly-8-gpu-h200": ["8-gpu-h200"] + "stress": ["8-gpu-h200"] }, "deny": [ "tok/req", diff --git a/scripts/ci/utils/compute_partitions.py b/scripts/ci/utils/compute_partitions.py index 0694bf60a..6d2a54b4e 100644 --- a/scripts/ci/utils/compute_partitions.py +++ b/scripts/ci/utils/compute_partitions.py @@ -128,8 +128,11 @@ def compute_partitions( in-source `est_time` / `(1.0, 0.0)`. `full_parallel=True` lifts the matrix-fanout throttle. """ - # Allowlist: stages pr-test.yml dispatches. Stress / weekly / - # nightly-* live in test/registered/ but pr-test doesn't run them. + # Allowlist of the stages this workflow dispatches -- what keeps stress / + # weekly / nightly out, since CUDA scheduled suites no longer carry + # `nightly=True`. The nightly filter still matters for CPU: some tests sit on + # a dispatched suite with the flag set, so run_suite.py skips them and their + # est_time must not inflate the shard count. dispatched_suites = set(run_timeouts) | set(_BASE_A_OVERRIDES) suite_tests = defaultdict(list) for t in tests: diff --git a/scripts/ci/utils/slash_command_handler.py b/scripts/ci/utils/slash_command_handler.py index fb9a8fb64..aad725e6d 100644 --- a/scripts/ci/utils/slash_command_handler.py +++ b/scripts/ci/utils/slash_command_handler.py @@ -702,11 +702,8 @@ def _extract_runner_configs(content): def _extract_legacy_suites(content): - """Pull every legacy single-string `suite=` from `register_cuda_ci(...)` calls. - - Mirrors _extract_runner_configs for the legacy nightly/weekly shape: a file - may register on multiple pools, so collect all of them rather than the first. - """ + """Pull every legacy single-string `suite=` from `register_cuda_ci(...)` + calls. Used only to report why such a file is not dispatchable.""" out = [] for args in re.finditer( r"^[^#\n]*register_cuda_ci\s*\(([^)]*)\)", content, re.MULTILINE @@ -717,38 +714,6 @@ def _extract_legacy_suites(content): return out -# Legacy nightly/weekly CUDA suites register with a single-string `suite=` -# instead of `runner_config=`, so they carry no runner metadata of their own. -# Map each to the runner_config in scripts/ci/runner_configs.yml whose hardware -# matches the runner the nightly/weekly pipeline actually uses (see -# .github/workflows/{nightly,weekly}-test-nvidia.yml), so /rerun-test can still -# dispatch a single nightly/weekly test. The runner label, install script, -# timeout and rdma_devices are then resolved from -# runner_configs.yml as usual, keeping that file the single source of truth for -# runner details. -# -# Suites on hardware with no matching runner_config (e.g. nightly-4-gpu-gb300) -# and non-CUDA suites (npu/amd) are intentionally absent and stay -# non-dispatchable until a matching runner_config exists. -_LEGACY_SUITE_TO_RUNNER_CONFIG = { - "nightly-1-gpu": "1-gpu-large", - "nightly-kernel-1-gpu": "1-gpu-large", - "nightly-eval-text-2-gpu": "2-gpu-large", - "nightly-perf-text-2-gpu": "2-gpu-large", - "nightly-eval-vlm-2-gpu": "2-gpu-large", - "nightly-perf-vlm-2-gpu": "2-gpu-large", - "nightly-4-gpu": "4-gpu-h100", - "nightly-4-gpu-b200": "4-gpu-b200", - "nightly-8-gpu-common": ["8-gpu-h200", "8-gpu-b200"], - "nightly-8-gpu-h200": "8-gpu-h200", - "nightly-kernel-8-gpu-h200": "8-gpu-h200", - "nightly-precision-8-gpu-h200": "8-gpu-h200", - "nightly-8-gpu-h20": "8-gpu-h20", - "nightly-8-gpu-b200": "8-gpu-b200", - "weekly-8-gpu-h200": "8-gpu-h200", -} - - def _dispatch_err(suite, msg): """Build a detect_suite error result for the given suite.""" return { @@ -811,11 +776,10 @@ def detect_suite(file_path_from_test): pool it should run on — so this returns a *list* of dispatch dicts, one per registration. Runner label, install script, timeout, and rdma_devices are all resolved from scripts/ci/runner_configs.yml — the - same single source of truth that drives the main PR test pipeline. - - Legacy nightly/weekly CUDA suites (single-string `suite=`) are dispatchable - too: each suite name is mapped to the matching runner_config via - _LEGACY_SUITE_TO_RUNNER_CONFIG, then resolved the same way. + same single source of truth that drives the main PR test pipeline. Every + dispatchable CUDA suite, per-commit and scheduled alike, goes through that + one path; the legacy single-string `suite=` carries no runner_config and is + reported as non-dispatchable. CPU files yield a single-element list. A file with no recognised (or no dispatchable) registration yields a one-element list whose dict has an @@ -837,19 +801,7 @@ def detect_suite(file_path_from_test): results.append(_resolve_runner_config(rc, full_path, suite)) return results - # Legacy nightly/weekly CUDA suites: single-string `suite=`, no - # runner_config. Map each mappable suite to its runner_config and resolve. legacy_suites = _extract_legacy_suites(content) - mappable = [s for s in legacy_suites if s in _LEGACY_SUITE_TO_RUNNER_CONFIG] - if mappable: - results = [] - for s in mappable: - rcs = _LEGACY_SUITE_TO_RUNNER_CONFIG[s] - if isinstance(rcs, str): - rcs = [rcs] - for rc in rcs: - results.append(_resolve_runner_config(rc, full_path, s)) - return results if re.search(r"^[^#\n]*register_cpu_ci\s*\(", content, re.MULTILINE): return [ @@ -869,11 +821,11 @@ def detect_suite(file_path_from_test): return [ _dispatch_err( suite, - f"Suite `{suite}` in `{full_path}` is not dispatchable via " - f"/rerun-test. It has no entry in _LEGACY_SUITE_TO_RUNNER_CONFIG " - f"— either it is a non-CUDA suite (npu/amd) or it runs on " - f"hardware with no matching runner_config in " - f"scripts/ci/runner_configs.yml.", + f"Suite `{suite}` in `{full_path}` is registered with the legacy " + f"single-string `suite=`, which carries no runner_config and so " + f"is not dispatchable via /rerun-test. Re-register it with " + f"`stage=`/`runner_config=` (CUDA), or dispatch its own " + f"workflow (npu/amd).", ) ] diff --git a/test/README.md b/test/README.md index b11a7671d..1502ea1f1 100644 --- a/test/README.md +++ b/test/README.md @@ -50,8 +50,8 @@ python3 test/registered/jit/test_add_constant.py python3 test/run_suite.py --hw cpu --suite base-a-test-cpu python3 test/run_suite.py --hw cuda --suite base-a-test-1-gpu-small -# Nightly tests -python3 test/run_suite.py --hw cuda --suite nightly-1-gpu --nightly +# Nightly tests (CUDA nightly suites take no --nightly; the stage is in the name) +python3 test/run_suite.py --hw cuda --suite nightly-test-1-gpu-large # With auto-partitioning (for parallel CI jobs) python3 test/run_suite.py --hw cuda --suite base-b-test-1-gpu-small \ diff --git a/test/registered/4-gpu-models/test_nvidia_nemotron_3_super_nvfp4.py b/test/registered/4-gpu-models/test_nvidia_nemotron_3_super_nvfp4.py index ba37a7ae9..faca4d38d 100644 --- a/test/registered/4-gpu-models/test_nvidia_nemotron_3_super_nvfp4.py +++ b/test/registered/4-gpu-models/test_nvidia_nemotron_3_super_nvfp4.py @@ -12,7 +12,7 @@ from sglang.test.test_utils import ( popen_launch_server, ) -register_cuda_ci(est_time=810, suite="nightly-4-gpu-b200", nightly=True) +register_cuda_ci(est_time=1200, stage="nightly", runner_config="4-gpu-b200") NEMOTRON_3_SUPER_NVFP4_MODEL = "nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4" diff --git a/test/registered/8-gpu-models/test_glm52_fp8.py b/test/registered/8-gpu-models/test_glm52_fp8.py index 47cb78ed5..6130fc4e7 100644 --- a/test/registered/8-gpu-models/test_glm52_fp8.py +++ b/test/registered/8-gpu-models/test_glm52_fp8.py @@ -6,8 +6,9 @@ from sglang.test.performance_test_runner import PerformanceTestParams from sglang.test.run_combined_tests import run_combined_tests from sglang.test.test_utils import ModelLaunchSettings -# Runs on both H200 and B200 via nightly-8-gpu-common suite -register_cuda_ci(est_time=1800, suite="nightly-8-gpu-common", nightly=True) +# Runs on both H200 and B200: registered once per runner_config below +register_cuda_ci(est_time=2880, stage="nightly", runner_config="8-gpu-h200") +register_cuda_ci(est_time=2880, stage="nightly", runner_config="8-gpu-b200") GLM_52_FP8_MODEL_PATH = "zai-org/GLM-5.2-FP8" diff --git a/test/registered/8-gpu-models/test_glm_46.py b/test/registered/8-gpu-models/test_glm_46.py index 71786fcfc..2d3bc0b08 100644 --- a/test/registered/8-gpu-models/test_glm_46.py +++ b/test/registered/8-gpu-models/test_glm_46.py @@ -6,8 +6,9 @@ from sglang.test.performance_test_runner import PerformanceTestParams from sglang.test.run_combined_tests import run_combined_tests from sglang.test.test_utils import ModelLaunchSettings -# Runs on both H200 and B200 via nightly-8-gpu-common suite -register_cuda_ci(est_time=1800, suite="nightly-8-gpu-common", nightly=True) +# Runs on both H200 and B200: registered once per runner_config below +register_cuda_ci(est_time=1320, stage="nightly", runner_config="8-gpu-h200") +register_cuda_ci(est_time=1320, stage="nightly", runner_config="8-gpu-b200") GLM_4_6_MODEL_PATH = "zai-org/GLM-4.6" diff --git a/test/registered/8-gpu-models/test_gpt_oss_120b.py b/test/registered/8-gpu-models/test_gpt_oss_120b.py index 1b6791dff..5a286a39d 100644 --- a/test/registered/8-gpu-models/test_gpt_oss_120b.py +++ b/test/registered/8-gpu-models/test_gpt_oss_120b.py @@ -5,9 +5,10 @@ from sglang.test.performance_test_runner import PerformanceTestParams from sglang.test.run_combined_tests import run_combined_tests from sglang.test.test_utils import ModelLaunchSettings -# Runs on both H200 and B200 via nightly-8-gpu-common suite +# Runs on both H200 and B200: registered once per runner_config below # Higher est_time due to 6 variants with both performance and accuracy tests -register_cuda_ci(est_time=1800, suite="nightly-8-gpu-common", nightly=True) +register_cuda_ci(est_time=690, stage="nightly", runner_config="8-gpu-h200") +register_cuda_ci(est_time=690, stage="nightly", runner_config="8-gpu-b200") GPT_OSS_120B_MXFP4_MODEL_PATH = "openai/gpt-oss-120b" GPT_OSS_120B_EAGLE3_DRAFT_MODEL_PATH = "lmsys/EAGLE3-gpt-oss-120b-bf16" diff --git a/test/registered/8-gpu-models/test_inkling_nvfp4_nightly.py b/test/registered/8-gpu-models/test_inkling_nvfp4_nightly.py index a3af2647e..d2e9e8460 100644 --- a/test/registered/8-gpu-models/test_inkling_nvfp4_nightly.py +++ b/test/registered/8-gpu-models/test_inkling_nvfp4_nightly.py @@ -10,7 +10,8 @@ from sglang.test.test_utils import ModelLaunchSettings, is_blackwell_system # NVFP4 needs Blackwell FP4 kernels, so this runs on the Blackwell leg of the # common 8-GPU suite (Hopper is skipped below). -register_cuda_ci(est_time=3600, suite="nightly-8-gpu-common", nightly=True) +register_cuda_ci(est_time=3600, stage="nightly", runner_config="8-gpu-h200") +register_cuda_ci(est_time=3600, stage="nightly", runner_config="8-gpu-b200") INKLING_NVFP4_MODEL = "thinkingmachines/Inkling-NVFP4" INKLING_SMALL_NVFP4_MODEL = "thinkingmachines/Inkling-Small-NVFP4" diff --git a/test/registered/8-gpu-models/test_kimi_k25.py b/test/registered/8-gpu-models/test_kimi_k25.py index fc6b75c90..e3b6b114d 100644 --- a/test/registered/8-gpu-models/test_kimi_k25.py +++ b/test/registered/8-gpu-models/test_kimi_k25.py @@ -6,8 +6,9 @@ from sglang.test.performance_test_runner import PerformanceTestParams from sglang.test.run_combined_tests import run_combined_tests from sglang.test.test_utils import ModelLaunchSettings -# Runs on both H200 and B200 via nightly-8-gpu-common suite -register_cuda_ci(est_time=3600, suite="nightly-8-gpu-common", nightly=True) +# Runs on both H200 and B200: registered once per runner_config below +register_cuda_ci(est_time=2820, stage="nightly", runner_config="8-gpu-h200") +register_cuda_ci(est_time=2820, stage="nightly", runner_config="8-gpu-b200") KIMI_K25_MODEL_PATH = "moonshotai/Kimi-K2.5" diff --git a/test/registered/8-gpu-models/test_ling_2_6_flash.py b/test/registered/8-gpu-models/test_ling_2_6_flash.py index 064c6b24a..4ee6bf3f0 100644 --- a/test/registered/8-gpu-models/test_ling_2_6_flash.py +++ b/test/registered/8-gpu-models/test_ling_2_6_flash.py @@ -12,7 +12,8 @@ from sglang.test.ci.ci_register import register_cuda_ci from sglang.test.kits.eval_accuracy_kit import GSM8KMixin from sglang.test.server_fixtures.default_fixture import DefaultServerBase -register_cuda_ci(est_time=600, suite="nightly-8-gpu-common", nightly=True) +register_cuda_ci(est_time=300, stage="nightly", runner_config="8-gpu-h200") +register_cuda_ci(est_time=300, stage="nightly", runner_config="8-gpu-b200") class TestLing26Flash(GSM8KMixin, DefaultServerBase): diff --git a/test/registered/8-gpu-models/test_llama4.py b/test/registered/8-gpu-models/test_llama4.py index d5e1e8eab..b76ca9604 100644 --- a/test/registered/8-gpu-models/test_llama4.py +++ b/test/registered/8-gpu-models/test_llama4.py @@ -6,8 +6,9 @@ from sglang.test.performance_test_runner import PerformanceTestParams from sglang.test.run_combined_tests import run_combined_tests from sglang.test.test_utils import ModelLaunchSettings -# Runs on both H200 and B200 via nightly-8-gpu-common suite -register_cuda_ci(est_time=1800, suite="nightly-8-gpu-common", nightly=True) +# Runs on both H200 and B200: registered once per runner_config below +register_cuda_ci(est_time=1800, stage="nightly", runner_config="8-gpu-h200") +register_cuda_ci(est_time=1800, stage="nightly", runner_config="8-gpu-b200") LLAMA4_MODEL_PATH = "meta-llama/Llama-4-Scout-17B-16E-Instruct" diff --git a/test/registered/8-gpu-models/test_longcat_flash_lite_fp8.py b/test/registered/8-gpu-models/test_longcat_flash_lite_fp8.py index be3b8e000..fe0ed5075 100644 --- a/test/registered/8-gpu-models/test_longcat_flash_lite_fp8.py +++ b/test/registered/8-gpu-models/test_longcat_flash_lite_fp8.py @@ -6,7 +6,7 @@ from sglang.test.performance_test_runner import PerformanceTestParams from sglang.test.run_combined_tests import run_combined_tests from sglang.test.test_utils import ModelLaunchSettings -register_cuda_ci(est_time=1200, suite="nightly-8-gpu-h200", nightly=True) +register_cuda_ci(est_time=1200, stage="nightly", runner_config="8-gpu-h200") # LongCat-Flash-Lite-FP8 is the smallest member of the LongCat family # (~138 GB FP8 weights, hidden=3072, 14 layers, 256 routed + 128 zero diff --git a/test/registered/8-gpu-models/test_minimax_m25.py b/test/registered/8-gpu-models/test_minimax_m25.py index b9e6d0adf..22fb25d81 100644 --- a/test/registered/8-gpu-models/test_minimax_m25.py +++ b/test/registered/8-gpu-models/test_minimax_m25.py @@ -6,8 +6,9 @@ from sglang.test.performance_test_runner import PerformanceTestParams from sglang.test.run_combined_tests import run_combined_tests from sglang.test.test_utils import ModelLaunchSettings -# Runs on both H200 and B200 via nightly-8-gpu-common suite -register_cuda_ci(est_time=1800, suite="nightly-8-gpu-common", nightly=True) +# Runs on both H200 and B200: registered once per runner_config below +register_cuda_ci(est_time=1860, stage="nightly", runner_config="8-gpu-h200") +register_cuda_ci(est_time=1860, stage="nightly", runner_config="8-gpu-b200") MINIMAX_M25_MODEL_PATH = "MiniMaxAI/MiniMax-M2.5" diff --git a/test/registered/8-gpu-models/test_mistral_large3.py b/test/registered/8-gpu-models/test_mistral_large3.py index bd40a0085..0531f8f93 100644 --- a/test/registered/8-gpu-models/test_mistral_large3.py +++ b/test/registered/8-gpu-models/test_mistral_large3.py @@ -8,9 +8,10 @@ from sglang.test.performance_test_runner import PerformanceTestParams from sglang.test.run_combined_tests import run_combined_tests from sglang.test.test_utils import ModelLaunchSettings, is_blackwell_system -# Runs on both H200 and B200 via nightly-8-gpu-common suite +# Runs on both H200 and B200: registered once per runner_config below # Note: trtllm_mla backend may have hardware-specific behavior -register_cuda_ci(est_time=3000, suite="nightly-8-gpu-common", nightly=True) +register_cuda_ci(est_time=3000, stage="nightly", runner_config="8-gpu-h200") +register_cuda_ci(est_time=3000, stage="nightly", runner_config="8-gpu-b200") MISTRAL_LARGE3_FP8_MODEL_PATH = "mistralai/Mistral-Large-3-675B-Instruct-2512" MISTRAL_LARGE3_NVFP4_MODEL_PATH = "mistralai/Mistral-Large-3-675B-Instruct-2512-NVFP4" diff --git a/test/registered/8-gpu-models/test_nvidia_nemotron_3_super_nightly.py b/test/registered/8-gpu-models/test_nvidia_nemotron_3_super_nightly.py index e8536cdcf..563e6f54e 100644 --- a/test/registered/8-gpu-models/test_nvidia_nemotron_3_super_nightly.py +++ b/test/registered/8-gpu-models/test_nvidia_nemotron_3_super_nightly.py @@ -7,8 +7,9 @@ from sglang.test.performance_test_runner import PerformanceTestParams from sglang.test.run_combined_tests import run_combined_tests from sglang.test.test_utils import ModelLaunchSettings, is_blackwell_system -# Runs on both Hopper and Blackwell via nightly-8-gpu-common suite -register_cuda_ci(est_time=5400, suite="nightly-8-gpu-common", nightly=True) +# Runs on both Hopper and Blackwell: registered once per runner_config below +register_cuda_ci(est_time=3360, stage="nightly", runner_config="8-gpu-h200") +register_cuda_ci(est_time=3360, stage="nightly", runner_config="8-gpu-b200") NEMOTRON_3_SUPER_BF16_MODEL = "nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16" NEMOTRON_3_SUPER_NVFP4_MODEL = "nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4" diff --git a/test/registered/8-gpu-models/test_qwen35.py b/test/registered/8-gpu-models/test_qwen35.py index 6d9f091f9..04d47321a 100644 --- a/test/registered/8-gpu-models/test_qwen35.py +++ b/test/registered/8-gpu-models/test_qwen35.py @@ -6,8 +6,9 @@ from sglang.test.performance_test_runner import PerformanceTestParams from sglang.test.run_combined_tests import run_combined_tests from sglang.test.test_utils import ModelLaunchSettings -# Runs on both H200 and B200 via nightly-8-gpu-common suite -register_cuda_ci(est_time=1800, suite="nightly-8-gpu-common", nightly=True) +# Runs on both H200 and B200: registered once per runner_config below +register_cuda_ci(est_time=3000, stage="nightly", runner_config="8-gpu-h200") +register_cuda_ci(est_time=3000, stage="nightly", runner_config="8-gpu-b200") QWEN35_MODEL_PATH = "Qwen/Qwen3.5-397B-A17B-FP8" diff --git a/test/registered/8-gpu-models/test_ring_2_5_1t.py b/test/registered/8-gpu-models/test_ring_2_5_1t.py index f42fa29f3..e204fb7ac 100644 --- a/test/registered/8-gpu-models/test_ring_2_5_1t.py +++ b/test/registered/8-gpu-models/test_ring_2_5_1t.py @@ -5,7 +5,8 @@ from sglang.test.ci.ci_register import register_cuda_ci from sglang.test.run_combined_tests import run_combined_tests from sglang.test.test_utils import ModelLaunchSettings -register_cuda_ci(est_time=510, suite="nightly-8-gpu-common", nightly=True) +register_cuda_ci(est_time=1680, stage="nightly", runner_config="8-gpu-h200") +register_cuda_ci(est_time=1680, stage="nightly", runner_config="8-gpu-b200") RING_2_5_1T_MODEL_PATH = "inclusionAI/Ring-2.5-1T" diff --git a/test/registered/attention/test_deepseek_v3_deterministic.py b/test/registered/attention/test_deepseek_v3_deterministic.py index 3c640b3c9..d38a7f799 100644 --- a/test/registered/attention/test_deepseek_v3_deterministic.py +++ b/test/registered/attention/test_deepseek_v3_deterministic.py @@ -12,7 +12,7 @@ from sglang.test.test_deterministic_utils import ( TestDeterministicBase, ) -register_cuda_ci(est_time=240, suite="nightly-1-gpu", nightly=True) +register_cuda_ci(est_time=240, stage="nightly", runner_config="1-gpu-large") DEEPSEEK_MODEL = "lmsys/sglang-ci-dsv3-test" diff --git a/test/registered/attention/test_qwen3_next_deterministic.py b/test/registered/attention/test_qwen3_next_deterministic.py index efaf6636e..b276f362b 100644 --- a/test/registered/attention/test_qwen3_next_deterministic.py +++ b/test/registered/attention/test_qwen3_next_deterministic.py @@ -12,7 +12,7 @@ from sglang.test.test_deterministic_utils import ( TestDeterministicBase, ) -register_cuda_ci(est_time=200, suite="nightly-4-gpu", nightly=True) +register_cuda_ci(est_time=200, stage="nightly", runner_config="4-gpu-h100") QWEN3_NEXT = "Qwen/Qwen3-Next-80B-A3B-Instruct" diff --git a/test/registered/backends/test_deepseek_v3_fp4_cutedsl_moe.py b/test/registered/backends/test_deepseek_v3_fp4_cutedsl_moe.py index c6484ca81..cea8fa5e6 100644 --- a/test/registered/backends/test_deepseek_v3_fp4_cutedsl_moe.py +++ b/test/registered/backends/test_deepseek_v3_fp4_cutedsl_moe.py @@ -17,7 +17,7 @@ from sglang.test.test_utils import ( write_github_step_summary, ) -register_cuda_ci(est_time=450, suite="nightly-4-gpu-b200", nightly=True) +register_cuda_ci(est_time=900, stage="nightly", runner_config="4-gpu-b200") FULL_DEEPSEEK_V3_FP4_MODEL_PATH = "nvidia/DeepSeek-V3-0324-FP4" SERVER_LAUNCH_TIMEOUT = 1000 diff --git a/test/registered/backends/test_flashinfer_trtllm_gen_moe_backend.py b/test/registered/backends/test_flashinfer_trtllm_gen_moe_backend.py index d51377016..72915e292 100644 --- a/test/registered/backends/test_flashinfer_trtllm_gen_moe_backend.py +++ b/test/registered/backends/test_flashinfer_trtllm_gen_moe_backend.py @@ -12,7 +12,7 @@ from sglang.test.test_utils import ( popen_launch_server, ) -register_cuda_ci(est_time=800, suite="nightly-4-gpu-b200", nightly=True) +register_cuda_ci(est_time=1770, stage="nightly", runner_config="4-gpu-b200") class FlashinferTrtllmGenMoeBackendFP8Base: diff --git a/test/registered/cuda_graph/piecewise/test_pcg_with_speculative_decoding.py b/test/registered/cuda_graph/piecewise/test_pcg_with_speculative_decoding.py index 0864131d7..b76c48111 100644 --- a/test/registered/cuda_graph/piecewise/test_pcg_with_speculative_decoding.py +++ b/test/registered/cuda_graph/piecewise/test_pcg_with_speculative_decoding.py @@ -11,7 +11,7 @@ import unittest from sglang.test.ci.ci_register import register_cuda_ci from sglang.test.server_fixtures.pcg_spec_fixture import PCGSpecBase -register_cuda_ci(est_time=531, suite="nightly-4-gpu", nightly=True) +register_cuda_ci(est_time=130, stage="nightly", runner_config="4-gpu-h100") class TestPCGWithEAGLE3(PCGSpecBase, unittest.TestCase): diff --git a/test/registered/cuda_graph/piecewise/test_pcg_with_speculative_decoding_dflash.py b/test/registered/cuda_graph/piecewise/test_pcg_with_speculative_decoding_dflash.py index 18dea0c49..e52548d04 100644 --- a/test/registered/cuda_graph/piecewise/test_pcg_with_speculative_decoding_dflash.py +++ b/test/registered/cuda_graph/piecewise/test_pcg_with_speculative_decoding_dflash.py @@ -15,7 +15,7 @@ from sglang.test.test_utils import ( CustomTestCase, ) -register_cuda_ci(est_time=531, suite="nightly-1-gpu", nightly=True) +register_cuda_ci(est_time=110, stage="nightly", runner_config="1-gpu-large") class TestPCGWithDFlash(PCGSpecBase, CustomTestCase): diff --git a/test/registered/cuda_graph/piecewise/test_pcg_with_speculative_decoding_extra.py b/test/registered/cuda_graph/piecewise/test_pcg_with_speculative_decoding_extra.py index cf5b1b64a..5aa0c4dd7 100644 --- a/test/registered/cuda_graph/piecewise/test_pcg_with_speculative_decoding_extra.py +++ b/test/registered/cuda_graph/piecewise/test_pcg_with_speculative_decoding_extra.py @@ -8,7 +8,7 @@ import unittest from sglang.test.ci.ci_register import register_cuda_ci from sglang.test.server_fixtures.pcg_spec_fixture import PCGSpecBase -register_cuda_ci(est_time=531, suite="nightly-4-gpu", nightly=True) +register_cuda_ci(est_time=450, stage="nightly", runner_config="4-gpu-h100") class TestPCGWithMTP(PCGSpecBase, unittest.TestCase): diff --git a/test/registered/cuda_graph/piecewise/test_piecewise_cuda_graph_support_1_gpu.py b/test/registered/cuda_graph/piecewise/test_piecewise_cuda_graph_support_1_gpu.py index 6f7742a75..b9d334251 100644 --- a/test/registered/cuda_graph/piecewise/test_piecewise_cuda_graph_support_1_gpu.py +++ b/test/registered/cuda_graph/piecewise/test_piecewise_cuda_graph_support_1_gpu.py @@ -19,7 +19,7 @@ from sglang.test.test_utils import ( ) # CI Registration -register_cuda_ci(est_time=180, suite="nightly-1-gpu", nightly=True) +register_cuda_ci(est_time=250, stage="nightly", runner_config="1-gpu-large") register_amd_ci(est_time=180, suite="stage-b-test-1-gpu-large-amd") diff --git a/test/registered/dcp/test_qwen3p5_triton_dcp.py b/test/registered/dcp/test_qwen3p5_triton_dcp.py index 9a6fe4d33..030faed71 100644 --- a/test/registered/dcp/test_qwen3p5_triton_dcp.py +++ b/test/registered/dcp/test_qwen3p5_triton_dcp.py @@ -20,7 +20,7 @@ register_amd_ci( suite="nightly-amd-accuracy-8-gpu-mi35x-qwen35-triton-dcp", nightly=True, ) -register_cuda_ci(est_time=4800, suite="nightly-4-gpu-b200", nightly=True) +register_cuda_ci(est_time=4800, stage="nightly", runner_config="4-gpu-b200") QWEN35_MODEL_PATH = os.environ.get("QWEN3_5_MODEL_PATH", "Qwen/Qwen3.5-397B-A17B-FP8") SERVER_LAUNCH_TIMEOUT = 4800 diff --git a/test/registered/debug_utils/test_crash_dump.py b/test/registered/debug_utils/test_crash_dump.py index 232c4386d..a716eb30d 100644 --- a/test/registered/debug_utils/test_crash_dump.py +++ b/test/registered/debug_utils/test_crash_dump.py @@ -21,7 +21,7 @@ from sglang.test.test_utils import ( popen_launch_server, ) -register_cuda_ci(est_time=40, suite="nightly-1-gpu", nightly=True) +register_cuda_ci(est_time=70, stage="nightly", runner_config="1-gpu-large") register_amd_ci(est_time=40, suite="nightly-amd-1-gpu", nightly=True) register_cpu_ci(est_time=225, suite="base-c-test-cpu") diff --git a/test/registered/debug_utils/test_dumper.py b/test/registered/debug_utils/test_dumper.py index d2767c2a9..f5158c086 100644 --- a/test/registered/debug_utils/test_dumper.py +++ b/test/registered/debug_utils/test_dumper.py @@ -50,7 +50,7 @@ from sglang.test.test_utils import ( run_distributed_test, ) -register_cuda_ci(est_time=30, suite="nightly-2-gpu", nightly=True) +register_cuda_ci(est_time=30, stage="nightly", runner_config="2-gpu-large") register_amd_ci(est_time=60, suite="nightly-amd", nightly=True) diff --git a/test/registered/debug_utils/test_engine_dumper_comparator_e2e.py b/test/registered/debug_utils/test_engine_dumper_comparator_e2e.py index 78df91bf6..e38173f37 100644 --- a/test/registered/debug_utils/test_engine_dumper_comparator_e2e.py +++ b/test/registered/debug_utils/test_engine_dumper_comparator_e2e.py @@ -35,7 +35,7 @@ from sglang.test.test_utils import ( popen_launch_server, ) -register_cuda_ci(est_time=300, suite="nightly-4-gpu", nightly=True) +register_cuda_ci(est_time=290, stage="nightly", runner_config="4-gpu-h100") register_amd_ci( est_time=300, suite="nightly-amd-4-gpu", diff --git a/test/registered/debug_utils/test_nightly_precision_regression.py b/test/registered/debug_utils/test_nightly_precision_regression.py index f658f1039..482b4435d 100644 --- a/test/registered/debug_utils/test_nightly_precision_regression.py +++ b/test/registered/debug_utils/test_nightly_precision_regression.py @@ -9,6 +9,8 @@ Env knobs: SGLANG_PRECISION_COMMIT override sglang sha (7-40 hex) tagged on push SGLANG_PRECISION_HF_REPO required HF dataset repo for cross-runner baseline storage; see precision_baseline_store + SGLANG_PRECISION_HF_TOKEN write token for that repo (not HF_TOKEN, which + carries the runner's gated-model read token) """ from __future__ import annotations @@ -47,7 +49,7 @@ try: except Exception: # pragma: no cover _hfs = None -register_cuda_ci(est_time=3600, suite="nightly-precision-8-gpu-h200", nightly=True) +register_cuda_ci(est_time=3600, stage="nightly", runner_config="8-gpu-h200") DEFAULT_MODELS_FOR_NIGHTLY_PRECISION = "zai-org/GLM-5.2-FP8" DEFAULT_DIFF_THRESHOLD = 1e-3 diff --git a/test/registered/debug_utils/test_soft_watchdog.py b/test/registered/debug_utils/test_soft_watchdog.py index 46f4bad2e..8a9030049 100644 --- a/test/registered/debug_utils/test_soft_watchdog.py +++ b/test/registered/debug_utils/test_soft_watchdog.py @@ -17,7 +17,7 @@ from sglang.test.test_utils import ( popen_launch_server, ) -register_cuda_ci(est_time=120, suite="nightly-1-gpu", nightly=True) +register_cuda_ci(est_time=240, stage="nightly", runner_config="1-gpu-large") register_amd_ci(est_time=120, suite="nightly-amd-1-gpu", nightly=True) register_cpu_ci(est_time=622, suite="base-c-test-cpu") diff --git a/test/registered/disaggregation/test_kimi_linear_pd_dcp4.py b/test/registered/disaggregation/test_kimi_linear_pd_dcp4.py index 0d55179ab..59f0e7697 100644 --- a/test/registered/disaggregation/test_kimi_linear_pd_dcp4.py +++ b/test/registered/disaggregation/test_kimi_linear_pd_dcp4.py @@ -19,7 +19,7 @@ from sglang.test.test_utils import ( popen_launch_server, ) -register_cuda_ci(est_time=1200, suite="nightly-8-gpu-b200", nightly=True) +register_cuda_ci(est_time=450, stage="nightly", runner_config="8-gpu-b200") KIMI_LINEAR_MODEL = "moonshotai/Kimi-Linear-48B-A3B-Instruct" PHYSICAL_PAGE_SIZE = 64 diff --git a/test/registered/ep/test_eplb_no_a2a.py b/test/registered/ep/test_eplb_no_a2a.py index 1a9e5d720..959cb3138 100644 --- a/test/registered/ep/test_eplb_no_a2a.py +++ b/test/registered/ep/test_eplb_no_a2a.py @@ -23,7 +23,7 @@ from sglang.test.test_utils import ( try_cached_model, ) -register_cuda_ci(est_time=420, suite="nightly-eval-text-2-gpu", nightly=True) +register_cuda_ci(est_time=200, stage="nightly", runner_config="2-gpu-large") # 72 routed experts + 48 replicas = 120 physical, 60 per rank, so two thirds of # the routed (token, expert) pairs get double-counted when ranks disagree. At 24 diff --git a/test/registered/eval/test_text_models_gsm8k_eval.py b/test/registered/eval/test_text_models_gsm8k_eval.py index cccba0337..db907175f 100644 --- a/test/registered/eval/test_text_models_gsm8k_eval.py +++ b/test/registered/eval/test_text_models_gsm8k_eval.py @@ -23,7 +23,7 @@ from sglang.test.test_utils import ( # downloading on cache miss. Use a longer timeout than the default 600s. NIGHTLY_EVAL_SERVER_TIMEOUT = 1800 -register_cuda_ci(est_time=3600, suite="nightly-eval-text-2-gpu", nightly=True) +register_cuda_ci(est_time=2880, stage="nightly", runner_config="2-gpu-large") MODEL_SCORE_THRESHOLDS = { # sgl-eval (zero-shot chat, \boxed{}, math_verify grading). Thresholds are diff --git a/test/registered/eval/test_vlms_mmmu_eval.py b/test/registered/eval/test_vlms_mmmu_eval.py index 7eb2c1498..939d3ed20 100644 --- a/test/registered/eval/test_vlms_mmmu_eval.py +++ b/test/registered/eval/test_vlms_mmmu_eval.py @@ -19,7 +19,7 @@ from sglang.test.test_utils import ( # Use a longer timeout than the default 600s. NIGHTLY_EVAL_SERVER_TIMEOUT = 1800 -register_cuda_ci(est_time=7200, suite="nightly-eval-vlm-2-gpu", nightly=True) +register_cuda_ci(est_time=7200, stage="nightly", runner_config="2-gpu-large") MODEL_THRESHOLDS = { # Conservative thresholds on 100 MMMU samples, especially for latency thresholds diff --git a/test/registered/gb300/test_deepseek_v4_pro_fp4.py b/test/registered/gb300/test_deepseek_v4_pro_fp4.py index 615e6bc83..6cb02bede 100644 --- a/test/registered/gb300/test_deepseek_v4_pro_fp4.py +++ b/test/registered/gb300/test_deepseek_v4_pro_fp4.py @@ -6,9 +6,7 @@ from sglang.test.performance_test_runner import PerformanceTestParams from sglang.test.run_combined_tests import run_combined_tests from sglang.test.test_utils import ModelLaunchSettings -register_cuda_ci( - est_time=7200, suite="nightly-4-gpu-gb300-deepseek-v4-pro-fp4", nightly=True -) +register_cuda_ci(est_time=7200, stage="nightly", runner_config="4-gpu-gb300") MODEL_PATH = "deepseek-ai/DeepSeek-V4-Pro" SERVER_LAUNCH_TIMEOUT = 3600 diff --git a/test/registered/gb300/test_glm52_nvfp4.py b/test/registered/gb300/test_glm52_nvfp4.py index 1795fb0a4..d193659e3 100644 --- a/test/registered/gb300/test_glm52_nvfp4.py +++ b/test/registered/gb300/test_glm52_nvfp4.py @@ -6,7 +6,7 @@ from sglang.test.performance_test_runner import PerformanceTestParams from sglang.test.run_combined_tests import run_combined_tests from sglang.test.test_utils import ModelLaunchSettings -register_cuda_ci(est_time=7200, suite="nightly-4-gpu-gb300-glm5-nvfp4", nightly=True) +register_cuda_ci(est_time=2280, stage="nightly", runner_config="4-gpu-gb300") MODEL_PATH = "nvidia/GLM-5.2-NVFP4" diff --git a/test/registered/gb300/test_kimi_k25_nvfp4.py b/test/registered/gb300/test_kimi_k25_nvfp4.py index 094654e7e..d28d62115 100644 --- a/test/registered/gb300/test_kimi_k25_nvfp4.py +++ b/test/registered/gb300/test_kimi_k25_nvfp4.py @@ -6,9 +6,7 @@ from sglang.test.performance_test_runner import PerformanceTestParams from sglang.test.run_combined_tests import run_combined_tests from sglang.test.test_utils import ModelLaunchSettings -register_cuda_ci( - est_time=7200, suite="nightly-4-gpu-gb300-kimi-k25-nvfp4", nightly=True -) +register_cuda_ci(est_time=7200, stage="nightly", runner_config="4-gpu-gb300") MODEL_PATH = "nvidia/Kimi-K2.5-NVFP4" DRAFT_MODEL_PATH = "lightseekorg/kimi-k2.5-eagle3-mla" diff --git a/test/registered/gb300/test_qwen35_fp8.py b/test/registered/gb300/test_qwen35_fp8.py index adc372586..1a48a4f68 100644 --- a/test/registered/gb300/test_qwen35_fp8.py +++ b/test/registered/gb300/test_qwen35_fp8.py @@ -6,7 +6,7 @@ from sglang.test.performance_test_runner import PerformanceTestParams from sglang.test.run_combined_tests import run_combined_tests from sglang.test.test_utils import ModelLaunchSettings -register_cuda_ci(est_time=7200, suite="nightly-4-gpu-gb300-qwen35-fp8", nightly=True) +register_cuda_ci(est_time=7200, stage="nightly", runner_config="4-gpu-gb300") MODEL_PATH = "Qwen/Qwen3.5-397B-A17B-FP8" diff --git a/test/registered/kernels/benchmark/kv_canary/bench_plan.py b/test/registered/kernels/benchmark/kv_canary/bench_plan.py index 9bf64fb67..9b952a994 100644 --- a/test/registered/kernels/benchmark/kv_canary/bench_plan.py +++ b/test/registered/kernels/benchmark/kv_canary/bench_plan.py @@ -25,7 +25,7 @@ from sglang.kernels.ops.kv_canary.verify import VerifyPlan from sglang.kernels.ops.kv_canary.write import WritePlan from sglang.test.ci.ci_register import register_amd_ci, register_cuda_ci -register_cuda_ci(est_time=900, suite="nightly-kernel-1-gpu", nightly=True) +register_cuda_ci(est_time=40, stage="nightly", runner_config="1-gpu-large") # AMD mirrors the CUDA nightly registration (nightly-only, no per-PR suite). register_amd_ci(est_time=900, suite="nightly-amd-kernel-1-gpu", nightly=True) diff --git a/test/registered/kernels/benchmark/kv_canary/bench_scatter_req_token_ids.py b/test/registered/kernels/benchmark/kv_canary/bench_scatter_req_token_ids.py index 5eea09a34..159b1bc93 100644 --- a/test/registered/kernels/benchmark/kv_canary/bench_scatter_req_token_ids.py +++ b/test/registered/kernels/benchmark/kv_canary/bench_scatter_req_token_ids.py @@ -16,7 +16,7 @@ from sglang.kernels.ops.kv_canary.scatter_req_token_ids import ( ) from sglang.test.ci.ci_register import register_amd_ci, register_cuda_ci -register_cuda_ci(est_time=180, suite="nightly-kernel-1-gpu", nightly=True) +register_cuda_ci(est_time=20, stage="nightly", runner_config="1-gpu-large") # AMD mirrors the CUDA nightly registration (nightly-only, no per-PR suite). # Note: amd_ci_exec.sh sets SGLANG_IS_IN_CI, so this runs the CI-reduced range # (_BS_AXIS_CI/_SEQ_LEN_AXIS_CI via get_benchmark_range), same as CUDA nightly. diff --git a/test/registered/kernels/benchmark/kv_canary/bench_verify.py b/test/registered/kernels/benchmark/kv_canary/bench_verify.py index 350754183..2d81fd8ef 100644 --- a/test/registered/kernels/benchmark/kv_canary/bench_verify.py +++ b/test/registered/kernels/benchmark/kv_canary/bench_verify.py @@ -32,7 +32,7 @@ from sglang.kernels.ops.kv_canary.verify import ( ) from sglang.test.ci.ci_register import register_amd_ci, register_cuda_ci -register_cuda_ci(est_time=900, suite="nightly-kernel-1-gpu", nightly=True) +register_cuda_ci(est_time=20, stage="nightly", runner_config="1-gpu-large") # AMD mirrors the CUDA nightly registration (nightly-only, no per-PR suite). # Note: amd_ci_exec.sh sets SGLANG_IS_IN_CI, so this runs the CI-reduced range # (build_fast_matrix_cases via get_benchmark_range), same as CUDA nightly. diff --git a/test/registered/kernels/benchmark/kv_canary/bench_write.py b/test/registered/kernels/benchmark/kv_canary/bench_write.py index 79963e0bd..71b22e33f 100644 --- a/test/registered/kernels/benchmark/kv_canary/bench_write.py +++ b/test/registered/kernels/benchmark/kv_canary/bench_write.py @@ -30,7 +30,7 @@ from sglang.kernels.ops.kv_canary.verify import ( from sglang.kernels.ops.kv_canary.write import WritePlan, launch_canary_write_kernel from sglang.test.ci.ci_register import register_amd_ci, register_cuda_ci -register_cuda_ci(est_time=900, suite="nightly-kernel-1-gpu", nightly=True) +register_cuda_ci(est_time=20, stage="nightly", runner_config="1-gpu-large") # AMD mirrors the CUDA nightly registration (nightly-only, no per-PR suite). # Note: amd_ci_exec.sh sets SGLANG_IS_IN_CI, so this runs the CI-reduced range # (build_fast_matrix_cases via get_benchmark_range), same as CUDA nightly. diff --git a/test/registered/kernels/ops/activation/test_activation.py b/test/registered/kernels/ops/activation/test_activation.py index 2a6afd52c..ba57d0667 100644 --- a/test/registered/kernels/ops/activation/test_activation.py +++ b/test/registered/kernels/ops/activation/test_activation.py @@ -14,7 +14,7 @@ from sglang.test.ci.ci_register import register_amd_ci, register_cuda_ci register_cuda_ci(est_time=20, stage="base-b-kernel-unit", runner_config="1-gpu-large") # Nightly is not redundant here: it sets SGLANG_JIT_KERNEL_RUN_FULL_TESTS=1 to expand get_ci_test_range sweeps. -register_cuda_ci(est_time=30, suite="nightly-kernel-1-gpu", nightly=True) +register_cuda_ci(est_time=20, stage="nightly", runner_config="1-gpu-large") register_amd_ci(est_time=20, stage="jit-kernel-unit", runner_config="amd") diff --git a/test/registered/kernels/ops/attention/test_cute_dsl_fp8_paged_mqa_logits.py b/test/registered/kernels/ops/attention/test_cute_dsl_fp8_paged_mqa_logits.py index 4abe52edb..112581f91 100644 --- a/test/registered/kernels/ops/attention/test_cute_dsl_fp8_paged_mqa_logits.py +++ b/test/registered/kernels/ops/attention/test_cute_dsl_fp8_paged_mqa_logits.py @@ -14,7 +14,7 @@ from sglang.srt.layers.attention.dsa.utils import ( from sglang.srt.utils import is_sm100_supported from sglang.test.ci.ci_register import register_cuda_ci -register_cuda_ci(est_time=180, suite="nightly-4-gpu-b200", nightly=True) +register_cuda_ci(est_time=180, stage="nightly", runner_config="4-gpu-b200") BLOCK_KV = 64 HEAD_DIM = 128 diff --git a/test/registered/kernels/ops/attention/test_deepgemm_paged_mqa_logits.py b/test/registered/kernels/ops/attention/test_deepgemm_paged_mqa_logits.py index d8fd96127..a28ebbd07 100644 --- a/test/registered/kernels/ops/attention/test_deepgemm_paged_mqa_logits.py +++ b/test/registered/kernels/ops/attention/test_deepgemm_paged_mqa_logits.py @@ -17,7 +17,7 @@ from sglang.srt.layers.attention.dsa.utils import ( from sglang.srt.utils import is_sm90_supported, is_sm100_supported from sglang.test.ci.ci_register import register_cuda_ci -register_cuda_ci(est_time=40, suite="nightly-4-gpu-b200", nightly=True) +register_cuda_ci(est_time=40, stage="nightly", runner_config="4-gpu-b200") BLOCK_KV = 64 HEAD_DIM = 128 diff --git a/test/registered/kernels/ops/attention/test_rope.py b/test/registered/kernels/ops/attention/test_rope.py index b34608857..d28a022b3 100644 --- a/test/registered/kernels/ops/attention/test_rope.py +++ b/test/registered/kernels/ops/attention/test_rope.py @@ -10,7 +10,7 @@ from sglang.test.ci.ci_register import register_amd_ci, register_cuda_ci register_cuda_ci(est_time=64, stage="base-b-kernel-unit", runner_config="1-gpu-large") # Nightly is not redundant here: it sets SGLANG_JIT_KERNEL_RUN_FULL_TESTS=1 to expand get_ci_test_range sweeps. -register_cuda_ci(est_time=256, suite="nightly-kernel-1-gpu", nightly=True) +register_cuda_ci(est_time=390, stage="nightly", runner_config="1-gpu-large") register_amd_ci(est_time=64, suite="jit-kernel-unit-test-amd") DEVICE = "cuda" diff --git a/test/registered/kernels/ops/communication/test_custom_all_reduce.py b/test/registered/kernels/ops/communication/test_custom_all_reduce.py index ad507b172..7ae2c2366 100644 --- a/test/registered/kernels/ops/communication/test_custom_all_reduce.py +++ b/test/registered/kernels/ops/communication/test_custom_all_reduce.py @@ -46,11 +46,7 @@ register_cuda_ci( runner_config="8-gpu-h200", ) # Nightly is not redundant here: it sets SGLANG_JIT_KERNEL_RUN_FULL_TESTS=1 to expand get_ci_test_range sweeps. -register_cuda_ci( - est_time=300, - suite="nightly-kernel-8-gpu-h200", - nightly=True, -) +register_cuda_ci(est_time=110, stage="nightly", runner_config="8-gpu-h200") # --------------------------------------------------------------------------- # Test parameters diff --git a/test/registered/kernels/ops/communication/test_symm_mem_all_gather.py b/test/registered/kernels/ops/communication/test_symm_mem_all_gather.py index bc73dde3d..d24472ec9 100644 --- a/test/registered/kernels/ops/communication/test_symm_mem_all_gather.py +++ b/test/registered/kernels/ops/communication/test_symm_mem_all_gather.py @@ -36,7 +36,7 @@ from sglang.test.kernels.utils import multigpu_pytest_main register_cuda_ci(est_time=240, stage="extra-b", runner_config="8-gpu-h200") # Nightly is not redundant here: it sets SGLANG_JIT_KERNEL_RUN_FULL_TESTS=1 to expand get_ci_test_range sweeps. -register_cuda_ci(est_time=240, suite="nightly-kernel-8-gpu-h200", nightly=True) +register_cuda_ci(est_time=70, stage="nightly", runner_config="8-gpu-h200") # --------------------------------------------------------------------------- # Test parameters diff --git a/test/registered/kernels/ops/diffusion/test_qknorm_rope.py b/test/registered/kernels/ops/diffusion/test_qknorm_rope.py index 98c96d092..264d1c0bd 100644 --- a/test/registered/kernels/ops/diffusion/test_qknorm_rope.py +++ b/test/registered/kernels/ops/diffusion/test_qknorm_rope.py @@ -10,7 +10,7 @@ from sglang.test.ci.ci_register import register_cuda_ci register_cuda_ci(est_time=44, stage="base-b-kernel-unit", runner_config="1-gpu-large") # Nightly is not redundant here: it sets SGLANG_JIT_KERNEL_RUN_FULL_TESTS=1 to expand get_ci_test_range sweeps. -register_cuda_ci(est_time=176, suite="nightly-kernel-1-gpu", nightly=True) +register_cuda_ci(est_time=220, stage="nightly", runner_config="1-gpu-large") DEVICE = "cuda" DTYPE = torch.bfloat16 diff --git a/test/registered/kernels/ops/diffusion/test_qwen_image_modulation.py b/test/registered/kernels/ops/diffusion/test_qwen_image_modulation.py index 5682bbadc..f0a5baa17 100644 --- a/test/registered/kernels/ops/diffusion/test_qwen_image_modulation.py +++ b/test/registered/kernels/ops/diffusion/test_qwen_image_modulation.py @@ -14,7 +14,7 @@ from sglang.test.ci.ci_register import register_amd_ci, register_cuda_ci register_cuda_ci(est_time=15, stage="base-b-kernel-unit", runner_config="1-gpu-large") # Nightly is not redundant here: it sets SGLANG_JIT_KERNEL_RUN_FULL_TESTS=1 to expand get_ci_test_range sweeps. -register_cuda_ci(est_time=120, suite="nightly-kernel-1-gpu", nightly=True) +register_cuda_ci(est_time=30, stage="nightly", runner_config="1-gpu-large") register_amd_ci(est_time=30, suite="nightly-amd-kernel-1-gpu", nightly=True) DEVICE = "cuda" diff --git a/test/registered/kernels/ops/diffusion/test_timestep_embedding.py b/test/registered/kernels/ops/diffusion/test_timestep_embedding.py index 59e13a672..d803b336f 100644 --- a/test/registered/kernels/ops/diffusion/test_timestep_embedding.py +++ b/test/registered/kernels/ops/diffusion/test_timestep_embedding.py @@ -18,7 +18,7 @@ from sglang.test.ci.ci_register import register_cuda_ci register_cuda_ci(est_time=16, stage="base-b-kernel-unit", runner_config="1-gpu-large") # Nightly is not redundant here: it sets SGLANG_JIT_KERNEL_RUN_FULL_TESTS=1 to expand get_ci_test_range sweeps. -register_cuda_ci(est_time=120, suite="nightly-kernel-1-gpu", nightly=True) +register_cuda_ci(est_time=20, stage="nightly", runner_config="1-gpu-large") CORRECTNESS_BATCH_SIZES = get_ci_test_range( [1, 2, 8, 128, 256, 512, 1536, 2048, 4096, 11008, 16384], diff --git a/test/registered/kernels/ops/diffusion/test_varlen_pack_pad.py b/test/registered/kernels/ops/diffusion/test_varlen_pack_pad.py index 222f1d1a1..dfdea7838 100644 --- a/test/registered/kernels/ops/diffusion/test_varlen_pack_pad.py +++ b/test/registered/kernels/ops/diffusion/test_varlen_pack_pad.py @@ -17,7 +17,7 @@ from sglang.test.ci.ci_register import register_amd_ci, register_cuda_ci register_cuda_ci(est_time=10, stage="base-b-kernel-unit", runner_config="1-gpu-large") # Nightly is not redundant here: it sets SGLANG_JIT_KERNEL_RUN_FULL_TESTS=1 to expand get_ci_test_range sweeps. -register_cuda_ci(est_time=60, suite="nightly-kernel-1-gpu", nightly=True) +register_cuda_ci(est_time=20, stage="nightly", runner_config="1-gpu-large") register_amd_ci(est_time=15, suite="nightly-amd-kernel-1-gpu", nightly=True) DEVICE = "cuda" diff --git a/test/registered/kernels/ops/diffusion/test_varlen_uspattn_equivalence.py b/test/registered/kernels/ops/diffusion/test_varlen_uspattn_equivalence.py index c6daac567..db67a6cf1 100644 --- a/test/registered/kernels/ops/diffusion/test_varlen_uspattn_equivalence.py +++ b/test/registered/kernels/ops/diffusion/test_varlen_uspattn_equivalence.py @@ -31,7 +31,7 @@ from sglang.test.ci.ci_register import register_cuda_ci register_cuda_ci(est_time=15, stage="base-b-kernel-unit", runner_config="1-gpu-large") # Nightly is not redundant here: it sets SGLANG_JIT_KERNEL_RUN_FULL_TESTS=1 to expand get_ci_test_range sweeps. -register_cuda_ci(est_time=60, suite="nightly-kernel-1-gpu", nightly=True) +register_cuda_ci(est_time=20, stage="nightly", runner_config="1-gpu-large") DEVICE = "cuda" DTYPES = get_ci_test_range([torch.bfloat16, torch.float16], [torch.bfloat16]) diff --git a/test/registered/kernels/ops/gemm/test_dsv3_router_gemm.py b/test/registered/kernels/ops/gemm/test_dsv3_router_gemm.py index d5d4d76c0..981fbd959 100644 --- a/test/registered/kernels/ops/gemm/test_dsv3_router_gemm.py +++ b/test/registered/kernels/ops/gemm/test_dsv3_router_gemm.py @@ -16,7 +16,7 @@ from sglang.test.ci.ci_register import register_cuda_ci register_cuda_ci(est_time=37, stage="base-b-kernel-unit", runner_config="1-gpu-large") # Nightly is not redundant here: it sets SGLANG_JIT_KERNEL_RUN_FULL_TESTS=1 to expand get_ci_test_range sweeps. -register_cuda_ci(est_time=148, suite="nightly-kernel-1-gpu", nightly=True) +register_cuda_ci(est_time=110, stage="nightly", runner_config="1-gpu-large") HIDDEN_DIMS = [1024, 4096, 5120, 6144, 7168] ROUTER_GEMM_CASES = get_ci_test_range( diff --git a/test/registered/kernels/ops/kimi_k3/test_collectives.py b/test/registered/kernels/ops/kimi_k3/test_collectives.py index dcaca8d5f..faab5e8bc 100644 --- a/test/registered/kernels/ops/kimi_k3/test_collectives.py +++ b/test/registered/kernels/ops/kimi_k3/test_collectives.py @@ -24,7 +24,7 @@ from sglang.test.ci.ci_register import register_cuda_ci from sglang.test.kernels.utils import multigpu_pytest_main register_cuda_ci(est_time=240, stage="base-c", runner_config="4-gpu-b200") -register_cuda_ci(est_time=480, suite="nightly-8-gpu-b200", nightly=True) +register_cuda_ci(est_time=480, stage="nightly", runner_config="8-gpu-b200") _HIDDEN_SIZE = 7168 _GEMM_AR_K_TOTAL = 12288 diff --git a/test/registered/kernels/ops/kvcache/test_store_cache.py b/test/registered/kernels/ops/kvcache/test_store_cache.py index a804a1ca6..6e0be27f3 100644 --- a/test/registered/kernels/ops/kvcache/test_store_cache.py +++ b/test/registered/kernels/ops/kvcache/test_store_cache.py @@ -10,7 +10,7 @@ from sglang.test.ci.ci_register import register_amd_ci, register_cuda_ci register_cuda_ci(est_time=28, stage="base-b-kernel-unit", runner_config="1-gpu-large") # Nightly is not redundant here: it sets SGLANG_JIT_KERNEL_RUN_FULL_TESTS=1 to expand get_ci_test_range sweeps. -register_cuda_ci(est_time=120, suite="nightly-kernel-1-gpu", nightly=True) +register_cuda_ci(est_time=40, stage="nightly", runner_config="1-gpu-large") register_amd_ci(est_time=55, stage="jit-kernel-unit", runner_config="amd") BS_LIST = [2**n for n in range(0, 15)] diff --git a/test/registered/kernels/ops/layernorm/test_fused_add_rmsnorm.py b/test/registered/kernels/ops/layernorm/test_fused_add_rmsnorm.py index 777e17715..b1afd5e1f 100644 --- a/test/registered/kernels/ops/layernorm/test_fused_add_rmsnorm.py +++ b/test/registered/kernels/ops/layernorm/test_fused_add_rmsnorm.py @@ -9,7 +9,7 @@ from sglang.test.ci.ci_register import register_cuda_ci register_cuda_ci(est_time=10, stage="base-b-kernel-unit", runner_config="1-gpu-large") # Nightly is not redundant here: it sets SGLANG_JIT_KERNEL_RUN_FULL_TESTS=1 to expand get_ci_test_range sweeps. -register_cuda_ci(est_time=120, suite="nightly-kernel-1-gpu", nightly=True) +register_cuda_ci(est_time=20, stage="nightly", runner_config="1-gpu-large") def sglang_jit_fused_add_rmsnorm( diff --git a/test/registered/kernels/ops/layernorm/test_qknorm.py b/test/registered/kernels/ops/layernorm/test_qknorm.py index d36b008fb..7b85ca675 100644 --- a/test/registered/kernels/ops/layernorm/test_qknorm.py +++ b/test/registered/kernels/ops/layernorm/test_qknorm.py @@ -10,7 +10,7 @@ from sglang.test.ci.ci_register import register_cuda_ci register_cuda_ci(est_time=37, stage="base-b-kernel-unit", runner_config="1-gpu-large") # Nightly is not redundant here: it sets SGLANG_JIT_KERNEL_RUN_FULL_TESTS=1 to expand get_ci_test_range sweeps. -register_cuda_ci(est_time=148, suite="nightly-kernel-1-gpu", nightly=True) +register_cuda_ci(est_time=130, stage="nightly", runner_config="1-gpu-large") def sglang_aot_qknorm( diff --git a/test/registered/kernels/ops/layernorm/test_qknorm_across_heads.py b/test/registered/kernels/ops/layernorm/test_qknorm_across_heads.py index c7dbf5007..414d18abc 100644 --- a/test/registered/kernels/ops/layernorm/test_qknorm_across_heads.py +++ b/test/registered/kernels/ops/layernorm/test_qknorm_across_heads.py @@ -10,7 +10,7 @@ from sglang.test.ci.ci_register import register_cuda_ci register_cuda_ci(est_time=15, stage="base-b-kernel-unit", runner_config="1-gpu-large") # Nightly is not redundant here: it sets SGLANG_JIT_KERNEL_RUN_FULL_TESTS=1 to expand get_ci_test_range sweeps. -register_cuda_ci(est_time=120, suite="nightly-kernel-1-gpu", nightly=True) +register_cuda_ci(est_time=20, stage="nightly", runner_config="1-gpu-large") def sglang_jit_qknorm_across_heads( diff --git a/test/registered/kernels/ops/layernorm/test_rmsnorm.py b/test/registered/kernels/ops/layernorm/test_rmsnorm.py index 148e40641..53a5c64cf 100644 --- a/test/registered/kernels/ops/layernorm/test_rmsnorm.py +++ b/test/registered/kernels/ops/layernorm/test_rmsnorm.py @@ -10,7 +10,7 @@ from sglang.test.ci.ci_register import register_amd_ci, register_cuda_ci register_cuda_ci(est_time=45, stage="base-b-kernel-unit", runner_config="1-gpu-large") # Nightly is not redundant here: it sets SGLANG_JIT_KERNEL_RUN_FULL_TESTS=1 to expand get_ci_test_range sweeps. -register_cuda_ci(est_time=240, suite="nightly-kernel-1-gpu", nightly=True) +register_cuda_ci(est_time=160, stage="nightly", runner_config="1-gpu-large") register_amd_ci(est_time=45, suite="jit-kernel-unit-test-amd") diff --git a/test/registered/kernels/ops/layernorm/test_rmsnorm_hf.py b/test/registered/kernels/ops/layernorm/test_rmsnorm_hf.py index ce00a5cbe..2a380f363 100644 --- a/test/registered/kernels/ops/layernorm/test_rmsnorm_hf.py +++ b/test/registered/kernels/ops/layernorm/test_rmsnorm_hf.py @@ -15,7 +15,7 @@ from sglang.test.ci.ci_register import register_amd_ci, register_cuda_ci register_cuda_ci(est_time=30, stage="base-b-kernel-unit", runner_config="1-gpu-large") # Nightly is not redundant here: it sets SGLANG_JIT_KERNEL_RUN_FULL_TESTS=1 to expand get_ci_test_range sweeps. -register_cuda_ci(est_time=120, suite="nightly-kernel-1-gpu", nightly=True) +register_cuda_ci(est_time=100, stage="nightly", runner_config="1-gpu-large") register_amd_ci(est_time=30, stage="jit-kernel-unit", runner_config="amd") EPS = 1e-5 diff --git a/test/registered/kernels/ops/moe/test_fp4_moe.py b/test/registered/kernels/ops/moe/test_fp4_moe.py index 8263cfa3b..1d7c375f4 100644 --- a/test/registered/kernels/ops/moe/test_fp4_moe.py +++ b/test/registered/kernels/ops/moe/test_fp4_moe.py @@ -16,7 +16,7 @@ from sglang.test.quant_ref_utils import ( dequantize_nvfp4_to_dtype, ) -register_cuda_ci(est_time=300, suite="nightly-4-gpu-b200", nightly=True) +register_cuda_ci(est_time=300, stage="nightly", runner_config="4-gpu-b200") if torch.cuda.get_device_capability() < (10, 0): pytest.skip( diff --git a/test/registered/kernels/ops/moe/test_fused_topk_deepseek.py b/test/registered/kernels/ops/moe/test_fused_topk_deepseek.py index ea7347a3c..1afd8744b 100644 --- a/test/registered/kernels/ops/moe/test_fused_topk_deepseek.py +++ b/test/registered/kernels/ops/moe/test_fused_topk_deepseek.py @@ -7,7 +7,7 @@ from sglang.srt.layers.moe.topk import biased_grouped_topk_gpu, biased_grouped_t from sglang.srt.utils import get_device from sglang.test.ci.ci_register import register_cuda_ci -register_cuda_ci(est_time=2, suite="nightly-1-gpu", nightly=True) +register_cuda_ci(est_time=40, stage="nightly", runner_config="1-gpu-large") @pytest.mark.parametrize( diff --git a/test/registered/kernels/ops/moe/test_moe_align_block_size.py b/test/registered/kernels/ops/moe/test_moe_align_block_size.py index 3809c4e43..326905ec6 100644 --- a/test/registered/kernels/ops/moe/test_moe_align_block_size.py +++ b/test/registered/kernels/ops/moe/test_moe_align_block_size.py @@ -12,7 +12,7 @@ from sglang.test.ci.ci_register import register_cuda_ci register_cuda_ci(est_time=28, stage="base-b-kernel-unit", runner_config="1-gpu-large") # Nightly is not redundant here: it sets SGLANG_JIT_KERNEL_RUN_FULL_TESTS=1 to expand get_ci_test_range sweeps. -register_cuda_ci(est_time=120, suite="nightly-kernel-1-gpu", nightly=True) +register_cuda_ci(est_time=50, stage="nightly", runner_config="1-gpu-large") def ceil_div(a, b): diff --git a/test/registered/kernels/ops/quantization/test_per_tensor_quant_fp8.py b/test/registered/kernels/ops/quantization/test_per_tensor_quant_fp8.py index 92cbf546e..1f013dc14 100644 --- a/test/registered/kernels/ops/quantization/test_per_tensor_quant_fp8.py +++ b/test/registered/kernels/ops/quantization/test_per_tensor_quant_fp8.py @@ -13,7 +13,7 @@ from sglang.test.ci.ci_register import register_cuda_ci register_cuda_ci(est_time=16, stage="base-b-kernel-unit", runner_config="1-gpu-large") # Nightly is not redundant here: it sets SGLANG_JIT_KERNEL_RUN_FULL_TESTS=1 to expand get_ci_test_range sweeps. -register_cuda_ci(est_time=120, suite="nightly-kernel-1-gpu", nightly=True) +register_cuda_ci(est_time=20, stage="nightly", runner_config="1-gpu-large") try: from sglang.srt.utils import is_hip diff --git a/test/registered/lora/test_lora_deepseek_v3_base_logprob_diff.py b/test/registered/lora/test_lora_deepseek_v3_base_logprob_diff.py index 8ae8dbd15..f73fcc4da 100644 --- a/test/registered/lora/test_lora_deepseek_v3_base_logprob_diff.py +++ b/test/registered/lora/test_lora_deepseek_v3_base_logprob_diff.py @@ -34,10 +34,7 @@ import sglang as sgl from sglang.test.ci.ci_register import register_cuda_ci from sglang.test.test_utils import CustomTestCase -register_cuda_ci( - est_time=300, - suite="nightly-8-gpu-b200", -) +register_cuda_ci(est_time=1800, stage="nightly", runner_config="8-gpu-b200") BASE_MODEL = "deepseek-ai/DeepSeek-V3.1-Base" LORA_HF_REPO = "yushengsu/lora-diff-DeepSeek-V3.1-Base" diff --git a/test/registered/lora/test_lora_eviction_policy.py b/test/registered/lora/test_lora_eviction_policy.py index 4b4923a32..0aedcdf3b 100644 --- a/test/registered/lora/test_lora_eviction_policy.py +++ b/test/registered/lora/test_lora_eviction_policy.py @@ -27,7 +27,7 @@ from sglang.test.ci.ci_register import ( register_xpu_ci, ) -register_cuda_ci(est_time=200, suite="nightly-1-gpu", nightly=True) +register_cuda_ci(est_time=10, stage="nightly", runner_config="1-gpu-large") register_amd_ci(est_time=200, suite="nightly-amd-1-gpu", nightly=True) register_cpu_ci(est_time=6, suite="base-c-test-cpu") register_xpu_ci(est_time=10, suite="stage-a-test-1-gpu-xpu") diff --git a/test/registered/lora/test_lora_kimi_k25_logprob_diff.py b/test/registered/lora/test_lora_kimi_k25_logprob_diff.py index b64b8e4b0..a62348aaf 100644 --- a/test/registered/lora/test_lora_kimi_k25_logprob_diff.py +++ b/test/registered/lora/test_lora_kimi_k25_logprob_diff.py @@ -34,10 +34,7 @@ import sglang as sgl from sglang.test.ci.ci_register import register_cuda_ci from sglang.test.test_utils import CustomTestCase -register_cuda_ci( - est_time=360, - suite="nightly-8-gpu-b200", -) +register_cuda_ci(est_time=420, stage="nightly", runner_config="8-gpu-b200") BASE_MODEL = "moonshotai/Kimi-K2.5" LORA_HF_REPO = "yushengsu/lora-diff-Kimi-K2.5" diff --git a/test/registered/lora/test_lora_openai_compatible.py b/test/registered/lora/test_lora_openai_compatible.py index 92188426a..79d1250a6 100644 --- a/test/registered/lora/test_lora_openai_compatible.py +++ b/test/registered/lora/test_lora_openai_compatible.py @@ -27,7 +27,7 @@ from sglang.test.test_utils import ( popen_launch_server, ) -register_cuda_ci(est_time=150, suite="nightly-1-gpu", nightly=True) +register_cuda_ci(est_time=180, stage="nightly", runner_config="1-gpu-large") register_amd_ci(est_time=150, suite="nightly-amd-1-gpu", nightly=True) diff --git a/test/registered/model_loading/test_prefetch_checkpoints_multi_gpu.py b/test/registered/model_loading/test_prefetch_checkpoints_multi_gpu.py index bad024095..b1778fb04 100644 --- a/test/registered/model_loading/test_prefetch_checkpoints_multi_gpu.py +++ b/test/registered/model_loading/test_prefetch_checkpoints_multi_gpu.py @@ -4,7 +4,7 @@ import sglang as sgl from sglang.test.ci.ci_register import register_cuda_ci from sglang.test.test_utils import CustomTestCase -register_cuda_ci(est_time=300, suite="nightly-4-gpu") +register_cuda_ci(est_time=300, stage="nightly", runner_config="4-gpu-h100") PROMPTS = [ "Hello, my name is", diff --git a/test/registered/model_loading/test_runai_model_loader.py b/test/registered/model_loading/test_runai_model_loader.py index b5a952ab2..e830671d2 100644 --- a/test/registered/model_loading/test_runai_model_loader.py +++ b/test/registered/model_loading/test_runai_model_loader.py @@ -5,7 +5,7 @@ from sglang.srt.utils.common import temp_set_env from sglang.test.ci.ci_register import register_cuda_ci from sglang.test.test_utils import CustomTestCase -register_cuda_ci(est_time=380, suite="nightly-1-gpu", nightly=True) +register_cuda_ci(est_time=50, stage="nightly", runner_config="1-gpu-large") TEST_GCS_MODEL = "gs://vertex-model-garden-public-us/codegemma/codegemma-2b/" diff --git a/test/registered/models_e2e/test_dsa_glm52_cache_layer_split.py b/test/registered/models_e2e/test_dsa_glm52_cache_layer_split.py index ea192f116..8fbe21aa6 100644 --- a/test/registered/models_e2e/test_dsa_glm52_cache_layer_split.py +++ b/test/registered/models_e2e/test_dsa_glm52_cache_layer_split.py @@ -20,11 +20,7 @@ from sglang.test.server_fixtures.disaggregation_fixture import ( PDDisaggregationServerBase, ) -register_cuda_ci( - est_time=450, - suite="nightly-8-gpu-b200", - nightly=True, -) +register_cuda_ci(est_time=750, stage="nightly", runner_config="8-gpu-b200") class TestGLM52DSACacheLayerSplit(PDDisaggregationServerBase, GSM8KMixin): diff --git a/test/registered/models_e2e/test_kimi_linear_unified_memory.py b/test/registered/models_e2e/test_kimi_linear_unified_memory.py index efec93bba..45320227a 100644 --- a/test/registered/models_e2e/test_kimi_linear_unified_memory.py +++ b/test/registered/models_e2e/test_kimi_linear_unified_memory.py @@ -32,7 +32,7 @@ from sglang.test.kits.eval_accuracy_kit import GSM8KMixin from sglang.test.kits.prefix_cache_branching_kit import PrefixCacheBranchingMixin from sglang.test.server_fixtures.default_fixture import DefaultServerBase -register_cuda_ci(est_time=600, suite="nightly-4-gpu", nightly=True) +register_cuda_ci(est_time=570, stage="nightly", runner_config="4-gpu-h100") KIMI_LINEAR_MODEL = "moonshotai/Kimi-Linear-48B-A3B-Instruct" diff --git a/test/registered/moe/test_hybrid_dp_ep_tp_mtp.py b/test/registered/moe/test_hybrid_dp_ep_tp_mtp.py index 49fbaa572..3b36d5eb4 100644 --- a/test/registered/moe/test_hybrid_dp_ep_tp_mtp.py +++ b/test/registered/moe/test_hybrid_dp_ep_tp_mtp.py @@ -17,7 +17,7 @@ from sglang.test.test_utils import ( # 60 test classes testing hybrid parallelism configurations # Each test launches server + runs MMLU eval (~90s per test) -register_cuda_ci(est_time=5400, suite="weekly-8-gpu-h200", nightly=True) +register_cuda_ci(est_time=8160, stage="weekly", runner_config="8-gpu-h200") class Test00(CustomTestCase): diff --git a/test/registered/perf/test_text_models_perf.py b/test/registered/perf/test_text_models_perf.py index d4f538950..9390622bc 100644 --- a/test/registered/perf/test_text_models_perf.py +++ b/test/registered/perf/test_text_models_perf.py @@ -9,7 +9,7 @@ from sglang.test.test_utils import ( parse_models, ) -register_cuda_ci(est_time=3600, suite="nightly-perf-text-2-gpu", nightly=True) +register_cuda_ci(est_time=450, stage="nightly", runner_config="2-gpu-large") RESULT_DIR = "performance_results_text_models" diff --git a/test/registered/perf/test_vlms_perf.py b/test/registered/perf/test_vlms_perf.py index 433805a77..ca09d5a28 100644 --- a/test/registered/perf/test_vlms_perf.py +++ b/test/registered/perf/test_vlms_perf.py @@ -11,7 +11,7 @@ from sglang.test.test_utils import ( parse_models, ) -register_cuda_ci(est_time=7200, suite="nightly-perf-vlm-2-gpu", nightly=True) +register_cuda_ci(est_time=750, stage="nightly", runner_config="2-gpu-large") RESULT_DIR = "performance_results_vlms" diff --git a/test/registered/radix_cache/unified_radix_tree/test_unified_radix_cache_kl_mimo.py b/test/registered/radix_cache/unified_radix_tree/test_unified_radix_cache_kl_mimo.py index 1eb02974e..1ce60c288 100644 --- a/test/registered/radix_cache/unified_radix_tree/test_unified_radix_cache_kl_mimo.py +++ b/test/registered/radix_cache/unified_radix_tree/test_unified_radix_cache_kl_mimo.py @@ -23,7 +23,7 @@ MIMO_LAUNCH_TIMEOUT = 3600 # MiMo V2.5 is pre-cached on the eight-H200 runner. The H200-only nightly suite # exercises the asymmetric MHA host pool end to end without adding PR CI cost. -register_cuda_ci(est_time=1200, suite="nightly-8-gpu-h200", nightly=True) +register_cuda_ci(est_time=270, stage="nightly", runner_config="8-gpu-h200") class TestUnifiedMiMoHiCacheLoadBackKL(CustomTestCase): diff --git a/test/registered/radix_cache/unified_radix_tree/test_unified_radix_cache_kl_nightly.py b/test/registered/radix_cache/unified_radix_tree/test_unified_radix_cache_kl_nightly.py index 7f1dc2175..8cb4aa29c 100644 --- a/test/registered/radix_cache/unified_radix_tree/test_unified_radix_cache_kl_nightly.py +++ b/test/registered/radix_cache/unified_radix_tree/test_unified_radix_cache_kl_nightly.py @@ -26,7 +26,7 @@ from sglang.test.test_utils import ( GLM5_MODEL = "zai-org/GLM-5.2-FP8" GLM5_LAUNCH_TIMEOUT = 3600 -register_cuda_ci(est_time=900, suite="nightly-8-gpu-h200", nightly=True) +register_cuda_ci(est_time=690, stage="nightly", runner_config="8-gpu-h200") class AccuracyTwoPassMixin: diff --git a/test/registered/rl/test_weight_checker_e2e.py b/test/registered/rl/test_weight_checker_e2e.py index bff4ad102..6a0b40030 100644 --- a/test/registered/rl/test_weight_checker_e2e.py +++ b/test/registered/rl/test_weight_checker_e2e.py @@ -34,7 +34,7 @@ from sglang.test.test_utils import ( popen_launch_server, ) -register_cuda_ci(est_time=150, suite="nightly-1-gpu", nightly=True) +register_cuda_ci(est_time=60, stage="nightly", runner_config="1-gpu-large") _MODEL_NAME = "Qwen/Qwen3-0.6B" # We address the up half via the HF-style unfused name "up_proj.weight". sglang's diff --git a/test/registered/scheduler/test_routing_key_scheduling.py b/test/registered/scheduler/test_routing_key_scheduling.py index c4411d53a..b39b5aa50 100644 --- a/test/registered/scheduler/test_routing_key_scheduling.py +++ b/test/registered/scheduler/test_routing_key_scheduling.py @@ -20,7 +20,7 @@ from sglang.test.test_utils import ( popen_launch_server, ) -register_cuda_ci(est_time=120, suite="nightly-1-gpu", nightly=True) +register_cuda_ci(est_time=50, stage="nightly", runner_config="1-gpu-large") register_amd_ci(est_time=120, suite="nightly-amd-1-gpu", nightly=True) register_cpu_ci(est_time=184, suite="base-c-test-cpu") diff --git a/test/registered/unit/batch_invariant_ops/test_batch_invariant_ops.py b/test/registered/unit/batch_invariant_ops/test_batch_invariant_ops.py index fc989ac53..caf46a986 100644 --- a/test/registered/unit/batch_invariant_ops/test_batch_invariant_ops.py +++ b/test/registered/unit/batch_invariant_ops/test_batch_invariant_ops.py @@ -11,7 +11,7 @@ from sglang.test.test_utils import CustomTestCase # Note: MI300 (gfx942) has 64KB shared memory limit but kernel needs 66KB # MI35x (gfx950/CDNA4) may have different limits - testing on MI35x only -register_cuda_ci(est_time=10, suite="nightly-1-gpu", nightly=True) +register_cuda_ci(est_time=20, stage="nightly", runner_config="1-gpu-large") register_amd_ci(est_time=10, suite="nightly-amd-1-gpu-mi35x", nightly=True) device_type = getattr(torch.accelerator.current_accelerator(), "type", "cpu") diff --git a/test/registered/vlm/test_encoder_dp.py b/test/registered/vlm/test_encoder_dp.py index 286ea2d08..195775005 100644 --- a/test/registered/vlm/test_encoder_dp.py +++ b/test/registered/vlm/test_encoder_dp.py @@ -7,7 +7,7 @@ from sglang.test.ci.ci_register import register_amd_ci, register_cuda_ci from sglang.test.kits.mmmu_vlm_kit import MMMUMultiModelTestBase from sglang.test.test_utils import is_in_ci -register_cuda_ci(est_time=500, suite="nightly-4-gpu", nightly=True) +register_cuda_ci(est_time=510, stage="nightly", runner_config="4-gpu-h100") register_amd_ci(est_time=500, suite="nightly-amd-4-gpu", nightly=True) MODELS = [ diff --git a/test/run_suite.py b/test/run_suite.py index b4ac59623..b4fb70de2 100644 --- a/test/run_suite.py +++ b/test/run_suite.py @@ -121,34 +121,16 @@ PER_COMMIT_SUITES = { # Nightly test suites (run nightly, organized by GPU configuration) NIGHTLY_SUITES = { HWBackend.CUDA: [ - "nightly-1-gpu", - "nightly-2-gpu", - "nightly-4-gpu", - "nightly-4-gpu-b200", - "nightly-8-gpu", - "nightly-8-gpu-h200", - "nightly-8-gpu-h20", - "nightly-8-gpu-b200", - "nightly-8-gpu-h200-basic", # Basic tests for large models on H200 - "nightly-8-gpu-b200-basic", # Basic tests for large models on B200 - "nightly-8-gpu-common", # Common tests that run on both H200 and B200 - "nightly-kernel-1-gpu", - "nightly-kernel-8-gpu-h200", - # Eval and perf suites (2-gpu) - "nightly-eval-text-2-gpu", - "nightly-eval-vlm-2-gpu", - "nightly-perf-text-2-gpu", - "nightly-perf-vlm-2-gpu", - # GB300 (4x GB300 NVL4) nightly suites - "nightly-4-gpu-gb300", - "nightly-4-gpu-gb300-deepseek-v4-pro-fp4", - "nightly-4-gpu-gb300-glm5-nvfp4", - "nightly-4-gpu-gb300-kimi-k25", - "nightly-4-gpu-gb300-kimi-k25-nvfp4", - "nightly-4-gpu-gb300-qwen35-fp8", - "nightly-4-gpu-gb300-qwen35-nvfp4", - # Nightly precision regression (per-layer hidden state comparison) - "nightly-precision-8-gpu-h200", + # `stage="nightly"` + a runner_config, same `{stage}-test-{runner_config}` + # shape as the per-commit suites. No `nightly=True`: the stage name + # carries the cadence; only the legacy suites below still need the flag. + "nightly-test-1-gpu-large", + "nightly-test-2-gpu-large", + "nightly-test-4-gpu-h100", + "nightly-test-4-gpu-b200", + "nightly-test-4-gpu-gb300", + "nightly-test-8-gpu-h200", + "nightly-test-8-gpu-b200", ], HWBackend.AMD: [ "nightly-amd", @@ -194,7 +176,9 @@ OTHER_SUITES = { ], HWBackend.CUDA: [ "stress", - "weekly-8-gpu-h200", + # `stage="weekly"` -- same shape. The three dicts group names for + # readability only; validation reads their union. + "weekly-test-8-gpu-h200", ], } @@ -244,14 +228,10 @@ def filter_tests( if t.backend == hw and t.effective_suite == suite and t.nightly == nightly ] - valid_suites = ( - NIGHTLY_SUITES.get(hw, []) if nightly else PER_COMMIT_SUITES.get(hw, []) - ) - - if suite not in valid_suites: - print( - f"Warning: Unknown suite {suite} for backend {hw.name}, nightly={nightly}" - ) + # Union of all three dicts, not just the per-commit or nightly half: + # CUDA nightly suites are selected by name alone, without --nightly. + if suite not in _valid_suites_by_backend().get(hw, set()): + print(f"Warning: Unknown suite {suite} for backend {hw.name}") enabled_tests = [t for t in ci_tests if t.disabled is None] skipped_tests = [t for t in ci_tests if t.disabled is not None] @@ -369,9 +349,11 @@ def run_a_suite(args): pretty_print_tests(args, ci_tests, skipped_tests) + # None hands the per-file budget over to est_time (see run_unittest_files). + timeout = None if args.timeout_from_est_time else args.timeout_per_file + # Add extra timeout when retry is enabled - timeout = args.timeout_per_file - if args.enable_retry: + if timeout is not None and args.enable_retry: timeout += args.retry_timeout_increase return run_unittest_files( @@ -399,7 +381,10 @@ def main(): parser.add_argument( "--nightly", action="store_true", - help="Run nightly tests instead of per-commit tests.", + help=( + "Include tests registered with nightly=True (AMD/CPU/NPU). CUDA " + "scheduled suites are selected by name and take no flag." + ), ) parser.add_argument( "--timeout-per-file", @@ -407,6 +392,14 @@ def main(): default=1200, help="The time limit for running one file in seconds (default: 1200).", ) + parser.add_argument( + "--timeout-from-est-time", + action="store_true", + help=( + "Derive each file's time limit from its own est_time instead of " + "the flat --timeout-per-file, for suites mixing fast and slow tests." + ), + ) parser.add_argument( "--continue-on-error", action="store_true",