docs: improve CI and testing documentation (#21202)
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Lianmin Zheng
co-authored by
Claude Opus 4.6
Claude Opus 4.6
parent
b4d3fb001d
commit
27ac831a84
+185
-77
@@ -1,40 +1,143 @@
|
||||
# Run Unit Tests
|
||||
# Test and Continuous Integration (CI) System in SGLang
|
||||
|
||||
SGLang uses the built-in library [unittest](https://docs.python.org/3/library/unittest.html) as the testing framework.
|
||||
This page introduces the test system, including the CI pipeline, file organization, and how to add and run tests.
|
||||
|
||||
## Test Backend Runtime
|
||||
## Three Stage CI Pipeline
|
||||
|
||||
The CI pipeline runs in three sequential stages after building the kernel:
|
||||
|
||||
- **Stage A** (pre-flight check, ~3 min): Quick smoke tests on small GPUs and CPU to catch obvious breakages early.
|
||||
- **Stage B** (basic tests, ~30 min): Core functional tests on both small GPUs (e.g., 5090) and large GPUs (e.g., H100), including 1-GPU and 2-GPU configurations. Kernel tests and multimodal generation tests also run in parallel at this stage.
|
||||
- **Stage C** (advanced tests, ~30 min): Multi-GPU and specialized hardware tests (H100, H200, B200), plus advanced features such as DeepEP, PD disaggregation, and GB300.
|
||||
|
||||
Here is an illustration
|
||||
```
|
||||
┌──────────────┐
|
||||
│ build kernel │
|
||||
└──────┬───────┘
|
||||
│
|
||||
├─────────────────────────────────────────────────────┐
|
||||
│ │
|
||||
▼ │
|
||||
┌─────────────────────────────────────┐ │
|
||||
│ Stage A (~3 min) │ │
|
||||
│ pre-flight check │ │
|
||||
│ │ │
|
||||
│ ┌─────────────────────────────┐ │ │
|
||||
│ │ stage-a-test-1-gpu-small │ │ │
|
||||
│ │ (small GPUs) │ │ │
|
||||
│ └─────────────────────────────┘ │ │
|
||||
│ ┌─────────────────────────────┐ │ │
|
||||
│ │ stage-a-test-cpu │ │ │
|
||||
│ │ (CPU) │ │ │
|
||||
│ └─────────────────────────────┘ │ │
|
||||
└──────┬──────────────────────────────┘ │
|
||||
│ │
|
||||
▼ ▼
|
||||
┌─────────────────────────────────────┐ ┌──────────────────────────┐
|
||||
│ Stage B (~30 min) │ │ kernel test │
|
||||
│ basic tests │ └──────────────────────────┘
|
||||
│ │ ┌──────────────────────────┐
|
||||
│ ┌─────────────────────────────┐ │ │ multimodal gen test │
|
||||
│ │ stage-b-test-1-gpu-small │ │ └──────────────────────────┘
|
||||
│ │ (small GPUs, e.g. 5090) │ │
|
||||
│ └─────────────────────────────┘ │
|
||||
│ ┌─────────────────────────────┐ │
|
||||
│ │ stage-b-test-1-gpu-large │ │
|
||||
│ │ (large GPUs, e.g. H100) │ │
|
||||
│ └─────────────────────────────┘ │
|
||||
│ ┌─────────────────────────────┐ │
|
||||
│ │ stage-b-test-2-gpu-large │ │
|
||||
│ │ (large GPUs, e.g. H100) │ │
|
||||
│ └─────────────────────────────┘ │
|
||||
└──────┬──────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────────────────────────┐
|
||||
│ Stage C (~30 min) │
|
||||
│ advanced tests │
|
||||
│ │
|
||||
│ ┌─────────────────────────────┐ │
|
||||
│ │ stage-c-test-1-gpu-h100 │ │
|
||||
│ │ (H100 GPUs) │ │
|
||||
│ └─────────────────────────────┘ │
|
||||
│ ┌─────────────────────────────┐ │
|
||||
│ │ stage-c-test-8-gpu-h200 │ │
|
||||
│ │ (8 x H200 GPUs) │ │
|
||||
│ └─────────────────────────────┘ │
|
||||
│ ┌─────────────────────────────┐ │
|
||||
│ │ stage-c-test-4-gpu-b200 │ │
|
||||
│ │ (4 x B200 GPUs) │ │
|
||||
│ └─────────────────────────────┘ │
|
||||
│ ┌─────────────────────────────┐ │
|
||||
│ │ Other advanced tests │ │
|
||||
│ │ (DeepEP, PD Disagg, GB300) │ │
|
||||
│ └─────────────────────────────┘ │
|
||||
└─────────────────────────────────────┘
|
||||
```
|
||||
|
||||
- Stage naming convention: `stage-{a,b,c}-test-{gpu_count}-gpu-{hardware}`
|
||||
- CI runner naming convention: `{gpu_count}-gpu-{hardware}` (e.g., `1-gpu-5090`, `4-gpu-h100`, `8-gpu-h200`)
|
||||
|
||||
|
||||
## Folder organization
|
||||
- `registered`: The registered test files. They are run in CI. Most tests should live in this folder. We use a custom registry system with a file as the basic unit.
|
||||
- `manual`: Test files that CI does not run; you run them manually. Typically, these are temporary tests, deprecated tests, or tests that are not suitable for CI—such as those that take too long or require special setup. We would still like to keep some files here for anyone who wants to run them locally.
|
||||
- `run_suite.py`: The launch script to run a test suite.
|
||||
- Other: utility scripts and metadata folders. The `srt` folder holds our legacy CI setup and should be deprecated as soon as possible.
|
||||
|
||||
Because the system uses a custom registry and the `run_suite.py` launcher, it supports both Python's built-in [unittest](https://docs.python.org/3/library/unittest.html) and the popular [pytest](https://docs.pytest.org/en/stable/) framework.
|
||||
The basic unit is a file, and you can use either framework in your file.
|
||||
The launcher runs `python filename.py` to execute tests, so make sure your file includes the following lines. Otherwise, CI will not run it.
|
||||
|
||||
```python
|
||||
# for unittest
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
```
|
||||
|
||||
```python
|
||||
# for pytest
|
||||
if __name__ == "__main__":
|
||||
import sys
|
||||
sys.exit(pytest.main([__file__]))
|
||||
```
|
||||
|
||||
## Run tests locally
|
||||
|
||||
### Run a single file or a single test
|
||||
```bash
|
||||
cd sglang/test/srt
|
||||
|
||||
# Run a single file
|
||||
python3 test_srt_endpoint.py
|
||||
python3 test/registered/core/test_srt_endpoint.py
|
||||
|
||||
# Run a single test
|
||||
python3 test_srt_endpoint.py TestSRTEndpoint.test_simple_decode
|
||||
|
||||
# Run a suite with multiple files
|
||||
python3 run_suite.py --suite per-commit
|
||||
python3 test/registered/core/test_srt_endpoint.py TestSRTEndpoint.test_simple_decode
|
||||
```
|
||||
|
||||
## Test Frontend Language
|
||||
### Run a suite with multiple files
|
||||
```bash
|
||||
cd sglang/test/lang
|
||||
# Run the CPU-only tests
|
||||
python3 test/run_suite.py --hw cpu --suite stage-a-test-cpu
|
||||
|
||||
# Run a single file
|
||||
python3 test_choices.py
|
||||
# Run the small GPU test
|
||||
python3 test/run_suite.py --hw cuda --suite stage-a-test-1-gpu-small
|
||||
```
|
||||
|
||||
## Adding or Updating Tests in CI
|
||||
### More examples
|
||||
```bash
|
||||
# Run nightly tests
|
||||
python test/run_suite.py --hw cuda --suite nightly-1-gpu --nightly
|
||||
|
||||
# With auto-partitioning (for parallel CI jobs)
|
||||
python test/run_suite.py --hw cuda --suite stage-b-test-1-gpu-small \
|
||||
--auto-partition-id 0 --auto-partition-size 4
|
||||
```
|
||||
|
||||
- Create new test files under `test/srt` or `test/lang` depending on the type of test.
|
||||
- For nightly tests, place them in `test/srt/nightly/`. Use the `NightlyBenchmarkRunner` helper class in `nightly_utils.py` for performance benchmarking tests.
|
||||
- Ensure they are referenced in the respective `run_suite.py` (e.g., `test/srt/run_suite.py`) so they are picked up in CI. For most small test cases, they can be added to the `per-commit-1-gpu` suite. Sort the test cases alphabetically by name.
|
||||
- Ensure you added `unittest.main()` for unittest and `sys.exit(pytest.main([__file__]))` for pytest in the scripts. The CI run them via `python3 test_file.py`.
|
||||
- The CI will run some suites such as `per-commit-1-gpu`, `per-commit-2-gpu`, and `nightly-1-gpu` automatically. If you need special setup or custom test groups, you may modify the workflows in [`.github/workflows/`](https://github.com/sgl-project/sglang/tree/main/.github/workflows).
|
||||
|
||||
## CI Registry System
|
||||
|
||||
Tests in `test/registered/` use a registry-based CI system for flexible backend/schedule configuration.
|
||||
For every test file you add, you need to register it in a suite and provide an estimate execution time in seconds.
|
||||
|
||||
### Registration Functions
|
||||
|
||||
@@ -59,85 +162,90 @@ register_cuda_ci(est_time=200, suite="stage-b-test-2-gpu-large")
|
||||
register_cuda_ci(est_time=200, suite="nightly-1-gpu", nightly=True)
|
||||
|
||||
# Multi-backend test
|
||||
register_cuda_ci(est_time=80, suite="stage-b-test-1-gpu-small")
|
||||
register_cuda_ci(est_time=80, suite="stage-a-test-1-gpu-small")
|
||||
register_amd_ci(est_time=120, suite="stage-a-test-1-gpu-small-amd")
|
||||
register_npu_ci(est_time=400, suite="nightly-8-npu-a3", nightly=True)
|
||||
|
||||
# Temporarily disabled test
|
||||
register_cuda_ci(est_time=80, suite="stage-b-test-1-gpu-small", disabled="flaky - see #12345")
|
||||
```
|
||||
|
||||
### Choosing Between 1-GPU Suites (5090 vs H100)
|
||||
## Available Suites
|
||||
|
||||
When adding 1-GPU tests, choose the appropriate suite based on hardware compatibility:
|
||||
You can find the available suites for each hardware backend at [`test/run_suite.py`](run_suite.py) (`PER_COMMIT_SUITES`, `NIGHTLY_SUITES`). Here we briefly describe some suites.
|
||||
|
||||
| Suite | Runner | GPU | When to Use |
|
||||
|-------|--------|-----|-------------|
|
||||
| `stage-a-test-1-gpu-small` | `1-gpu-5090` | RTX 5090 (32GB, SM120) | Stage A per-commit smoke on 5090 (CUDA) |
|
||||
| `stage-a-test-1-gpu-small-amd` | AMD CI runners | ROCm | Stage A per-commit smoke (AMD) |
|
||||
| `stage-b-test-1-gpu-small` | `1-gpu-5090` | RTX 5090 (32GB, SM120) | 5090-compatible tests (preferred) |
|
||||
| `stage-b-test-1-gpu-large` | `1-gpu-h100` | H100 (80GB, SM90) | Large models or 5090-incompatible tests |
|
||||
### Per-commit (CUDA)
|
||||
|
||||
**Use `stage-b-test-1-gpu-small` (5090) whenever possible** - this is the preferred suite for most 1-GPU tests.
|
||||
| Suite | Runner (label) | Description |
|
||||
| --- | --- | --- |
|
||||
| `stage-a-test-1-gpu-small` | `1-gpu-5090` | Quick checks on a small NVIDIA GPU before heavier stages |
|
||||
| `stage-b-test-1-gpu-small` | `1-gpu-5090` | Core engine tests that fit a 5090-class card |
|
||||
| `stage-b-test-1-gpu-large` | `1-gpu-h100` | Tests that need H100-class memory or kernels (e.g. FA3) |
|
||||
| `stage-b-test-2-gpu-large` | `2-gpu-h100` | Two-GPU correctness and parallelism (TP/PP-style workloads) on H100 |
|
||||
| `stage-b-test-4-gpu-b200` | `4-gpu-b200` | Early Blackwell coverage (e.g. SM100+ paths) on four GPUs |
|
||||
| `stage-c-test-4-gpu-h100` | `4-gpu-h100` | Large 4-GPU H100 integration and scaling tests |
|
||||
| `stage-c-test-8-gpu-h200` | `8-gpu-h200` | Large 8-GPU H200 runs for big models and parallelism |
|
||||
| `stage-c-test-8-gpu-h20` | `8-gpu-h20` | Large 8-GPU H20 runs for big models |
|
||||
| `stage-c-test-deepep-4-gpu-h100` | `4-gpu-h100` | DeepEP expert-parallel and related networking on four H100s. |
|
||||
| `stage-c-test-deepep-8-gpu-h200` | `8-gpu-h200` | DeepEP at 8-GPU H200 scale. |
|
||||
| `stage-c-test-4-gpu-b200` | `4-gpu-b200` | 4-GPU B200 suite for large models on blackwell |
|
||||
| `stage-c-test-4-gpu-gb200` | `4-gpu-gb200`| 4-GPU GB200 suite for large models on grace blackwell |
|
||||
|
||||
**Use `stage-b-test-1-gpu-large` (H100) if ANY of these apply:**
|
||||
Multimodal diffusion uses `python/sglang/multimodal_gen/test/run_suite.py`, not `test/run_suite.py`.
|
||||
|
||||
1. **Architecture incompatibility (SM120/Blackwell)**:
|
||||
- FA3 attention backend (requires SM≤90)
|
||||
- MLA with FA3 backend
|
||||
- FP8/MXFP4 quantization (not supported on SM120)
|
||||
- Certain Triton kernels (shared memory limits)
|
||||
### Per-commit (CPU)
|
||||
|
||||
2. **Memory requirements**:
|
||||
- Models >30B params or large MoE
|
||||
- Tests requiring >32GB VRAM
|
||||
| Suite | Runner (label) | Description |
|
||||
| --- | --- | --- |
|
||||
| `stage-a-test-cpu` | `ubuntu-latest` | CPU-only unit tests |
|
||||
|
||||
3. **Known 5090 failures**:
|
||||
- Weight update/sync tests
|
||||
- Certain spec decoding tests
|
||||
### Per-commit (AMD)
|
||||
|
||||
If a test cannot run on 5090 due to any of the above, use `stage-b-test-1-gpu-large` which runs on H100.
|
||||
| Suite | Runner (label) | Description |
|
||||
| --- | --- | --- |
|
||||
| `stage-a-test-1-gpu-small-amd` | `linux-mi325-1gpu-sglang` | Quick checks on one MI325-class GPU in the AMD CI container. |
|
||||
| `stage-b-test-2-gpu-large-amd` | `linux-mi325-2gpu-sglang` | 2-GPU ROCm correctness and parallel setups. |
|
||||
| `stage-b-test-large-8-gpu-35x-disaggregation-amd` | `linux-mi35x-gpu-8.fabric` | Prefill–decode disaggregation and RDMA-oriented tests on an 8×MI35x fabric runner. |
|
||||
| `stage-c-test-large-8-gpu-amd` | `linux-mi325-8gpu-sglang` | 8-GPU MI325 scaling and integration. |
|
||||
|
||||
### Available Suites
|
||||
### Nightly
|
||||
|
||||
**Per-Commit (CUDA)**:
|
||||
- Stage A: `stage-a-test-1-gpu-small` (5090), `stage-a-test-2`, `stage-a-test-cpu`
|
||||
- Stage B: `stage-b-test-1-gpu-small` (5090), `stage-b-test-1-gpu-large` (H100), `stage-b-test-2-gpu-large`
|
||||
- Stage C (4-GPU): `stage-c-test-4-gpu-h100`, `stage-c-test-4-gpu-b200`, `stage-c-test-4-gpu-gb200`, `stage-c-test-deepep-4-gpu-h100`
|
||||
- Stage C (8-GPU): `stage-c-test-8-gpu-h20`, `stage-c-test-8-gpu-h200`, `stage-c-test-8-gpu-b200`, `stage-c-test-deepep-8-gpu-h200`
|
||||
Nightly registry suites are listed in `NIGHTLY_SUITES` in [`test/run_suite.py`](run_suite.py). They are not driven by `pr-test.yml` / `pr-test-amd*.yml`; see workflows such as `nightly-test-nvidia.yml` and `nightly-test-amd.yml`. Examples:
|
||||
|
||||
**Per-Commit (AMD)**:
|
||||
- `stage-a-test-1-gpu-small-amd`, `stage-b-test-1-gpu-small-amd`, `stage-b-test-2-gpu-large-amd`
|
||||
- `nightly-1-gpu` (CUDA)
|
||||
- `nightly-8-gpu-h200` (CUDA)
|
||||
- `nightly-eval-vlm-2-gpu` (CUDA)
|
||||
- `nightly-amd` (AMD)
|
||||
- `nightly-amd-8-gpu-mi35x` (AMD)
|
||||
|
||||
**Nightly**:
|
||||
- `nightly-1-gpu`, `nightly-2-gpu`, `nightly-4-gpu`, `nightly-8-gpu`, etc.
|
||||
### Choosing a suite for your test
|
||||
|
||||
### Running Tests with run_suite.py
|
||||
Use the lightest suite that still meets your test's needs.
|
||||
|
||||
```bash
|
||||
# Run per-commit tests
|
||||
python test/run_suite.py --hw cuda --suite stage-b-test-1-gpu-small
|
||||
- Prefer the CPU suite (`stage-a-test-cpu`) when no GPU is required.
|
||||
- For most small GPU workloads that fit a 5090-class card in CI, use `stage-b-test-1-gpu-small`. Most tests should go here.
|
||||
- If you really need more GPU memory capacity or Hopper-specific features, use `stage-b-test-1-gpu-large`.
|
||||
- Use multi-GPU suites only when the test actually needs multiple GPUs or other advanced multi-GPU behavior.
|
||||
|
||||
# Run nightly tests
|
||||
python test/run_suite.py --hw cuda --suite nightly-1-gpu --nightly
|
||||
In rare cases, if you need a new runner or custom setup, you might need to add a new suite.
|
||||
|
||||
# With auto-partitioning (for parallel CI jobs)
|
||||
python test/run_suite.py --hw cuda --suite stage-b-test-1-gpu-small \
|
||||
--auto-partition-id 0 --auto-partition-size 4
|
||||
```
|
||||
## Steps for Adding a Test
|
||||
Please refer to [.claude/skills/write-sglang-test/SKILL.md](../.claude/skills/write-sglang-test/SKILL.md)
|
||||
|
||||
## Writing Elegant Test Cases
|
||||
## Multi-hardware backends
|
||||
This README mostly describes the CI pipeline for NVIDIA GPU backends.
|
||||
Other hardware backends should follow the same practices, use the multi-backend registry system, and build their own pipelines.
|
||||
A scheduled job summarizes test coverage across all backends; [here is an example run](https://github.com/sgl-project/sglang/actions/runs/23424304300).
|
||||
|
||||
- Learn from existing examples in [sglang/test/srt](https://github.com/sgl-project/sglang/tree/main/test/srt).
|
||||
- Reduce the test time by using smaller models and reusing the server for multiple test cases. Launching a server takes a lot of time.
|
||||
- Use as few GPUs as possible. Do not run long tests with 8-gpu runners.
|
||||
- If the test cases take too long, considering adding them to nightly tests instead of per-commit tests.
|
||||
- Keep each test function focused on a single scenario or piece of functionality.
|
||||
- Give tests descriptive names reflecting their purpose.
|
||||
- Use robust assertions (e.g., assert, unittest methods) to validate outcomes.
|
||||
- Clean up resources to avoid side effects and preserve test independence.
|
||||
- Reduce the test time by using smaller models and reusing the server for multiple test cases.
|
||||
## Tips for Writing Elegant Test Cases
|
||||
- Learn from existing examples in [test/registered](https://github.com/sgl-project/sglang/tree/main/test/registered).
|
||||
- Reduce the test time by using smaller models and reusing the server for multiple test cases. Launching a server takes a lot of time, so please reuse a single server for many tests instead of launching many servers.
|
||||
- Use as few GPUs as possible. Use 1-GPU runners whenever possible. Do not run long tests with 8-gpu runners.
|
||||
- If the test cases take too long, consider adding them to nightly tests instead of per-commit tests.
|
||||
- Each test file `test_xxx.py` should take less than 500 seconds. If a single file takes longer than that, split it into multiple files.
|
||||
- Each GitHub Actions job should take less than 30 minutes. If a single job takes longer than that, split it into multiple jobs.
|
||||
|
||||
|
||||
## Adding New Models to Nightly CI
|
||||
- **For text models**: extend [global model lists variables](https://github.com/sgl-project/sglang/blob/85c1f7937781199203b38bb46325a2840f353a04/python/sglang/test/test_utils.py#L104) in `test_utils.py`, or add more model lists
|
||||
- **For vlms**: extend the `MODEL_THRESHOLDS` global dictionary in `test/srt/nightly/test_vlms_mmmu_eval.py`
|
||||
## Other Notes
|
||||
### Adding New Models to Nightly CI
|
||||
- **For text models**: Extend the [global model list variables](https://github.com/sgl-project/sglang/blob/85c1f7937781199203b38bb46325a2840f353a04/python/sglang/test/test_utils.py#L104) in `test_utils.py`, or add more model lists.
|
||||
- **For VLMs**: Extend the `MODEL_THRESHOLDS` global dictionary in `test/srt/nightly/test_vlms_mmmu_eval.py`.
|
||||
|
||||
@@ -7,6 +7,7 @@ including batch efficiency, timeout handling, and error cases.
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
import sys
|
||||
import time
|
||||
from unittest.mock import Mock
|
||||
|
||||
@@ -292,4 +293,4 @@ class TestAsyncDynamicbatchTokenizer:
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
pytest.main([__file__])
|
||||
sys.exit(pytest.main([__file__]))
|
||||
|
||||
@@ -11,6 +11,7 @@ Covers:
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
import sys
|
||||
import threading
|
||||
import time
|
||||
from unittest.mock import Mock
|
||||
@@ -361,4 +362,4 @@ class TestAsyncMMDataProcessor:
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
pytest.main([__file__])
|
||||
sys.exit(pytest.main([__file__]))
|
||||
|
||||
@@ -4,6 +4,7 @@ Test script to verify SGLang config file integration.
|
||||
|
||||
import argparse
|
||||
import os
|
||||
import sys
|
||||
import tempfile
|
||||
|
||||
import pytest
|
||||
@@ -162,4 +163,4 @@ def test_error_handling():
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
pytest.main([__file__])
|
||||
sys.exit(pytest.main([__file__]))
|
||||
|
||||
@@ -15,6 +15,7 @@ so the YAML only needs ``dumper.dump(...)`` calls.
|
||||
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
@@ -341,4 +342,4 @@ def _save_comparator_output(*, stdout: str, stderr: str) -> Path:
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
pytest.main([__file__, "-v"])
|
||||
sys.exit(pytest.main([__file__, "-v"]))
|
||||
|
||||
@@ -1,3 +1,5 @@
|
||||
import sys
|
||||
|
||||
import pytest
|
||||
import torch
|
||||
|
||||
@@ -94,4 +96,4 @@ def test_fused_topk_deepseek(seq_length, params, apply_routed_scaling_factor_on_
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
pytest.main([__file__])
|
||||
sys.exit(pytest.main([__file__]))
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import socket
|
||||
import sys
|
||||
from dataclasses import dataclass
|
||||
|
||||
import pytest
|
||||
@@ -391,4 +392,4 @@ def _layernorm_guard_misc_worker(
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
pytest.main([__file__])
|
||||
sys.exit(pytest.main([__file__]))
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
# Temporarily adapted from https://github.com/vllm-project/vllm/blob/main/tests/lora/test_fused_moe_lora_kernel.py, will optimize in future refactor
|
||||
import random
|
||||
import sys
|
||||
|
||||
import pytest
|
||||
import torch
|
||||
@@ -377,4 +378,4 @@ def test_fused_moe_lora_kernel(
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
pytest.main([__file__])
|
||||
sys.exit(pytest.main([__file__]))
|
||||
|
||||
@@ -2,16 +2,47 @@
|
||||
Unit tests for the OpenAIServingEmbedding class from serving_embedding.py.
|
||||
"""
|
||||
|
||||
import importlib
|
||||
import importlib.abc
|
||||
import importlib.machinery
|
||||
import sys
|
||||
import types
|
||||
import unittest
|
||||
import uuid
|
||||
from unittest.mock import MagicMock, Mock
|
||||
|
||||
|
||||
# Stub out sgl_kernel (and all submodules) before any sglang import so
|
||||
# the test runs on CPU-only runners without the real CUDA library.
|
||||
for _mod in ("sgl_kernel", "sgl_kernel.kvcacheio"):
|
||||
if _mod not in sys.modules:
|
||||
sys.modules[_mod] = MagicMock()
|
||||
class _SglKernelMockLoader(importlib.abc.Loader):
|
||||
def create_module(self, spec):
|
||||
mod = types.ModuleType(spec.name)
|
||||
mod.__path__ = []
|
||||
mod.__package__ = spec.name
|
||||
mod.__loader__ = self
|
||||
mod.__getattr__ = lambda name: MagicMock()
|
||||
return mod
|
||||
|
||||
def exec_module(self, module):
|
||||
pass
|
||||
|
||||
|
||||
class _SglKernelMockFinder(importlib.abc.MetaPathFinder):
|
||||
"""Import hook that intercepts all sgl_kernel.* imports and returns mocks."""
|
||||
|
||||
_PREFIX = "sgl_kernel"
|
||||
_loader = _SglKernelMockLoader()
|
||||
|
||||
def find_spec(self, fullname, path, target=None):
|
||||
if fullname == self._PREFIX or fullname.startswith(self._PREFIX + "."):
|
||||
return importlib.machinery.ModuleSpec(
|
||||
fullname, self._loader, is_package=True
|
||||
)
|
||||
return None
|
||||
|
||||
|
||||
if "sgl_kernel" not in sys.modules:
|
||||
sys.meta_path.insert(0, _SglKernelMockFinder())
|
||||
|
||||
from fastapi import Request
|
||||
|
||||
|
||||
+8
-2
@@ -1,5 +1,6 @@
|
||||
import argparse
|
||||
import glob
|
||||
import os
|
||||
import sys
|
||||
from typing import List
|
||||
|
||||
@@ -39,13 +40,14 @@ PER_COMMIT_SUITES = {
|
||||
"stage-b-test-1-gpu-small",
|
||||
"stage-b-test-1-gpu-large",
|
||||
"stage-b-test-2-gpu-large",
|
||||
"stage-b-test-4-gpu-b200",
|
||||
"stage-c-test-4-gpu-h100",
|
||||
"stage-c-test-4-gpu-b200",
|
||||
"stage-c-test-4-gpu-gb200",
|
||||
"stage-c-test-deepep-4-gpu-h100",
|
||||
"stage-c-test-8-gpu-h20",
|
||||
"stage-c-test-8-gpu-h200",
|
||||
"stage-c-test-8-gpu-b200",
|
||||
"stage-c-test-deepep-4-gpu-h100",
|
||||
"stage-c-test-deepep-8-gpu-h200",
|
||||
],
|
||||
HWBackend.NPU: [
|
||||
@@ -169,9 +171,13 @@ def run_a_suite(args):
|
||||
auto_partition_size = args.auto_partition_size
|
||||
|
||||
# All tests (per-commit and nightly) are now in registered/
|
||||
# Use absolute paths so the script works from any working directory
|
||||
script_dir = os.path.dirname(os.path.abspath(__file__))
|
||||
files = [
|
||||
f
|
||||
for f in glob.glob("registered/**/*.py", recursive=True)
|
||||
for f in glob.glob(
|
||||
os.path.join(script_dir, "registered", "**", "*.py"), recursive=True
|
||||
)
|
||||
if not f.endswith("/conftest.py") and not f.endswith("/__init__.py")
|
||||
]
|
||||
# Strict: all registered files must have proper registration
|
||||
|
||||
@@ -1,97 +0,0 @@
|
||||
import argparse
|
||||
import os
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
from sglang.test.ci.ci_utils import TestFile, run_unittest_files
|
||||
|
||||
# Nightly test suites
|
||||
suites = {
|
||||
"nightly-1-gpu": [
|
||||
TestFile("test_nsa_indexer.py", 2),
|
||||
TestFile("test_lora_qwen3.py", 97),
|
||||
TestFile("test_lora_radix_cache.py", 200),
|
||||
TestFile("test_lora_eviction_policy.py", 200),
|
||||
TestFile("test_lora_openai_api.py", 30),
|
||||
TestFile("test_lora_openai_compatible.py", 150),
|
||||
TestFile("test_lora_hf_sgl_logprob_diff.py", 300),
|
||||
TestFile("test_batch_invariant_ops.py", 10),
|
||||
TestFile("test_cpp_radix_cache.py", 60),
|
||||
TestFile("test_deepseek_v3_deterministic.py", 240),
|
||||
],
|
||||
"nightly-4-gpu-b200": [
|
||||
TestFile("test_flashinfer_trtllm_gen_moe_backend.py", 300),
|
||||
TestFile("test_gpt_oss_4gpu_perf.py", 600),
|
||||
TestFile("test_flashinfer_trtllm_gen_attn_backend.py", 300),
|
||||
TestFile("test_fp4_moe.py", 300),
|
||||
TestFile("test_qwen3_fp4_trtllm_gen_moe.py", 300),
|
||||
TestFile("test_eagle_infer_beta_dp_attention_large.py", 600),
|
||||
],
|
||||
"nightly-8-gpu-b200": [
|
||||
TestFile("test_deepseek_r1_fp8_trtllm_backend.py", 3600),
|
||||
TestFile("test_deepseek_v32_gpqa.py", 3600),
|
||||
TestFile("test_mistral_large3_basic.py", 600),
|
||||
],
|
||||
"nightly-4-gpu": [
|
||||
TestFile("test_encoder_dp.py", 500),
|
||||
TestFile("test_qwen3_next_deterministic.py", 200),
|
||||
],
|
||||
"nightly-8-gpu": [],
|
||||
"nightly-8-gpu-h200": [
|
||||
TestFile("test_deepseek_v32_nsabackend.py", 600),
|
||||
],
|
||||
"nightly-8-gpu-h20": [],
|
||||
}
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument(
|
||||
"--suite",
|
||||
type=str,
|
||||
required=True,
|
||||
help="Test suite to run (e.g., nightly-1-gpu, nightly-4-gpu, etc.).",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--timeout-per-file",
|
||||
type=int,
|
||||
default=1200,
|
||||
help="The time limit for running one file in seconds (default: 1200).",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--continue-on-error",
|
||||
action="store_true",
|
||||
default=False,
|
||||
help="Continue running remaining tests even if one fails (default: False, useful for nightly tests).",
|
||||
)
|
||||
args = parser.parse_args()
|
||||
|
||||
if args.suite not in suites:
|
||||
print(f"Error: Suite '{args.suite}' not found in available suites")
|
||||
print(f"Available suites: {list(suites.keys())}")
|
||||
exit(1)
|
||||
|
||||
files = suites[args.suite]
|
||||
|
||||
# Change directory to test/nightly where the test files are located
|
||||
nightly_dir = Path(__file__).parent / "nightly"
|
||||
os.chdir(nightly_dir)
|
||||
|
||||
# Add test/ to PYTHONPATH so tests can import shared utils
|
||||
test_dir = str(Path(__file__).parent)
|
||||
pythonpath = os.environ.get("PYTHONPATH", "")
|
||||
os.environ["PYTHONPATH"] = f"{test_dir}:{pythonpath}" if pythonpath else test_dir
|
||||
|
||||
print(f"Running {len(files)} tests from suite: {args.suite}")
|
||||
print(f"Test files: {[f.name for f in files]}")
|
||||
|
||||
exit_code = run_unittest_files(
|
||||
files,
|
||||
timeout_per_file=args.timeout_per_file,
|
||||
continue_on_error=args.continue_on_error,
|
||||
)
|
||||
sys.exit(exit_code)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,369 +0,0 @@
|
||||
import argparse
|
||||
import logging
|
||||
import os
|
||||
import queue
|
||||
import re
|
||||
import subprocess
|
||||
import threading
|
||||
import time
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime
|
||||
from typing import List, Optional, Tuple
|
||||
|
||||
import psutil
|
||||
import yaml
|
||||
|
||||
from sglang.utils import wait_for_http_ready
|
||||
|
||||
|
||||
@dataclass
|
||||
class ServerConfig:
|
||||
command: str
|
||||
process_names: List[str]
|
||||
default_port: int
|
||||
|
||||
|
||||
@dataclass
|
||||
class TaskConfig:
|
||||
server_cmd: str
|
||||
client_cmd: str
|
||||
name: Optional[str] = None
|
||||
server_type: Optional[str] = None
|
||||
|
||||
|
||||
@dataclass
|
||||
class TaskResult:
|
||||
name: str
|
||||
success: bool
|
||||
output: str
|
||||
runtime: float
|
||||
timestamp: str
|
||||
|
||||
|
||||
SERVER_DEFAULTS = {
|
||||
"sglang": ServerConfig(
|
||||
command="sglang.launch_server",
|
||||
process_names=["sglang.launch_server"],
|
||||
default_port=30000,
|
||||
),
|
||||
"vllm": ServerConfig(
|
||||
command="vllm.entrypoints.openai.api_server",
|
||||
process_names=["vllm.entrypoints.openai.api_server"],
|
||||
default_port=8000,
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
def parse_key_info(output: str) -> str:
|
||||
"""Extract and format key information from the output"""
|
||||
key_info = []
|
||||
|
||||
# Extract Args namespace
|
||||
args_match = re.search(r"Namespace\(.*?\)", output, re.DOTALL)
|
||||
if args_match:
|
||||
key_info.append(args_match.group(0))
|
||||
|
||||
# Extract input/output token counts
|
||||
token_matches = re.findall(r"#(Input|Output) tokens: \d+", output)
|
||||
key_info.extend(token_matches)
|
||||
|
||||
# Extract benchmark result section
|
||||
result_match = re.search(
|
||||
r"============ Serving Benchmark Result ============.*?={50,}",
|
||||
output,
|
||||
re.DOTALL,
|
||||
)
|
||||
if result_match:
|
||||
key_info.append(result_match.group(0))
|
||||
|
||||
return "\n\n".join(key_info)
|
||||
|
||||
|
||||
def extract_port_from_command(cmd: str, server_type: str) -> int:
|
||||
port_match = re.search(r"--port[= ](\d+)", cmd)
|
||||
if port_match:
|
||||
return int(port_match.group(1))
|
||||
return SERVER_DEFAULTS.get(server_type, ServerConfig("", [], 8000)).default_port
|
||||
|
||||
|
||||
def detect_server_type(cmd: str) -> str:
|
||||
for server_type, config in SERVER_DEFAULTS.items():
|
||||
if config.command in cmd:
|
||||
return server_type
|
||||
return "unknown"
|
||||
|
||||
|
||||
def stream_output(
|
||||
process: subprocess.Popen, prefix: str, logger: logging.Logger
|
||||
) -> queue.Queue:
|
||||
output_queue = queue.Queue()
|
||||
|
||||
def stream_pipe(pipe, prefix):
|
||||
for line in iter(pipe.readline, ""):
|
||||
if prefix == "CLIENT":
|
||||
output_queue.put(line.rstrip())
|
||||
logger.debug(f"{prefix} | {line.rstrip()}")
|
||||
|
||||
stdout_thread = threading.Thread(
|
||||
target=stream_pipe, args=(process.stdout, prefix), daemon=True
|
||||
)
|
||||
stderr_thread = threading.Thread(
|
||||
target=stream_pipe, args=(process.stderr, prefix), daemon=True
|
||||
)
|
||||
|
||||
stdout_thread.start()
|
||||
stderr_thread.start()
|
||||
return output_queue, (stdout_thread, stderr_thread)
|
||||
|
||||
|
||||
class ProcessManager:
|
||||
def __init__(self):
|
||||
self.server_process: Optional[subprocess.Popen] = None
|
||||
self.client_process: Optional[subprocess.Popen] = None
|
||||
self.logger = logging.getLogger(__name__)
|
||||
|
||||
def start_process(
|
||||
self, command: str, prefix: str
|
||||
) -> Tuple[subprocess.Popen, queue.Queue]:
|
||||
process = subprocess.Popen(
|
||||
command,
|
||||
shell=True,
|
||||
stdout=subprocess.PIPE,
|
||||
stderr=subprocess.PIPE,
|
||||
text=True,
|
||||
bufsize=1,
|
||||
)
|
||||
|
||||
output_queue, threads = stream_output(process, prefix, self.logger)
|
||||
return process, output_queue, threads
|
||||
|
||||
def kill_process_tree(self, process: subprocess.Popen):
|
||||
try:
|
||||
parent = psutil.Process(process.pid)
|
||||
children = parent.children(recursive=True)
|
||||
|
||||
for child in children:
|
||||
try:
|
||||
child.kill()
|
||||
except psutil.NoSuchProcess:
|
||||
pass
|
||||
|
||||
parent.kill()
|
||||
gone, alive = psutil.wait_procs(children + [parent], timeout=3)
|
||||
|
||||
for p in alive:
|
||||
try:
|
||||
p.kill()
|
||||
except psutil.NoSuchProcess:
|
||||
pass
|
||||
|
||||
except psutil.NoSuchProcess:
|
||||
pass
|
||||
|
||||
def cleanup(self, process_names: List[str]):
|
||||
if self.client_process:
|
||||
self.kill_process_tree(self.client_process)
|
||||
self.client_process = None
|
||||
|
||||
if self.server_process:
|
||||
self.kill_process_tree(self.server_process)
|
||||
self.server_process = None
|
||||
|
||||
for proc in psutil.process_iter(["pid", "name", "cmdline"]):
|
||||
try:
|
||||
cmdline = " ".join(proc.cmdline())
|
||||
if any(name in cmdline for name in process_names):
|
||||
proc.kill()
|
||||
except (psutil.NoSuchProcess, psutil.AccessDenied):
|
||||
continue
|
||||
|
||||
|
||||
class ExperimentRunner:
|
||||
def __init__(self):
|
||||
self.process_manager = ProcessManager()
|
||||
self.logger = logging.getLogger(__name__)
|
||||
|
||||
def wait_for_server(
|
||||
self, port: int, timeout: int = 300, process: Optional[subprocess.Popen] = None
|
||||
) -> bool:
|
||||
try:
|
||||
wait_for_http_ready(
|
||||
url=f"http://localhost:{port}/health",
|
||||
timeout=timeout,
|
||||
process=process,
|
||||
)
|
||||
self.logger.debug(f"Server ready on port {port}")
|
||||
return True
|
||||
except (RuntimeError, TimeoutError) as e:
|
||||
self.logger.error("Server failed to become ready: %s", e)
|
||||
return False
|
||||
|
||||
def run_task(self, config: TaskConfig) -> TaskResult:
|
||||
start_time = time.perf_counter()
|
||||
client_output = []
|
||||
|
||||
try:
|
||||
if not config.server_type:
|
||||
config.server_type = detect_server_type(config.server_cmd)
|
||||
|
||||
server_config = SERVER_DEFAULTS.get(config.server_type)
|
||||
if not server_config:
|
||||
raise ValueError(f"Unknown server type: {config.server_type}")
|
||||
|
||||
port = extract_port_from_command(config.server_cmd, config.server_type)
|
||||
|
||||
self.process_manager.cleanup(server_config.process_names)
|
||||
|
||||
self.logger.debug(f"Starting server: {config.name}")
|
||||
self.process_manager.server_process, _, server_threads = (
|
||||
self.process_manager.start_process(config.server_cmd, "SERVER")
|
||||
)
|
||||
|
||||
if not self.wait_for_server(
|
||||
port, process=self.process_manager.server_process
|
||||
):
|
||||
raise TimeoutError("Server startup timeout")
|
||||
|
||||
time.sleep(10)
|
||||
|
||||
self.logger.debug("Starting client")
|
||||
self.process_manager.client_process, output_queue, client_threads = (
|
||||
self.process_manager.start_process(config.client_cmd, "CLIENT")
|
||||
)
|
||||
|
||||
returncode = self.process_manager.client_process.wait()
|
||||
|
||||
while True:
|
||||
try:
|
||||
line = output_queue.get_nowait()
|
||||
client_output.append(line)
|
||||
except queue.Empty:
|
||||
break
|
||||
|
||||
if returncode != 0:
|
||||
raise RuntimeError(f"Client failed with code {returncode}")
|
||||
|
||||
# Parse and format the output
|
||||
full_output = "\n".join(client_output)
|
||||
formatted_output = parse_key_info(full_output)
|
||||
|
||||
return TaskResult(
|
||||
name=config.name,
|
||||
success=True,
|
||||
output=formatted_output,
|
||||
runtime=time.perf_counter() - start_time,
|
||||
timestamp=datetime.now().isoformat(),
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
return TaskResult(
|
||||
name=config.name,
|
||||
success=False,
|
||||
output=str(e),
|
||||
runtime=time.perf_counter() - start_time,
|
||||
timestamp=datetime.now().isoformat(),
|
||||
)
|
||||
|
||||
finally:
|
||||
if config.server_type in SERVER_DEFAULTS:
|
||||
self.process_manager.cleanup(
|
||||
SERVER_DEFAULTS[config.server_type].process_names
|
||||
)
|
||||
time.sleep(10)
|
||||
|
||||
|
||||
def load_config(config_path: str) -> List[TaskConfig]:
|
||||
with open(config_path, "r") as f:
|
||||
config_data = yaml.safe_load(f)
|
||||
|
||||
configs = []
|
||||
for idx, entry in enumerate(config_data.get("tasks", [])):
|
||||
if not isinstance(entry, dict):
|
||||
raise ValueError(f"Invalid entry at index {idx}")
|
||||
|
||||
config = TaskConfig(
|
||||
server_cmd=entry.get("server_cmd"),
|
||||
client_cmd=entry.get("client_cmd"),
|
||||
name=entry.get("name", f"task-{idx+1}"),
|
||||
server_type=entry.get("server_type"),
|
||||
)
|
||||
|
||||
if not config.server_cmd or not config.client_cmd:
|
||||
raise ValueError(f"Missing commands in {config.name}")
|
||||
|
||||
configs.append(config)
|
||||
|
||||
return configs
|
||||
|
||||
|
||||
def setup_logging(debug: bool = False):
|
||||
level = logging.DEBUG if debug else logging.INFO
|
||||
logging.basicConfig(
|
||||
level=level,
|
||||
format="%(asctime)s - %(levelname)s - %(message)s",
|
||||
handlers=[logging.StreamHandler(), logging.FileHandler("experiment.log")],
|
||||
)
|
||||
|
||||
|
||||
def format_results(results: List[TaskResult]) -> str:
|
||||
"""Format experiment results in Markdown for GitHub step summary."""
|
||||
output = ["# Experiment Results\n"]
|
||||
|
||||
for result in results:
|
||||
output.append(f"## {result.name}")
|
||||
output.append(f"**Status**: {'✅ Success' if result.success else '❌ Failed'}")
|
||||
output.append(f"**Runtime**: {result.runtime:.2f} seconds")
|
||||
output.append(f"**Timestamp**: {result.timestamp}")
|
||||
output.append("\n**Output**:\n```")
|
||||
output.append(result.output)
|
||||
output.append("```\n")
|
||||
|
||||
return "\n".join(output)
|
||||
|
||||
|
||||
def get_bool_env_var(name: str, default: str = "false") -> bool:
|
||||
value = os.getenv(name, default)
|
||||
return value.lower() in ("true", "1")
|
||||
|
||||
|
||||
def write_in_github_step_summary(results: List[TaskResult]):
|
||||
"""Write formatted results to GitHub step summary."""
|
||||
if not os.environ.get("GITHUB_STEP_SUMMARY"):
|
||||
logging.warning("GITHUB_STEP_SUMMARY environment variable not set")
|
||||
return
|
||||
|
||||
formatted_content = format_results(results)
|
||||
with open(os.environ["GITHUB_STEP_SUMMARY"], "a") as f:
|
||||
f.write(formatted_content)
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description="Experiment Runner")
|
||||
parser.add_argument(
|
||||
"--config", type=str, required=True, help="Path to YAML config file"
|
||||
)
|
||||
parser.add_argument("--debug", action="store_true", help="Enable debug output")
|
||||
args = parser.parse_args()
|
||||
|
||||
setup_logging(args.debug)
|
||||
logger = logging.getLogger(__name__)
|
||||
results = []
|
||||
|
||||
try:
|
||||
configs = load_config(args.config)
|
||||
runner = ExperimentRunner()
|
||||
|
||||
for config in configs:
|
||||
logger.info(f"Running {config.name}")
|
||||
result = runner.run_task(config)
|
||||
results.append(result)
|
||||
|
||||
if get_bool_env_var("SGLANG_IS_IN_CI"):
|
||||
write_in_github_step_summary(results)
|
||||
except Exception as e:
|
||||
logger.error(f"Error: {e}")
|
||||
raise
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,52 +0,0 @@
|
||||
"""
|
||||
used for debug using tensor comparison
|
||||
dump {name: tensor} into "log_hf.jsonl" and "log_srt.jsonl"
|
||||
use the same name for two tensors that supposed to be close
|
||||
recommend name like: "layer 2 after mlp"
|
||||
"""
|
||||
|
||||
import json
|
||||
import sys
|
||||
|
||||
import torch
|
||||
|
||||
if len(sys.argv) > 1:
|
||||
assert sys.argv[1] == "base"
|
||||
hf_log = "base_log_hf.jsonl"
|
||||
srt_log = "base_log_srt.jsonl"
|
||||
else:
|
||||
hf_log = "log_hf.jsonl"
|
||||
srt_log = "log_srt.jsonl"
|
||||
|
||||
|
||||
def load_data(filepath):
|
||||
tensors = {}
|
||||
with open(filepath, "r") as f:
|
||||
lines = f.readlines()
|
||||
for line in lines:
|
||||
data = json.loads(line)
|
||||
for k, v in data.items():
|
||||
tensors[k] = torch.tensor(v)
|
||||
return tensors
|
||||
|
||||
|
||||
hf_tensors = load_data(hf_log)
|
||||
srt_tensors = load_data(srt_log)
|
||||
|
||||
|
||||
def get_diff(t1, t2):
|
||||
t1 = t1.reshape(t2.shape)
|
||||
max_diff = torch.max(abs(t1.reshape(t2.shape) - t2))
|
||||
l2_dis = torch.dist(t1, t2, p=2)
|
||||
return l2_dis, max_diff
|
||||
|
||||
|
||||
for k, _ in srt_tensors.items():
|
||||
l2_dis, max_diff = get_diff(hf_tensors[k], srt_tensors[k])
|
||||
print(f"{k} {l2_dis=} {max_diff=}")
|
||||
if k == "layer 1 attn":
|
||||
print(hf_tensors[k])
|
||||
print(srt_tensors[k])
|
||||
if k == "layer 0 prefill k":
|
||||
print(srt_tensors[k].shape)
|
||||
print(hf_tensors[k].shape)
|
||||
@@ -1,57 +0,0 @@
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
|
||||
import pandas as pd
|
||||
from tabulate import tabulate
|
||||
|
||||
# Parse command-line arguments
|
||||
parser = argparse.ArgumentParser(description="Parse JSONL benchmark and summarize.")
|
||||
parser.add_argument("input_file", type=str, help="Path to input JSONL file")
|
||||
parser.add_argument(
|
||||
"--md",
|
||||
action="store_true",
|
||||
help="If set, print the summary table in Markdown format (GitHub style)",
|
||||
)
|
||||
args = parser.parse_args()
|
||||
|
||||
input_file = args.input_file
|
||||
base_name = os.path.splitext(os.path.basename(input_file))[0]
|
||||
output_file = f"{base_name}_summary.csv"
|
||||
|
||||
fields = [
|
||||
"max_concurrency",
|
||||
"input_throughput",
|
||||
"output_throughput",
|
||||
"mean_ttft_ms",
|
||||
"median_ttft_ms",
|
||||
"p99_ttft_ms",
|
||||
"mean_tpot_ms",
|
||||
"median_tpot_ms",
|
||||
"p99_tpot_ms",
|
||||
]
|
||||
|
||||
# Read JSONL and parse
|
||||
results = []
|
||||
with open(input_file, "r") as f:
|
||||
for line in f:
|
||||
data = json.loads(line)
|
||||
row = {field: data.get(field, None) for field in fields}
|
||||
max_conc = data.get("max_concurrency")
|
||||
out_tp = data.get("output_throughput")
|
||||
row["per_user_throughput"] = out_tp / max_conc if max_conc else None
|
||||
results.append(row)
|
||||
|
||||
# Convert to DataFrame
|
||||
df = pd.DataFrame(results)
|
||||
|
||||
# Save to CSV
|
||||
df.to_csv(output_file, index=False)
|
||||
print(f"\nSaved summary to: {output_file}\n")
|
||||
|
||||
if args.md:
|
||||
# Print Markdown table
|
||||
print(tabulate(df, headers="keys", tablefmt="github", floatfmt=".3f"))
|
||||
else:
|
||||
# Print ASCII table
|
||||
print(tabulate(df, headers="keys", tablefmt="grid", floatfmt=".3f"))
|
||||
@@ -96,7 +96,7 @@ suite_ascend = {
|
||||
TestFile("ascend/test_ascend_tp1_bf16.py", 400),
|
||||
TestFile("ascend/test_ascend_compile_graph_tp1_bf16.py", 400),
|
||||
TestFile("ascend/test_ascend_w8a8_quantization.py", 400),
|
||||
TestFile("test_embed_interpolate_unittest.py", 400),
|
||||
TestFile("ascend/test_embed_interpolate_unittest.py", 400),
|
||||
],
|
||||
"per-commit-2-npu-a2": [
|
||||
TestFile("ascend/test_ascend_graph_tp2_bf16.py", 400),
|
||||
@@ -326,4 +326,9 @@ def main():
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
print(
|
||||
"DEPRECATION NOTICE: The folder `test/srt` should be deprecated as soon as possible. "
|
||||
"Migrate tests to the new CI registry system described in `test/README.md`.",
|
||||
flush=True,
|
||||
)
|
||||
main()
|
||||
|
||||
Reference in New Issue
Block a user